Hey everyone, so as I’m sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.

We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS’er, and is causing these disconnects to our backend server, problematic for sure.

We’ve opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out https://status.lemmy.world and https://dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates

In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.

Cheers

  • LW Infra Team
  • Cthuwu@lemmy.worldEnglish
    2703·
    10 months ago

    As always, the transparency is appreciated. Some growing pains are certainly to be expected

  • Boozilla@lemmy.worldEnglish
    2426·
    10 months ago

    Whenever I get frustrated by the outages I remind myself: still better than reddit.

    • Shady_Shiroe@lemmy.world
      913·
      10 months ago

      Thanks to lemmy I don’t doom scroll anymore because either there is an outage or I read all the new content I’m subscribed to in 10-20 minutes

      • GONADS125@lemmy.world
        30·
        10 months ago

        That’s mostly how I redditted for years. It was mostly for those moments in between things, on the toilet, laying in bed at night. Not something I did for long periods of time.

      • kratoz29@lemm.eeEnglish
        102·
        10 months ago

        Huh, I could keep up with my daily feeds before, now not so much.

        • grue@lemmy.worldEnglish
          111·
          10 months ago

          It’s your fault for not pitching in and making lemmy more stable.

          Plot twist: unlike Reddit, with Lemmy that would be a legitimate option! Ain’t Free Software and federated networks grand?

        • shastaxc@lemmy.world
          6·
          10 months ago

          What is the burden if I wanted to host a node and limit access just to myself? Is it just a portal into the rest of the fediverse or is there a large maintenance burden or storage requirement?

          • maor@lemmy.org.il
            51·
            10 months ago

            Yeah, pretty big storage requirement due to the way pictrs works. Pictrs is the piece of software Lemmy relies upon to manage image storage, uploads, and most importantly: caches pictures from other instances. This takes up a HUGE amount of storage space, and there’s no official way to clear this up, see these posts I recently made: first one, second one. The solution I resorted to is renting a 1TB storage box from Hetzner for 3 euros per month, pretty sweet deal but I was kinda annoying by it. So the cheapest deal I could find costs me 6 euros per month: 3 for an Alma Linux ARM VPS from Hetzner, and 3 for that storage box. If you’re in for the fun in tinkering (I sure as hell am in), then get ready for a good time. Other than that, if your main line of reasoning is to take burden off of lemmy.world, then I think just go ahead and join another instance. Better yet: join croud funding of another instnace:)

            • Draconic NEO@lemmy.world
              1·
              10 months ago

              I don’t think this is correct, the images from one instance are hosted on that instance, they don’t get copied over. What’s federated and copied over is text and references to those images (like this ![](https://image.url/image.filename)). The only images that are hosted on your own Lemmy instance are the ones that you and your users upload, which sure if you have open sign ups and a bunch of people using your instance uploading images then it will become a problem very quickly, but for personal use it shouldn’t unless you’re uploading a lot of images or even videos.

              Just use external image hosts for posts and comments, and only use image uploads for your own profile and banner and everything should be fine (if you want extra assurance you can disable uploads in case you forget).

              • maor@lemmy.org.il
                1·
                10 months ago

                Nah I thought the same but then I manually checked it. In most of the image posts I see, the image URL starts with lemmy.org.il, which made me wonder whether they’re actually downloaded or it’s some kind of whacky proxy. So I downloaded some of these pics and looked for files of identical size and hash digest, and indeed they were on my disk!

                It’s not a bad decision to cache pics, because it does make the experience really smooth, and I’m not complaining about it. Mastodon does this as well

                • Draconic NEO@lemmy.world
                  2·
                  10 months ago

                  Weird because I’ve checked it before and it doesn’t seem to, one notable example is that on the lemmy.zip communities I can’t see any of the images at all because something on my network or with my ISP blocks zip domains. The posts, comments and all their text gets copied over to lemmy.world but the images don’t seem to and thus they don’t display for me due to their domains being blocked.

                  (Alternate DNS doesn’t seem to bypass whatever they’re doing to block it, I can only connect to .zip domains through Tor or VPN).

                  It’s not a bad decision to cache pics, because it does make the experience really smooth, and I’m not complaining about it. Mastodon does this as well

                  I agree it’s definitely a good idea to create a more smooth user experience, would also eliminate the problems that I pointed out with the images not loading due to their domain being blocked.

    • malloc@lemmy.worldEnglish
      9·
      10 months ago

      I fallback to one of the many instances that federate with lw. Don’t get frustrated at all bro

  • Machefi@lemmy.world
    811·
    10 months ago

    I don’t blame you for this, but the uptime records are incomplete at best. I’ve experienced the site being down (and confirmed with Down for Everyone or Just Me), yet status.lemmy.world showed all systems operational. As I’m writing this, status.lemmy.world is missing most data up to yesterday and dash.lemmy.world shows 16 days uptime.

    I have lots of respect to you for even having these. I also remember status.lemmy.world work mostly fine some time ago. But as of right now, both uptime monitors fail to serve their purpose.

    • lwadmin@lemmy.worldOPMEnglish
      811·
      10 months ago

      You need to hover over the status bar to see if there is any down time for that day. We can enable it to log incidents every time there is a burp, but we are still tuning alerts as we only have it create a incident when we ACK it in PagerDuty. You can always check the dashboard for up to the minute stats, as well as https://lemmy-status.org/endpoints/_lemmy-world We’ll add this info to make things clearer <3

      EDIT: Added more info to our status page, thanks for the feedback Machefi!

      EDIT2: Also the missing data is due to us removing and adding more specific monitors for the different infra services.

      • Obinice@lemmy.world
        263·
        10 months ago

        Excuse me stop being so cool, you’re raising the bar too high for everyone else thank you

  • perfectra1n@lemmy.worldEnglish
    421·
    10 months ago

    On your Cloudflare account, if there was a change in the CNAME/A record being proxied vs. DNS only, that could cause an issue, as Cloudflare would then strip headers off the request that your Apache/Nginx would be looking for.

    If you enabled HTTP DDoS protection in your Security -> WAF tab (I think that’s where it is) that could do this too. Might be worth disabling.

    Also check for any headers your HTTP load balancer might be expecting, that Cloudflare could be stripping.

    Might be worth tailing the webserver logs to see what happens to requests coming in from Cloudflare.

  • Dagnet@lemmy.world
    37·
    10 months ago

    Thanks for keeping us up to date, I don’t mind downtime personally as long as I’m aware of what’s going on

  • drewthejedi@lemmy.world
    371·
    10 months ago

    Everyone talking about the downtime including lemme and me just enjoying lemmy like never before. I’ve experienced no downtime so far

    • malloc@lemmy.worldEnglish
      181·
      10 months ago

      The beauty of a federated app is that I do not have to rely on lw or lemme. Use another instance and go on with my day 😃

      • RisingSwell@lemmy.worldEnglish
        93·
        10 months ago

        Down for you is not the same as down. If it’s up when you aren’t using it, it’s still up

  • BellaDonna@mujico.org
    5226·
    10 months ago

    Let me be real. I never noticed outages stopping. It feels like it’s daily, I’m used to it, but I think it happens so often that lemmy.world has lost its growth opportunity, and we alienated the normies. I’m still going to stay on Lemmy, and I believe you’re doing the best you can, but we lost for the time being, the migration to Lemmy from Reddit is stunted.

    • danielton@lemmy.world
      695·
      10 months ago

      It didn’t help that almost every other general purpose instance blocked sign-ups in June and early July either, or required an essay on the application. Lemmy.world was the only one that was even trying at all, and I will commend them for that.

      Hopefully things will get better by the next time spez screws up. Because there will be a next time.

      • gabe [he/him]@literature.cafe
        121·
        10 months ago

        Next time the lemmy join page needs to be improved so people can spread and don’t try to centralize into a single instance and break the purpose of lemmy in the first place.

        • danielton@lemmy.world
          24·
          10 months ago

          Sure, but to be fair, there weren’t really many general-purpose instances that were accepting sign-ups from anybody when the Reddit bullshit went down in June. That’s part of why lemmy.world got as big as it did.

          Most people who ended up on Squabblr and Discuit instead went there because they didn’t have to write an essay to join or try to find a server that was accepting sign-ups and wasn’t down a lot.

          • Blaze
            5·
            10 months ago

            Squabblr and Discuit

            Discuit is a place where 4142 people get together to find cool stuff and discuss things.

            Squabblr doesn’t have a count of active users (33k registered users)

            I know we are low on the numbers, but still a bit higher than them.

            • danielton@lemmy.world
              1·
              10 months ago

              OK, but I didn’t say more people ended up there than here. I was just stating the main reason people chose them over Lemmy.

              • Blaze
                1·
                10 months ago

                To be honest, with such low numbers, I guess after a while they just came to Lemmy once instances were a bit more stable or went something else altogether

                • danielton@lemmy.world
                  3·
                  10 months ago

                  Squabblr decided to become a “Free Speech” platform and remove rules against LGBT hate speech, so they shot themselves in the foot. Almost everybody who was active on Squabblr moved to Discuit, but even Discuit has nowhere near the activity of Lemmy and kbin.

      • HTTP_404_NotFound@lemmyonline.comEnglish
        9·
        10 months ago

        Partly due to the fact… lemmy itself, basically has no moderation or administration features at all…

        So, the only way to assist with that issue, is stricter enforcement up-front.

        Besides, if someone doesn’t wanna take the time to have a verified email, and literally type 49 when registering an application… I really don’t wanna take the time to worry about having to potentially worry about them being spammers/etc.

    • PM_ME_YOUR_ZOD_RUNES@sh.itjust.works
      124·
      10 months ago

      We shouldn’t be trying to grow a single instance. That defeats the whole point of Lemmy. I started on Lemmy world and switched once I got fed up with the constant connection issues. Plus, Lemmy world blocked piracy communities so fuck that. I’m happy that I am able to quickly create an account on another instance.

    • HTTP_404_NotFound@lemmyonline.comEnglish
      22·
      10 months ago

      I never noticed outages stopping

      I am in the same boat. If it were not for these posts, I’d have never noticed lemmy world was down.

      If I post to, or put a comment on something from lemmy world, it will just federate over when it’s back online.

      If someone posts to something on lemmy world, it will eventually federate over my way.

      But, hey, everyone is gonna fuss when they decided to put all of their eggs in one basket and now, that one basket gets targeted. (metaphor for lemmy world.)

  • Shadywack@lemmy.worldEnglish
    252·
    10 months ago

    Only thing that needs to be said here is: thank you for the best Lemmy instance (in my opinion) out there!

  • webhead@lemmy.world
    20·
    10 months ago

    Could look into Dacentec if you need more cheap servers. I use them for my stuff. YMMV since you’re getting a hell of a lot more traffic than I am but they haven’t blocked Cloudflare on mine yet so that’s a plus. :)