Hello there friendly users!

Update 3:50pm CEST
The container has been moved, most work on the new host has been completed. There are some permanent data errors on the original storage pool, but it does not like it affected this linux container. Please give DNS and federation a moment.

Some (but only a few) of you have experienced an outage today around 5-8am CEST. This – as far as my current findings go – happened due to a filesystem panic. So far, i only experienced such a thing on virtual servers with a certain amount of load.

Even tho relatively speaking, i didn’t feel like there was much to worry about with the current setup and its utilization, i feel like when looking at a potential second wave from Reddit, this instance should be prepared – so i just did a thing:

What we are moving to

We will be moving from a Netcup RS 4000 G9.5 KVM with 32 GB memory and 10 dedicated EPYC cores, (that was meant for various smaller tchncs services such as XMPP, Email, Clouds, Websites and stuff and lots of headroom), to a Hetzner serverauction baremetal thing with an Intel Core i7-7700, 2x SSD M.2 NVMe 512 GB, 64 GB DDR4 Mem, 1 Gbit Networking in Finland.

I plan to migrate today as soon as the thing is ready and configured. There will be a downtime of up to 1 hour.

Lemmy will share its seat on the machine with the tiny Calckey instance that was deployed around the same time.

Increased monthly cost: around 40 EUR

Note on the thing Meta is developing with ActivityPub federation support:
I am not a friend of Meta and am happy to keep it out of my life an off of my servers. However there are lots of rumors and barely any confirmed informations around the project. I will act accordingly once more is known. I also won’t make any exceptions in terms of enforcing our rules. However i won’t participate in signature lists for pre-blocking everything Meta and run around in circles, screaming in panic.

  • Ruapho
    12·
    2 years ago

    Thanks for investing your time and money. Btw: the Donate page is down (502 Error)

    • MilanOPMA
      9·
      2 years ago

      sorry about that, the website is back up – and thanks for considering :)

  • thomholwerda@kbin.social
    6·
    2 years ago

    Excellent position on the Meta thing, Milan! It’s the only sane thing to do at the moment, I think.

    Thanks for all the hard work!

  • BeliefPropagator
    4·
    2 years ago

    Is there a metrics/grafana page where I can see the bandwidth and compute that the server uses? Do you have insights into that? Where do you see the bottlenecks in the current system?

    • MilanOPMA
      3·
      2 years ago

      Not right now, sorry, but i might add something like this (again) in the near future. There was some memory leaky behavior and a crying ZFS of this kind never happened to me on my dedicated hosts so far. Otherwise the previous system was fine… hard to predict the next incoming wave and stuff tho.

      • BeliefPropagator
        1·
        2 years ago

        Np. Don’t worry about it, I don’t want to hassle you into doing extra (potentially) unnecessary work. Interesting that ZFS was leaking - I have no experience with it, I just heard that ZFS in general requires a lot of RAM to be a reasonable filesystem choice. Are you sure it was a leak, or could it have been e.g. caches filling up as well?

  • heimchen
    2·
    2 years ago

    How much does the server cost all in all monthly (or are the 40€ after the increase and not just the difference between previous and after)

    • MilanOPMA
      4·
      2 years ago

      The server that was just rented is around 38 € per month. I am not sure if this was including tax or not… (maybe with a setup fee tho). There are also dynamic costs for media storage, but there are no per-container statistics of this kind… it shouldn’t be much more than an euro or something right now, i don’t know.

      …the old server will remain in the tchncs infra.

  • ivanhoe
    2·
    2 years ago

    Was there any data loss caused by the outage? I can’t find a post anymore that I made a while ago. It’s nothing important, I’m just curious.

    • MilanOPMA
      3·
      2 years ago

      there shouldn’t be data loss from before the incident…

  • elektronaut
    1·
    2 years ago

    Seems like I cannot post notes with attached images on Calckey. Text-only posts do work though. @elektronaut@procial.tchncs.de

    • MilanOPMA
      1·
      2 years ago

      sorry about that. it should now be fixed

      • elektronaut
        1·
        2 years ago

        Never mind. Posting notes with attachments works again now. 👍🏻

        • MilanOPMA
          2·
          2 years ago

          No never mind needed: missed a little thing in the domain config. Glad that you can confirm it being working now :)