Hello there friendly users!
Update 3:50pm CEST
The container has been moved, most work on the new host has been completed. There are some permanent data errors on the original storage pool, but it does not like it affected this linux container. Please give DNS and federation a moment.
Some (but only a few) of you have experienced an outage today around 5-8am CEST. This – as far as my current findings go – happened due to a filesystem panic. So far, i only experienced such a thing on virtual servers with a certain amount of load.
Even tho relatively speaking, i didn’t feel like there was much to worry about with the current setup and its utilization, i feel like when looking at a potential second wave from Reddit, this instance should be prepared – so i just did a thing:
What we are moving to
We will be moving from a Netcup RS 4000 G9.5 KVM with 32 GB memory and 10 dedicated EPYC cores, (that was meant for various smaller tchncs services such as XMPP, Email, Clouds, Websites and stuff and lots of headroom), to a Hetzner serverauction baremetal thing with an Intel Core i7-7700, 2x SSD M.2 NVMe 512 GB, 64 GB DDR4 Mem, 1 Gbit Networking in Finland.
I plan to migrate today as soon as the thing is ready and configured. There will be a downtime of up to 1 hour.
Lemmy will share its seat on the machine with the tiny Calckey instance that was deployed around the same time.
Increased monthly cost: around 40 EUR
Note on the thing Meta is developing with ActivityPub federation support:
I am not a friend of Meta and am happy to keep it out of my life an off of my servers. However there are lots of rumors and barely any confirmed informations around the project. I will act accordingly once more is known. I also won’t make any exceptions in terms of enforcing our rules. However i won’t participate in signature lists for pre-blocking everything Meta and run around in circles, screaming in panic.
Thank you for your time and effort.
sorry about that, the website is back up – and thanks for considering :)
deleted by creator
For me, it seemed to be all of tchncs at the same time.
deleted by creator
Tchncs offers almost everything federated. :D
everything thats tagged with H1 on the status page, nothing else was affected
sure :)
Excellent position on the Meta thing, Milan! It’s the only sane thing to do at the moment, I think.
Thanks for all the hard work!
deleted by creator
Is there a metrics/grafana page where I can see the bandwidth and compute that the server uses? Do you have insights into that? Where do you see the bottlenecks in the current system?
Not right now, sorry, but i might add something like this (again) in the near future. There was some memory leaky behavior and a crying ZFS of this kind never happened to me on my dedicated hosts so far. Otherwise the previous system was fine… hard to predict the next incoming wave and stuff tho.
Np. Don’t worry about it, I don’t want to hassle you into doing extra (potentially) unnecessary work. Interesting that ZFS was leaking - I have no experience with it, I just heard that ZFS in general requires a lot of RAM to be a reasonable filesystem choice. Are you sure it was a leak, or could it have been e.g. caches filling up as well?
Wohoo, we are back
How much does the server cost all in all monthly (or are the 40€ after the increase and not just the difference between previous and after)
The server that was just rented is around 38 € per month. I am not sure if this was including tax or not… (maybe with a setup fee tho). There are also dynamic costs for media storage, but there are no per-container statistics of this kind… it shouldn’t be much more than an euro or something right now, i don’t know.
…the old server will remain in the tchncs infra.
Was there any data loss caused by the outage? I can’t find a post anymore that I made a while ago. It’s nothing important, I’m just curious.
there shouldn’t be data loss from before the incident…
Seems like I cannot post notes with attached images on Calckey. Text-only posts do work though. @elektronaut@procial.tchncs.de
sorry about that. it should now be fixed
Never mind. Posting notes with attachments works again now. 👍🏻
No never mind needed: missed a little thing in the domain config. Glad that you can confirm it being working now :)
Didnt experience outage but there are some Errors time to time.
