Hey everyone. If you want to post links or discuss the Reddit blackout, please localize it to this thread in order to keep things tidy!
Hey everyone. If you want to post links or discuss the Reddit blackout, please localize it to this thread in order to keep things tidy!
Too much load? Reddit is down.
Not enough load? Believe it or not, also down.
I’d love to know what it is about subreddits going private that caused issues.
Maybe some overload caused by a process having to dig deeper to find best/top posts?
apparently that’s exactly the case.
That is an interesting aspect no engineer could have foreseen!
You’d be surprised how much critical infrastructure was implemented through trial and error and has just been left like that for years…
Anything less than 99% of infrastructure working that way would be surprising. Everything is held together with scotch tape and scotch whisky.
I’ll be sure to repeat that last line to my fellow team members :D
I like this idea. I imagine that with the top subs being dark the automated top posts that get scrounged up may be too terrifying for the front page and they hit the panic button while they scramble to curate through the absolute worst filth they’ve ever seen.
“It’s merely coincidence. But starting Wednesday, our servers will be more robust and you can browse the site using our official app.” - Spez, while sniffing a decanter of human shit
God we need indefinite blackouts.
It’s entirely possible that they’ve made some assumptions about what a “normal” level of traffic looks like when writing code for their backend, which has caused some things to break when that has changed.
Not our fault if their code is shit.
How is that an example of bad code?
Honestly, it’s probably not - if I’m actually right this is likely an issue that Reddit’s engineers never predicted would happen so never planned for it. I was being hyperbolic.
It’s not reactive. A proper reactive system can handle fluctuations in usage patterns more robustly.
I’m having a hard time believing the claim that Reddit’s code isn’t reactive.
Wouldn’t be surprised if it’s just a gigantic mess of nested if-else statements.
Gotos all the way down
Maybe, but this was a huge increase in usage. Reddit never expected to deal with anywhere near thousands of subs going private simultaneously.
Tildes’ dev Deimos used to work for Reddit and had this guess https://tildes.net/~tech/163e/reddit_appears_to_be_down_during_blackout_day_1#comment-87v1.
The servers run on the tears of bitter whiny CEOs.
Reddit is hosted on AWS after all…
They’re lying. Fish swim, birds fly, sun shines, Reddit lies.
Probably a drop in usage flagged some internal test
Another comment suggested that whatever cluster management automation they have in place may be configured to suspend services during low load to reduce costs. But if usage drops unexpectedly low then you enter a state that has never been tested. It may be that some services are (incorrectly) suspended across the board and causing failures.
deleted by creator
Want Free API? Straight to down status.
Want cheaper API? Also straight to down status
Not enough people on Reddit because of protests? Also straight to down status
This comment is so good an upvote won’t do justice (without awards, a classic comment such as this now has some merit… it’s a new day boys & girls, a good day)
If Beehaw offered awards I would actually buy them, at least the money would be going towards keeping the lights on for a project that isn’t actively trying to screw over users for profit.
Give them some gold. Oh wait…
Rebelling moderators, we have a special jail for rebelling moderators.
thank you, this comment made my day
Lol, this made me chuckle out loud. Good job Sausage man!