Oh gosh, I’m so sorry… 😔 Meanwhile our backup drive is showing ext4 block errors so I’ve been checking it over. SMART says everything is good but I’m checking the integrity of the backup VM images just to be sure. Going to setup a crontab to start rsync’ing them to a spare SSD I have in the server box on a regular basis.
After checking the logs I realized that drive has been spitting bad block error for the last two months but I never noticed it, not even checking the SMART numbers. This one is full on me :(. Check your logs and backup, folks.
A lot of the stuff that I absolutely cannot lose at all are backed up on a private Gitlab repo. SMART unfortunately cannot predict when a hard disk mechanically fails. It could happen instantaneously without giving you any signs whatsoever.
It sucks because redundancy is expensive if you actually invest in a RAID setup. And it’s not even considered a backup solution since filesystem corruption can happen, accidental deletions can fuck you over, etc.
It sucks because redundancy is expensive if you actually invest in a RAID setup.
Yep, buying that 2nd 10TB drive for the media center was painful… but less painful than it’d be to rip all those Blu-rays and torrent junk from nyaa again, so it was still worth it. And of course mdadm is working great.
Evidently that backup image was corrupted when I checked it in Proxmox Backup Server’s panel. Redid the backup and rechecked it and it said all was fine. I added that crontab that automatically moves those files to a spare SSD I have sitting around on the server.
I think that minor corruption happened because I originally had the hard disk mounted on the hypervisor level before it gets pulled away and passed through to the VM when it boots up. That’s what I’m hoping to be the case anyways.
I used to have it set to backup full VM images on Proxmox itself. Now I remotely backup the files to a proxmox backup server that I run in a VM for which I passed the hard disk through to, because the backups there can be deduplicated and take soooo much less space.
I’ll check on it here in a week or so to see if the VM image backup ends up corrupted again. I need to find a way to have it ping me if a backup fails too. Currently Proxmox only supports email afaik, but I think I can write a bash script that polls my logs and then pings my phone using ntfy or whatever is the cool new hip software in town for notification needs.
Oh gosh, I’m so sorry… 😔 Meanwhile our backup drive is showing ext4 block errors so I’ve been checking it over. SMART says everything is good but I’m checking the integrity of the backup VM images just to be sure. Going to setup a crontab to start rsync’ing them to a spare SSD I have in the server box on a regular basis.
After checking the logs I realized that drive has been spitting bad block error for the last two months but I never noticed it, not even checking the SMART numbers. This one is full on me :(. Check your logs and backup, folks.
A lot of the stuff that I absolutely cannot lose at all are backed up on a private Gitlab repo. SMART unfortunately cannot predict when a hard disk mechanically fails. It could happen instantaneously without giving you any signs whatsoever.
It sucks because redundancy is expensive if you actually invest in a RAID setup. And it’s not even considered a backup solution since filesystem corruption can happen, accidental deletions can fuck you over, etc.
Yep, buying that 2nd 10TB drive for the media center was painful… but less painful than it’d be to rip all those Blu-rays and torrent junk from nyaa again, so it was still worth it. And of course mdadm is working great.
Forgot to give an update:
Evidently that backup image was corrupted when I checked it in Proxmox Backup Server’s panel. Redid the backup and rechecked it and it said all was fine. I added that crontab that automatically moves those files to a spare SSD I have sitting around on the server.
I think that minor corruption happened because I originally had the hard disk mounted on the hypervisor level before it gets pulled away and passed through to the VM when it boots up. That’s what I’m hoping to be the case anyways.
I used to have it set to backup full VM images on Proxmox itself. Now I remotely backup the files to a proxmox backup server that I run in a VM for which I passed the hard disk through to, because the backups there can be deduplicated and take soooo much less space.
I’ll check on it here in a week or so to see if the VM image backup ends up corrupted again. I need to find a way to have it ping me if a backup fails too. Currently Proxmox only supports email afaik, but I think I can write a bash script that polls my logs and then pings my phone using ntfy or whatever is the cool new hip software in town for notification needs.