From this post, I got the idea to remove and replace the wheels altogether using a hacksaw.
Here’s what it looks like right now. My problem now is that I can’t remove the leftover metal shaft from the caster wheel.
I’ve tried using some lubricant and hammering it with a screwdriver but it didn’t budge.
I suspect a press is your best option, which is really a slightly more controlled variation of your hammering it with a screwdriver effort.