You can turn off swipe on the remote:

Go to Settings > Remote and Devices > Click Pad

Set it to “click only.”

Now you use the directional buttons, and the center is just a button. Use triple-clicks on arrows for some things, like closing apps.

I feel like the new update has fixed the swiping issue for me though. It feels like it has better “brakes”. Might just be my imagination though.