No manner of handwashing will prevent this. Skin sloughs off pretty constantly and over-washing will spur overproduction of sebum. Some people produce more than others.

But it is a gaming controller and there’s some likelihood of food being involved in the area, so you’re not entirely wrong.