When I was a Brit in school my teachers would often say to me ‘take your hands out of your pockets while speaking to me!’, so its more an etiquette thing rather than being rude, I’d say.