Culinarily, you can define vegetables using the basic food groups. Grains and starches are a distinct group and not part of the vegetable food group, despite the fact that they come from plants. It is easy to see that not all food that is plant-based is, culinarily, a “vegetable” when you consider things like fruits and nuts, which people have no trouble distinguishing from vegetables.

And yes, many things we culinarily consider vegetables actually fall under the scientific definition of fruit, and some “fruits” do not fall under that definition.