These AI systems do make up bullshit often enough that there’s even a term for it: Hallucination.

Kind of a euphemistic term, like how religious people made up the word ‘faith’ to cover for the more honest term: gullible.