Biggest problem with it is that it lies with the exact same confidence it tells the truth. Or, put another way, it’s confidently incorrect as often as it is confidently correct - and there’s no way to tell the difference unless you already know the answer.
That’s the way. I’ve been programming for nigh on four decades, and it’s almost a daily occurrence with junior devs going to stack overflow or chatGPT to solve an issue instead of just searching the code where nine times out of ten the problem (or a very similar one) is already solved.