The most reasonable to me is that megafauna come and wreck your farms, so you have to hunt them to extinction before farming is viable. Humans can stop a horse from eating their crops, but stopping an elephant or mammoth is not easy at all, and we see that humans hunted those to extinction right before they started agriculture.
There used to be elephants and mammoths all over the world, but almost all places where humans developed agriculture they had already killed all of them.
All the farming for millenia on the Indian subcontinent and southeast Asia in general seems to disagree with that. The Indus Valley civilization certainly had elephants around.
And African elephants form enormous herds, there used to be herds of hundreds of elephants roaming around, no way stone age level societies can defend against those all you can do is try to avoid them.
It is really hard to kill an elephant, you need a lot of people to do it or big traps. You can lookup videos of people doing that, you have like 100 people standing around throwing spears at it as it slowly bleeds to death and that is when its alone. If its a herd you can't even do that.
So when farming requires you to have a big hunting party constantly ready to take down an elephant, why not just live off elephant meat? The farm doesn't add any value and is very difficult to start.
There used to be elephants and mammoths all over the world, but almost all places where humans developed agriculture they had already killed all of them.