Why are people building their software in Python when they need performance? There are other languages that excel at this. Python strengths lie elsewhere and they are increasingly being diminished with all these "upgrades". Splitting the ecosystem with async was already a major blow and now we are looking at another split along gil/non-gil axis. It's just sad.
Python is the de-facto default and main language in data science and ML community. You start because it is easy and have a great ecosystem but then after some time you start to worry about performance at some point. And it takes time, efforts and money to re-write in rust,C++.. etc.
I think running inference engines is one particular case of that. You can train and tweak your model using python with its wonderful pytorch and numpy. Then at some point when scale and performance matter then this becomes a potential problem.
To be a bit pedantic, I guess may be it is fairer to say python is the main glue language for scientific computing where DS and ML are part of. The “big bang” seems to have started from Numpy/Scipy and friends and cascading from there.
Companies like Facebook also have huge interests in speeding up python as they have a lot of stacks written in python.
Mostly projects / products that start off with very low usage thus python is perfectly fine (Why overoptimize). And then it becomes useful - and a rewrite isn't worth it.
How is that Python’s problem though? If you paint yourself into a corner by choosing the wrong language then eat the rewrite or eat the hardware costs.
There was a consensus after the Python 3 debacle of “No major breaking changes”. We seem to have lost that because of moneyed interests and that’s sad.
That's such a strangely distorted world view. If a car company releases a, idk, trunk-extension in response to customer feedback would you go "How is this Ford's problem? If you didn't think about the trunk size eat the loss and buy a new car" ? Python developers want python to remain useful to the developers who want to keep using the language. It's not an incomprehensible motivation.
If I attach something to my car it doesn’t affect your car.
Who does no-GIL benefit? For the majority who use Python for single threaded code, no-GIL will make their code slower because a thread-safe interpreter is always going to be slower than one that can assume ST operation.
For the minority who want parallelism, there’s two other options: OS processes and subinterpreters. If you can use either of these then you will get better performance with a GIL for the same reason.
So no-GIL will only be faster for a minority of a minority who want parallelism but can’t use OS processes or subinterpreters.
Meanwhile everybody else writing libraries has to make sure that their code is no-GIL safe, to support this tiny minority, and if no-GIL ever becomes default then everybody else has to do something to turn it off.
> If I attach something to my car it doesn’t affect your car.
Yes, that was my point.
> For the majority who use Python for single threaded code, no-GIL will make their code slower because a thread-safe interpreter is always going to be slower than one that can assume ST operation.
I'm almost sure the python developers said that they will compensate the slow down with other optimizations, so that you'd never have single-threaded performance degradation version-to-version.
> So no-GIL will only be faster for a minority of a minority who want parallelism but can’t use OS processes or subinterpreters.
One would hope to 1. open new use cases for python thus attracting developers that would have otherwise not given the language consideration and 2. other users could benefit from new optimizations that could be implemented further down the dependency stack.
Of course there's no guarantee that that will materialize, but the idea the adding support to an established, lightweight and well-supported concurrency primitive is so obviously a "stupid idea" shows to me that your (rudely expressed) opinion is entirely self-centered and nearsighted.
I might add that the move from python 2 to 3 was incredibly painful, but I assume most agree (with the benefit of hindsight) that it was entirely correct.
