Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

But why is it such a big problem in Python compared to other stacks? Why does all python projects end up depending on you having those exact tools of things installed locally and the planets aligned a certain way, when other stacks do not?


It's not a big problem in Python in general, only in scientific computing / number crunching projects, because of the dependencies on huge complex software, some of it ancient, written in C, Fortran, and C++. So why do we hear about this problem in Python a lot? Well, because it's what's used for the glue/frontend, which is what users work with directly. It's selection bias. Sure, another language might fare somewhat better or worse for this or that reason, but at the end of the day it's gonna be a pain in the ass (at least until next-generation, complete, deterministic, language-agnostic solutions like Nix/GUIX really gain traction).


There are 365k projects in the "official" package index. While not all of these are important, it's a tip-off to the magnitude of the problem. The habit of blowing past a problem by grabbing a random library and moving on to the next problem leaves us with a mess of dependencies. And many of those were either written by amateurs like us, not maintained, etc.

Maybe other languages have fewer libraries, or maybe the habit of grabbing libraries at random evolved concurrently with the rise of Python.

My team has a rule that we don't let a project get past a certain stage without proving that it can be installed and run on a clean machine and archiving all of the necessary repo's with the project. It's easily forgotten that testing your installer is part of testing your program.


I'm not sure that it is a problem in Python more than other languages.

It might look worse because many Python projects use tools such as CUDA, which are notoriously dependent on the specific OS, architecture, method of installation etc. But that same issue will exist in most languages - if you're linking against CUDA, you will sometimes have problems with the package installation. Particularly if you try to run the code on a different OS, CPU architecture, using a different GPU, etc.

I don't think it really has anything to do with Python. It just happens that most people doing work that depends on tricky packages such as CUDA also happen to be using Python.


Python is for all intents and purposes a "glue" language. You don't do the heavy computing in Python, you just pull in a C++ library that has a Python interface. This adds a ton of friction because these dependencies will often not be precompiled so you need to have the right system libraries to build the module before using it.

It's not much of a problem for other stacks because they either are fast enough that they have a library written in the same language for problem X (C#/Java/Rust) or they aren't targeting the same type of work (JS, Ruby, etc...). C++ has the exact same problem as Python and I'd argue that it's even worse.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: