The reason you do stuff in a venv is to isolate that environment from other python projects on your system, so one Python project doesn’t break another. I use Docker for similar reasons for a lot of non-Python projects.

A lot of Python projects involve specific versions of libraries, because things break. I’ve had similar issues with non-Python projects. I’m not sure I’d say Python is particularly worse about it.

There are tools in place that can make the sharing of Python projects incredibly easy and portable and consistent, but I only ever see the best maintained projects using them unfortunately.

No, it’s not just you, Python’s tooling is a mess. It’s not necessarily anyone’s fault, but there are a ton of options and a lot of very similarly named things that accomplish different (but sometimes similar) tasks. (pyenv, venv, and virtualenv come to mind.) As someone who considers themselves between beginner and intermediate proficiency in Python, this is my biggest hurdle right now.

Python is the only programming language that has forced me to question what the difference is between an egg and a wheel.

Python developer here. Venv is good, venv is life. Every single project I create starts with

python3 -m venv venv

source venv/bin/activate

pip3 install {everything I need}

pip3 freeze > requirements.txt

Now write code!

Don’t forget to update your requirements.txt using pip3 freeze again anytime you add a new library with pip.

If you installed a lot of packages before starting to develop with virtual environments, some libraries will be in your OS python install and won’t be reflected in pip freeze and won’t get into your venv. This is the root of all evil. First of all, don’t do that. Second, you can force libraries to install into your venv despite them also being in your system by installing like so:

pip3 install --ignore-installed mypackage

If you don’t change between Linux and windows most libraries will just work between systems, but if you have problems on another system, just recreate the whole venv structure

rm -rf venv (…make a new venv, activate it) pip3 install -r requirements.txt

Once you get the hang of this you can make Python behave without a lot of hassle.

This is a case where a strength can also be a weakness.

You re not stupid, python’s packaging & versionning is PITA. as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem

Python never had much of a central design team. People mostly just scratched their own itch, so you get lots of different tools that do only a small part each, and aren’t necessarily compatible.

I agree. Python is my language of choice 80% or so of the time.

But my god, it does packaging badly! Especially if it’s dependent on linking to compiled code!

Why it is like that, I couldn’t tell. The language is older than git, so that might be part of it.

However, you’re installing python libraries from github? I very very rarely have to do that. In what context do you have to do that regularly?

It’s something of a “14 competing standards” situation, but uv seems to be the nerd favourite these days.

Yes it’s terrible. The only hope on the horizon is uv. It’s significantly better than all the other tooling (Poetry, pip, pipenv, etc.) so I think it has a good chance of reducing the options to just Pip or uv at least.

But I fully expect the Python Devs to ignore it, and maybe even make life deliberately difficult for it like they did for static analysers. They have some strange priorities sometimes.

With all the hype surrounding Python it’s easy to forget that it’s a really old language. And, in my opinion, the leadership is a bit of a mess so there hasn’t been any concerted effort on standardizing tooling.

Some unsolicited advice from somebody who is used more refined build environments but is doing a lot of Python these days:

The whole venv thing isn’t too bad once you get the hang of it. But be prepared for people to tell you that you’re using the wrong venv for reasons you’ll never quit understand or likely need to care about. Just use the bundled “python -m venv venv” and you’ll be fine despite other “better” alternatives. It’s bundled so it’s always available to you. And feel free to just drop/recreate your venv whenever you like or need. They’re ephemeral and pretty large once you’ve installed a lot of things.

Use “pipx” to install python applications you want to use as programs rather than libraries. It creates and manages venvs for them so you don’t get library conflicts. Something like “pip-tools” for example (pipx install pip-tools).

Use “pyenv” to manage installed python versions - it’s a bit like “sdkman” for the JVM ecosystem and makes it easy to deal with the “specific versions of python” stuff.

For dependencies for an app - I just create a requirements.txt and “pip install -r requirements.txt” for the most part… Though I should use one of the 80 better ways to do it because they can help with updating versions automatically. Those tools mostly also just spit out a requirements.txt in the end so it’s pretty easy to migrate to them. pip-tools is what my team is moving towards and it seems a reasonable option. YMMV.

I’ve started using poetry and the experience has improved.

Python is hacky, because it hacks. There’s a bunch of ways you can do anything. You can run it on numerous platforms, or even on web assembly. It’s not maintained centrally. Each “app” you find is just somebodies hack project they’re sharing with you for fun.

How to improve Python packaging, or why fourteen tools are at least twelve too many

Still relevant.

Yeah the tooling sucks. The only tooling I’ve liked is Poetry, I never have trouble installing or packaging the apps that use it.

The difficulty with python tooling is that you have to learn which tools you can and should completely ignore.

Unless you are a 100x engineer managing 500 projects with conflicting versions, build systems, docker, websites, and AAAH…

• you don’t really need venvs
• you should not use more than on package manager (I recommend pip) and you should cling to it with all your might and never switch. Mixing e.g. conda, on linux system installers like apt, is the problem. Just using one is fine.
• You don’t “need” need any other tools. They are bonuses that you should use and learn how to use, exactly when you need them and not before. (type hinting checker, linting, testing, etc…)

Why is it like this?

Isolation for reliability, because it costs the businesses real $$$ when stuff goes down.

venvs exists to prevent the case that “project 1” and “project 2” use the same library “foobar”. Except, “project 1” is old, the maintainer is held up and can’t update as fast and “project 2” is a cutting edge start up that always uses the newest tech.

When python imports a library it would use “the libary” that is installed. If project 2 uses foobar version 15.9 which changed functionality, and project 1 uses foobar uses version 1.0, you get a bug, always, in either project 1 or project 2. Venvs solve this by providing project specific sets of libraries and interpreters.

In practice for many if not most users, this is meaningless, because if you’re making e.g. a plot with matplotlib, that won’t change. But people have “best practices” so they just do stuff even if they don’t need it.

It is a tradeoff between being fine with breakage and fixing it when it occurs and not being fine with breakage. The two approaches won’t mix.

very specific (often outdated) version of python,

They are giving you the version that they know worked. Often you can just remove the specific version pinning and it will work fine, because again, it doesn’t actually change that much. But still, the project that’s online was the working state.