Skip to main content

Using virtualenv for more than Python projects

Sorry, it's not a complete instruction, just a thought. It occurred to me (some time ago) that Python's virtualenv is, essentially, a simplified version of system "prefix", it has bin, lib, include, and can have more stuff when needed. If you're willing to experiment (you'll probably have to set a few additional environment variables and/or build flags but that's no big deal), you can install various other tools there up until you have a complete system with its own compiler and complete set of libraries although it's much simpler to keep using system compiler and libraries only complimenting them when needed.

Granted, prefixes are nothing new, people were using /opt (and their home directory) this way since the beginning of time. But with little help of virtualenv-wrapper or pyenv you can easily switch between them and isolate environments better. Binaries and stuff installed in virtualenv would override system defaults but only when venv is activated. Sounds like a good thing to me.

What use-cases are there? I'll start with ones that originally occurred to me. First is installing libs and tools in user directory in a controllable way. Sometimes you need to install stuff (especially pythonic stuff) for very basic things. Neovim, for one, needs several Python packages to function normally. The official recommendation is to install them with sudo pip install but I can hardly recommend it, installing anything system-wide with third-party package managers is generally a bad idea. Even though pip installs its packages in a separate site-packages directory, you'll still have a hard time maintaining it, it's hard to list all packages that are installed that way, it's hard to upgrade everything (pip-tools can help but still), it's hard to uninstall stuff without traces. So why not install it all into a neovim-specific virtualenv? Maybe even compile neovim itself there so that you don't have to rely on other repositories (their official PPA is mostly ok, but it's harder in other distributions). Same with any other tools, especially Python-related ones. You can keep requirements files in your dotfiles and put some shims into your ~/bin so you don't have to activate/deactivate all the time (rcm hooks can be used to run the actual install commands, I love rcm, by the way).

Second possible use-case is system libraries. Many Python packages require system libraries (and headers!) to build. lxml, psycopg2, pillow, the list goes on. If you don't have required headers yet, hopefully you'll get some informative error message on install. Or you'll see information in project's README. After a while, you just remember to always install libxml2-dev, libjpeg-dev, python-all-dev, etc. on any new server. But why not just install them into virtualenv? Admittedly, it would require a bit longer time for compilation but you'll at least be sure that you use the same versions of libraries across your systems (ok, this one is less important now that everyone and their father uses docker or some similar tools, and even use of virtualenv is possibly diminished with the rise of containers, although there's no particular reason why you can't containerize virtualenvs... but I digress).

Somewhat related to both, I had an idea to experiment with implementing a window manager and other GUI tools using pygtk (or is it pygobject now? I haven't been following) but installing it into virtualenv is a serious problem. Even if you can do most of the stuff using system's site-packages, things like testing against multiple versions of libraries and python would be a compete mess. Containers would probably help if I get serious about it, but some lighter isolation sounds much better.

It's all might be not very perfect for distributing stuff to users, of course, but when you're developing something or when you can't rely on distribution' repositories for any reason it sounds like a better idea than most others. In particular I hate it when people I don't know and have no reasons to trust say me "run this long and obscure script as root so it installs our stuff deep in your system in a way that makes it hard to remove", user-directory installs which you can easily list/review/update/purge are way safer. So, something to replace those notorious curl -s | sudo bash is yet another use-case. Most new systems now come with python3 and virtualenv anyway. It might even grow up to become another linuxbrew some day (great part is it can deliver binary packages as well so it won't necessarily require much compilation; and probably we'd want to use a separate package index for non-python-related ports) or even replace system package manager at least for some stuff (which would probably require environment multiplexing, meta-packages, and probably better dependencies control), but that's getting ahead of time.

(On a separate note, it would be great to use github as a replacement for pypi. Some trusted build service could be used to build binary packages (and generate a signature containing both git commit hash and binary tarball hash) so pip could verify that package wasn't tampered with and a particular revision of source code was used. Trust doesn't necessarily require hosting actual files yourself. I always thought that hosting a whole linux distribution entirely on github is an interesting idea. Although I'll probably never time time or community. But that's largely unrelated.)

In any case, virtualenv is great and we should do more of it. I think critics who think it's too hacky and complicated and unnecessary just don't understand it.

Popular posts from this blog

Russia becomes an internet outcast for some reason

It's starting to happen more and more often to me. I'm trying to use some service and all of a sudden I see an error page, sometimes a custom one but more often generated by cloudflare or a similar service, with error message effectively saying that IP addresses from Russia are banned on the service. And what is even more interesting, most of this services don't have any particular policy reasons not to allow Russian users, it's not not like they are publishing prescribed materials or doing anything interesting whatsoever. A payment processor, a popular online course provider, some random forums and information sites, one site with educational videos that I like (that one is especially funny because I'm banned from the subscription page and so can't pay them but their cdn does't care and I can actually download any video I want). But why does it happen? Is there some recommendation in some popular security guideline "block Russia, Cambodia, and Afghan

ORICO — small things that stand out

More than once I've written about stuff made by Big Brand manufacturers so I decided to balance things out by writing about a mostly-unknown but good things. In short, if you need something small like HDD enclosure or some cable or USB hub or a dongle of some kind, check if ORICO makes it and chances are you won't be disappointed. It's usually slightly more expensive than low-end stuff from aliexpress but, in my opinion, it's usually worth it. Disclaimer. This post is a shameless promotion of their brand although, unfortunately, nobody paid me for it or even sent me any free stuff. Guys from ORICO and not, if you read this, I'll gladly review whatever free goodies you send me :) I only do honest reviews so better make sure to send the good stuff. At this point I own a USB hub, three different 2.5" HDD enclosures, and a precision screwdriver set from them. And probably something else I don't remember. Here are the pics: USB hub in its natural habitat

Terrible experience with bluetooth headsets and what to do about it

If you own a Bluetooth A2DP headset and any kind of personal computer, good chance is you tried connecting them together. Isn't it nice to watch movies and listen to music free from cluttering wires? Well, good chance is you hated that experience and if that was the reason for buying the headset, you might've even considered throwing it away (or using it only with your phone). Why do they suck so much? Everything points to the fact that it's not a hardware problem. The same headset most likely will work flawlessly paired with an android device or even the same computer under a different operating system (windows users report huge difference between, like, 8.0 and 8.1, and not in favor of the latter, surprisingly, I think only mac users report good stuff about their macbooks and beats although I didn't try it myself so they might all be faking it :) ). And most likely it's not even the drivers or other low-level stuff, android and desktop linux has mostly the same