Marc Richter: Using pyenv to manage your Python interpreters

This article was published at Using pyenv to manage your Python interpreters .
If you are reading this on any other page, which is not some “planet” or aggregator, you are reading stolen content. Please read this article at its source, which is linked before to ensure to get the best reading experience; thank you! ❤

When I started to learn Python a few years ago, I often wondered about what’s the “correct” or “best” way to prepare your system’s Python environment for the requirements your software project or some Python-based application you’d like to start using may have: Should I install modules using the package manager of my OS? Or by using Python tools for it like pip? What are “virtual environments” and how do I utilize these for my projects? What’s all this pyenv, pip, pipenv, easy_install, setuptools, anaconda, conda, miniconda …

In this article series, I’d like to introduce the most common tools and techniques on how to do this in the Python world.
At the end of the series, I will share some of my thoughts, doubts, and questions I had back then, tell about some experiences I gathered in the meantime and generally share the outcome of this journey and what my Python-Workflow looks like, nowadays.

Introduction to pyenv

This first article is about pyenv, a lightweight, yet powerful, Python version management tool that works in user – scope and does stay out of the way of systems global Python interpreters.

Installing pyenv

On my development workstations, the first thing I usually do when preparing my Python development environment is to install pyenv. pyenv lets you easily install and switch between multiple versions of Python. There are no administrative permissions required, since everything happens in your user context and

$HOME

  directory (

~

). This way, it is even an option for multi-user environments like a shared system at work or one of those you get when renting cheap hosting for your homepage, in which you do not have root permissions.
It can be installed with one single command like this (make sure to met prerequisites and build dependencies first):

curl https://pyenv.run | bash

Without any changes, this clones the very latest version of pyenv into the directory

~/.pyenv

.
Next, this needs to be loaded in any newly launched shell. pyenv is sharing advice on how to do this on its own just after the install-command has been executed:

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="/home/mrichter/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Note that the PATH defined is prefixed with this new folder. This way, it has precedence over any global interpreter version installed in a different location (like those installed by the PMS) for your own shell only, making sure that executing

python

and

pip

 named commands will always use the versions pointed to from pyenv.

As soon as that code has been entered into your shell configuration file, spawn a subshell to have these loaded in your current environment:

$ exec $SHELL

Now the command

pyenv

is available in your

PATH

, as well as any Python interpreter installed by it in the future.
It is recommended to search for updates right away; even when there should be none after a fresh install we just did, since you see that

pyenv

is working fine at least:

mrichter@ww-arcade:~$ pyenv update
Updating /home/mrichter/.pyenv...
...

Using pyenv to install some Python interpreters

Let’s install a few Python interpreters, shouldn’t we?

“A few?!?” – I can hear you say, already
Yeah – welcome to the easy-as-f**k – world of pyenv! Let’s go for having these versions as an example:

• 2.7.17
• 3.6.9
• 3.8.0

For a list of all available interpreters, execute

pyenv install --list

.

Thanks to pyenv, this is as easy as this now:

$ pyenv install 2.7.17
Downloading Python-2.7.17.tar.xz...
-> https://www.python.org/ftp/python/2.7.17/Python-2.7.17.tar.xz
Installing Python-2.7.17...
Installed Python-2.7.17 to /home/mrichter/.pyenv/versions/2.7.17

$

BAM – you now have Python 2.7.17 available in your environment! And none is cluttering the global environment since they got installed to

~/.pyenv/versions

.
Repeat that for the formerly mentioned versions and you are done.

Watch out for the dependencies!

Note: You still need to provide Python’s build dependencies for your OS yourself! If you see something like this during interpreter installation, you are most certainly missing any of them (

zlib

 in this example):

$ pyenv install 3.7.5
Downloading Python-3.7.5.tar.xz...
-> https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tar.xz
Installing Python-3.7.5...

BUILD FAILED (Ubuntu 18.04 using python-build 20180424)

Inspect or clean up the working tree at /tmp/python-build.20191126215827.6156
Results logged to /tmp/python-build.20191126215827.6156.log

Last 10 log lines:
    sys.exit(ensurepip._main())
  File "/tmp/python-build.20191126215827.6156/Python-3.7.5/Lib/ensurepip/__init__.py", line 204, in _main
    default_pip=args.default_pip,
  File "/tmp/python-build.20191126215827.6156/Python-3.7.5/Lib/ensurepip/__init__.py", line 117, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/tmp/python-build.20191126215827.6156/Python-3.7.5/Lib/ensurepip/__init__.py", line 27, in _run_pip
    import pip._internal
zipimport.ZipImportError: can't decompress data; zlib not available
Makefile:1141: recipe for target 'install' failed
make: *** [install] Error 1
$

Installing them by adding the appropriate “source” URI in Ubuntu plus executing

apt-get build-dep python3.6

, as the Python’s build dependencies page suggests, solves this issue in Ubuntu Linux. Please help yourself with any other OS distribution.

Switching between Python Interpreters

You now have several Python interpreters available – and now? How to use and switch between them? Seems a bit unhandy to change all your shebangs to something like

~/.pyenv/versions/3.8.0/bin/python

, isn’t it?

Don’t worry – it’s far easier than that!
Let’s say, you are about to start your Python project in the empty/new directory

~/project1

. All you need to do is to switch to that directory and enable the interpreter of your choice “locally” (explained in a minute) using pyenv:

~$ cd ~/project1
~/project1$ ls -al
total 0
drwxrwxrwx 1 mrichter mrichter 512 Nov 26 23:39 .
drwxr-xr-x 1 mrichter mrichter 512 Nov 27 00:01 ..
~/project1$ python -V
Python 2.7.15+
~/project1$ pyenv local 3.6.9
~/project1$ python -V
Python 3.6.9
~/project1$ ls -al
total 0
drwxrwxrwx 1 mrichter mrichter 512 Nov 27 00:18 .
drwxr-xr-x 1 mrichter mrichter 512 Nov 27 00:01 ..
-rw-rw-rw- 1 mrichter mrichter   6 Nov 27 00:18 .python-version
~/project1$ cat .python-version
3.6.9
~/project1$

How cool is that?? You can switch your pre-installed environments now with a single command, that only creates a single text file, called

.python-version

in your projects folder!
That pretty much reduces the shebang – issue to use

#!/usr/bin/env python

for all your scripts; no matter which version your projects are using and without the need to change it for future version changes

About pyenv scopes

Note: pyenv supports two scopes:

1. local
   The scope of the current project/directory and its subdirectories is called “the local scope”. When you change the directory to a directory for which no local version is defined, this scope automatically changes to what is defined globally.
   As shown in a minute, this scope’s version can be set using the

   pyenv local

   command.
2. global
   The scope if no specific local scope has been defined (default) is called “the global scope”. It is defined in the file

   ~/.pyenv/version

   and can be set and changed using the

   pyenv global

     command.

The difference of these defined scopes is shown in this short shell-session:

mrichter@ww-arcade:~/test$ pyenv versions
* system (set by /home/mrichter/.pyenv/version)
  2.7.17
  3.6.9
  3.7.5
  3.8.0
mrichter@ww-arcade:~/test$ pyenv global
system
mrichter@ww-arcade:~/test$ pyenv local
pyenv: no local version configured for this directory
mrichter@ww-arcade:~/test$ python -V
Python 2.7.15+
mrichter@ww-arcade:~/test$ /usr/bin/python -V
Python 2.7.15+
mrichter@ww-arcade:~/test$ pyenv local 3.8.0
mrichter@ww-arcade:~/test$ python -V
Python 3.8.0
mrichter@ww-arcade:~/test$ cd ..
mrichter@ww-arcade:~$ python -V
Python 2.7.15+
mrichter@ww-arcade:~$ cd test/
mrichter@ww-arcade:~/test$ python -V
Python 3.8.0
mrichter@ww-arcade:~/test$ mkdir subdir
cd mrichter@ww-arcade:~/test$ cd subdir/
mrichter@ww-arcade:~/test/subdir$ python -V
Python 3.8.0
mrichter@ww-arcade:~/test/subdir$

• In the output of

  pyenv versions,

  we see all Python interpreters installed by pyenv. The global one (default) is marked with an asterisk (*);

  system

  in this example.

  • system

    is a special case: It circumvents pyenv installed interpreters and results in the executable

    python

    found in the remaining parts of the

    $PATH

    variable. This means, that it normally resolves to the Python interpreter that didn’t get installed by pyenv but by your PMS or similar.

• pyenv global

  again shows that global scope is set to

  system

  .

• pyenv local

  prints what the current directory’s local scope is set to (nothing so far; falling back to global).
• The first 

  python -V

  shows the version of Python in effect. It is the system’s default version of Python (

  2.7.15+

  ).

  • /usr/bin/python -V

    shows that this is identical to the full path to the system’s default

    python

    executable.

• pyenv local 3.8.0

  sets the local scope to be Python 3.8.0, as the next execution of

  python -V

  shows.
  Note that this very same command yielded a different result before we set the local scope!
• As soon as we leave the directory again, we are back to the global scope’s version

  2.7.15+

  .
• Going back into the project directory with its local version defined (containing the file

  .python-version

  ) immediately enables that locally defined version again, without anything else to care about.
  • This even stays active for subdirectories.

This is pretty much all you need to know about pyenv to get started! It covers 100% of what I need in my day-to-day work, so there should not be missing too much in order to get you started with it
Feel free to investigate additional details from the project’s documentation resources.

Next time

I hope you enjoyed this first article!
I always welcome comments of any kind, so feel invited to leave yours in the comment section below

In the next part of this series, I will introduce pipenv – an advanced package-, dependency- and virtualenv-manager for Python.

Stay tuned to get your head wrapped around the workflow this awesome piece of software brings to your Python dependency management workflow!

Marc Richter

Born in 1982, Marc Richter is an IT enthusiastic since 1994. He became addicted when he first put hands on their family’s pc and never stopped investigating and exploring new things since then.
He is married to Jennifer Richter and proud father of two wonderful children, Lotta and Linus.
His current professional focus is DevOps and Python development.

An exhaustive bio can be found at this blog post.

Found my articles useful? Maybe you would like to support my efforts and give me a tip then?

Marc Richter's personal site - About Linux, programming in Python and Music

from Planet Python
via read more