Managing Python Virtual Environments with pyenv

Maros Kukan

Published in

Python in Plain English

9 min readAug 21, 2023

Foreword

Beautiful is better than ugly.
— Zen of Python by Tim Peters

Let me start by admitting that I love working with Python. This love is rooted in the fact that Python as a language is very flexible and versatile. It allows me to go very quickly from an idea to a working prototype. This includes simple automation scripts and more complex web applications. It also comes with community support and ever-expanding list of libraries.

Yet, as the Python ecosystem expands with an abundance of libraries, frameworks, and tools, the challenges of maintaining a coherent and isolated development environment have become increasingly pronounced.

The need to work seamlessly across different projects, each with its unique set of dependencies, often leads to version conflicts, code inconsistencies, and unexpected behavior.

There are number of solutions which can address the above challenges:

Full OS virtualization — resource intensive
Application and runtime containerization — often used in production
Python virtual environments — lightweight and great during development

In this article I would like to take you on a journey where we explore the third option - Python virtual environments.

What are virtual environments

Python virtual environments are isolated and self-contained spaces where you can work on different Python projects with their own sets of dependencies. These environments allow you to create a controlled environment for each project, ensuring that the packages and libraries you install for one project do not conflict with those in another. This helps prevent version conflicts and makes it easier to manage project-specific dependencies.

Virtual environments enable you to have multiple versions of Python and different packages installed on the same system without affecting each other. They are particularly useful when you’re working on various projects simultaneously, each with its own requirements.

Prerequisites

To have smooth experience while following the examples below, I would recommend having a standard Linux Distribution installed.

Environment Setup

The environment for this exercise is very simple and consist of single Fedora 38 virtual machine. We use Vagrant to abstract the underlying VM lifecycle.

💡Tip: Not familiar with Vagrant? Learn how to save time managing local development environments at scale in Managing Dev Environments with Vagrant.

💡Tip: Curious how this virtual machine template was build in the first place? Learn how to automated golden image creation in Automating Golden Image builds with Packer.

# Create new project directory
New-Item -ItemType Directory -Force -Path "$HOME\projects\python-venv"

# Move to this directory
cd $HOME\projects\python-venv

# Download the Vagrant file
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/maroskukan/linux-cookbook/main/Vagrantfiles/fedora38/Vagrantfile" `
                  -OutFile Vagrantfile

📝Note: The Vagrant Box used in this demo is maroskukan/fedora38. This virtual machine template comes with EUFI firmware and supports Microsoft Hyper-V, VMware Workstation and Oracle VirtualBox.

Next, we create and start the virtual machine using vagrant up command. If you have multiple backends (Hyper-V, VMware Workstation, Oracle VirtualBox) available, we can use the --provider option.

We can chain multiple commands together with &&.

# Provision and connect to the Virtual Machine using Default Provider
vagrant up --no-provision && vagrant ssh

Bringing machine 'fedora38' up with 'hyperv' provider...                                                     ==> fedora38: Verifying Hyper-V is enabled...                                                                ==> fedora38: Verifying Hyper-V is accessible...                                                             ==> fedora38: Importing a Hyper-V instance
    fedora38: Creating and registering the VM...
    fedora38: Successfully imported VM
    fedora38: Configuring the VM...
    fedora38: Setting VM Enhanced session transport type to disabled/default (VMBus)
==> fedora38: Starting the machine...
==> fedora38: Waiting for the machine to report its IP address...
    fedora38: Timeout: 120 seconds
    fedora38: IP: 172.30.203.247
==> fedora38: Waiting for machine to boot. This may take a few minutes...
    fedora38: SSH address: 172.30.203.247:22
    fedora38: SSH username: vagrant
    fedora38: SSH auth method: private key
    fedora38:
    fedora38: Vagrant insecure key detected. Vagrant will automatically replace
    fedora38: this with a newly generated keypair for better security.
    fedora38:
    fedora38: Inserting generated public key within guest...
==> fedora38: Machine booted and ready!
==> fedora38: Setting hostname...
==> fedora38: Installing rsync to the VM...
==> fedora38: Rsyncing folder: /cygdrive/c/Users/Maros_Kukan/projects/python-venv/ => /vagrant
==> fedora38: Machine not provisioned because `--no-provision` is specified.
[vagrant@fedora38 ~]$

If everything worked out well, we should land in bash shell which is our starting point.

In the following section, we take a look at the Python environment.

Python Environment

In this section, we are going to first review the current Python environment that Fedora 38 provides. Afterwards we install, configure and test the pyenv tool.

Assessment

Before we start making any changes to our environment, lets quickly review our current Python arsenal.

# Search for files in /usr/bin for which name starts with python*
find /usr/bin -name python* -exec file {} \; 2>/dev/null
/usr/bin/python: symbolic link to ./python3
/usr/bin/python3: symbolic link to python3.11
/usr/bin/python3.11: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=3e6eae34c82de9e112e48289c49532ee80ab3929, for GNU/Linux 3.2.0, stripped

# Verify Python version
python --version
Python 3.11.4

From the above output, we can see that Python 3 binary is indeed available by default. Two symbolic links are also available to make it easier to execute.

Also, let us check if Python package manager (pip) is also available.

# Search for files /usr/bin for which name starts with pip*
find /usr/bin -name pip* -exec file {} \; 2>/dev/null

And we see that pip is not installed in this environment. That’s fine.

Pyenv

Over the years I have used plethora of tools for managing virtual environments. From not using any, through the use of virtualenv and venv to pyenv with pyenv-virtualenv extension.

Pyenv is a versatile and lightweight utility that enables the easy installation and management of multiple Python versions on a single system. This makes switching between different versions very seamless an natural.

Installation

The easiest way to install pyenv is through pyenv-installer. This is a shell script and as with all download scripts I encourage you to review it before execution.

# Install supporting packages
sudo dnf install -y git

# Install pyenv by download and redirecting the installation script
curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

This shell script with leverage git to clone the project repository to $HOME/.pyenv. Since this location is not included in the $PATH variable, we need to update our shell’s configuration in next step.

Configuration

To use pyenv and virtual-env plugin to manage Python version and virtual environments respectively, we first need to update our shell’s configuration file. For bash, that would be ~/.bashrc.

# Create a backup of the .bashrc file with the current date
cp ~/.bashrc{,_$(date +%Y-%m-%d)}

# Update the .bashrc configuration to load pyenv automatically
echo '# Custom configuration for pyenv' >> ~/.bashrc
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc

# Update the .bashrc configuration to load pyenv-virtualenv automatically
echo '# Custom configuration for pyenv-virtualenv ' >> ~/.bashrc
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc

# Apply changes by restarting the current shell
exec "$SHELL"

Validation

Next, verify that pyenv is working by listing default Python versions as well as virtual environments.

# List default Python versions available to pyenv
pyenv versions
* system (set by /home/vagrant/.pyenv/version)

# List all Python virtualenvs found in `$PYENV_ROOT/versions/*'.
pyenv virtualenvs

Updating

It is best practice to update pyenv and plugins from time to time. We can do this by using the pyenv update command.

# Update pyenv and plugins
pyenv udpate

Updating /home/vagrant/.pyenv...
From https://github.com/pyenv/pyenv
 * branch              master     -> FETCH_HEAD
Already up to date.
Updating /home/vagrant/.pyenv/plugins/pyenv-doctor...
From https://github.com/pyenv/pyenv-doctor
 * branch            master     -> FETCH_HEAD
Already up to date.
Updating /home/vagrant/.pyenv/plugins/pyenv-update...
From https://github.com/pyenv/pyenv-update
 * branch            master     -> FETCH_HEAD
Already up to date.
Updating /home/vagrant/.pyenv/plugins/pyenv-virtualenv...
From https://github.com/pyenv/pyenv-virtualenv
 * branch            master     -> FETCH_HEAD
Already up to date.

This concludes the pyenv installation and configuration parts. In the next section, we learn how we can actually manage multiple Python versions.

Managing Python versions

The big benefit that pyenv brings is it allows us to have multiple different Python versions installed at the same time without any conflicts.

Pyenv will download and compile the desired Python version. In order to do that, it needs build tools available. Let us install those first.

# Install build tools
sudo dnf install -y tar \
                    make \
                    gcc \
                    patch \
                    zlib-devel \
                    bzip2 \
                    bzip2-devel \
                    readline-devel \
                    sqlite \
                    sqlite-devel \
                    openssl-devel \
                    tk-devel \
                    libffi-devel \
                    xz-devel \
                    libuuid-devel \
                    gdbm-devel \
                    libnsl2-devel

Now we are ready to install Python version 3.10.12.

# Download and install Python 3.10.12
pyenv install 3.10.12

Downloading Python-3.10.12.tar.xz...
-> https://www.python.org/ftp/python/3.10.12/Python-3.10.12.tar.xz
Installing Python-3.10.12...
Installed Python-3.10.12 to /home/vagrant/.pyenv/versions/3.10.12

And two more, Python 3.9.17 and 3.8.17

# Download and install Python 3.9.17
pyenv install 3.9.17

# Download and install Python 3.8.17
pyenv install 3.8.17

💡Tip: After installing a new Python version, it is good practice to run pyenv rehash. This will ensure that all shim files are updated. Without this, it may happen that for example python3 will point to correct version but python will not.

Once done installing, lets verify the available Python versions.

# List all Python versions available to pyenv
pyenv versions

* system (set by /home/vagrant/.pyenv/version)
  3.8.17
  3.9.17
  3.10.12

To switch to different Python version in current shell we can use the shell option followed by desired version.

pyenv shell 3.9.17


python --version
Python 3.9.17

To switch back to system version, use python shell system command.

To remove a specific Python version, we can use the pyenv uninstall command.

# Remove Python 3.8.17
pyenv uninstall -f 3.8.17

pyenv: 3.8.17 uninstalled

However in practice we usually create a virtual environment and map it to a specific Python version based on our requirements. Let us see how we can do that.

Managing Python virtual environments

The pyenv-virtualenv plugin is included in the base pyenv installation and provides a convenient way for managing Python virtual environments.

Imagine that we have two projects, one called Prod and other called Dev.

For Production we need to run on Python 3.9.17 with some minimum number of application packages installed.

For Development we need to run on newer Python 3.10.12 with some minimum number of application packages installed as well as linting packages installed for development purpose.

Lets transform this though into reality using by using pyenv’s virtualenv.

# Create Production environment
pyenv virtualenv 3.9.17 prod

Requirement already satisfied: setuptools in /home/vagrant/.pyenv/versions/3.9.17/envs/prod/lib/python3.9/site-packages (58.1.0)
Requirement already satisfied: pip in /home/vagrant/.pyenv/versions/3.9.17/envs/prod/lib/python3.9/site-packages (23.0.1)

# Create Development environment
pyenv virtualenv 3.10.12 dev

# List all available environments
pyenv virtualenvs

  3.10.12/envs/dev (created from /home/vagrant/.pyenv/versions/3.10.12)
  3.9.17/envs/prod (created from /home/vagrant/.pyenv/versions/3.9.17)
  dev (created from /home/vagrant/.pyenv/versions/3.10.12)
  prod (created from /home/vagrant/.pyenv/versions/3.9.17)

Next, we create directories for two environments and use the pyenv local command to create a special file .python-version which will contain the environment name.

# Create directories for prod and dev
mkdir {prod,dev}

# Create .python-version file inside prod directory
cd prod
pyenv local prod

# Verify the current environment
pyenv version

prod (set by /home/vagrant/prod/.python-version)


# Create .python-version file inside dev directory
cd ../dev
pyenv local dev

# Verify the current environment
pyenv version

dev (set by /home/vagrant/dev/.python-version)

💡Tip: Pyenv will update the shell prompt to include the environment name. For example (prod) [vagrant@fedora38 prod]$.

While still in dev directory install the pylint package. Afterwards list all installed packages for this environment.

# Install linting packages
pip install pylint

# List packages in the dev environment
pip list
Package           Version
----------------- -------
astroid           2.15.6
dill              0.3.7
isort             5.12.0
lazy-object-proxy 1.9.0
mccabe            0.7.0
pip               23.0.1
platformdirs      3.10.0
pylint            2.17.5
setuptools        65.5.0
tomli             2.0.1
tomlkit           0.12.1
typing_extensions 4.7.1
wrapt             1.15.0

💡Tip: It is best practice to keep all your dependencies in a file, for example requirements.txt that is part of the project directory which is checked in the version control system.

And contrast this with prod environment.

cd ../prod
pip list

pip list
Package    Version
---------- -------
pip        23.0.1
setuptools 58.1.0

I think you get the idea. Finally, let us delete the dev virtual environment.

# Delete dev environment
pyenv virtualenv-delete dev
pyenv-virtualenv: remove /home/vagrant/.pyenv/versions/3.10.12/envs/dev? (y/N) y

This concludes the workflow around managing the virtual environments.

Cleanup

When done exploring the environment, we can clean up by removing the vagrant virtual machine.

# Move to project directory
cd $HOME\projects\python-venv

# Delete the Virtual Machine
vagrant destroy -f

# Remove the project directory
cd $HOME
Remove-Item -Path $HOME\projects\python-venv -Recurse

💡Tip: The Vagrant box template will remain available on your host machine for future use. If you would like to remove it you need to use the vagrant box remove maroskukan/fedora38.

Closing thoughts

In summary, Python virtual environments provide a controlled space where you can install dependencies specific to a project without interfering with the global Python environment or other projects. This isolation prevents conflicts between package versions and ensures that changes made to one project won’t affect others.

I encourage you to start using virtual environments today for a more efficient and organized coding journey. Your Python projects will thank you! Until next time, thank you for reading!

In Plain English

Thank you for being a part of our community! Before you go:

Be sure to clap and follow the writer! 👏
You can find even more content at PlainEnglish.io 🚀
Sign up for our free weekly newsletter. 🗞️
Follow us on Twitter, LinkedIn, YouTube, and Discord.