Python Development Environment set up with Virtual Environment

Real world applications requires number of services, micro services with different service having different kind of dependencies. So It is very essential to setup separate development environment for separate projects and even services to make the development flexible. Otherwise you may and in wiping out the OS to clear out the mess created by conflicting dependencies :)

There are couple of options to isolate the development environments while working on python. I mostly use Docker and Venv depending on the requirement for creating development environments for my projects.

PS: venv is a package shipped with Python 3, which you can run using python3 -m venv (although for some reason some distros separate it out into a separate distro package, such as python3-venv on Ubuntu/Debian). It serves a similar purpose to virtualenv, and works in a very similar way, but it doesn't need to copy Python binaries around (except on Windows). Use this if you don't need to support Python 2. At the time of writing, the Python community seems to be happy with virtualenv and I haven't heard much talk of venv.

Docker vs Venv

1. Docker is isolation of the OS itself whereas Venv is isolating only the python language and modules while sharing the same OS. So you can fiddle with the OS itself and create versions of them.Docker gives you portability from one machine to another.
2. With Venv you will have to completely redo the installation in another place, unless you want to manually copy all the files and folder. In case of docker the portability becomes very easier.

While Docker seems a go-to option for production phase, I find it simpler to carry development with Venv when you only need separate versions of python packages and versions.

Setting Up the Venv

Installation

Before installing Venv for python, we need few pre-requisites to be fulfilled. First, we need to install build-essential, libssl-dev, libffi-dev and python-dev packages.

sudo apt-get install build-essential libssl-dev libffi-dev python-dev

Pip package has to be installed.

sudo apt-get update && sudo apt install python3-pip

Now, Python3-venv package can be installed which will create virtual environment for python.

sudo apt install -y python3-venv

Setting up a new virtual environment

Now we are ready to set-up a new virtual environment. We will go through an example where I currently used Venv to create a deep learning development environment using recently released tensorflow2.

First, we create a directory for our new environment and go to that location.

mkdir tf2 && cd tf2

Now, we will be creating a new virtual environment where our Python programs can be written and projects can be created.

python3 -m venv tf2

Now if we visualise, the directory structure at the current location upto level 2:

tree -L 2 tf2
op:-
tf2
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── easy_install
│ ├── easy_install-3.6
│ ├── pip
│ ├── pip3
│ ├── pip3.6
│ ├── python -> python3
│ └── python3 -> /usr/bin/python3
├── include
├── lib
│ └── python3.6
├── lib64 -> lib
├── pyvenv.cfg
└── share
 └── python-wheels

Now when we want to use the new environment, first it is needed to be activated.

source tf2/bin/activate

Now the tf2 environment is activated, we can install tensorflow2 here.

sudo apt update
sudo apt install python3-dev python3-pip (pre-requisitte for tensorflow2)
pip3 install — upgrade pip (require pip 19.0 or later)
pip3 install — upgrade tensorflow

Now to check, the tensorflow version:

(tf2) iamsam@iamsam-ThinkPad-L480:~/sam_devpro/tf2$ python
Python 3.6.8 (default, Oct 7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import tensorflow
>>> tensorflow.__version__
‘2.0.0’

Now, if we want to come out of that environment then we have to deactivate it. To check, if now we are out of that new environment or not:

iamsam@iamsam-ThinkPad-L480:~/sam_devpro/tf2$ python
Python 3.6.8 (default, Oct 7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import tensorflow
>>> tensorflow.__version__
‘1.14.0’

Which is the default version in my system.

As we can see, Venv or Virtual environment is very handy in developing different projects. Many raise if we have docker which gives OSlevel isolation, why do we need Venv then? I believe, Venv is still a good idea in that case. They protect you against any bad system-wide Python packages your OS image might have. They don’t introduce any extra overhead, while allowing to have a clean environment and the ability to re-create it outside of Docker (eg. for local development without Docker).

References:

Install Python3 on Ubuntu 18.04 and Set Up a Virtual Programming Environment

Virtualenv the Docker way

Install TensorFlow with pip | TensorFlow