Texas Tech University

Installing a local copy of Python

Introduction

This tutorial will cover installing Python (via Miniconda) locally into your $HOME folder so that you may install Python versions and packages different from those maintained by the HPCC.  The instructions to perform this installation can be found below.

This method is intended as a replacement for the system versions available through modules, so if you have previously loaded one of these within your session, you must first issue a "module unload python" or "module purge" command. 

 

Table of Contents:

  1. Installing a Local Copy of Python
  2. Installing Python Packages
  3. Creating Virtual Environments
  4. Using Python2 over Python3

 

Installing a Local Copy of Python

The following script can be used to install Python 3 (using Miniconda) to your $HOME directory.  You can then use the "environments" feature of Conda to create, activate, deactivate, and move between separate self-consistent environments to manage your Python and related code sets.

For further information on using the conda command line to control and select between your environments, please see the documentation on the following page: https://conda.io/docs/user-guide/getting-started.html

 

Automated Python Installation Script:

#  Running the following script will automate the process of installing a local copy of Miniconda v3
/lustre/work/examples/InstallPython.sh


Manual Python Installation Script:

#  Download the most recent version of Miniconda v3
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh


# Execute the installation script to install to $HOME/conda.
bash ~/miniconda.sh -b -p ~/conda


# Add Miniconda/Python to your PATH
# You will likely want to do this as part of every job submission
# script you use that will use your local copy of Python.
export PATH="$HOME/conda/bin:$PATH"


# Post-installation check: make sure the correct pip is being used
which python       #  Expected output: ~/conda/bin/python

 


 

Installing Python Packages

Using the Conda installer comes with the added benefit that it comes pre-loaded with a number of popular and powerful python packages, such as numpy and scipy.  However, you may find that you need additional packages to perform your research.  Conda makes installing many python packages easy with the use of the "conda install" command. For instance, if you needed to install the BioPython package (a popular package among bioinformaticians and biologists), you could use the following command:

conda install biopython

To use these packages in practice, be sure to read the "Getting started with conda" tutorial and learn how to use the "conda create", "source activate", and "source deactivate" commands to get the most out of the use of these powerful tools for organizing your Python and related software environments.

 


 

  

Creating Virtual Environments

Another added bonus of the Conda package manager is the ability to separate your Python dependency stacks into separate virtual environments.  This is particularly useful when you have different workflows that require different versions of python libraries or even different versions of python itself.  For instance, if you have some applications that require Python v3.x and another application that requires Python v2.7, then you can simply create a Python 2.7 environment and switch to that environment as needed without causing any problems for your Python v3 environment.

Below is a very brief tutorial for how to create a virtual environment that uses different versions of some software stacks.  For this demonstration we will be installing QIIME v1.9, a bioinformatics software known for having odd dependencies that often create difficulties when being installed on modern installations of Python.  Please feel free to look over the "Getting started with conda" tutorial as well as the "Managing environments" documentation provided by the developers as they do provide a more in-depth look at managing environments.

Tutorial:

If you haven't already done so, please install Miniconda using the installation instructions above.  Then make sure the Conda binary directory has been added to your path.  If you installed using the instructions above, then this can be done using the following:

export PATH="$HOME/conda/bin:$PATH"

Next, ensure your conda application is on your path by running the following command:

which conda

If this does not return either "~/conda/bin/conda" or the path you selected to install conda to, then you will need to go back up a step and ensure you exported the correct directory to your PATH variable.

Now we will create a conda environment.  This is done by giving conda the "create" command and then passing it an argument "-n <name>".  This <name> will be what you use to reference your environment later, so it is often best to use something descriptive to ensure you know what the environment contains in the future.  After giving it a name you can then list the packages you want to install, at this point we should note you can also use an "=" (equal sign) to force Conda to install a specific version.  The following example command will create a new environment called "qiime1" and install into that environment python v2.7,  qiime, matplotlib v1.4.3, mock, nose, h5py and all of their dependencies:

conda create -n python2 python=2.7 scipy=0.15.0 numpy
#As a reminder:
#     create -> instructs conda to create a new virtual environment
#     -n python2 -> instructs conda to name this new environment "python2"
#     python=2.7 -> instructs conda to install Python 2.7 instead of Python v3 (default)
#     scipy=0.15.0 -> instructs conda to install v0.15.0 of scipy instead of the newest version.
# numpy -> instructs conda to install the most recent version of numpy

Once the installation is complete (takes about 1 minute) you will now have a new conda environment named python2.  To see a list of all environments you have access to, use the following command:

conda env list

Switching to one of these environments requires you to run the source command.  For the environment we created above we would need to source the python2 environment.  Run the following commands to see how loading an environment can change the applications you can access.

#View the current version of Python
python --version #This will likely return Python 3.X.Y :: Anaconda, Inc.

#Load the python2 environment
source activate python2

#View the current version of Python
python --version #This will likely return Python 2.7.14 :: Anaconda, Inc.

As you can see, the version of Python changed when you changed environments.  Any work you do with Python will now use this specific version of Python and only the packages you have installed into this environment - nothing installed by the HPCC or that you have installed into other environments will be visible.  To leave this environment you can run the following command:

source deactivate

This returns you to your base conda installation and unloads everything from your virtual environment.

 


 

 

Using Python2 over Python3

Many users would still prefer to use Python v2 over Python v3, however the installation script above will install Python v3.  To switch to Python v2, you will need to create a Python2 environment and then source that environment whenever you need access to Python2.  This can done using the tutorial from the "Creating Virtual Environments" section above, however the relevant commands are reproduced below:

conda create -n python2 python=2.7 scipy=0.15.0 numpy
conda env list   #Now you should see an environment called "python2"
source activate python2  #This will load the python2 environment
#Once you have finished using the environment you will need to deactivate/unload it.
source deactivate

High Performance Computing Center