Python Modules

Python modules can be generalized to

  • libraries - things you use in python code
  • programs - thigns you run outside of python

You've been using them all day. Can you think of an example?

In [1]:
import numpy
import matplotlib
import multiprocessing
import sys

Modules like multiprocessing and sys are standard modules, but packages like numpy and matplotlib are non-standard. You can usually tell when they have their own external site.

Standard

External

Notice that the standard ones both come from the python.org site.

Installing modules

Lets say you wanted to use the pysam module

http://pysam.readthedocs.io/en/latest/installation.html

to work with a sam file in python. The documentation says that we only need to run

pip install pysam

We can run that on the shell through our notebook by prepending a !.

In [2]:
!pip search pysam

pysamstats (0.24.3)            - A Python utility for calculating statistics
                                 against genome position based on sequence
                                 alignments from a SAM, BAM or CRAM file.
djangosaml2-bernii (0.11.2)    - pysaml2 integration in Django
djangosaml2 (0.16.0)           - pysaml2 integration for Django
pysaml2 (4.4.0)                - Python implementation of SAML Version 2
pysample (0.1)                 - A python package than can be used as a
                                 model/template for writing, testing and
                                 deploying python packages.
djangosaml2-dj16 (0.10.0)      - pysaml2 integration in Django 1.6
pysam-arm-linux (0.8.3)        - pysam for ARM (armv7l) Linux GNUroot Debian
                                 on Nvidia Shield Tablet [Tegra K1]
pysamlsp (0.1.4)               - A service provider implementation for
                                 SAML2.0.
djangosaml2_tenant (0.22.0)    - pysaml2 integration for multi-tenant in
                                 Django
pysam (0.11.2.1)               - pysam
djangosaml2-knaperek (0.14.1)  - pysaml2 integration for Django
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
In [3]:
!pip install pysam

Collecting pysam
  Using cached pysam-0.11.2.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: pysam
Exception:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/opt/conda/lib/python3.5/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/opt/conda/lib/python3.5/site-packages/pip/req/req_set.py", line 732, in install
    **kwargs
  File "/opt/conda/lib/python3.5/site-packages/pip/req/req_install.py", line 835, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/opt/conda/lib/python3.5/site-packages/pip/req/req_install.py", line 1030, in move_wheel_files
    isolated=self.isolated,
  File "/opt/conda/lib/python3.5/site-packages/pip/wheel.py", line 344, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/opt/conda/lib/python3.5/site-packages/pip/wheel.py", line 315, in clobber
    ensure_dir(destdir)
  File "/opt/conda/lib/python3.5/site-packages/pip/utils/__init__.py", line 83, in ensure_dir
    os.makedirs(path)
  File "/opt/conda/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/conda/lib/python3.5/site-packages/pysam'
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Why didn't that work?

If you take a look at the error, you can see that we are getting a permission error.

PermissionError: [Errno 13] Permission denied: '/opt/conda/lib/python3.5/site-packages/pysam'

This type of error happens when you are trying to write files in a location you do not have access to. Remember that we're all sharing the same computer system, so we want to provide a consistent environment between users and sessions. Earlier this week, you did learn that you can compile and run code from your own directories, and python has the same capability. Python even makes it simple.

Local Modules

By default, python will look for local modules installed in

$HOME/.local/lib/pythonX.Y/site-packages

You can also ask pip to install packages there instead of system-wide directories that you don't own by using the --user flag.

(https://pip.readthedocs.io/en/latest/user_guide/#user-installs)

In [4]:
!pip install --user pysam

Collecting pysam
  Using cached pysam-0.11.2.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: pysam
Successfully installed pysam
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
In [1]:
!pip list --user

pysam (0.11.2.1)
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
In [5]:
import pysam

Uninstalling pysam-0.11.2.1:
  Successfully uninstalled pysam-0.11.2.1
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pysam
  Using cached pysam-0.11.2.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: pysam
Successfully installed pysam
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

After Installation

The package is installed, but you will need to restart your kernel for the your local path to be loaded. Please

  • File -> Close and Halt
  • Logout
  • Login
  • Restart Notebook

You only need to do this once since your $HOME/.local folder did not exit before.

In [6]:
import pysam

Give it a try with biopython or another package

In [7]:
!pip search biopython

A3MIO (0.0.1)                  - A3M/A2M I/O for BioPython
bioscripts.convert (0.4)       - Biopython scripts for converting molecular
                                 sequences between formats.
gbgb (0.1.0)                   - Goodbye, GenBank is a package for use with
                                 Biopython that gives feature annotations from
                                 GenBank records a new and better life.
bcbio-gff (0.6.4)              - Read and write Generic Feature Format (GFF)
                                 with Biopython integration.
biopython (1.69)               - Freely available tools for computational
                                 molecular biology.
biopython-extensions (0.18.6)  - Misc utilities and definitions not included
                                 or hidden in BioPython
Bio_Eutils (1.65)              - Standalone version of the Biopython Eutils
                                 modules
bioext (0.19.1)                - Misc utilities and definitions not included
                                 or hidden in BioPython
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
In [8]:
!pip install --user biopython

Collecting biopython
  Downloading biopython-1.69.tar.gz (15.4MB)
    100% |████████████████████████████████| 15.4MB 86kB/s 
[?25hBuilding wheels for collected packages: biopython
  Running setup.py bdist_wheel for biopython ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | done
[?25h  Stored in directory: /home/jupyter/.cache/pip/wheels/11/15/60/25711f23f903630be11f25fa6284e8ecb6a73b227ba9c9b799
Successfully built biopython
Installing collected packages: biopython
Successfully installed biopython-1.69
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
In [9]:
import Bio

Works!

Standard Packages

If you ever need to do a manual install of a python package outside of pip, the --user flag exists there as well.

(https://docs.python.org/2/install/#alternate-installation-the-user-scheme)

Uninstalling

While not necessary at all, you can clean up your environment by uninstalling these packages with

In [11]:
!pip uninstall -y biopython pysam

Uninstalling biopython-1.69:
  Successfully uninstalled biopython-1.69
Uninstalling pysam-0.11.2.1:
  Successfully uninstalled pysam-0.11.2.1
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.