Install gensim and nltk into Virtualenv

This post introduces how to install gensim and nltk into a virtualenv. It is always a good strategy to install some package(s)/library(s) you often use (together) into a separate virtualenv, so it will not be interrupted by other libraries (because different libraries may depend on different versions of another libraries).

$ sudo apt-get install python-pip python-dev python-virtualenv
  • Create a Virtualenv environment in the directory for python  ~/gensim-venv:

Note that you can setup your virtualenv in any directory you want — just replace the ~/gensim-venv with your preferred directory path. But be sure to change it accordingly when you follow the rest of the tutorial.

$ virtualenv --system-site-packages -p python ~/gensim-venv # for python 

$ virtualenv --system-site-packages -p python3 ~/gensim-venv3 # for python 3, it is better to add "venv3" when naming your virtualenv, so you know it is for python3

The --system-site-packages Option

If you build with virtualenv --system-site-packages ENV, your virtual environment will inherit packages from /usr/lib/python2.7/site-packages(or wherever your global site-packages directory is).

This can be used if you have control over the global site-packages directory, and you want to depend on the packages there. If you want isolation from the global system, do not use this flag (and note that if you do not use the “system-site-packages” flag, you NEED to install the version of the python you need in the virtualenv that you just created)

  • Activate the virtual environment:
$ source ~/gensim-venv/bin/activate  # If using bash
(gensim-venv)$  # Your prompt should change
  • Install gensim in the virtualenv for python:
pip install --upgrade gensim
  • After the install you will activate the Virtualenv environment each time you want to use gensim.
  • With the Virtualenv environment activated, you can now test your gensim installation.
  •  When you are done using gensim, deactivate the environment.
(gensim-venv)$ deactivate

$  # Your prompt should change back

To use gensim later you will have to activate the Virtualenv environment again:

$ source ~/gensim-venv/bin/activate  # If using bash.

(gensim-venv)$  # Your prompt should change.
# Run Python programs that use gensim.
...
# When you are done using gensim, deactivate the environment.
(gensim-venv)$ deactivate
  • To delete a virtual environment, just delete its folder. (In this case, it would be rm -rf gensim-venv.)
  • To install nltk into the virtualenv gensim-venv, just issue this command after you activated the gensim-venv
(gensim-venv)$ pip install -U nltk

Note: you may notice that those commands issued inside a virtualenv (i.e., after you create a virtualenv and activated it), you do not need to add sudo before the commands. Because all those commands will all be confined within that virtualenv folder you created – that is why  it will not affect other packages installed in other virtualenv or system-wide, and it will not be affected by other packages installed in another virtualenv(s). You can create as many as virtualenv you want, so please naming your virtualenv with meaning name, otherwise, over time, we will forget what was actually being installed in some virtualenv:)

 

Books and Courses and Tutorials: Python

This page provides some useful resources (books and courses) with Python. (See this post for text processing related resources, and this post for a collection of python packages and related resources)

(I have created a page for this topic, so stop editing this post, BUT KEEP this post, because not every book on this page has been moved to the page.)

Read file from line 2 or skip header row in Python

This post introduces how to read file from line 2 in Python.

  • method 1:
with open(fname) as f:
  next(f)
  for line in f:
    #do something

Note: If you need the header later, instead of next(f) use f.readline() and store it as a variable.
Or use header_line = next(f).

  • method 2
f = open(fname,'r')
lines = f.readlines()[1:]
f.close()

This will skip 1 line. for example, [‘a’, ‘b’, ‘c’][1:] => [‘b’, ‘c’]

 

Find which version of TensorFlow was installed

This depends on how you installed TensorFlow.

Pip installation

Run:

python -c 'import tensorflow as tf; print(tf.__version__)'  # for Python 2
python3 -c 'import tensorflow as tf; print(tf.__version__)'  # for Python 3

Note that python is symlinked to /usr/bin/python3 in some Linux distributions, so use pythoninstead of python3 in these cases.

pip list | grep tensorflow for Python 2 or pip3 list | grep tensorflow for Python 3 will also show the version of Tensorflow installed.

Virtualenv installation

Run:

python -c 'import tensorflow as tf; print(tf.__version__)'  # for both Python 2 and Python 3

pip list | grep tensorflow will also show the version of Tensorflow installed.

Examples

For example, I have installed TensorFlow 0.12.1 in a virtualenv for Python 3. So, I get:

and installed TensorFlow from source for Python 2 (not in a virtualenv).

Referenced:

Install TensorFlow for Python 2.7 and Python 3.5 on the same machine (Ubuntu 16.04)

I already installed GPU TensorFlow from source for Python 2 (see this post), and now I would like to also install GPU TensorFlow for Python 3 on the same machine using Virtualenv.

Virtualenv is a tool to keep the dependencies required by different Python projects in separate places. The Virtualenv installation of TensorFlow will not override pre-existing version of the Python packages needed by TensorFlow. See here for a detailed introduction of how virtualenv works and some basic usage.

With Virtualenv the installation is as follows:

  • Install pip and Virtualenv:
$ sudo apt-get update
$ sudo apt-get install python-pip python-dev python-virtualenv
  • Create a Virtualenv environment in the directory for python 3 ~/tensorflow-venv3:
$ virtualenv --system-site-packages -p python3 ~/tensorflow-venv3 

#for python 2
$ virtualenv --system-site-packages -p python ~/tensorflow-venv 

The --system-site-packages Option

If you build with virtualenv --system-site-packages ENV, your virtual environment will inherit packages from /usr/lib/python2.7/site-packages (or wherever your global site-packages directory is).

This can be used if you have control over the global site-packages directory, and you want to depend on the packages there. If you want isolation from the global system, do not use this flag.

  • Activate the virtual environment:
$ source ~/tensorflow-venv3/bin/activate  # If using bash
(tensorflow-venv3)$  # Your prompt should change
  • Install TensorFlow in the virtualenv for python 3:

Now, install TensorFlow just as you would for a regular Pip installation. First select the correct binary to install (from this page):

# Ubuntu/Linux 64-bit, GPU enabled, Python 3.5
# Requires CUDA toolkit 8.0 and CuDNN v5. For other versions, see "Installing from sources" below.
(tensorflow-venv3)$ export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-0.12.1-cp35-cp35m-linux_x86_64.whl
# Python 2
(tensorflow-venv2)$ pip install --upgrade $TF_BINARY_URL

# Python 3
(tensorflow-venv3)$ pip3 install --upgrade $TF_BINARY_URL

Or you can choose the .whl file you built from source by yourself. Like the one I built in the post GPU tensorflow installation from source

# Python 2 pip install /path to/the .whl file you built from source/tensorflow-0.12.1-cp27-cp27mu-linux_x86_64.whl # Python 3 pip3 install /path to/the .whl file you built from source/tensorflow-0.12.1-cp27-cp27mu-linux_x86_64.whl

Note that when I used the .whl file I built to intall tensorflow into the virtualenv, I met this error. So I ended up installing the binary file from this page.

pip3 install ~/tensorflow_pkg/tensorflow-0.12.1-cp27-cp27mu-linux_x86_64.whl 
tensorflow-0.12.1-cp27-cp27mu-linux_x86_64.whl is not a supported wheel on this platform.
# Ubuntu/Linux 64-bit, GPU enabled, Python 3.5
# Requires CUDA toolkit 8.0 and CuDNN v5. For other versions, see "Installing from sources" below.
(tensorflow-venv3)$ export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-0.12.1-cp35-cp35m-linux_x86_64.whl
  • After the install you will activate the Virtualenv environment each time you want to use TensorFlow.
  • With the Virtualenv environment activated, you can now test your TensorFlow installation.

In your virtualenv, open a python session and type import tensorflow as tf.

If all went well, you should see the following output:

Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
>>>
  •  When you are done using TensorFlow, deactivate the environment.
    (tensorflow-venv3)$ deactivate
    
    $  # Your prompt should change back
    

To use TensorFlow later you will have to activate the Virtualenv environment again:

$ source ~/tensorflow-venv3/bin/activate  # If using bash.

(tensorflow-venv3)$  # Your prompt should change.
# Run Python programs that use TensorFlow.
...
# When you are done using TensorFlow, deactivate the environment.
(tensorflow-venv3)$ deactivate
  • To delete a virtual environment, just delete its folder. (In this case, it would be rm -rf tensorflow-venv3.)

You can test whether both TensorFlow installed in python 2 and python 3 works. See below for my example.

 

Posts I referenced:

Open multiple files using “open ” and “with open” in Python

This post introduces two ways to open multiple files in Python.

  • “with open” # do not need to bother to close the file(s) if use “with open”
with open("datafile.csv" , "r") as f:
  f.read()
with open("datafile2.csv" , "r") as f2:
  f2.read()

Or

try:
  with open('file.csv', 'w') as f, open('file2.csv', 'w') as f2:
    do_something()
except IOError as e:
  print 'Operation failed: %s' % e.strerror
  • “open”  # need to close the file when use open.
f = open("datafile.csv" , "r")
f2 = open("datafile2.csv" , "r")
f.read()
f2.read()
f.close()
f2.close()

 

References:

 

Remove a character from a string using Python

This post introduces how to remove / replace a character from a string using Python.

In Python, strings are immutable, so we need to create a new string. If you want to remove the ‘;’ wherever it appears. See below for an example.

Remove character”;”

>>> original_str = "good; ok"
>>> new_str = original_str.replace(";","")
>>> print(new_str)
good ok
>>>

See the examples below for different ways to check whether a string contains a specific character.

>>> s = "good; ok"
>>> ";" in s
True
>>> ";" not in s
False
>>> s.find(";") == -1
False
>>> s. find(";") != -1
True
>>> chars = set(";o")
>>> if any((c in chars) for c in s):
>>> print("Found")
Found
>>> import re
>>> pattern = re.compile(r";o")
>>> if pattern.findall(s):
>>> print("Found")
Found
>>>

 

References:

How to delete a character from a string using python?

Remove all special characters, punctuation and spaces from string

How to check a string for specific characters? (pdf)

Access PostgreSQL from Python using pscopg2

psycopg2 is a python module for PostgreSQL.

  • Basic module usage

http://initd.org/psycopg/docs/usage.html#basic-module-usage

  • with statement

Starting from version 2.5, psycopg2’s connections and cursors are context managers and can be used with the with statement:

http://initd.org/psycopg/docs/usage.html#with-statement

  • Server side cursors

http://initd.org/psycopg/docs/usage.html#server-side-cursors

When a database query is executed, the Psycopg cursor usually fetches all the records returned by the backend, transferring them to the client process. If the query returned an huge amount of data, a proportionally large amount of memory will be allocated by the client.

If the dataset is too large to be practically handled on the client side, it is possible to create a server side cursor. Using this kind of cursor it is possible to transfer to the client only a controlled amount of data, so that a large dataset can be examined without keeping it entirely in memory.

 

Posts Referenced:

 

Ways to use double quotes inside a string variable in Python

To use double quotes inside a string variable in Python, see below for the three ways:

1) Use single and double quotes together:

>>> print '"words to be double-quoted"'
"words to be double-quoted"

2) Escape the double quotes within the string:

>>> print "\"words to be double-quoted\""
"words to be double-quoted" 

3) Use triple-quoted strings:

>>> print """ "words to be double-quoted" """
"words to be double-quoted" 

Note: when you need to use single quote within a single quote, the above methods work for single quote case as well.

References:

Create a hash table for large data in python

This post introduces how to create a hash table in python.

Text files can have duplicates which will overwrite existing keys in your dictionary (the python name for a hash table). We can create a unique set of the keys, and then use a dictionary comprehension to populate the dictionary.

sample_file.txt:

a
b
c
c

 

Python code:

with open("sample_file.txt") as f:
  keys = set(line.strip() for line in f.readlines())
my_dict = {key: 1 for key in keys if key}

>>> my_dict
{'a': 1, 'b': 1, 'c': 1}