[django] Sending matplotlib generated figure(s) to django web app

This post introduces how to serve the figures generated by matplotlib to django web app without saving on the server.

In your django views py file, import the following libraries

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

from io import BytesIO
import base64

Note: the two lines of code above in blue need to be placed at the very beginning of the py script; otherwise, you would meet the following error: _tkinter.TclError: no display name and no $DISPLAY environment variable.

In the views py file, in the function that you defined to pass the image data to the front end template file, add the following code:

buf = BytesIO()
plt.savefig(buf, format='png', dpi=300)
image_base64 = base64.b64encode(buf.getvalue()).decode('utf-8').replace('\n', '')
buf.close()

Note: in the function that you defined to pass the image data to the front end template file in your views py file, remember to send the value of the variable image_base64 via, for example, json.

Now, in your front end template file, you can add the following image tag.

<img src="data:image/png;base64,{{image_base64}}" alt="some text to display to your users when the image does not show correctly" width=500 height=auto />

You now should be able to see the figure displayed on your web app page:)

For more details about using data url to pass image data to front end html file, check here (pdf).

 

 

Run Jupyter notebook from terminal with tmux

This post will walk you through how to run a Jupyter notebook script from terminal with tmux (check here for my post about tmux usage).

When you are running Jupyter on a remote server or on cluster/ cloud resources, there are situations where you would like the Jupyter on the remote server or cluster continue running without termination when you shut down your laptop or desktop that you used to access the remote server. tmux will help with this.

In this post, we cover how to let your jupyter notebook running on a remote server continue running without termination via tmux.

Step 1: connect to your remote server with port forwarding

check the Step 5-2 in my post here about setting up Jupyter notebook for how to access your remote server with port forwarding, if you are not familiar with it.

Step 2: install tmux 

 check here for my post about tmux installation and usage

Step 3: install runipy  python package

Check here for runipy installation and usage.

Step 4: in your terminal type the following command, then it will go into tmux window

$ tmux

Step 5: Start jupyter notebook within your tmux session with the following command

$ jupyter notebook --no-browser

The –no-browser option prevents Jupyter from automatically opening a browser window.

Let this terminal stay running.

Step 6: from your laptop, ssh to your remote server (does not need port forwarding this time)

Step 7: cd to where the jupyter notebook script located that you would like to run from terminal

If you do not know what does cd mean and do, check my post for a list of commonly used Linux commands.

Step 8:  use the following command to run your ipynb script (this will save the output of each cell back to the notebook file)

$ runipy -o MyNotebook.ipynb 

To save the notebook output as a new notebook, run:

$ runipy MyNotebook.ipynb OutputNotebook.ipynb

If your ipynb script without any error itself, it should be running on the server now.

Step 9:  Things to pay attention to:

Do not close the terminal where you run the ipynb script within tmux session on your computer that you used to connect to the remote server, that will cause the termination of running the ipynb. But you can make your laptop in sleep or even shut down the computer, the tmux session will keep the ipynb running on your remote server and save the output in the ipynb.

References:

https://www.datasciencebytes.com/bytes/2015/12/18/using-jupyter-notebooks-securely-on-remote-linux-machines/ 

http://forums.fast.ai/t/ipython-notebook-on-a-remote-server-with-tmux/10044/2

Checking from command line if Jupyter server is running and kill if needed

This post provides instructions on how to check whether a Jupyter server is running from command line and kill if needed.

Normally, you can kill a Jupyter server from the same terminal window where you launched your Jupyter notebook by hit CTRL + C, then type yes, to shut down the kernels of Your jupyter notebook.

But, there are situations where you want to know whether a Jupyter-notebook running on your remote server, but the Jupyter notebook was started on another desktop (e.g., your office desktop), (and now you are working at home from your laptop, and want to check whether the notebook is still running).

After you login to your Server where you Jupyter notebook was installed and running, you can use the following command to list runing notebooks.

$ jupyter notebook list

You will see a list of running notebooks in the terminal, if you have several running ones.

You can use the following command to kill specific notebook (identified by the port it runs the jupyter) that you would like to stop.

$ jupyter notebook stop 8888

P.S.:

Each server should start on a new port. jupyter notebook list is reading a set of data files – each notebook server you run writes a file when it starts up, and attempts to remove it when it shuts down. If you see different listed servers on the same port, that means some of them exited without successful removal of the file when it created (for example, unexpected shut down of the notebook would cause this happens).

 

References:

https://github.com/jupyter/notebook/issues/1950

https://github.com/jupyter/notebook/issues/2844

 

 

Run Jupyter Notebook script from terminal

Normally people run jupyter notebook via browser, but in some situation, we will need to run it from terminal, for example, when running the script takes long time.

This post introduces how to run a jupyter notebook script from terminal.

Solution  I:

runipy can do this. runipy will run all cells in a notebook. If an error occurs, the process will stop.

  • Install  runipy package
$ pip3 install runipy # for python 3.x 
$ pip install runipy  # for python 2.x
  • runipy command-line usages

 

  • To run a .ipynb file as a script, run:
$ runipy MyNotebook.ipynb
  • To save the output of each cell back to the notebook file, run:
$ runipy -o MyNotebook.ipynb
  • To save the notebook output as a new notebook, run:
$ runipy MyNotebook.ipynb OutputNotebook.ipynb
  • To run a .ipynb file and generate an HTML report, run:
$ runipy MyNotebook.ipynb --html report.html

 

Solution  II:

The latest versions of jupyter comes with  the nbconvert command tool for notebook conversion allows us  to do this without any extra packages.

Just go to your terminal and type:

$ jupyter nbconvert --to notebook --execute mynotebook.ipynb --output mynotebook.ipynb

This will open the notebook, execute it, capture new output, and save the result in mynotebook.nbconvert.ipynb. By default, nbconvert will abort conversion if any exceptions occur during execution of a cell. If you specify --allow-errors (in addition to the --execute flag) then conversion will continue and the output from any exception will be included in the cell output.

if you meet this error,

raise exception(“Cell execution timed out”)

$ jupyter nbconvert --to notebook --execute --allow-errors --ExecutePreprocessor.timeout=180 mynotebook.ipynb 

You can use the –inplace flag as well:

$ jupyter nbconvert --to notebook --execute --inplace mynotebook.ipynb

 

check here for more (updated) usages about nbconvert jupyter command tool.

 

References:

https://pypi.python.org/pypi/runipy

http://nbconvert.readthedocs.io/en/latest/usage.html#convert-notebook 

Can I run Jupyter notebook cells in commandline?

 

 

 

 

[Python] get a list of sorted directories and/or files

This posts provides a piece of Python code to sort files, folders, and the combination of files and folders in a given directory. It works for Python 3.x. (It should work for Python 2.x, if you change the syntax of print statement to that of Python 2.x.)

Return the oldest and newest file(s), folder(s), or file(s) +folder(s) in a given directory and sort them by modified time.

import os

# change this as the parent directory name of the files you would like to sort

path = 'parent_directory_name'

if (os.path.isdir(path) and (not os.path.exists(path))):

   print("the directory does not exist")
else:
   os.chdir(path)

   # files varialbe contains all files and folders under the path directory

   files = sorted(os.listdir(os.getcwd()), key=os.path.getmtime)

   if len(files) == 0:

      print("there are no regular files or folders in the given directory!")

   else:

      #folder list

      directory_list = []

      #regular file list

      file_list = []

      for f in files:

          if (os.path.isdir(f)):

              directory_list.append(f)

      elif (os.path.isfile(f)):

          file_list.append(f)

      if len(directory_list) == 0:

         print("there are no folders in the given directory!")

    else:

        oldest_folder = directory_list[0]

        newest_folder = directory_list[-1]

        print("Oldest folder:", oldest_folder)

        print("Newest folder:", newest_folder)

        print("All folders sorted by modified time -- oldest to newest:", directory_list) 

    if len(file_list) == 0:

        print("there are no (regular) files in the given directory!")

    else:

        oldest_file = file_list[0]

        newest_file = file_list[-1]

        print("Oldest file:", oldest_file)

        print("Newest file:", newest_file)

        print("All (regular) files sorted by modified time -- oldest to newest:", file_list)

    if len(file_list) > 0 and len(directory_list) > 0:

        oldest = files[0]

        newest = files[-1]

        print("Oldest (file/folder):", oldest)

        print("Newest (file/folder):", newest)

        print("All (file/folder) sorted by modified time -- oldest to newest:", files)

See below for a pic of the code.

Saving IPython/Jupyter notebook as PDF on Ubuntu

When you would like to save your Jupyter notebook as a PDF file, and you encouter the following problems on Ubuntu OS. This post is for you.

The solution:

XeLatex is a part of texlive-xetex package.

To install on Ubuntu, run the following command: 

$ sudo apt-get install texlive-xetex

Now you can download  your ipynb file as PDF!

Install and run IPython and Jupyter Notebook in virtualenv on Ubuntu 16.04 (Desktop and Remote Server)

This post introduces how to install IPython and Jupyter Notebook in virtualenv on Ubuntu 16.04 (both local Desktop and remote server.)

Step 0: install virtualenv and setup virtualenv environment

If you have not installed virtualenv yet, you need to do so before proceed.

Check my post for more details about how to setup python virtual environment and why it is better to install python libraries in Python virtual environment.

  • Install pip and Virtualenv for python 2.x and python 3.x:
$ sudo apt-get update
$ sudo apt-get install openjdk-8-jdk git python-dev python3-dev python-numpy python3-numpy build-essential python-pip python3-pip python-virtualenv swig python-wheel libcurl3-dev
  • Create a Virtualenv environment in the directory for python2.x and python 3.x:
#for python 2.x
virtualenv --system-site-packages -p python ~/ipy-jupyter-venv

# for python 3.x 
virtualenv --system-site-packages -p python3 ~/ipy-jupyter-venv3

(Note: To delete a virtual environment, just delete the corresponding folder.  For example, In our cases, it would be rm -rf ipy-jupyter-venv or rm -rf ipy-jupyter-venv3.)

Step 1: Install IPython

Before installing IPython and Jupyter, be sure to activate your python virtual environment first.

# for python 2.x
$ source ~/ipy-jupyter-venv/bin/activate  # If using bash
(ipy-jupyter-venv)$  # Your prompt should change

# for python 3.x
$ source ~/ipy-jupyter-venv3/bin/activate  # If using bash
(ipy-jupyter-venv3)$  # Your prompt should change

Use the following command to install IPython

#for python 2.x

(ipy-jupyter-venv) liping:~$ pip install ipython

#for python 3.x

(ipy-jupyter-venv3) liping:~$ pip3 install ipython

Step 2: Install Jupyter

Use the following command to install Jupyter Notebook

#for python 2.x 

(ipy-jupyter-venv) liping:~$ pip install jupyter

#for python 3.x 

(ipy-jupyter-venv3) liping:~$ pip3 install jupyter

Step 3: Test

#for python 2.x


(ipy-jupyter-venv) liping:~$ which python
/home/liping/ipy-jupyter-venv/bin/python


#for python 3.x
(ipy-jupyter-venv3) liping:~$ which python3

/home/liping/ipy-jupyter-venv3/bin/python3


#for python 2.x
(ipy-jupyter-venv) liping:~$ which ipython

/home/liping/ipy-jupyter-venv/bin/ipython

#for python 3.x
(ipy-jupyter-venv3) liping:~$ which ipython3

/home/liping/ipy-jupyter-venv3/bin/ipython3

#for python 2.x
(ipy-jupyter-venv) liping:~$ which jupyter-notebook

/home/liping/ipy-jupyter-venv/bin/jupyter-notebook

#for python 3.x
(ipy-jupyter-venv3) liping:~$ which jupyter-notebook

/home/liping/ipy-jupyter-venv3/bin/jupyter-notebook

Step 4: Add Kernel

The Jupyter Notebook and other frontends automatically ensure that the IPython kernel is available. However, if you want to use a kernel with a different version of Python, or in a virtualenv or conda environment, you’ll need to install that manually. 

We are using virutalenv, so we need to install IPython kernel in the virtualenv we created in Step 0 above.

(ipy-jupyter-venv) liping:~$  python -m ipykernel install --user --name myipy_jupter_env --display-name "ipy-jupyter-venv"

Installed kernelspec myipy_jupter_env in /home/liping/.local/share/jupyter/kernels/myipy_jupter_env

(ipy-jupyter-venv3) liping:~$  python3 -m ipykernel install --user --name myipy_jupter_env3 --display-name "ipy-jupyter-venv3"

Installed kernelspec myipy_jupter_env3 in /home/liping/.local/share/jupyter/kernels/myipy_jupter_env3

Step 5: Run Jupyter Notebook

#for python2.x

 (ipy-jupyter-venv) liping:~$ jupyter-notebook

#for python 3.x

 (ipy-jupyter-venv3) liping:~$ jupyter-notebook

If you are running Jupyter Notebook on a local Linux computer (not on a remote server), you can simply navigate to localhost:8888 to connect to Jupyter Notebook. If you are running Jupyter Notebook on a remote server, you will need to connect to the server using SSH tunneling as outlined in the Step 5-2 below.

At this point, you can keep the SSH connection open and keep Jupyter Notebook running or can exit the app and re-run it once you set up SSH tunneling. Let’s keep it simple and stop the Jupyter Notebook process. We will run it again once we have SSH tunneling working. To stop the Jupyter Notebook process, press CTRL+C, type Y, and hit ENTER to confirm. The following will be displayed:

[C 08:08:04.232 NotebookApp] Shutdown confirmed

[I 08:08:04.232 NotebookApp] Shutting down 0 kernels

Step 5-2: Connecting to a remote Server Using SSH Tunneling

This step is only for those who are connecting to a Jupyter Notebook installed on a remote server.

Now we will learn how to connect to the Jupyter Notebook web interface using SSH tunneling. Since Jupyter Notebook is running on a specific port on the remote server (such as :8888:8889 etc.), SSH tunneling enables you to connect to the remote server’s port securely.

Below I will describe how to create an SSH tunnel from  a Mac or Linux (Windows users can check step 4 introduced at here). Note the following instructions in this step refer to your local computer (not the remote server).

SSH Tunneling with a Mac or Linux

Open a new terminal window on your Mac or Linux.

Issue the following ssh command to start SSH tunneling:

$ ssh -L 8000:localhost:8888 your_server_username@your_server_ip

Note: The ssh command opens an SSH connection, but -L specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote server. This means that whatever is running on the second port number (i.e. 8888) on the remote server will appear on the first port number (i.e. 8000) on your local computer. You should change 8888 to the port which Jupyter Notebook is running on. Optionally change port 8000 to one of your choosing (for example, if 8000 is used by another process). Use a port greater or equal to 8000 (ie 80018002, etc.) to avoid using a port already in use by another process. 

If no error shows up after running the ssh -L command, now be sure to activate your python virtual environment first.

# for python 2.x 

$ source ~/ipy-jupyter-venv/bin/activate  # If using bash (ipy-jupyter-venv)$  # Your prompt should change 

# for python 3.x 

$ source ~/ipy-jupyter-venv3/bin/activate  # If using bash (ipy-jupyter-venv3)$  # Your prompt should change

you can run Jupyter Notebook by issuing the following command:

#for python2.x  
(ipy-jupyter-venv) liping:~$ jupyter-notebook 

#for python 3.x  
(ipy-jupyter-venv3) liping:~$ jupyter-notebook

Now, from a web browser on your local machine, open the Jupyter Notebook web interface with http://localhost:8000 (or whatever port number you chose above when you ssh -L into your remote server).

Note: see the following instructions for how to change the startup folder of your Jupyter notebook.

  • In your terminal window
  • Enter the startup folder by typing cd /some_folder_name.
  • Type jupyter notebook to launch the Jupyter Notebook App (it will appear in a new browser window or tab).

Step 6 — Using Jupyter Notebook

By this point you should have Jupyter Notebook running, and you should be connected to it using a web browser. Jupyter Notebook is very powerful and has many features. Below I will outline a few of the basic features to get you started using the notebook. Automatically, Jupyter Notebook will show all of the files and folders in the directory it is run from.

To create a new notebook file, select New > Python 3 or New > ipy-jupyter-venv3 from the top right pull-down menu (Note: this is the so called kernel we installed in Step 4 above):

This will open a notebook. We can now run Python code in the cell or change the cell to markdown. For example, change the first cell to accept Markdown by clicking Cell > Cell Type > Markdown from the top navigation bar. We can now write notes using Markdown and even include equations written in LaTeX by putting them between the $$ symbols. For example, type the following into the cell after changing it to markdown:

# Simple Equation

Let us now implement the following equation:
$$ y = x^2$$

where $x = 2$

To turn the markdown into rich text, press CTRL+ENTER, and the following should be the results:

Note: You can use the markdown cells to make notes and document your code.

Now Let’s implement that simple equation and print the result. Select Insert > Insert Cell Below to insert and cell and enter the following code:

#for python 2.x
x = 2
y = x*x
print y

#for pyton 3.x
x = 2
y = x*x
print (y)

To run the code, press CTRL+ENTER. The following should be the results:

You now can include python libraries and use the notebook as you would with any other Python development environment! 

You should be now able to write reproducible Python code and notes using markdown using Jupyter notebook running on a remote server. To get a quick tour of Jupyter notebook, select Help > User Interface Tour from the top navigation menu.

Happy learning and coding!

Step 7: Deactivate your virtualenv

Each time you would like to use iPython and Jupyter, you need to activate the virtual environment into which it installed, and when you are done using iPython and Jupyter, deactivate the environment.

# for python 2
(ipy-jupyter-venv)$ deactivate
$  # Your prompt should change back

#for python 3
(ipy-jupyter-venv3)$ deactivate
$  # Your prompt should change back

Note: To delete a virtual environment, just delete its folder. (In this case, it would be rm -rf ipy-jupyter-venv or rm -rf ipy-jupyter-venv3.)

References:

Intalling Jupyter in a virtualenv (pdf)

Running iPython cleanly inside a virtualenv (pdf)

Using a virtualenv in an IPython notebook

Installing the IPython kernel

How To Set Up a Jupyter Notebook to Run IPython on Ubuntu 16.04 (pdf)

 Running the Jupyter Notebook (Change Jupyter Notebook startup folder (OS X))

 

 

 

Save figure as a pdf file in Python

This post introduces how to save a figure as a pdf file in Python using Matplotlib.

When using savefig, the file format can be specified by the extension:

savefig('foo.png')
savefig('foo.pdf')

The first line of code above will give a rasterized  output and the second will give a vectorized output.

In addition, you’ll find that pylab leaves a generous, often undesirable, whitespace around the image. You can remove it with:

savefig('foo.png', bbox_inches='tight')

You can also use  figure to set dpi of the figure:

import numpy as np
import matplotlib.pyplot as plt

fig  = plt.figure(figsize=(1,4),facecolor = 'red', dpi=100)
plt.savefig('test.png', dpi=100)

plt.show(fig)

 

 

Matplotlib default figure size

This post introduces how to check the default figure  size in Matplotlib, and how to change the figure size.

The default value is [8.0, 6.0] which can be changed of course.
To know all the default values just inspect the value of ‘rcParams’

print(plt.rcParams)  # it will tell you all default setting in Matplotlib
 
print(plt.rcParams.get('figure.figsize'))

To change the figure size.

you can use the following:

plt.figure(figsize=(20,10))

fig, ax = plt.subplots(figsize=(20, 10))

Using Apache Solr with Python

This post provides the instructions to use Apache Solr with Python in different ways.

======using Pysolr

Below are two small python snippets that the author of the post used for testing writing to and reading from a new SOLR server.

The script below will attempt to add a document to the SOLR server.

# Using Python 2.X
from __future__ import print_function  
import pysolr

# Setup a basic Solr instance. The timeout is optional.
solr = pysolr.Solr('http://some-solr-server.com:8080/solr/', timeout=10)

# How you would index data.
solr.add([  
    {
        "id": "doc_1",
        "title": "A very small test document about elmo",
    }
])

The snippet below will attempt to search for the document that was just added from the snippet above.

# Using Python 2.X
from __future__ import print_function  
import pysolr

# Setup a basic Solr instance. The timeout is optional.
solr = pysolr.Solr('http://some-solr-server.com:8080/solr/', timeout=10)

results = solr.search('elmo')

print("Saw {0} result(s).".format(len(results)))  

 

======GitHub repos

pysolr is a lightweight Python wrapper for Apache Solr. It provides an interface that queries the server and returns results based on the query.

install Pysolr using pip

pip install pysolr

Multicore Index

Simply point the URL to the index core:

# Setup a Solr instance. The timeout is optional.
solr = pysolr.Solr('http://localhost:8983/solr/core_0/', timeout=10)

SolrClient is a simple python library for Solr; built in python3 with support for latest features of Solr.

Components of SolrClient

 

References: