[django] Sending matplotlib generated figure(s) to django web app

This post introduces how to serve the figures generated by matplotlib to django web app without saving on the server.

In your django views py file, import the following libraries

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

from io import BytesIO
import base64

Note: the two lines of code above in blue need to be placed at the very beginning of the py script; otherwise, you would meet the following error: _tkinter.TclError: no display name and no $DISPLAY environment variable.

In the views py file, in the function that you defined to pass the image data to the front end template file, add the following code:

buf = BytesIO()
plt.savefig(buf, format='png', dpi=300)
image_base64 = base64.b64encode(buf.getvalue()).decode('utf-8').replace('\n', '')
buf.close()

Note: in the function that you defined to pass the image data to the front end template file in your views py file, remember to send the value of the variable image_base64 via, for example, json.

Now, in your front end template file, you can add the following image tag.

<img src="data:image/png;base64,{{image_base64}}" alt="some text to display to your users when the image does not show correctly" width=500 height=auto />

You now should be able to see the figure displayed on your web app page:)

For more details about using data url to pass image data to front end html file, check here (pdf).

 

 

Tips about Jupyter Notebook

This post provides some tips about Jupyter Notebook.

  • When you open a pdf file in a browser via your running Jupyter Notebook.

You may encounter some error similar to this “Error! (file) is not UTF-8 encoded”,

the solution:

check the address in your browser, for example, for the following case,

http://localhost:8000/edit/confusionmatrix.pdf

if I change the edit in the url address given above to  view

http://localhost:8000/view/confusionmatrix.pdf

You will see your beautiful fig or document in pdf format.

  • Another similar situation:

When you open a html file in your browser via your running Jupyter Notebook, and you would like to edit the html file, instead of seeing its display,

the solution:

change view to edit in the url address of the file, see the following for an example, then you should be able to edit your html file.

http://localhost:8000/view/index.html 

to

http://localhost:8000/edit/index.html

Happy Jupyter Notebooking!

Dynamic GPU usage monitoring (CUDA)

To dynamically  monitor NVIDIA GPU usage, here I introduce two methods:

method 1: use nvidia-smi

in your terminal, issue the following command:

$ watch -n 1 nvidia-smi

It will continually update the gpu usage info (every second, you can change the 1 to 2 or the time interval you want the usage info to be updated).

method 2: use the open source monitoring program glances with its GPU monitoring plugin

in your terminal, issue the following command to install glances with its GPU monitoring plugin

$ sudo pip install glances[gpu]

to launch it, in your terminal, issue the following command:

 $ sudo glances

Then you should see your GPU usage etc. It also monitors the CPU, disk IO, disk space, network, and a few other things

For more commonly used Linux commands, check my other posts at here  and here .

Resources about Azure Batch AI

This post provides some resources about Microsoft Auzre Batch AI.

Introduction:

Batch AI is a managed service that enables data scientists and AI researchers to train AI and other machine learning models on clusters of Azure virtual machines, including VMs with GPU support. You describe the requirements of your job, where to find the inputs and store the outputs, and Batch AI handles the rest.

Hands-on tutorials:

Videos:

 

Run Jupyter notebook from terminal with tmux

This post will walk you through how to run a Jupyter notebook script from terminal with tmux (check here for my post about tmux usage).

When you are running Jupyter on a remote server or on cluster/ cloud resources, there are situations where you would like the Jupyter on the remote server or cluster continue running without termination when you shut down your laptop or desktop that you used to access the remote server. tmux will help with this.

In this post, we cover how to let your jupyter notebook running on a remote server continue running without termination via tmux.

Step 1: connect to your remote server with port forwarding

check the Step 5-2 in my post here about setting up Jupyter notebook for how to access your remote server with port forwarding, if you are not familiar with it.

Step 2: install tmux 

 check here for my post about tmux installation and usage

Step 3: install runipy  python package

Check here for runipy installation and usage.

Step 4: in your terminal type the following command, then it will go into tmux window

$ tmux

Step 5: Start jupyter notebook within your tmux session with the following command

$ jupyter notebook --no-browser

The –no-browser option prevents Jupyter from automatically opening a browser window.

Let this terminal stay running.

Step 6: from your laptop, ssh to your remote server (does not need port forwarding this time)

Step 7: cd to where the jupyter notebook script located that you would like to run from terminal

If you do not know what does cd mean and do, check my post for a list of commonly used Linux commands.

Step 8:  use the following command to run your ipynb script (this will save the output of each cell back to the notebook file)

$ runipy -o MyNotebook.ipynb 

To save the notebook output as a new notebook, run:

$ runipy MyNotebook.ipynb OutputNotebook.ipynb

If your ipynb script without any error itself, it should be running on the server now.

Step 9:  Things to pay attention to:

Do not close the terminal where you run the ipynb script within tmux session on your computer that you used to connect to the remote server, that will cause the termination of running the ipynb. But you can make your laptop in sleep or even shut down the computer, the tmux session will keep the ipynb running on your remote server and save the output in the ipynb.

References:

https://www.datasciencebytes.com/bytes/2015/12/18/using-jupyter-notebooks-securely-on-remote-linux-machines/ 

http://forums.fast.ai/t/ipython-notebook-on-a-remote-server-with-tmux/10044/2

Checking from command line if Jupyter server is running and kill if needed

This post provides instructions on how to check whether a Jupyter server is running from command line and kill if needed.

Normally, you can kill a Jupyter server from the same terminal window where you launched your Jupyter notebook by hit CTRL + C, then type yes, to shut down the kernels of Your jupyter notebook.

But, there are situations where you want to know whether a Jupyter-notebook running on your remote server, but the Jupyter notebook was started on another desktop (e.g., your office desktop), (and now you are working at home from your laptop, and want to check whether the notebook is still running).

After you login to your Server where you Jupyter notebook was installed and running, you can use the following command to list runing notebooks.

$ jupyter notebook list

You will see a list of running notebooks in the terminal, if you have several running ones.

You can use the following command to kill specific notebook (identified by the port it runs the jupyter) that you would like to stop.

$ jupyter notebook stop 8888

P.S.:

Each server should start on a new port. jupyter notebook list is reading a set of data files – each notebook server you run writes a file when it starts up, and attempts to remove it when it shuts down. If you see different listed servers on the same port, that means some of them exited without successful removal of the file when it created (for example, unexpected shut down of the notebook would cause this happens).

 

References:

https://github.com/jupyter/notebook/issues/1950

https://github.com/jupyter/notebook/issues/2844

 

 

Run Jupyter Notebook script from terminal

Normally people run jupyter notebook via browser, but in some situation, we will need to run it from terminal, for example, when running the script takes long time.

This post introduces how to run a jupyter notebook script from terminal.

Solution  I:

runipy can do this. runipy will run all cells in a notebook. If an error occurs, the process will stop.

  • Install  runipy package
$ pip3 install runipy # for python 3.x 
$ pip install runipy  # for python 2.x
  • runipy command-line usages

 

  • To run a .ipynb file as a script, run:
$ runipy MyNotebook.ipynb
  • To save the output of each cell back to the notebook file, run:
$ runipy -o MyNotebook.ipynb
  • To save the notebook output as a new notebook, run:
$ runipy MyNotebook.ipynb OutputNotebook.ipynb
  • To run a .ipynb file and generate an HTML report, run:
$ runipy MyNotebook.ipynb --html report.html

 

Solution  II:

The latest versions of jupyter comes with  the nbconvert command tool for notebook conversion allows us  to do this without any extra packages.

Just go to your terminal and type:

$ jupyter nbconvert --to notebook --execute mynotebook.ipynb --output mynotebook.ipynb

This will open the notebook, execute it, capture new output, and save the result in mynotebook.nbconvert.ipynb. By default, nbconvert will abort conversion if any exceptions occur during execution of a cell. If you specify --allow-errors (in addition to the --execute flag) then conversion will continue and the output from any exception will be included in the cell output.

if you meet this error,

raise exception(“Cell execution timed out”)

$ jupyter nbconvert --to notebook --execute --allow-errors --ExecutePreprocessor.timeout=180 mynotebook.ipynb 

You can use the –inplace flag as well:

$ jupyter nbconvert --to notebook --execute --inplace mynotebook.ipynb

 

check here for more (updated) usages about nbconvert jupyter command tool.

 

References:

https://pypi.python.org/pypi/runipy

http://nbconvert.readthedocs.io/en/latest/usage.html#convert-notebook 

Can I run Jupyter notebook cells in commandline?

 

 

 

 

[Paper published] Check out our new machine/deep learning paper

New machine/deep learning paper led by Liping: Visually-Enabled Active Deep Learning for (Geo) Text and Image Classification: A Review https://t.co/kSF3O71tbD

Through the synthesis of multiple rapidly developing research areas, this systematic review is relevant to multiple research domains, including but not limited to GIScience, computer science, data science, information science, visual analytics, information visualization, image analysis, and computational linguistics, as well as any domains that need to leverage machine learning and deep learning .

 

[Python] get a list of sorted directories and/or files

This posts provides a piece of Python code to sort files, folders, and the combination of files and folders in a given directory. It works for Python 3.x. (It should work for Python 2.x, if you change the syntax of print statement to that of Python 2.x.)

Return the oldest and newest file(s), folder(s), or file(s) +folder(s) in a given directory and sort them by modified time.

import os

# change this as the parent directory name of the files you would like to sort

path = 'parent_directory_name'

if (os.path.isdir(path) and (not os.path.exists(path))):

   print("the directory does not exist")
else:
   os.chdir(path)

   # files varialbe contains all files and folders under the path directory

   files = sorted(os.listdir(os.getcwd()), key=os.path.getmtime)

   if len(files) == 0:

      print("there are no regular files or folders in the given directory!")

   else:

      #folder list

      directory_list = []

      #regular file list

      file_list = []

      for f in files:

          if (os.path.isdir(f)):

              directory_list.append(f)

      elif (os.path.isfile(f)):

          file_list.append(f)

      if len(directory_list) == 0:

         print("there are no folders in the given directory!")

    else:

        oldest_folder = directory_list[0]

        newest_folder = directory_list[-1]

        print("Oldest folder:", oldest_folder)

        print("Newest folder:", newest_folder)

        print("All folders sorted by modified time -- oldest to newest:", directory_list) 

    if len(file_list) == 0:

        print("there are no (regular) files in the given directory!")

    else:

        oldest_file = file_list[0]

        newest_file = file_list[-1]

        print("Oldest file:", oldest_file)

        print("Newest file:", newest_file)

        print("All (regular) files sorted by modified time -- oldest to newest:", file_list)

    if len(file_list) > 0 and len(directory_list) > 0:

        oldest = files[0]

        newest = files[-1]

        print("Oldest (file/folder):", oldest)

        print("Newest (file/folder):", newest)

        print("All (file/folder) sorted by modified time -- oldest to newest:", files)

See below for a pic of the code.

Saving IPython/Jupyter notebook as PDF on Ubuntu

When you would like to save your Jupyter notebook as a PDF file, and you encouter the following problems on Ubuntu OS. This post is for you.

The solution:

XeLatex is a part of texlive-xetex package.

To install on Ubuntu, run the following command: 

$ sudo apt-get install texlive-xetex

Now you can download  your ipynb file as PDF!