Dynamic GPU usage monitoring (CUDA)

To dynamically  monitor NVIDIA GPU usage, here I introduce two methods:

method 1: use nvidia-smi

in your terminal, issue the following command:

$ watch -n 1 nvidia-smi

It will continually update the gpu usage info (every second, you can change the 1 to 2 or the time interval you want the usage info to be updated).

method 2: use the open source monitoring program glances with its GPU monitoring plugin

in your terminal, issue the following command to install glances with its GPU monitoring plugin

$ sudo pip install glances[gpu]

to launch it, in your terminal, issue the following command:

 $ sudo glances

Then you should see your GPU usage etc. It also monitors the CPU, disk IO, disk space, network, and a few other things

For more commonly used Linux commands, check my other posts at here  and here .

Resources about Azure Batch AI

This post provides some resources about Microsoft Auzre Batch AI.


Batch AI is a managed service that enables data scientists and AI researchers to train AI and other machine learning models on clusters of Azure virtual machines, including VMs with GPU support. You describe the requirements of your job, where to find the inputs and store the outputs, and Batch AI handles the rest.

Hands-on tutorials:



Run Jupyter notebook from terminal with tmux

This post will walk you through how to run a Jupyter notebook script from terminal with tmux (check here for my post about tmux usage).

When you are running Jupyter on a remote server or on cluster/ cloud resources, there are situations where you would like the Jupyter on the remote server or cluster continue running without termination when you shut down your laptop or desktop that you used to access the remote server. tmux will help with this.

In this post, we cover how to let your jupyter notebook running on a remote server continue running without termination via tmux.

Step 1: connect to your remote server with port forwarding

check the Step 5-2 in my post here about setting up Jupyter notebook for how to access your remote server with port forwarding, if you are not familiar with it.

Step 2: install tmux 

 check here for my post about tmux installation and usage

Step 3: install runipy  python package

Check here for runipy installation and usage.

Step 4: in your terminal type the following command, then it will go into tmux window

$ tmux

Step 5: Start jupyter notebook within your tmux session with the following command

$ jupyter notebook --no-browser

The –no-browser option prevents Jupyter from automatically opening a browser window.

Let this terminal stay running.

Step 6: from your laptop, ssh to your remote server (does not need port forwarding this time)

Step 7: cd to where the jupyter notebook script located that you would like to run from terminal

If you do not know what does cd mean and do, check my post for a list of commonly used Linux commands.

Step 8:  use the following command to run your ipynb script (this will save the output of each cell back to the notebook file)

$ runipy -o MyNotebook.ipynb 

To save the notebook output as a new notebook, run:

$ runipy MyNotebook.ipynb OutputNotebook.ipynb

If your ipynb script without any error itself, it should be running on the server now.

Step 9:  Things to pay attention to:

Do not close the terminal where you run the ipynb script within tmux session on your computer that you used to connect to the remote server, that will cause the termination of running the ipynb. But you can make your laptop in sleep or even shut down the computer, the tmux session will keep the ipynb running on your remote server and save the output in the ipynb.




Checking from command line if Jupyter server is running and kill if needed

This post provides instructions on how to check whether a Jupyter server is running from command line and kill if needed.

Normally, you can kill a Jupyter server from the same terminal window where you launched your Jupyter notebook by hit CTRL + C, then type yes, to shut down the kernels of Your jupyter notebook.

But, there are situations where you want to know whether a Jupyter-notebook running on your remote server, but the Jupyter notebook was started on another desktop (e.g., your office desktop), (and now you are working at home from your laptop, and want to check whether the notebook is still running).

After you login to your Server where you Jupyter notebook was installed and running, you can use the following command to list runing notebooks.

$ jupyter notebook list

You will see a list of running notebooks in the terminal, if you have several running ones.

You can use the following command to kill specific notebook (identified by the port it runs the jupyter) that you would like to stop.

$ jupyter notebook stop 8888


Each server should start on a new port. jupyter notebook list is reading a set of data files – each notebook server you run writes a file when it starts up, and attempts to remove it when it shuts down. If you see different listed servers on the same port, that means some of them exited without successful removal of the file when it created (for example, unexpected shut down of the notebook would cause this happens).







Run Jupyter Notebook script from terminal

Normally people run jupyter notebook via browser, but in some situation, we will need to run it from terminal, for example, when running the script takes long time.

This post introduces how to run a jupyter notebook script from terminal.

Solution  I:

runipy can do this. runipy will run all cells in a notebook. If an error occurs, the process will stop.

  • Install  runipy package
$ pip3 install runipy # for python 3.x 
$ pip install runipy  # for python 2.x
  • runipy command-line usages


  • To run a .ipynb file as a script, run:
$ runipy MyNotebook.ipynb
  • To save the output of each cell back to the notebook file, run:
$ runipy -o MyNotebook.ipynb
  • To save the notebook output as a new notebook, run:
$ runipy MyNotebook.ipynb OutputNotebook.ipynb
  • To run a .ipynb file and generate an HTML report, run:
$ runipy MyNotebook.ipynb --html report.html


Solution  II:

The latest versions of jupyter comes with  the nbconvert command tool for notebook conversion allows us  to do this without any extra packages.

Just go to your terminal and type:

$ jupyter nbconvert --to notebook --execute mynotebook.ipynb --output mynotebook.ipynb

This will open the notebook, execute it, capture new output, and save the result in mynotebook.nbconvert.ipynb. By default, nbconvert will abort conversion if any exceptions occur during execution of a cell. If you specify --allow-errors (in addition to the --execute flag) then conversion will continue and the output from any exception will be included in the cell output.

if you meet this error,

raise exception(“Cell execution timed out”)

$ jupyter nbconvert --to notebook --execute --allow-errors --ExecutePreprocessor.timeout=180 mynotebook.ipynb 

You can use the –inplace flag as well:

$ jupyter nbconvert --to notebook --execute --inplace mynotebook.ipynb


check here for more (updated) usages about nbconvert jupyter command tool.





Can I run Jupyter notebook cells in commandline?





[Paper published] Check out our new machine/deep learning paper

New machine/deep learning paper led by Liping: Visually-Enabled Active Deep Learning for (Geo) Text and Image Classification: A Review https://t.co/kSF3O71tbD

Through the synthesis of multiple rapidly developing research areas, this systematic review is relevant to multiple research domains, including but not limited to GIScience, computer science, data science, information science, visual analytics, information visualization, image analysis, and computational linguistics, as well as any domains that need to leverage machine learning and deep learning .


[Python] get a list of sorted directories and/or files

This posts provides a piece of Python code to sort files, folders, and the combination of files and folders in a given directory. It works for Python 3.x. (It should work for Python 2.x, if you change the syntax of print statement to that of Python 2.x.)

Return the oldest and newest file(s), folder(s), or file(s) +folder(s) in a given directory and sort them by modified time.

import os

# change this as the parent directory name of the files you would like to sort

path = 'parent_directory_name'

if (os.path.isdir(path) and (not os.path.exists(path))):

   print("the directory does not exist")

   # files varialbe contains all files and folders under the path directory

   files = sorted(os.listdir(os.getcwd()), key=os.path.getmtime)

   if len(files) == 0:

      print("there are no regular files or folders in the given directory!")


      #folder list

      directory_list = []

      #regular file list

      file_list = []

      for f in files:

          if (os.path.isdir(f)):


      elif (os.path.isfile(f)):


      if len(directory_list) == 0:

         print("there are no folders in the given directory!")


        oldest_folder = directory_list[0]

        newest_folder = directory_list[-1]

        print("Oldest folder:", oldest_folder)

        print("Newest folder:", newest_folder)

        print("All folders sorted by modified time -- oldest to newest:", directory_list) 

    if len(file_list) == 0:

        print("there are no (regular) files in the given directory!")


        oldest_file = file_list[0]

        newest_file = file_list[-1]

        print("Oldest file:", oldest_file)

        print("Newest file:", newest_file)

        print("All (regular) files sorted by modified time -- oldest to newest:", file_list)

    if len(file_list) > 0 and len(directory_list) > 0:

        oldest = files[0]

        newest = files[-1]

        print("Oldest (file/folder):", oldest)

        print("Newest (file/folder):", newest)

        print("All (file/folder) sorted by modified time -- oldest to newest:", files)

See below for a pic of the code.

Install and use htop on Ubuntu 16.04 Desktop and Server

This post introduces an interactive tool for visually monitoring the memory and process usages of your Ubuntu 16.04 Desktop or Server in real time.

  • What is Htop?

Htop is an interactive system-monitor process-viewer and process-manager. It is designed as an alternative to the Unix program top. It shows a frequently updated list of the processes running on a computer, normally ordered by the amount of CPU usage. Unlike top, htop provides a full list of processes running, instead of the top resource-consuming processes. Htop uses color and gives visual information about processor, swap and memory status.

  • Install Htop on Ubuntu 16.04 LTS Desktop and Server

(This works on both an Ubuntu 16.04 Desktop and  Server.)

Installing htop package on Ubuntu 16.04 (Xenial Xerus) is as easy as running the following command on terminal:

Step 1. First make sure that all your system packages are up-to-date by running these following apt-get commands in the terminal.

$ sudo apt-get update

Step 2. Installing Htop. Install htop process monitoring tool using apt-get command:

$ sudo apt-get install htop
  • Use Htop to monitor your Ubuntu 16.04 LTS Desktop and Server in real-time

Now that htop is installed on your server you’ll want to start the program by running the following in a command prompt:

$ htop

This will open the program and you’ll see something similar to the following:

Leave this terminal open, you can use CTRL + ALT + T  to open another new terminal for your other work. Then htop will help you monitor your memory usage in real-time:)  Enjoy!

Install and run IPython and Jupyter Notebook in virtualenv on Ubuntu 16.04 (Desktop and Remote Server)

This post introduces how to install IPython and Jupyter Notebook in virtualenv on Ubuntu 16.04 (both local Desktop and remote server.)

Step 0: install virtualenv and setup virtualenv environment

If you have not installed virtualenv yet, you need to do so before proceed.

Check my post for more details about how to setup python virtual environment and why it is better to install python libraries in Python virtual environment.

  • Install pip and Virtualenv for python 2.x and python 3.x:
$ sudo apt-get update
$ sudo apt-get install openjdk-8-jdk git python-dev python3-dev python-numpy python3-numpy build-essential python-pip python3-pip python-virtualenv swig python-wheel libcurl3-dev
  • Create a Virtualenv environment in the directory for python2.x and python 3.x:
#for python 2.x
virtualenv --system-site-packages -p python ~/ipy-jupyter-venv

# for python 3.x 
virtualenv --system-site-packages -p python3 ~/ipy-jupyter-venv3

(Note: To delete a virtual environment, just delete the corresponding folder.  For example, In our cases, it would be rm -rf ipy-jupyter-venv or rm -rf ipy-jupyter-venv3.)

Step 1: Install IPython

Before installing IPython and Jupyter, be sure to activate your python virtual environment first.

# for python 2.x
$ source ~/ipy-jupyter-venv/bin/activate  # If using bash
(ipy-jupyter-venv)$  # Your prompt should change

# for python 3.x
$ source ~/ipy-jupyter-venv3/bin/activate  # If using bash
(ipy-jupyter-venv3)$  # Your prompt should change

Use the following command to install IPython

#for python 2.x

(ipy-jupyter-venv) liping:~$ pip install ipython

#for python 3.x

(ipy-jupyter-venv3) liping:~$ pip3 install ipython

Step 2: Install Jupyter

Use the following command to install Jupyter Notebook

#for python 2.x 

(ipy-jupyter-venv) liping:~$ pip install jupyter

#for python 3.x 

(ipy-jupyter-venv3) liping:~$ pip3 install jupyter

Step 3: Test

#for python 2.x

(ipy-jupyter-venv) liping:~$ which python

#for python 3.x
(ipy-jupyter-venv3) liping:~$ which python3


#for python 2.x
(ipy-jupyter-venv) liping:~$ which ipython


#for python 3.x
(ipy-jupyter-venv3) liping:~$ which ipython3


#for python 2.x
(ipy-jupyter-venv) liping:~$ which jupyter-notebook


#for python 3.x
(ipy-jupyter-venv3) liping:~$ which jupyter-notebook


Step 4: Add Kernel

The Jupyter Notebook and other frontends automatically ensure that the IPython kernel is available. However, if you want to use a kernel with a different version of Python, or in a virtualenv or conda environment, you’ll need to install that manually. 

We are using virutalenv, so we need to install IPython kernel in the virtualenv we created in Step 0 above.

(ipy-jupyter-venv) liping:~$  python -m ipykernel install --user --name myipy_jupter_env --display-name "ipy-jupyter-venv"

Installed kernelspec myipy_jupter_env in /home/liping/.local/share/jupyter/kernels/myipy_jupter_env

(ipy-jupyter-venv3) liping:~$  python3 -m ipykernel install --user --name myipy_jupter_env3 --display-name "ipy-jupyter-venv3"

Installed kernelspec myipy_jupter_env3 in /home/liping/.local/share/jupyter/kernels/myipy_jupter_env3

Step 5: Run Jupyter Notebook

#for python2.x

 (ipy-jupyter-venv) liping:~$ jupyter-notebook

#for python 3.x

 (ipy-jupyter-venv3) liping:~$ jupyter-notebook

If you are running Jupyter Notebook on a local Linux computer (not on a remote server), you can simply navigate to localhost:8888 to connect to Jupyter Notebook. If you are running Jupyter Notebook on a remote server, you will need to connect to the server using SSH tunneling as outlined in the Step 5-2 below.

At this point, you can keep the SSH connection open and keep Jupyter Notebook running or can exit the app and re-run it once you set up SSH tunneling. Let’s keep it simple and stop the Jupyter Notebook process. We will run it again once we have SSH tunneling working. To stop the Jupyter Notebook process, press CTRL+C, type Y, and hit ENTER to confirm. The following will be displayed:

[C 08:08:04.232 NotebookApp] Shutdown confirmed

[I 08:08:04.232 NotebookApp] Shutting down 0 kernels

Step 5-2: Connecting to a remote Server Using SSH Tunneling

This step is only for those who are connecting to a Jupyter Notebook installed on a remote server.

Now we will learn how to connect to the Jupyter Notebook web interface using SSH tunneling. Since Jupyter Notebook is running on a specific port on the remote server (such as :8888:8889 etc.), SSH tunneling enables you to connect to the remote server’s port securely.

Below I will describe how to create an SSH tunnel from  a Mac or Linux (Windows users can check step 4 introduced at here). Note the following instructions in this step refer to your local computer (not the remote server).

SSH Tunneling with a Mac or Linux

Open a new terminal window on your Mac or Linux.

Issue the following ssh command to start SSH tunneling:

$ ssh -L 8000:localhost:8888 your_server_username@your_server_ip

Note: The ssh command opens an SSH connection, but -L specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote server. This means that whatever is running on the second port number (i.e. 8888) on the remote server will appear on the first port number (i.e. 8000) on your local computer. You should change 8888 to the port which Jupyter Notebook is running on. Optionally change port 8000 to one of your choosing (for example, if 8000 is used by another process). Use a port greater or equal to 8000 (ie 80018002, etc.) to avoid using a port already in use by another process. 

If no error shows up after running the ssh -L command, now be sure to activate your python virtual environment first.

# for python 2.x 

$ source ~/ipy-jupyter-venv/bin/activate  # If using bash (ipy-jupyter-venv)$  # Your prompt should change 

# for python 3.x 

$ source ~/ipy-jupyter-venv3/bin/activate  # If using bash (ipy-jupyter-venv3)$  # Your prompt should change

you can run Jupyter Notebook by issuing the following command:

#for python2.x  
(ipy-jupyter-venv) liping:~$ jupyter-notebook 

#for python 3.x  
(ipy-jupyter-venv3) liping:~$ jupyter-notebook

Now, from a web browser on your local machine, open the Jupyter Notebook web interface with http://localhost:8000 (or whatever port number you chose above when you ssh -L into your remote server).

Note: see the following instructions for how to change the startup folder of your Jupyter notebook.

  • In your terminal window
  • Enter the startup folder by typing cd /some_folder_name.
  • Type jupyter notebook to launch the Jupyter Notebook App (it will appear in a new browser window or tab).

Step 6 — Using Jupyter Notebook

By this point you should have Jupyter Notebook running, and you should be connected to it using a web browser. Jupyter Notebook is very powerful and has many features. Below I will outline a few of the basic features to get you started using the notebook. Automatically, Jupyter Notebook will show all of the files and folders in the directory it is run from.

To create a new notebook file, select New > Python 3 or New > ipy-jupyter-venv3 from the top right pull-down menu (Note: this is the so called kernel we installed in Step 4 above):

This will open a notebook. We can now run Python code in the cell or change the cell to markdown. For example, change the first cell to accept Markdown by clicking Cell > Cell Type > Markdown from the top navigation bar. We can now write notes using Markdown and even include equations written in LaTeX by putting them between the $$ symbols. For example, type the following into the cell after changing it to markdown:

# Simple Equation

Let us now implement the following equation:
$$ y = x^2$$

where $x = 2$

To turn the markdown into rich text, press CTRL+ENTER, and the following should be the results:

Note: You can use the markdown cells to make notes and document your code.

Now Let’s implement that simple equation and print the result. Select Insert > Insert Cell Below to insert and cell and enter the following code:

#for python 2.x
x = 2
y = x*x
print y

#for pyton 3.x
x = 2
y = x*x
print (y)

To run the code, press CTRL+ENTER. The following should be the results:

You now can include python libraries and use the notebook as you would with any other Python development environment! 

You should be now able to write reproducible Python code and notes using markdown using Jupyter notebook running on a remote server. To get a quick tour of Jupyter notebook, select Help > User Interface Tour from the top navigation menu.

Happy learning and coding!

Step 7: Deactivate your virtualenv

Each time you would like to use iPython and Jupyter, you need to activate the virtual environment into which it installed, and when you are done using iPython and Jupyter, deactivate the environment.

# for python 2
(ipy-jupyter-venv)$ deactivate
$  # Your prompt should change back

#for python 3
(ipy-jupyter-venv3)$ deactivate
$  # Your prompt should change back

Note: To delete a virtual environment, just delete its folder. (In this case, it would be rm -rf ipy-jupyter-venv or rm -rf ipy-jupyter-venv3.)


Intalling Jupyter in a virtualenv (pdf)

Running iPython cleanly inside a virtualenv (pdf)

Using a virtualenv in an IPython notebook

Installing the IPython kernel

How To Set Up a Jupyter Notebook to Run IPython on Ubuntu 16.04 (pdf)

 Running the Jupyter Notebook (Change Jupyter Notebook startup folder (OS X))




The Data Science Venn Diagram by Drew Conway

What is Data Science?

Data Science is a surprisingly hard definition to nail down, especially given the fact that how ubiquitous the term has become.

Vocal critics have variously dismissed the term as a superfluous label (after all, what science doesn’t involve data?)But, these critiques miss something important.

Data science, is perhaps the best label we have for the cross-disciplinary set of skills that are becoming increasingly important in many applications across industry and academia. This cross-disciplinary piece is the key.

In VanderPlas’s opinion, the best existing definition of data science is illustrated by Drew Conway’s Data Science Venn Diagram (see the figure below), first published on Drew Conway’s blog in September 2010.

The Data Science Venn Diagram above captures the essence of what people mean when they say “data science”:

it is fundamentally an interdisciplinary subject. Data science comprises three distinct and overlapping areas:

the skills of a statistician who knows how to model and summarize (big) datasets;

the skills of a computer scientist who can design and use algorithms to efficiently store, process, and visualize this data; and

the domain expertise — what we might think of as “classical” training in a subject — necessary both to formulate the right questions and to put their answers in context.

With this in mind, it would be better to think of data science not as a new domain of knowledge to learn, but as a new set of skills that you can apply within your current area of expertise.

(If you want to get started with your data science journey and apply it in your area of expertise, check out this page for some useful resources that I have collected for you.)


References and Further Reading List: