Setting up Python and Jupyter Notebook on Mac

This page provides the instructions for how to install and run IPython and  Jupyter Notebook in a virtualenv  on Mac.

Most probably your Mac has already come with Python installed (see step 1 and step 2 below to check whether Python and Python 3 is installed on your mac, because my Mac book air has both Python and Python3.6 installed, I will go ahead to step 3 to install virtualenv).  

Open a terminal on your mac and type in Python and Python3 to see whether Python and Python3 installed on your mac. See below for example.

Follow the steps below for detailed instructions.

Step 1: Check whether Python is installed on your Mac

To test whether your Mac has Python 2.x or 3.x, look at the instructions below.

Open a terminal  (in your launchpad, type in terminal, and then click and open it.)

Type in python, if you see similar thing as the pic shown below, it means python 2.7 was installed on your Mac

type exit() to exit from the Python 2.7 environment

Type python3 in your terminal, if you see similar warning as the pic shown below, it means you will need to install Python 3 (see the steps below for installation). If you see older version (< 3.5) of python 3 installed on your mac, follow step 2 to install Python 3.5

Step 2: Install Python 3.5

I recommend you download Python 3.5.4 from here, if you do not have Python 3.x pre-installed on your mac.

https://www.python.org/downloads/mac-osx/ 

Click the installer and install it following the wizard.

When you finish the installation, type python3 in your terminal, if you see similar stuff in the pic below, Python 3.5 is installed successfully and you are ready to proceed to step 3.

Step 3: Install virtualenv 

Note: the command text in blue is the exact command you should type into your terminal, the same through this whole instruction.

Lipings-MacBook-Pro:~ Liping$ pip3 install virtualenv

Step 4: Setup virtualenv environment

Issue the command below to set up a virtualenv environment that we will use later, using the vitualenvwe installed in step 3 above.

$ virtualenv --system-site-packages -p python3 ~/ipy-jupyter-venv3

Step 5: Install IPython

Before installing IPython and Jupyter Notebook, be sure to activate your python virtual environment first.

$ source ~/ipy-jupyter-venv3/bin/activate 
(ipy-jupyter-venv3)$  # Your prompt should change

Type the following command to install IPython

(ipy-jupyter-venv3) liping:~pip3 install ipython

Step 6: Install Jupyter Notebook

Use the following command to install Jupyter Notebook

(ipy-jupyter-venv3) liping:~$ pip3 install jupyter

Step 7: Test jupypter notebook installation

(ipy-jupyter-venv3) liping:~$ which python3

/Users/Liping/ipy-jupyter-venv3/bin/python3
#for python 3.x
(ipy-jupyter-venv3) liping:~$ which ipython3

/Users/Liping/ipy-jupyter-venv3/bin/ipython3
#for python 3.x
(ipy-jupyter-venv3) liping:~$ which jupyter-notebook

/Users/Liping/ipy-jupyter-venv3/bin/jupyter-notebook

Step 8: Add Kernel

The Jupyter Notebook and other frontends automatically ensure that the IPython kernel is available. However, if you want to use a kernel with a different version of Python, or in a virtualenv environment, you’ll need to install that manually. 

We are using virutalenv, so we need to install IPython kernel in the virtualenv we created in Step 4 above.

(ipy-jupyter-venv3) liping:~$  python3 -m ipykernel install --user --name myipy_jupter_env3 --display-name "ipy-jupyter-venv3"

Installed kernelspec myipy_jupter_env3 in /Users/Liping/Library/Jupyter/kernels/myipy_jupter_env3

 

Step 9: Before we are running our jupyter Notebook, let us create a folder from which we will start our jupyter notebook.

The following command will change our directory to Desktop (cd refers to change directory)

(ipy-jupyter-venv3) Lipings-MacBook-Pro:~ Liping$ cd Desktop 

the following command will help us create a folder named Geog597_ML_session under our Desktop folder

(ipy-jupyter-venv3) Lipings-MacBook-Pro:Desktop Liping$ mkdir test_jupyter

the following command will change our directory into the folder we just created.

(ipy-jupyter-venv3) Lipings-MacBook-Pro:Desktop Liping$ cd test_jupyter/

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$ 

Step 10: Run Jupyter Notebook

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$ jupyter-notebook

Then your default browser should automatically open a web page similar to the one shown below.

If the web page does not show up automatically, just type localhost:8888 into your browser, you should see the page shown above.

 

Step 11: Testing and Using Jupyter Notebook

By this point you should have Jupyter Notebook running, and you should be connected to it using a web browser. Jupyter Notebook is very powerful and has many features. Below I will outline a few of the basic features to get you started using the notebook. Automatically, Jupyter Notebook will show all of the files and folders in the directory it is run from (for our case, it is empty now, because we have not put anything in that folder Desktop/test_jupyter we just created in step 9 ).

To create a new notebook file, select New > ipy-jupyter-venv3 from the top right pull-down menu (Note: this is the so called kernel we installed in Step 8 above):

This will open a notebook. We can now run Python code in the cell or change the cell to markdown (markdown is for note, not for code). For example, change the first cell to accept Markdown by clicking Cell > Cell Type > Markdown from the top navigation bar, or by click the Markdown shown in the pic below. We can now write notes using Markdown,  for example, type the following into the cell after changing it to markdown:

# Testing

Hit Ctrl + Enter, you will see it the text changed into heading style.

          Then insert one cell by the menu shown below (insert Cell Below).

type in the following

a = 5
b = 10
print (a+b)

and then hit Ctrl + Enter.  You should see the following.

Step 12: Stop jupyter Notebook.

To stop the Jupyter Notebook process, press CTRL+C,

type Y, and hit ENTER to confirm. The following will be displayed:

Step 13: Exit virtualenv environment

Once you are done, remember to exit your virtualenv using the following command:

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$

deactivate

Note that your prompt changes back (see the pic below)

Step 14: Instructions for using Jupyter Notebook next time

Next time when you need to use you jupyther notebook, following the steps summarized below.

Lipings-MacBook-Pro:~ Liping$ source ~/ipy-jupyter-venv3/bin/activate  

(ipy-jupyter-venv3)$  # Your prompt should change

Change directory to where you would like to start your jupyter-notebook

(ipy-jupyter-venv3) Lipings-MacBook-Pro:~ Liping$ cd Desktop/test_jupyter

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$ 

Run jupyter notebook:

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$ jupyter-notebook

 

Congratulations, you have now set up ipython and jupyter notebook on your mac!

Timing how long a python script runs

This post introduces several ways to find out how long a python script takes to complete its execution.

  • If you are using Linux or Mac OS, in your terminal
$ time ./your_script.py
  • Several ways to do the task by adding a few lines of code in your py script.
import time
startTime = time.time()

your_func() #python3: print ("It took", time.time() - startTime, "seconds.")

See the following for an example in python 3. 

import time
import functools

startTime = time.time()

print(functools.reduce(lambda x,y: x+y, [47,11,42,13]))

#python3:
print ("It took", time.time() - startTime, "seconds.")

Another way to do the same thing:

from datetime import datetime
startTime = datetime.now()

#do something

#Python 2: 
print datetime.now() - startTime 

#Python 3: 
print(datetime.now() - startTime)

One more way to do the same thing with a nicely formatted output.

import sys
import timeit

startTime = timeit.default_timer()

#do some nice things...

stopTime = timeit.default_timer()
totalRunningTime = stopTime - startTime

# output running time in a nice format.
mins, secs = divmod(totalRunningTime, 60)
hours, mins = divmod(mins, 60)

sys.stdout.write("Total running time: %d:%d:%d.\n" % (hours, mins, secs))

If you want to compare two blocks of code / functions quickly you can do the following:

import timeit

startTime = timeit.default_timer()
your_func1()
#python3
print(timeit.default_timer() - startTime)

startTime2 = timeit.default_timer()
your_func2()
#python3
print(timeit.default_timer() - starTime2)

[django] Sending matplotlib generated figure(s) to django web app

This post introduces how to serve the figures generated by matplotlib to django web app without saving on the server.

In your django views py file, import the following libraries

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

from io import BytesIO
import base64

Note: the two lines of code above in blue need to be placed at the very beginning of the py script; otherwise, you would meet the following error: _tkinter.TclError: no display name and no $DISPLAY environment variable.

In the views py file, in the function that you defined to pass the image data to the front end template file, add the following code:

buf = BytesIO()
plt.savefig(buf, format='png', dpi=300)
image_base64 = base64.b64encode(buf.getvalue()).decode('utf-8').replace('\n', '')
buf.close()

Note: in the function that you defined to pass the image data to the front end template file in your views py file, remember to send the value of the variable image_base64 via, for example, json.

Now, in your front end template file, you can add the following image tag.

<img src="data:image/png;base64,{{image_base64}}" alt="some text to display to your users when the image does not show correctly" width=500 height=auto />

You now should be able to see the figure displayed on your web app page:)

For more details about using data url to pass image data to front end html file, check here (pdf).

 

 

Tips about Jupyter Notebook

This post provides some tips about Jupyter Notebook.

  • When you open a pdf file in a browser via your running Jupyter Notebook.

You may encounter some error similar to this “Error! (file) is not UTF-8 encoded”,

the solution:

check the address in your browser, for example, for the following case,

http://localhost:8000/edit/confusionmatrix.pdf

if I change the edit in the url address given above to  view

http://localhost:8000/view/confusionmatrix.pdf

You will see your beautiful fig or document in pdf format.

  • Another similar situation:

When you open a html file in your browser via your running Jupyter Notebook, and you would like to edit the html file, instead of seeing its display,

the solution:

change view to edit in the url address of the file, see the following for an example, then you should be able to edit your html file.

http://localhost:8000/view/index.html 

to

http://localhost:8000/edit/index.html

Happy Jupyter Notebooking!

How to Deploy a Django Application on a Server Running Ubuntu

This post provides tutorials on how to deploy a Django application on a server running Ubuntu.

Softwares needed to set up a server for Django application:

  • Virtualenv & Virtualenvwrapper / Conda
  • Django
  • Gunicorn
  • Nginx
  • Supervisor

Concept

Nginx will face the outside world. It will serve media files (images, CSS, etc.) directly from the file system. However, it can’t talk directly to Django applications; it needs something that will run the application, feed it requests from the web, and return responses.

That’s Gunicorn‘s job. Gunicorn will create a Unix socket, and serve responses to Nginx via the wsgi protocol – the socket passes data in both directions:

The outside world <-> Nginx <-> The socket <-> Gunicorn

All this family will live into a Virtualenv. Already wondered why Virtualenv is so useful when you develop Pythons’s applications? Continue to read and you will understand.

Before you start

References

I assume you have a server available on which you have root privileges. I am using a server running Debian 7, so everything here should also work on an Ubuntu server or other Debian-based distribution. If you’re using an RPM-based distro (such as CentOS), you will need to replace the aptitude commands by their yum counterparts and if you’re using FreeBSD you can install the components from ports.

Apache and Nginx together on Ubuntu

Is it possible to run  Nginx and Apache at the same time on the same machine?

The answer is YES.

This post provides instructions on how to configure Apache and Nginx  to work together on the same machine running Ubuntu.

Overview

Nginx and Apache are great and powerful web servers. However, they both have drawbacks; Apache uses up server memory while Nginx (best used for static files) require the help of php-fpm to process dynamic content.

Nginx is an excellent lightweight web server designed to serve high traffic while Apache is another popular web server serving more than half of all active websites in the world. One can combine the two web servers to significant effect, with Nginx serving as static web server front and while Apache is processing the back end. So let’s look into how to configure your Nginx to work with Apache side by side.

Configuring Apache and Nginx

Apache and Nginx can definitely run simultaneously. The default config will not allow them to start at the same time because they will both try to listen on the same port and the same IP.However, you can easily either change the ports or ports and IPs, or IPs. There are various ways to make them run either one behind the other(usually Apache behind Nginx because Nginx will be the first entry point in the chain since it’s faster for static resources, and Apache will then only be triggered for some advanced dynamic rendering/processing) or just side by side.

Set different ports for each server. That means you can leave port 80 for Nginx and assign Apache a different port.

Nginx

Install Nginx

Install and configure Nginx that will serve as the front end of your site.

$ sudo apt-get install nginx

Once it has downloaded, configure the virtual host to run on the front end. However, a few changes are required on the configuration file.

Configure Nginx

Open up the nginx configuration file

$ sudo nano /etc/nginx/sites-available/example

For example, you could tell apache to listen on 127.0.0.1:8080 and instruct Nginx to reverse –proxy traffic to Apache while still serving static content.

Those text in blue is where you should edit according to your server info.

 server {
listen 127.0.0.1:80;
server_name some.name another.dname;
access_log /var/log/nginx/something-access.log;
location / {
proxy_pass http://localhost:8080;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ~* ^.+\.(jpg|js|jpeg|png)$ {
root /some/where/on/your/disks;
}
# put your static hosting config here.
}

Activate the virtual host.

$ sudo ln -s /etc/nginx/sites-available/example /etc/nginx/sites-enabled/example

Delete the default Nginx server block.

$ sudo rm /etc/nginx/sites-enabled/default

Apache

Install Apache

Install the backend which is Apache

$ sudo apt-get install apache2

Apache starts running on port 80 as Nginx is not started.Let’s make Apache listen on a different port so that they can work together.

Configuring Apache

Open Apache ports.conf file using the below command

$ sudo nano /etc/apache2/ports.conf

Look for the following line;

Listen 127.0.0.1:80

and  change it to

Listen 127.0.0.1:8080

Save and Exit.

Next, edit the default virtual host file in Apache.

The <VirtualHost> in this file is set to serve sites only on port 80

$ sudo nano /etc/apache2/sites-available/000-default.conf

Look for the following line,

<VirtualHost 127.0.0.1:80>

then, change it to;

<VirtualHost 127.0.0.1:8080>

Save the file and reload Apache.

$ sudo service apache2 reload

Verify that Apache is now listening on 8080.

$ sudo netstat -tlpn

The output is shown below, with apache2 listening on :::8080.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1086/sshd
tcp6 0 0 :::8080 :::* LISTEN 4678/apache2

Nginx is configured and running as the front web server on port while Apache is configured to run on the backend on port 8080. Nginx redirects proxy traffic as it still serves static content.

Conclusion

The most important thing we take from this simple configuration is that Apache and Nginx can and do work together. A problem may arise when they both listen to the same ports. By giving them different ports to listen to, your system functionality is assured.

References

 

 

 

Dynamic GPU usage monitoring (CUDA)

To dynamically  monitor NVIDIA GPU usage, here I introduce two methods:

method 1: use nvidia-smi

in your terminal, issue the following command:

$ watch -n 1 nvidia-smi

It will continually update the gpu usage info (every second, you can change the 1 to 2 or the time interval you want the usage info to be updated).

method 2: use the open source monitoring program glances with its GPU monitoring plugin

in your terminal, issue the following command to install glances with its GPU monitoring plugin

$ sudo pip install glances[gpu]

to launch it, in your terminal, issue the following command:

 $ sudo glances

Then you should see your GPU usage etc. It also monitors the CPU, disk IO, disk space, network, and a few other things

For more commonly used Linux commands, check my other posts at here  and here .

Resources about Azure Batch AI

This post provides some resources about Microsoft Auzre Batch AI.

Introduction:

Batch AI is a managed service that enables data scientists and AI researchers to train AI and other machine learning models on clusters of Azure virtual machines, including VMs with GPU support. You describe the requirements of your job, where to find the inputs and store the outputs, and Batch AI handles the rest.

Hands-on tutorials:

Videos:

 

Run Jupyter notebook from terminal with tmux

This post will walk you through how to run a Jupyter notebook script from terminal with tmux (check here for my post about tmux usage).

When you are running Jupyter on a remote server or on cluster/ cloud resources, there are situations where you would like the Jupyter on the remote server or cluster continue running without termination when you shut down your laptop or desktop that you used to access the remote server. tmux will help with this.

In this post, we cover how to let your jupyter notebook running on a remote server continue running without termination via tmux.

Step 1: connect to your remote server with port forwarding

check the Step 5-2 in my post here about setting up Jupyter notebook for how to access your remote server with port forwarding, if you are not familiar with it.

Step 2: install tmux 

 check here for my post about tmux installation and usage

Step 3: install runipy  python package

Check here for runipy installation and usage.

Step 4: in your terminal type the following command, then it will go into tmux window

$ tmux

Step 5: Start jupyter notebook within your tmux session with the following command

$ jupyter notebook --no-browser

The –no-browser option prevents Jupyter from automatically opening a browser window.

Let this terminal stay running.

Step 6: from your laptop, ssh to your remote server (does not need port forwarding this time)

Step 7: cd to where the jupyter notebook script located that you would like to run from terminal

If you do not know what does cd mean and do, check my post for a list of commonly used Linux commands.

Step 8:  use the following command to run your ipynb script (this will save the output of each cell back to the notebook file)

$ runipy -o MyNotebook.ipynb 

To save the notebook output as a new notebook, run:

$ runipy MyNotebook.ipynb OutputNotebook.ipynb

If your ipynb script without any error itself, it should be running on the server now.

Step 9:  Things to pay attention to:

Do not close the terminal where you run the ipynb script within tmux session on your computer that you used to connect to the remote server, that will cause the termination of running the ipynb. But you can make your laptop in sleep or even shut down the computer, the tmux session will keep the ipynb running on your remote server and save the output in the ipynb.

References:

https://www.datasciencebytes.com/bytes/2015/12/18/using-jupyter-notebooks-securely-on-remote-linux-machines/ 

http://forums.fast.ai/t/ipython-notebook-on-a-remote-server-with-tmux/10044/2