Run Jupyter notebook from terminal with tmux

This post will walk you through how to run a Jupyter notebook script from terminal with tmux (check here for my post about tmux usage).

When you are running Jupyter on a remote server or on cluster/ cloud resources, there are situations where you would like the Jupyter on the remote server or cluster continue running without termination when you shut down your laptop or desktop that you used to access the remote server. tmux will help with this.

In this post, we cover how to let your jupyter notebook running on a remote server continue running without termination via tmux.

Step 1: connect to your remote server with port forwarding

check the Step 5-2 in my post here about setting up Jupyter notebook for how to access your remote server with port forwarding, if you are not familiar with it.

Step 2: install tmux 

 check here for my post about tmux installation and usage

Step 3: install runipy  python package

Check here for runipy installation and usage.

Step 4: in your terminal type the following command, then it will go into tmux window

$ tmux

Step 5: Start jupyter notebook within your tmux session with the following command

$ jupyter notebook --no-browser

The –no-browser option prevents Jupyter from automatically opening a browser window.

Let this terminal stay running.

Step 6: from your laptop, ssh to your remote server (does not need port forwarding this time)

Step 7: cd to where the jupyter notebook script located that you would like to run from terminal

If you do not know what does cd mean and do, check my post for a list of commonly used Linux commands.

Step 8:  use the following command to run your ipynb script (this will save the output of each cell back to the notebook file)

$ runipy -o MyNotebook.ipynb 

To save the notebook output as a new notebook, run:

$ runipy MyNotebook.ipynb OutputNotebook.ipynb

If your ipynb script without any error itself, it should be running on the server now.

Step 9:  Things to pay attention to:

Do not close the terminal where you run the ipynb script within tmux session on your computer that you used to connect to the remote server, that will cause the termination of running the ipynb. But you can make your laptop in sleep or even shut down the computer, the tmux session will keep the ipynb running on your remote server and save the output in the ipynb.

References:

https://www.datasciencebytes.com/bytes/2015/12/18/using-jupyter-notebooks-securely-on-remote-linux-machines/ 

http://forums.fast.ai/t/ipython-notebook-on-a-remote-server-with-tmux/10044/2

tmux resources

This post provides a brief introduction to tmux and some commonly used commands and useful resources about tmux.

(Thanks Davide for recommending such a handy tool to me.)

(Stay tuned — I will update this post while I am gaining new skills about tmux.)

======What is tmux?

According to the tmux authors:

tmux is a terminal multiplexer. What is a terminal multiplexer? It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal. And do a lot more.

 

====== tmux sessions, windows, and panes explained

One of these features is the ability to break your session into more discreet components, called windows and panes. These are good for organizing multiple varied activities in a logical way.

Let’s look at how they relate to each other.

Nesting

tmuxnesting

tmux sessions have windows, and windows have panes. Below you can see how how they are conceptualized:

  • Sessions are for an overall theme, such as work, or experimentation, or sysadmin.
  • Windows are for projects within that theme. So perhaps within your experimentation session you have a window titled noderestapi, and one titled lua sample.
  • Panes are for views within your current project. So within your sysadmin session, which has a logs window, you may have a few panes for access logs, error logs, and system logs.

It’s also possible to create panes within a session without first creating a separate window. I do this sometimes. Hopefully it isn’t as horrible as it sounds right after reading about nesting.

Q about differences between sessions and windows and panes:

Can someone explain how they use sessions and windows and panes?
It feels like one too many levels, why both sessions AND windows?
I’m willing to believe there is a use case, but for the life of me I can’t come up with one.

A: 

I use sessions for different projects, and windows and panes within a project. For example, I’ll have a session for my puppet code, and a session for a bash script I’m working on. Within the puppet session, if I’m going to work on a new module, I open a new window. If I’m writing code in vim, I usually split off a pane so I can test running it next to it. Having the error output next to the code makes debugging really fast.

In a lot of ways, it basically acts like a tiling window manager for the terminal. The advantage over a window manager is that the whole layout can be accessed remotely. So if I’m working on a project from work, I can quickly resume working on the project from my home computer after SSH’ing in and reattaching to the tmux session.

 

====== see below for commonly used tmux comands

By default, tmux uses Ctrl-b as its shortcut activation chord, which enables you perform a number of functions quickly.

  • Switch between different windows

Ctrl – b (presee the  ctrl and b keys at the same time and release them at the same time, and then immediately press window number)  window number

  • Cursor move within an active window

Ctrl-b (release the keys, and immediately press the “[” key ) the key with “[“.

when you see the cursor becomes a solid flashing diamond, you can use arrow key to move your cursor in the active terminal window.

Note: Press “q” to exit the cursor move mode.

  • Session management
  • s list sessions
  • $ rename the current session # default session name and window name is number, can rename session names to meaningful ones
  • d detach from the current session

tmux is developed on a client-server model which means that the session is stored on the server and persist beyond ssh logout.

The following command will create a new session called mysession:

tmux new-session -s mysession

To attach to a session run:

tmux attach -t mysession

To list all session run:

tmux ls

You can kill a session using the following command:

tmux kill-session -t mysession

you can grossly kill all tmux processes with the following command:

pkill -f tmux

Frequently used sessions commands

Ctrl-b d	  Detach from the current session 
Ctrl-b (          Go to previous session
Ctrl-b )          Go to next session
Ctrl-b L          Go to previously used session
Ctrl-b s          Choose a session from the sessions list

Ctrl + D    — exit tmux from terminal.

  • Windows (tabs) Management

Each session can have multiple windows. By default all windows are numbered starting from zero.

Frequently used windows (tabs) commands

Ctrl-b 1  Switch to window 1
Ctrl-b c  Create new window
Ctrl-b w  List all windows
Ctrl-b n  Go to next window
Ctrl-b p  Go to previous window
Ctrl-b f  Find window
Ctrl-b ,  Name window
Ctrl-b w  Choose a window from the windows list
Ctrl-b &  Kill the current window

 

  • One of the handy things about tmux is how easy it is to resize panes:

Ctrl +b, followed by holding down Alt, and using the arrow keys to resize.

 

======things about how to save sessions and recover sessions.

If you reboot you computer you will lose the sessions. Sessions cannot be saved. But, they can be scripted. What most do in fact is to script some sessions so that you can re-create them.

======Installing tmux

  • Installation with sudo privilege

Installation is pretty straightforward if you have Ubuntu or any other Debian-based distribution you can install tmux with:

sudo apt-get install tmux

on CentOS/Fedora:

yum install tmux

and on MacOS:

brew install tmux

Note: check the comments for some up-to-date scripts.

 

References and further reading list

This tutorial covers installation of tmux and some commonly used commands.

This is a pretty good introduction to tmux, it includes why tmux and the comaprision between tmux and screen, as well as some tmux shortcuts