The Data Science Venn Diagram by Drew Conway

What is Data Science?

Data Science is a surprisingly hard definition to nail down, especially given the fact that how ubiquitous the term has become.

Vocal critics have variously dismissed the term as a superfluous label (after all, what science doesn’t involve data?)But, these critiques miss something important.

Data science, is perhaps the best label we have for the cross-disciplinary set of skills that are becoming increasingly important in many applications across industry and academia. This cross-disciplinary piece is the key.

In VanderPlas’s opinion, the best existing definition of data science is illustrated by Drew Conway’s Data Science Venn Diagram (see the figure below), first published on Drew Conway’s blog in September 2010.

The Data Science Venn Diagram above captures the essence of what people mean when they say “data science”:

it is fundamentally an interdisciplinary subject. Data science comprises three distinct and overlapping areas:

the skills of a statistician who knows how to model and summarize (big) datasets;

the skills of a computer scientist who can design and use algorithms to efficiently store, process, and visualize this data; and

the domain expertise — what we might think of as “classical” training in a subject — necessary both to formulate the right questions and to put their answers in context.

With this in mind, it would be better to think of data science not as a new domain of knowledge to learn, but as a new set of skills that you can apply within your current area of expertise.

(If you want to get started with your data science journey and apply it in your area of expertise, check out this page for some useful resources that I have collected for you.)


References and Further Reading List:



List Node process and kill specific process if needed

This post provides instructions on how to list running nodeJS application, and  how to kill specific process.

  • List node process:
$ ps -e|grep node
  • If you want know, how many nodejs processes running, you can use the following command
$ ps -aef | grep node

It will give list of nodejs process with it’s project name. This is helpful when you are running multipe nodejs application & you want kill specific process for the specific project.

  • Kill  specific process using the following command:
$kill -9 XXXX

XXXX is the process number to be killed.

  • You can kill all node processes using 
$ pkill node

If all those kill process commands mentioned above don’t work for you, probably you will need to check if you were using any other packages to run your node process.

For example, if you were running your node process using PM2(a NPM package), the kill [processID] command indeed will disable the process but it will keep the port that you ran the nodejs app occupied. You will  need to go into PM2 and dump all node process to free up the port again.


You can use the following comment to check all running process, including node and java apps etc.

$ ps -ef

you will see a list with the headers as the following line:


 Among the line,

STIME means the time your app starts,

TIME   means total CPU usage of the corresponding process


CMD is the app’s name running (i.e., name of the process, including arguments, if any)


if you only want to list node process,

use the following command,

$ ps -ef | grep node

if you only want to list java process,

use the following command,

$ ps -ef | grep java

You got the idea…

-e and -f are options to the ps command, and pipes  (i.e., | ) take the output of one command and pass it as the input to another. Here is a full breakdown of this command:

ps – list processes

-e – show all processes, not just those belonging to the user

-f – show processes in full format (more detailed than default)

command 1 | command 2 – pass output of command 1 as input to command 2

grep find lines containing a pattern

processname – the pattern for grep to search for in the output of ps -ef

So, the following command means:

look for lines containing processname in a detailed overview/snapshot of all current processes, and display those lines

ps -ef | grep processname


Commonly used commands for Node.js (Ubuntu)

This post provides some commonly used commands for Node.js

  • Start a Node.js application
$node server.js

Note: After you start your node application, go to the host name address (e.g., of your server and then a semicolon, and then give port name. (For example, if it is running on port 8081, then type the server host name address:8081/) then you should see your node application web page.

  • Stop a Node.js application
$Ctrl + C



Install Keras with GPU TensorFlow as backend on Ubuntu 16.04

This post introduces how to install Keras with TensorFlow as backend on Ubuntu Server 16.04 LTS with CUDA 8 and a NVIDIA TITAN X (Pascal) GPU, but it should work for Ubuntu Desktop 16.04 LTS.

We gratefully acknowledge the support of NVIDIA Corporation with awarding one Titan X Pascal GPU used for our machine learning and deep learning based research.

Keras is a great choice to learn machine learning and deep learning. Keras has easy syntax and can use Google TensorFlow or Microsoft CNTK or Theano as its backend.  Keras is simply a wrapper around more complex numerical computation engines such as TensorFlow and Theano.

Keras abstracts away much of the complexity of building a deep neural network, leaving us with a very simple, nice, and easy to use interface to rapidly build, test, and deploy deep learning architectures.

TensorFlow is extremely flexible, allowing you to deploy network computation to multiple CPUs, GPUs, servers, or even mobile systems without having to change a single line of code.

This makes TensorFlow an excellent choice for training distributed deep learning networks in an architecture agnostic way.

Now Let’s start on the installation of Keras with TensorFlow as its backend.

1: Setup Python virtual environment

Check my post about more details about how to setup python virtual environment and why it is better to install python libraries in Python virtual environment.

  • Install pip and Virtualenv for python and python 3:
$ sudo apt-get update
$ sudo apt-get install openjdk-8-jdk git python-dev python3-dev python-numpy python3-numpy build-essential python-pip python3-pip python-virtualenv swig python-wheel libcurl3-dev
  • Create a Virtualenv environment in the directory for python and python 3:
#for python 2
virtualenv --system-site-packages -p python ~/keras-tf-venv

# for python 3 
virtualenv --system-site-packages -p python3 ~/keras-tf-venv3

(Note: To delete a virtual environment, just delete its folder.  For example, In our cases, it would be rm -rf keras-tf-venv or rm -rf keras-tf-venv3.)

2: Update & Install NVIDIA Drivers (skip this if you do not need to TensorFlow GPU version)

Check another post I wrote(steps 1-4 in that post) for detailed instructions about how to update and install NVIDIA Drive and CUDA 8.0 and cuDNN for the requirements of GPU TensorFlow installation.

Notes: If you have old version of NVIDIA driver installed used the following to remove it first before installation of new driver.

Step 1: Remove older version of NVIDIA
sudo apt-get purge nvidia*

Step 2: Reboot the system

test whesther it is removed

$ sudo nvidia-smi 
$ sudo: nvidia-smi: command not found  # this means the old driver was uninstalled.

(Note: If you have older version of CUDA and cuDNN installed, check the post for uninstallation.  How to uninstall CUDA Toolkit and cuDNN under Linux? (02/16/2017) (pdf). Actually to uninstall (older version) of CUDA, it tells you how to uninstall it when you install, see the Install cuda 8.0 below.)

(Note: I tried to install the latest Nvidia drive, latest cuda and latest cudnn (i.e., v6.0), but it did not work for me when I installed TensorFlow. After a few testing, I found when I install Nvidia drive 375.82,, cudnn-8.0-linux-x64-v5.1.tgz. it works.)

Install cuda 8.0:

Toolkit: Installed in /usr/local/cuda-8.0
Samples: Installed in /home/liping, but missing recommended libraries

Please make sure that
– PATH includes /usr/local/cuda-8.0/bin
– LD_LIBRARY_PATH includes /usr/local/cuda-8.0/lib64, or, add /usr/local/cuda-8.0/lib64 to /etc/ and run ldconfig as root

To uninstall the CUDA Toolkit, run the uninstall script in /usr/local/cuda-8.0/bin

Please see CUDA_Installation_Guide_Linux.pdf in /usr/local/cuda-8.0/doc/pdf for detailed information on setting up CUDA.

***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 361.00 is required for CUDA 8.0 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
sudo <CudaInstaller>.run -silent -driver

Logfile is /tmp/cuda_install_3813.log


3: Install TensorFlow

Before installing TensorFlow and Keras, be sure to activate your python virtual environment first.

# for python 2
$ source ~/keras-tf-venv/bin/activate  # If using bash
(keras-tf-venv)$  # Your prompt should change

# for python 3
$ source ~/keras-tf-venv3/bin/activate  # If using bash
(keras-tf-venv3)$  # Your prompt should change

 (keras-tf-venv)$ pip install --upgrade tensorflow   # Python 2.7; CPU support (no GPU support)
 (keras-tf-venv3)$ pip3 install --upgrade tensorflow   # Python 3.n; CPU support (no GPU support)
 (keras-tf-venv)$ pip install --upgrade tensorflow-gpu  # Python 2.7;  GPU support
 (keras-tf-venv3)$ pip3 install --upgrade tensorflow-gpu # Python 3.n; GPU support

Note: If the commands for installing TensorFlow given above failed (typically because you invoked a pip version lower than 8.1), install TensorFlow in the active virtualenv environment by issuing a command of the following format:

 (keras-tf-venv)$ pip install --upgrade TF_PYTHON_URL   # Python 2.7
 (keras-tf-venv3)$ pip3 install --upgrade TF_PYTHON_URL  # Python 3.N

where TF_PYTHON_URL identifies the URL of the TensorFlow Python package. The appropriate value of TF_PYTHON_URLdepends on the operating system, Python version, and GPU support. Find the appropriate value for TF_PYTHON_URL for your system here. For example, if you are installing TensorFlow for Linux, Python 2.7, and CPU-only support, issue the following command to install TensorFlow in the active virtualenv environment: (see below for examples. Note that check here to get the latest version for your system.)

#for python 2.7 -- CPU only
(keras-tf-venv)$pip install --upgrade \

#for python 2.7 -- GPU support
(keras-tf-venv)$pip install --upgrade \

# for python 3.5 -- CPU only
(keras-tf-venv3)$ pip3 install --upgrade \

# for python 3.5 -- GPU support
(keras-tf-venv3)$ pip3 install --upgrade \
  • Validate your TensorFlow installation. (as I just installed GPU tensorflow, so if you install CPU TensorFlow, the output might be slightly different.)

#For Python 2.7

(keras-tf-venv) :~$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
2017-08-01 14:28:31.257054: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 14:28:31.257090: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 14:28:31.257103: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 14:28:31.257114: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 14:28:31.257128: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 14:28:32.253475: I tensorflow/core/common_runtime/gpu/] Found device 0 with properties: 
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:03:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
2017-08-01 14:28:32.253512: I tensorflow/core/common_runtime/gpu/] DMA: 0 
2017-08-01 14:28:32.253519: I tensorflow/core/common_runtime/gpu/] 0: Y 
2017-08-01 14:28:32.253533: I tensorflow/core/common_runtime/gpu/] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
>>> print(
Hello, TensorFlow!
>>> exit()
(keras-tf-venv) :~$ 

#for python 3

(keras-tf-venv3) :~$ python3
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
2017-08-01 13:54:30.458376: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 13:54:30.458413: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 13:54:30.458425: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 13:54:30.458436: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 13:54:30.458448: W tensorflow/core/platform/] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 13:54:31.420661: I tensorflow/core/common_runtime/gpu/] Found device 0 with properties: 
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:03:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
2017-08-01 13:54:31.420692: I tensorflow/core/common_runtime/gpu/] DMA: 0 
2017-08-01 13:54:31.420699: I tensorflow/core/common_runtime/gpu/] 0: Y 
2017-08-01 13:54:31.420712: I tensorflow/core/common_runtime/gpu/] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
>>> print(
b'Hello, TensorFlow!'
>>> exit() 
(keras-tf-venv3) :~$

If you see the output as below, it indicates your TensorFlow was installed correctly.

Hello, TensorFlow!

3: Install Keras

(Note: Be sure that you activated your python virtual environment before you install Keras.)

Installing Keras is even easier than installing TensorFlow.

First, let’s install a few dependencies:

#for python 2
$ pip install numpy scipy
$ pip install scikit-learn
$ pip install pillow
$ pip install h5py

#for python 3
$ pip3 install numpy scipy
$ pip3 install scikit-learn
$ pip3 install pillow
$ pip3 install h5py

4: Verify that your keras.json file is configured correctly

Let’s now check the contents of our keras.json  configuration file. You can find this file at ~/.keras/keras.json .

use nano to open and edit the file.

$ nano ~/.keras/keras.json

The default values should be something like this:

 "epsilon": 1e-07,
 "backend": "tensorflow",
 "floatx": "float32",
 "image_data_format": "channels_last"

Can’t find your keras.json file?

On most systems the keras.json  file (and associated subdirectories) will not be created until you open up a Python shell and directly import the keras  package itself.

If you find that the ~/.keras/keras.json  file does not exist on your system, simply open up a shell, (optionally) access your Python virtual environment (if you are using virtual environments), and then import Keras:

#for python 2
$ python
>>> import keras
>>> quit()

#for python 3
$ python3
>>> import keras
>>> quit()

From there, you should see that your keras.json  file now exists on your local disk.

If you see any errors when importing keras  go back to the top of step 4 and ensure your keras.json  configuration file has been properly updated.

5: Test Keras + TensorFlow installation

To verify that Keras + TensorFlow have been installed, simply access the keras_tf  environment using the workon  command, open up a Python shell, and import keras :

Specifically, you can see the text Using TensorFlow backend  display when importing Keras — this successfully demonstrates that Keras has been installed with the TensorFlow backend.

Note: each time you would like to use Keras, you need to activate the virtual environment into which it installed, and when you are done using Keras, deactivate the environment.

# for python 2
(keras-tf-venv)$ deactivate
$  # Your prompt should change back

#for python 3
(keras-tf-venv3)$ deactivate
$  # Your prompt should change back

Note: To delete a virtual environment, just delete its folder. (In this case, it would be rm -rf keras-tf-venv or rm -rf keras-tf-venv3.)


Installing Keras with TensorFlow backend (by on November 14, 2016 in Deep Learning, Libraries, Tutorials)

Installing keras makes tensorflow can’t find GPU

Installing Nvidia, Cuda, CuDNN, TensorFlow and Keras

Keras as a simplified interface to TensorFlow: tutorial

I: Calling Keras layers on TensorFlow tensors

II: Using Keras models with TensorFlow

III: Multi-GPU and distributed training

IV: Exporting a model with TensorFlow-serving

tmux resources

This post provides a brief introduction to tmux and some commonly used commands and useful resources about tmux.

(Thanks Davide for recommending such a handy tool to me.)

(Stay tuned — I will update this post while I am gaining new skills about tmux.)

======What is tmux?

According to the tmux authors:

tmux is a terminal multiplexer. What is a terminal multiplexer? It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal. And do a lot more.


====== tmux sessions, windows, and panes explained

One of these features is the ability to break your session into more discreet components, called windows and panes. These are good for organizing multiple varied activities in a logical way.

Let’s look at how they relate to each other.



tmux sessions have windows, and windows have panes. Below you can see how how they are conceptualized:

  • Sessions are for an overall theme, such as work, or experimentation, or sysadmin.
  • Windows are for projects within that theme. So perhaps within your experimentation session you have a window titled noderestapi, and one titled lua sample.
  • Panes are for views within your current project. So within your sysadmin session, which has a logs window, you may have a few panes for access logs, error logs, and system logs.

It’s also possible to create panes within a session without first creating a separate window. I do this sometimes. Hopefully it isn’t as horrible as it sounds right after reading about nesting.

Q about differences between sessions and windows and panes:

Can someone explain how they use sessions and windows and panes?
It feels like one too many levels, why both sessions AND windows?
I’m willing to believe there is a use case, but for the life of me I can’t come up with one.


I use sessions for different projects, and windows and panes within a project. For example, I’ll have a session for my puppet code, and a session for a bash script I’m working on. Within the puppet session, if I’m going to work on a new module, I open a new window. If I’m writing code in vim, I usually split off a pane so I can test running it next to it. Having the error output next to the code makes debugging really fast.

In a lot of ways, it basically acts like a tiling window manager for the terminal. The advantage over a window manager is that the whole layout can be accessed remotely. So if I’m working on a project from work, I can quickly resume working on the project from my home computer after SSH’ing in and reattaching to the tmux session.


====== see below for commonly used tmux comands

By default, tmux uses Ctrl-b as its shortcut activation chord, which enables you perform a number of functions quickly.

  • Switch between different windows

Ctrl – b (presee the  ctrl and b keys at the same time and release them at the same time, and then immediately press window number)  window number

  • Cursor move within an active window

Ctrl-b (release the keys, and immediately press the “[” key ) the key with “[“.

when you see the cursor becomes a solid flashing diamond, you can use arrow key to move your cursor in the active terminal window.

Note: Press “q” to exit the cursor move mode.

  • Session management
  • s list sessions
  • $ rename the current session # default session name and window name is number, can rename session names to meaningful ones
  • d detach from the current session

tmux is developed on a client-server model which means that the session is stored on the server and persist beyond ssh logout.

The following command will create a new session called mysession:

tmux new-session -s mysession

To attach to a session run:

tmux attach -t mysession

To list all session run:

tmux ls

You can kill a session using the following command:

tmux kill-session -t mysession

Frequently used sessions commands

Ctrl-b d	  Detach from the current session 
Ctrl-b (          Go to previous session
Ctrl-b )          Go to next session
Ctrl-b L          Go to previously used session
Ctrl-b s          Choose a session from the sessions list

Ctrl + D    — exit tmux from terminal.

  • Windows (tabs) Management

Each session can have multiple windows. By default all windows are numbered starting from zero.

Frequently used windows (tabs) commands

Ctrl-b 1  Switch to window 1
Ctrl-b c  Create new window
Ctrl-b w  List all windows
Ctrl-b n  Go to next window
Ctrl-b p  Go to previous window
Ctrl-b f  Find window
Ctrl-b ,  Name window
Ctrl-b w  Choose a window from the windows list
Ctrl-b &  Kill the current window


  • One of the handy things about tmux is how easy it is to resize panes:

Ctrl +b, followed by holding down Alt, and using the arrow keys to resize.


======things about how to save sessions and recover sessions.

If you reboot you computer you will lose the sessions. Sessions cannot be saved. But, they can be scripted. What most do in fact is to script some sessions so that you can re-create them.

======Installing tmux

  • Installation with sudo privilege

Installation is pretty straightforward if you have Ubuntu or any other Debian-based distribution you can install tmux with:

sudo apt-get install tmux

on CentOS/Fedora:

yum install tmux

and on MacOS:

brew install tmux

Note: check the comments for some up-to-date scripts.


References and further reading list

This tutorial covers installation of tmux and some commonly used commands.

This is a pretty good introduction to tmux, it includes why tmux and the comaprision between tmux and screen, as well as some tmux shortcuts




Some quotes about AI, machine learning, and deep learning

(Stay tuned, I keep updating this post while I plow in my deep learning garden:))

(If you do not know the differences and relations among AI, machine learning, and deep learning, I suggest you read my post at here.)


Artificial Intelligence, deep learning, machine learning — whatever you’re doing if you don’t understand it — learn it. Because otherwise you’re going to be a dinosaur within 3 years. 

— Mark Cuban 

source: Upfront Summit 2017 (pdf)

I deeply believe that any business that doesn’t make an investment in machine learning in 2017 will fall behind their competition.

When data volume swells beyond a human’s ability to discern the patterns in it—when a company is faced with truly big data—we need a new form of intelligence. GIS, infused with artificial intelligence, can help executives make better decisions.

source: A new business intelligence emerges: Geo.AI ()  -- pdf

There are definitely some academic statisticians who just don’t understand why what I do is statistics, but basically I think they are all wrong. What I do is fundamentally statistics. The fact that data science exists as a field is a colossal failure of statistics. To me, that is what statistics is all about. It is gaining insight from data using modelling and visualization. Data munging and manipulation is hard and statistics has just said that’s not our domain.

-- Hadley Wickham

source: Hadley Wickham, the Man Who Revolutionized R (Jul 24, 2015) -- pdf

[M]achines of this character can behave in a very complicated manner when the number of units is large.

Alan Turing (1948) “Intelligent Machines”, page 6
Simple models and a lot of data trump more elaborate models based on less data.

Peter Norvig
More data beats clever algorithms, but better data beats more data. 

Peter Norvig
No programming language is perfect. There is not even a single best language; there are only languages well suited or perhaps poorly suited for particular purposes. Understanding the problem and associated programming requirements is necessary for choosing the language best suited for the solution.

--Herbert Mayer
Source: Mayer, Herbert (1989). Advanced C programming on the IBM PC. Windcrest Books. p. xii (preface).

(See here for some quotes about data analysis.)


Install Node.js on Ubuntu 16.04 LTS

This post provides instructions about how to install Node.js on Ubuntu 16.04 LTS. See this post for Node.js resources. (Node.js Offical Github Repo.)

NPM (Node Package Manager) is the default package manager for the JavaScript runtime environment Node.js. NPM hosts thousands of free node packages. In general, NPM is installed on your computer after you install Node.js.

There are several ways to install Node.js on Ubuntu:

  • Method #1 (our choice in this tutorial): Install Node.js with Node Version Manager (NVM) to manage multiple active Node.js versions

Using nvm, we can install multiple, self-contained versions of Node.js, which will allow us to control our environment, and get access to the newest versions of Node.js, but will also allow us to keep previous releases that our applications may depend on. (nvm is just like Virtualenv in Python, if you are familiar with it, which allows us to install multiple version of the same Python library into “virtual folders” by pip.)

This is the method we will cover later in this tutorial.

  • Method #2: Install the bundled Distro-Stable Version Node.js (version 4.2.6) – it is very simple to install, just one or two commands.

Ubuntu 16.04 contains a version of Node.js in its default repositories that can be used to easily provide a consistent experience across multiple systems. At the time of writing, the version in the repositories is version 4.2.6. This will not be the latest version, but it should be quite stable, and should be sufficient for quick experimentation with the language.

This tutorial picked the Node Version Manager (nvm) based method, because it is much more flexible.

See below for the step by step instructions. (Check out the reading list below if you need the install instructions for other methods listed above.)

Step 0: (Before we get started) Remove old Node package to avoid conflicts

Open a terminal (Ctrl + Alt + T), and type the following command. 

$ dpkg --get-selections | grep node

# If it says install in the right column, Node is on your system:
#ax25-node                                       install

#node                                            install

Step 1: Install prerequisite packages

We’ll need to get the software packages from our Ubuntu repositories that will allow us to build source packages. The nvm script will leverage these tools to build the necessary components.

First, we need to make sure we have a C++ compiler. Open a terminal window (Ctrl + Alt + T) and install the build-essential and libssl-dev packages. By default, Ubuntu does not come with these tools — but they can be installed by the following commands.

$ sudo apt-get update

$ sudo apt-get install build-essential libssl-dev

Step 2: Install nvm

Once the prerequisite packages are installed, we can install and update NVM(Node Version Manager) using cURL. (Note: to get the latest installation version link, on the page scroll down to “Install script”.) 

$ curl -o- | bash

Inspect the installation script with nano:

$ nano

#Note that we DO NOT need to add anything in the opened nano text editor window. We just need to create the .sh file. 
# Use Ctrl+O to save the file, and then hit Enter, and then Ctrl +X to close the file.

Run the script with bash (Note that run the following command in your terminal):

$ bash

It will install the software into a sub-directory under our home directory at ~/.nvm. It will also add some necessary lines to our ~/.profile file in order to use it.

To have access to the nvm functionality, we need to source the ~/.profile file so that our current session knows about the changes:

$ source ~/.profile

Now that we have nvm installed, we can install isolated Node.js versions.

Step 3: Install  Node.js

The following command will tell us which versions of Node.js are available for us to install:

$ nvm ls-remote
    v6.10.3 (Latest LTS: Boron)

The newest version when I write this post is v7.10.0. We can install it by the following command:

$ nvm install 7.10.0

By default, nvm will switch to use the most recently installed version. We can explicitly tell nvm to use the version we just installed by the following command:

$ nvm use 7.10.0

When we install Node.js using nvm, the executable is called node (NOT nodejs that you may see in other tutorials). We can check the currently used Node and npm version by the following commands:

$ node -v  
# OR 
$ node --version

# Output
# v7.10.0

$ npm -v
# OR 
$ npm --version

# output
# 4.2.0

Step 4: using nvm to manage different versions of installed Node.js 

If you have multiple Node.js versions, you can see what are installed by the following command:

$ nvm ls

To set a default Node.js version to be used in any new shell, use the alias default command

$ nvm alias default 7.10.0

# This version will be automatically selected when a new session spawns. You can also reference it by the alias like this:

$ nvm use default

To learn more  about the options available to use with nvm, run the following command in your terminal:

$ nvm help

Step 5: using npm to install Node.js modules

Each version of Node.js will keep track of its own packages and has npm available to manage these.

We can use npm to install packages to the Node.js project’s ./node_modules directory by using the normal format. For example, for the express module:

$ npm install express

If you’d like to install it globally (i.e., making it available to other projects using the same Node.js version), you can add the -g flag:

$ npm install -g express

This will install the package in:


Note that installing globally will allow us to run the commands from the command line, but we will have to link the package within a project in order to use it in that project:

$ npm link express


References and further reading list:

This tutorial covers two methods:

Method #1: Install the bundled distro specif Node.js version 4.2.6

Method #2: Install the latest version of Node.js version 6.x or 7.x

This post is very good — it covers the following ways to install Node.js:

Installing using the Official Repository
Installing using the Github Source Code Clone
Installing using Node Version Manager (NVM)

It covers How to Install Multiple Nodejs version with NVM and also covers how to remove Node.js 

This tutorial covers:

  • How To Install the Distro-Stable Version for Ubuntu
  • How To Install Using a PPA
  • How To Install Using NVM

there are a quite a few ways to get up and running with Node.js on your Ubuntu 16.04 server. Your circumstances will dictate which of the above methods is the best idea for your circumstance. While the packaged version in Ubuntu’s repository is the easiest, the nvm method is definitely much more flexible.

This is a pretty good tutorial. It covers 4 Ways to Install Node.js on Ubuntu. There are several ways to do this, but The author of this post recommended  Option 1: Node Version Manager (nvm). Here is the full list of options:



Save figure as a pdf file in Python

This post introduces how to save a figure as a pdf file in Python using Matplotlib.

When using savefig, the file format can be specified by the extension:


The first line of code above will give a rasterized  output and the second will give a vectorized output.

In addition, you’ll find that pylab leaves a generous, often undesirable, whitespace around the image. You can remove it with:

savefig('foo.png', bbox_inches='tight')

You can also use  figure to set dpi of the figure:

import numpy as np
import matplotlib.pyplot as plt

fig  = plt.figure(figsize=(1,4),facecolor = 'red', dpi=100)
plt.savefig('test.png', dpi=100)



Run Java program on terminal with external library JAR on Ubuntu

This post provides the instructions how to run a Java program from Terminal with external library JAR.

When using Eclipse to code Java program, which imports some external JAR library, we can use Eclipse to compile/build/run the program.

But if we would like to run our Java program that used External library Jars from Terminal, where should we put those JAR files, and how to build and run the program.

  • For compiling the java file having dependency on a jar
$ javac -cp /path/to/jar/file
  • For executing the class file
$ java -cp .:/path/to/jar/file Myprogram

Note: cp in the above commands refers to classpath, which is a parameter in Java Virtual Machine or the Java compiler that specifies the location of user-defined classes and packages. The parameter may be set either on the command-line, or through an environment variable.

For example,  if your current working directory in terminal is src/report/

$ javac -cp src/external/myImportedJarfile.jar

$ java -cp .:src/external/myImportedJarfile.jar myJavaProgram

If you have multiple jar files a.jar,b.jar and c.jar. To add them to classpath

$javac -cp .:a.jar:b.jar:c.jar

$java -cp .:a.jar:b.jar:c.jar HelloWorld

Note: on Windows, use “;” instead of “:”

Using Java 6 or later, the classpath option supports wildcards. Note the following:

  • Use straight quotes (")
  • Use *, not *.jar

Wild cards were introduced from Java 6. Class path entries can contain the basename wildcard character *, which is considered equivalent to specifying a list of all the files in the directory with the extension .jar or .JAR.

java -cp "lib/*" -jar %MAINJAR%

If you need only specific jars, you will need to add them individually. The classpath string does not accept generic wildcards like Jar*, .jar, hiber etc.


The following entry does not work:

java -cp "Halo.jar;lib/*.jar" ni.package.MainClass

Correct entry is :

java -cp "Halo.jar;lib/*" ni.package.MainClass



Install Ubuntu 16.04 on Oracle VirtualBox that runs on Windows or Mac

This post provides some notes and useful resources about installing Ubuntu 16.04 on Oracle VirtualBox that runs on your Mac or Windows.

Note: check the RAM and hard disk size of your machine before creating a virtual machine on it.

Notes about which version of Ubuntu to download and install:

For Ubuntu, it is not always a wise choice to pick the newest version. My suggestion is that (unless you are aware that you need to install a particular version), download and install the latest LTS (Long Term Support) version (see the picture below from Ubuntu wiki page). Every two years, a Ubuntu LTS version is released, which will be supported for updates for five years. For example, as of now, Ubuntu 16.04 LTS is the latest LTS version.

The two main things you need to pay attention to when you create a virtual machine:

  • Memory allocation for your virtual machine.

You can set it as half of your RAM (e.g., if your RAM is 8 G, set it as 4 G or 5G for your virtual machine should be fine.)

  • Storage type:  Select “Dynamically allocated” if you are not sure how large storage you actually will need.

There are already several very good tutorials about this along with snapshots, so I won’t create a tutorial for this. See below for some useful resources I collected. (See some notes I wrote below for some posts.)

My notes: This one is very good (with snapshots), including  Guest additions and Shared folders settings. (Note that Guest additions are required if you want to set Shared folder, so be sure to install Guest additions first).

You can use the following command to check whether Guest additions were installed on your Ubuntu virtual machine if you are not sure because you installed your Ubuntu VM a while ago. (Note: even though you may find Guest additions was installed, you will still need to install Guest additions for your newly installed VM, otherwise the Shared folders wont work for you.)

Use lsmod from the command line, as it will tell you not only if it’s installed, but properly loaded:

$ lsmod | grep vboxguest
vboxguest             282624  6 vboxsf

I have tested Shared folders instructions (with pictures) in this tutorial on my Ubuntu 16.04 VM, and it works. The only difference is that on Ubuntu 16.04 VM, after you issued the following command on your terminal and  restart the Ubuntu guest machine, you do not need to do anything as the tutorial said, the shared folder is automatically mounted each time you start you Ubuntu VM. (After you restart, click the Files icon on the task bar, and you will see the shared folder you just set just now is automatically mounted there:))

  • sudo adduser brb vboxsf   # Replace 'brb' with your account name on Ubuntu. 

One more note: Although Shared Folder setting in VM is very convenient, using VirtualBox shared folder directly for fastq data, annotation or output directory can significantly reduce the performance compared to a native (Ubuntu) system or VirtualBox native system, so my recommendation is only use the folder to transfer files between windows/mac and your Ubuntu VM.

P.S. If you see some tutorials tell you that you need to enter some command like “sudo mount -t vboxsf sharing /mnt/share” to automatically mount the shared folder each time you start your Ubuntu VM, that is outdated instructions.

Fortunately, new VirtualBox version (4.x +) has a (GUI) Auto-mount option (see pics below) when you set your shared folder. (Note that you can choose your customized folder to share, instead of using a system predefined folder such as Documents or Downloads.)

If you want to share the clipboard between your host and your virtual machine, check out the picture below.


Answers to some frequently asked questions:

Q: Do I need to backup my files when I upgrade my VirtualBox to newer version.

A: just install the latest version and you will have all your files in the new one. You need not have to uninstall the old virtual machine.

Q: After I install the updates of Windows 10, my VirtualBox won’t start…

A: just install the latest version and you will have all your files in the new one. You need not have to uninstall the old virtual machine.


My notes:  this one is very good (with snapshots) on Mac. My notes above about VM settings running on Windows work the same for VM settings running on Mac.