Tag: Mac

Install Miniconda on Mac

This post introduces how to install Miniconda on Mac.

(For installing Miniconda on Linux OS, check out this post.)

(For commonly used conda commands check this post.)

(For the comparison among Anaconda, Miniconda, and Virtualenv with pip, check this post.)

Step 1: download  Miniconda (bash installer) from

https://docs.conda.io/en/latest/miniconda.html 

see the highlighted in the pic below.

A file called Miniconda3-latest-MacOSX-x86_64.sh will be shown in your Downloads folder.

Step 2: Open a Terminal window.

(If you don’t know how to open a terminal window,  through lauchpad type in Terminal, you will see the application.)

In the terminal window, type in

$ cd Downloads

Step 3: run the bash “shell” script to install Miniconda

In the terminal window, type in the following.

$ bash Miniconda3-latest-MacOSX-x86_64.sh

Scroll through the license (press the Space bar or Enter to move through quickly), type ‘yes’ to approve the terms, and then accept all the installation defaults.

Step 4: Close the Terminal window, and open a new Terminal window.

in the newly opened Terminal window.

Type the following:

$ conda -V

If you see something like the following, it means you have successfully installed conda via miniconda on your Mac.

conda 4.5.11

Step 5: Uninstalling Miniconda

To uninstall Python Anconda/Miniconda, we just simply remove the installation folder and remove the environment variables set in the hidden file .bash_profile in your home directory. For my installation, it will be just like this.

$ rm -rf /users/my-user-name/miniconda/
$ rm -rf /users/my-user-name/anaconda/

Then, you can edit the .bash_profile file and remove the following entries added for Anaconda/Miniconda directory from your PATH environment variable.

# added by Miniconda3 installer
$ export PATH="/Users/my-user-name/miniconda3/bin:$PATH"
# added by Anaconda3 installer
$ export PATH="/Users/my-user-name/anaconda3/bin:$PATH"

If you do not know where the hidden .bash_profile is located and how to edit it, see below for detailed instructions.

(1) Open a new terminal and go to your home directory. You can do this by using the command below.

$ cd

(2) use the following command to list all files, including hidden files in your home directory.

$ ls -a 
# you should see there is a file called .bash_profile.

(3) Use the cat command to see the contents of the hidden file .bash_profile. Type the following command into your terminal.

$ cat .bash_profile

You will see something like the following (depends on what you installed, if you installed Miniconda3, you will only see the first two lines. If you installed Anaconda3, you will see the bottom two lines.

# added by Miniconda3 installer 
$ export PATH="/Users/my-user-name/miniconda3/bin:$PATH" 

# added by Anaconda3 installer 
$ export PATH="/Users/my-user-name/anaconda3/bin:$PATH"

(4) To remove installed Miniconda/Anaconda from your .bash_profile use the command below to edit the file using the nano editor.

$ nano .bash_profile

Remove the Miniconda /Anoconda path in your .bash_profile hidden file.

Then Type control + X to exit out of nano

Save changes by typing Y.

Close the terminal, and now Miniconda/Anaconda should be successfully uninstalled from your Mac.

 

(Tested on macOS Mojave. Note that you can install Miniconda onto your Mac even when you are not an admin user.)

For commonly used conda commands check this post.

For the comparison among Anaconda, Miniconda, and Virtualenv with pip, check this post.

 

 

 

 

 

 

Install OpenCV3 into virtualenv on Mac

This post introduces how to install OpenCV3 into a virtualenv on Mac.

If you have not setup virtualenv on your mac, check my post here to do that before you proceed the tutorial in this post.

Let us get started.

Step 1: Activate your virtualenv in your terminal

for example:

$ source ~/ipy-jupyter-venv3/bin/activate  

(ipy-jupyter-venv3)$  # Your prompt should change

Step 2: Install OpenCV (modules) according to your needs

  1. If you need only main modules, in your activated virtualenv in your terminal, run the following
(ipy-jupyter-venv3)$ pip3 install opencv-python 

2. If you need both main and contrib modules (check extra modules listing from OpenCV documentation),  in your activated virtualenv in your terminal, run the following

(ipy-jupyter-venv3)$ pip3 install opencv-contrib-python 

Step 3: Test whether openCV is installed correctly

To test whether OpenCV installed correctly into your virtualenv, in your terminal type in those command below in bold.

(ipy-jupyter-venv3) liping$ python3.6

Python 3.6.5 (default, Mar 30 2018, 06:41:53)

[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin

Type “help”, “copyright”, “credits” or “license” for more information.

>>> import cv2

>>> cv2.__version__

‘3.4.3’

>>

Step 4 (optional): To uninstall opencv inside virtualenv, following the command below according to how you installed it.

(ipy-jupyter-venv3) Liping:~$ pip3 uninstall opencv-python 

or

(ipy-jupyter-venv3) Liping:~$ pip3 uninstall opencv-contrib-python

 

Summary:

In this post, you learned how to install and uninstall OpenCV into your virtualenv.

Setting up Python and Jupyter Notebook on Mac

This page provides the instructions for how to install and run IPython and  Jupyter Notebook in a virtualenv  on Mac.

Most probably your Mac has already come with Python installed (see step 1 and step 2 below to check whether Python and Python 3 is installed on your mac, because my Mac book air has both Python and Python3.6 installed, I will go ahead to step 3 to install virtualenv).  

Open a terminal on your mac and type in Python and Python3 to see whether Python and Python3 installed on your mac. See below for example.

Follow the steps below for detailed instructions.

Step 1: Check whether Python is installed on your Mac

To test whether your Mac has Python 2.x or 3.x, look at the instructions below.

Open a terminal  (in your launchpad, type in terminal, and then click and open it.)

Type in python, if you see similar thing as the pic shown below, it means python 2.7 was installed on your Mac

type exit() to exit from the Python 2.7 environment

Type python3 in your terminal, if you see similar warning as the pic shown below, it means you will need to install Python 3 (see the steps below for installation). If you see older version (< 3.5) of python 3 installed on your mac, follow step 2 to install Python 3.5

Step 2: Install Python 3.5

I recommend you download Python 3.5.4 from here, if you do not have Python 3.x pre-installed on your mac.

https://www.python.org/downloads/mac-osx/ 

Click the installer and install it following the wizard.

When you finish the installation, type python3 in your terminal, if you see similar stuff in the pic below, Python 3.5 is installed successfully and you are ready to proceed to step 3.

Step 3: Install virtualenv 

Note: the command text in blue is the exact command you should type into your terminal, the same through this whole instruction.

Lipings-MacBook-Pro:~ Liping$ pip3 install virtualenv

Step 4: Setup virtualenv environment

Issue the command below to set up a virtualenv environment that we will use later, using the vitualenvwe installed in step 3 above.

$ virtualenv --system-site-packages -p python3 ~/ipy-jupyter-venv3

Step 5: Install IPython

Before installing IPython and Jupyter Notebook, be sure to activate your python virtual environment first.

$ source ~/ipy-jupyter-venv3/bin/activate 
(ipy-jupyter-venv3)$  # Your prompt should change

Type the following command to install IPython

(ipy-jupyter-venv3) liping:~$ pip3 install ipython

Step 6: Install Jupyter Notebook

Use the following command to install Jupyter Notebook

(ipy-jupyter-venv3) liping:~$ pip3 install jupyter

Step 7: Test jupypter notebook installation

(ipy-jupyter-venv3) liping:~$ which python3

/Users/Liping/ipy-jupyter-venv3/bin/python3
#for python 3.x
(ipy-jupyter-venv3) liping:~$ which ipython3

/Users/Liping/ipy-jupyter-venv3/bin/ipython3
#for python 3.x
(ipy-jupyter-venv3) liping:~$ which jupyter-notebook

/Users/Liping/ipy-jupyter-venv3/bin/jupyter-notebook

Step 8: Add Kernel

The Jupyter Notebook and other frontends automatically ensure that the IPython kernel is available. However, if you want to use a kernel with a different version of Python, or in a virtualenv environment, you’ll need to install that manually. 

We are using virutalenv, so we need to install IPython kernel in the virtualenv we created in Step 4 above.

(ipy-jupyter-venv3) liping:~$  python3 -m ipykernel install --user --name myipy_jupter_env3 --display-name "ipy-jupyter-venv3"

Installed kernelspec myipy_jupter_env3 in /Users/Liping/Library/Jupyter/kernels/myipy_jupter_env3

 

Step 9: Before we are running our jupyter Notebook, let us create a folder from which we will start our jupyter notebook.

The following command will change our directory to Desktop (cd refers to change directory)

(ipy-jupyter-venv3) Lipings-MacBook-Pro:~ Liping$ cd Desktop 

the following command will help us create a folder named Geog597_ML_session under our Desktop folder

(ipy-jupyter-venv3) Lipings-MacBook-Pro:Desktop Liping$ mkdir test_jupyter

the following command will change our directory into the folder we just created.

(ipy-jupyter-venv3) Lipings-MacBook-Pro:Desktop Liping$ cd test_jupyter/

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$

Step 10: Run Jupyter Notebook

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$ jupyter-notebook

Then your default browser should automatically open a web page similar to the one shown below.

If the web page does not show up automatically, just type localhost:8888 into your browser, you should see the page shown above.

 

Step 11: Testing and Using Jupyter Notebook

By this point you should have Jupyter Notebook running, and you should be connected to it using a web browser. Jupyter Notebook is very powerful and has many features. Below I will outline a few of the basic features to get you started using the notebook. Automatically, Jupyter Notebook will show all of the files and folders in the directory it is run from (for our case, it is empty now, because we have not put anything in that folder Desktop/test_jupyter we just created in step 9 ).

To create a new notebook file, select New > ipy-jupyter-venv3 from the top right pull-down menu (Note: this is the so called kernel we installed in Step 8 above):

This will open a notebook. We can now run Python code in the cell or change the cell to markdown (markdown is for note, not for code). For example, change the first cell to accept Markdown by clicking Cell > Cell Type > Markdown from the top navigation bar, or by click the Markdown shown in the pic below. We can now write notes using Markdown,  for example, type the following into the cell after changing it to markdown:

# Testing

Hit Ctrl + Enter, you will see it the text changed into heading style.

          Then insert one cell by the menu shown below (insert Cell Below).

type in the following

a = 5
b = 10
print (a+b)

and then hit Ctrl + Enter.  You should see the following.

Step 12: Stop jupyter Notebook.

To stop the Jupyter Notebook process, press CTRL+C,

type Y, and hit ENTER to confirm. The following will be displayed:

Step 13: Exit virtualenv environment

Once you are done, remember to exit your virtualenv using the following command:

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$

deactivate

Note that your prompt changes back (see the pic below)

Step 14: Instructions for using Jupyter Notebook next time

Next time when you need to use you jupyther notebook, following the steps summarized below.

Lipings-MacBook-Pro:~ Liping$ source ~/ipy-jupyter-venv3/bin/activate  

(ipy-jupyter-venv3)$  # Your prompt should change

Change directory to where you would like to start your jupyter-notebook

(ipy-jupyter-venv3) Lipings-MacBook-Pro:~ Liping$ cd Desktop/test_jupyter

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$

Run jupyter notebook:

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$ jupyter-notebook

 

Congratulations, you have now set up ipython and jupyter notebook on your mac!

Timing how long a python script runs

This post introduces several ways to find out how long a python script takes to complete its execution.

  • If you are using Linux or Mac OS, in your terminal
$ time ./your_script.py
  • Several ways to do the task by adding a few lines of code in your py script.
import time
startTime = time.time()

your_func() #python3: print ("It took", time.time() - startTime, "seconds.")

See the following for an example in python 3. 

import time
import functools

startTime = time.time()

print(functools.reduce(lambda x,y: x+y, [47,11,42,13]))

#python3:
print ("It took", time.time() - startTime, "seconds.")

Another way to do the same thing:

from datetime import datetime
startTime = datetime.now()

#do something

#Python 2: 
print datetime.now() - startTime 

#Python 3: 
print(datetime.now() - startTime)

One more way to do the same thing with a nicely formatted output.

import sys
import timeit

startTime = timeit.default_timer()

#do some nice things...

stopTime = timeit.default_timer()
totalRunningTime = stopTime - startTime

# output running time in a nice format.
mins, secs = divmod(totalRunningTime, 60)
hours, mins = divmod(mins, 60)

sys.stdout.write("Total running time: %d:%d:%d.\n" % (hours, mins, secs))

If you want to compare two blocks of code / functions quickly you can do the following:

import timeit

startTime = timeit.default_timer()
your_func1()
#python3
print(timeit.default_timer() - startTime)

startTime2 = timeit.default_timer()
your_func2()
#python3
print(timeit.default_timer() - starTime2)

tmux resources

This post provides a brief introduction to tmux and some commonly used commands and useful resources about tmux.

(Thanks Davide for recommending such a handy tool to me.)

(Stay tuned — I will update this post while I am gaining new skills about tmux.)

======What is tmux?

According to the tmux authors:

tmux is a terminal multiplexer. What is a terminal multiplexer? It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal. And do a lot more.

 

====== tmux sessions, windows, and panes explained

One of these features is the ability to break your session into more discreet components, called windows and panes. These are good for organizing multiple varied activities in a logical way.

Let’s look at how they relate to each other.

Nesting

tmuxnesting

tmux sessions have windows, and windows have panes. Below you can see how how they are conceptualized:

  • Sessions are for an overall theme, such as work, or experimentation, or sysadmin.
  • Windows are for projects within that theme. So perhaps within your experimentation session you have a window titled noderestapi, and one titled lua sample.
  • Panes are for views within your current project. So within your sysadmin session, which has a logs window, you may have a few panes for access logs, error logs, and system logs.

It’s also possible to create panes within a session without first creating a separate window. I do this sometimes. Hopefully it isn’t as horrible as it sounds right after reading about nesting.

Q about differences between sessions and windows and panes:

Can someone explain how they use sessions and windows and panes?
It feels like one too many levels, why both sessions AND windows?
I’m willing to believe there is a use case, but for the life of me I can’t come up with one.

A: 

I use sessions for different projects, and windows and panes within a project. For example, I’ll have a session for my puppet code, and a session for a bash script I’m working on. Within the puppet session, if I’m going to work on a new module, I open a new window. If I’m writing code in vim, I usually split off a pane so I can test running it next to it. Having the error output next to the code makes debugging really fast.

In a lot of ways, it basically acts like a tiling window manager for the terminal. The advantage over a window manager is that the whole layout can be accessed remotely. So if I’m working on a project from work, I can quickly resume working on the project from my home computer after SSH’ing in and reattaching to the tmux session.

 

====== see below for commonly used tmux comands

By default, tmux uses Ctrl-b as its shortcut activation chord, which enables you perform a number of functions quickly.

  • Switch between different windows

Ctrl – b (presee the  ctrl and b keys at the same time and release them at the same time, and then immediately press window number)  window number

  • Cursor move within an active window

Ctrl-b (release the keys, and immediately press the “[” key ) the key with “[“.

when you see the cursor becomes a solid flashing diamond, you can use arrow key to move your cursor in the active terminal window.

Note: Press “q” to exit the cursor move mode.

  • Session management
  • s list sessions
  • $ rename the current session # default session name and window name is number, can rename session names to meaningful ones
  • d detach from the current session

tmux is developed on a client-server model which means that the session is stored on the server and persist beyond ssh logout.

The following command will create a new session called mysession:

tmux new-session -s mysession

To attach to a session run:

tmux attach -t mysession

To list all session run:

tmux ls

You can kill a session using the following command:

tmux kill-session -t mysession

you can grossly kill all tmux processes with the following command:

pkill -f tmux

Frequently used sessions commands

Ctrl-b d	  Detach from the current session 
Ctrl-b (          Go to previous session
Ctrl-b )          Go to next session
Ctrl-b L          Go to previously used session
Ctrl-b s          Choose a session from the sessions list

Ctrl + D    — exit tmux from terminal.

  • Windows (tabs) Management

Each session can have multiple windows. By default all windows are numbered starting from zero.

Frequently used windows (tabs) commands

Ctrl-b 1  Switch to window 1
Ctrl-b c  Create new window
Ctrl-b w  List all windows
Ctrl-b n  Go to next window
Ctrl-b p  Go to previous window
Ctrl-b f  Find window
Ctrl-b ,  Name window
Ctrl-b w  Choose a window from the windows list
Ctrl-b &  Kill the current window

 

  • One of the handy things about tmux is how easy it is to resize panes:

Ctrl +b, followed by holding down Alt, and using the arrow keys to resize.

 

======things about how to save sessions and recover sessions.

If you reboot you computer you will lose the sessions. Sessions cannot be saved. But, they can be scripted. What most do in fact is to script some sessions so that you can re-create them.

======Installing tmux

  • Installation with sudo privilege

Installation is pretty straightforward if you have Ubuntu or any other Debian-based distribution you can install tmux with:

sudo apt-get install tmux

on CentOS/Fedora:

yum install tmux

and on MacOS:

brew install tmux

Note: check the comments for some up-to-date scripts.

 

References and further reading list

This tutorial covers installation of tmux and some commonly used commands.

This is a pretty good introduction to tmux, it includes why tmux and the comaprision between tmux and screen, as well as some tmux shortcuts