Tag: Python

(Geo) VIZ Resources

This page provides some resources about VIZ or GeoVIZ.

(Stay tuned, as I will update the content on this  page while I plow and grow in my deep learning garden:))

(Geo) VIZ Scholars/Groups

Selected Talks from Prof. Tamara Munzner (see complete list of Talks HERE).

(There are videos and demos for some VIZ tools on Dr. Shixia Liu’s home page under Publications)

Some selected papers from Dr. Shixia Liu’s research group:

Liu, S., Wang, X., Collins, C., Dou, W., Ouyang, F., El-Assady, M., … & Keim, D. (2018). Bridging text visualization and mining: A task-driven survey. IEEE transactions on visualization and computer graphics. (PDF, tool — a visualization tool that is part of a survey of text visualization & mining) 

Liu, S., Wang, X., Liu, M., & Zhu, J. (2017). Towards better analysis of machine learning models: A visual analytics perspective. Visual Informatics, 1(1), 48-56.

Jiang, L., Liu, S., & Chen, C. (2018). Recent research advances on interactive machine learning. Journal of Visualization, 1-17.

Liu, S., Chen, C., Lu, Y., Ouyang, F., & Wang, B. (2019). An interactive method to improve crowdsourced annotations. IEEE transactions on visualization and computer graphics, 25(1), 235-245.

Liu, S., Andrienko, G., Wu, Y., Cao, N., Jiang, L., Shi, C., … & Hong, S. (2018). Steering data quality with visual analytics: The complexity challenge. Visual Informatics.

Liu, M., Shi, J., Li, Z., Li, C., Zhu, J., & Liu, S. (2017). Towards better analysis of deep convolutional neural networks. IEEE transactions on visualization and computer graphics, 23(1), 91-100.

Selected papers:

Sacha, D., Sedlmair, M., Zhang, L., Lee, J. A., Weiskopf, D., North, S., & Keim, D. (2016, August). Human-centered machine learning through interactive visualization. ESANN. (PDF)

Selected papers:

Bernard, J., Hutter, M., Zeppelzauer, M., Fellner, D., & Sedlmair, M. (2018). Comparing visual-interactive labeling with active learning: An experimental study. IEEE transactions on visualization and computer graphics, 24(1), 298-308. (PDF)

 

  • TBA

 

(Geo) VIZ Charts

The Most Searched for Visualization Types, Tools, and Books

Finding D3 plugins with ease.

  • TBA

(Geo) VIZ Tools

  • Coming soon.

Good (Geo) VIZ  / GIS / geospatial Python libraries:

  • Cartopy : Support for geographical data (using a wide range of other libraries)

  • Coming soon.

Good (Geo) VIZ  Posts:

Great post. If you do any visualization or computer vision, these are things you need to know (but most people still don’t know:))

 

Good (GeoAI) VIZ  Posts:

 

Credits:

Prof. Alan M. MacEachren, Dr. Nai Yang, Dr. Teresa Onorati

Many thanks go to Professor Alan M. MacEachren letting me entering the interesting field of (Geo)VIZ!

 

References and Further Reading List:

 

Conda commands (create virtual environments for python with conda)

This post provides some commonly used conda command.

(I use Mac and Linux OS, so the commands here assume that you use Mac and Linux OS.  If you are using Windows, most of the conda commands should be the same, but some command might be slightly different. For example, the one I am aware of is that the command for activating and deactivating conda environment is a bit different, you do not need to add “source” in the command like Mac and Linux OS do).

(To use conda, you need to have either Miniconda or Anaconda installed on your machine. Check out here for installing Miniconda on Mac, and here for installing Miniconda on CentOS7/RedHat7. Check here if you would like to know the difference between Miniconda and Anaconda.)

The first part of this post introduces common conda commands, and the second part of this post provides some package installation ( e.g., OpenCV, scikit-image, jupyter notebook etc.) using conda.

Common conda commands

Check whether conda is installed / check conda version info

In your terminal, type in the following command

$ conda -V 

#if you see something like below it means conda is installed, and it provides its version info. 

conda 4.5.11

Check conda is up to date

In your terminal, type in

$ conda update conda

# Upadate any packages if necessary by typing y to proceed.

Create a virtual environment using conda for your project

with conda, we can create virtual environment for different versions of pythons.

To see a list of available python versions available in conda repository, type the following command with regular expression and then press enter.

$ conda search "^python$"  # you should see a list of python versions, including python2.X and python3.X

Now, let us create a virtual environment with conda

use the following command to create a virtual environment for a python version you specified, replace x.x with the Python version you would like to use.

$ conda create -n yourenvname python=x.x 


# for example, the following command will create a Python 3.6 conda virtual environment called "conda-venv3_py36".

$ conda create -n conda-venv3_py36 python=3.6 

# the following command will create a Python 2.7 conda virtual environment called "conda-venv_py27".

$ conda create -n conda-venv_py27 python=2.7

Press y to proceed. This will install the Python version (and all the associated anaconda packaged libraries if you installed conda via Anaconda) at “path_to_your_anaconda_location/anaconda/envs/yourenvname” or at “path_to_your_miniconda_location/miniconda/envs/yourenvname”

Activate your virtual environment

Once we created a virtual environment using conda, before we start to using it, we need to activate it first each time we need to use the virtual environment.

To activate or switch into your virtual environment, simply type the following where yourenvname is the name you gave to your environement when creating.

$ source activate yourenvname

Activating a conda environment modifies the PATH and shell variables to point to the specific isolated Python  you created. Note that the command prompt will change to indicate which conda environemnt you are currently in by prepending (yourenvname). 

If you do not remember your virtualenv or do not want to type it, you can use the following command to see a list of all your environments,

$ conda info -e

Install (additional) Python packages to a virtual environment

To install packages only to your virtual environment (not system wide), enter the following command

$ conda install -n yourenvname package-name

# yourenvname is the name of your environment, and package-name is the name of the package you would like to install.

# Note that if not specify “-n yourenvname” will install the package to the root Python installation.

Or you can simply first activate and into the virtual environment you would like to install packages [see the command above in (4)], and then use the following command

$ conda install package-name

(For some specific packages installation, check section 2 below in this post.)

Deactivate your virtual environment.

Each time once we finish working in your virtual environment created using conda, we will need to deactivate the virtual environment to exit from it.

To end a session in the current virtual environment, enter the following command .

$ source deactivate

# Note that we do not need to specify the envname - whichever is currently active will be deactivated, and the PATH and shell variables will return to normal.

Delete a virtual environment

When we do not need a virtual environment created by conda any more, we can simply remove it by the following command.

conda remove -n yourenvname -all # yourenvname is the name of the environment you would like to delete. # (You may instead use $ conda env remove -n myenv.)

To verify that the environment was removed, in your Terminal window or an conda Prompt, run the following

$ conda info -e

The environments list that displays should not show the removed environment.

Cloning a conda virtual environment

To make  an exact copy of an environment by creating a clone of it, using the following command,

$ conda create --name myclone --clone myenv


# NOTE: replace myclone with the name of the new environment. Replace myenv with the name of the existing environment that you want to copy. # see the following for an example of cloning py35 and naming the new copy as py35-2 $ conda create --clone py35 --name py35-2

To verify that the copy was made:

conda info -e

In the environments list that displays, you should see both the source environment and the new copy.

Viewing a list of your environments

To see a list of all of your conda virtual environments, in your Terminal or in one of your conda virtual environment, run one of the following commands:

$ conda info -e

OR

$ conda info --envs 

OR

$ conda env list

You will see a list of all your conda environments, and the active environment is shown with *.

Viewing a list of the packages in a conda environment

To see a list of all packages installed in a specific environment,

  • If the environment is not activated, in your Terminal window or a conda prompt, run the following:
$ conda list -n myenv
  • If the environment is activated, in your Terminal window or a conda prompt, run the following:
$ conda list

To see whether a specific package is installed in a conda environment, 

  • If the environment is not activated, in your Terminal window or a conda prompt, run the following:
$ conda list -n myenv package-name

# for example, the following command will list the opencv versions installed in the conda environment you specified

$ conda list -n myenv opencv
  • If the environment is activated, in your Terminal window or a conda prompt, run the following:
$ conda list package-name

# for example, the following command will list the opencv versions installed in the current active conda environment you are in

$ conda list opencv

 

Package installation using conda

Before installing packages using conda, make sure to first create a conda virtual environment [see the command at (3) in section 1 above in this post] and then  activate and into the environment you would like to install the packages into [see the command for (4) in section 1 above in this post].

Install Numpy

  $ conda install numpy

Install Matplotlib

  $ conda install matplotlib

Install Keras 

$ conda install keras

This should also install tensorflow

Install h5py

  $ conda install h5py

Install Jupyter Notebook

$ conda install jupyter

Install IPython

  $ conda install ipython

Install OpenCV3 (https://opencv.org/)

  $ conda install -c conda-forge opencv

The command above will install (by default) the latest version of opencv available in the conda repository. if you would like to specifiy which version of openCV to install, you can first use the following comamnd to check OpenCV versions available.

$ conda search "^openCV$"   # you should see a list of openCV versions.

Then you could use the following command to install the version of OpenCV you would like to install,

 $ conda install -c conda-forge opencv=x.x # for example, the following command will install openCV 3..4.1, instead of the current lastest version 3.4.2. Note that is opencv==3.4.1, not opencv=3.4, if opencv=3.4, it will install openCV3.4.2 (the latest version in 3.4 series).

$ conda install -c conda-forge opencv==3.4.1

Install Scikit-image

$ conda install -c conda-forge scikit-image

Install Django

use the following command to search what vesion of django is available in your conda environment.

$ conda search "^django$"

use the following command to install specific version of django you would like to install into your conda environment.

$ conda install -c conda-forge django==1.11.8

use the following to test whether the django is installed successfully in your conda environment.

$ python
Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44)
[GCC 7.3.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import django
>>> print(django.__version__)
1.11.8
>>>

 

Installing non-conda packages

If a package is not available from conda or Anaconda.org, you may be able to find and install the package via conda-forge or with another package manager like pip.

Pip packages do not have all the features of conda packages and we recommend first trying to install any package with conda. If the package is unavailable through conda, try finding and installing it with conda-forge.

If you still cannot install the package, you can try installing it with pip. The differences between pip and conda packages cause certain unavoidable limits in compatibility but conda works hard to be as compatible with pip as possible.

Removing packages using conda

  • To remove a package such as SciPy in an environment such as myenv: 
    conda remove -n myenv scipy
  • To remove a package such as SciPy in the current environment: 
    conda remove scipy
  • To remove multiple packages at once, such as SciPy and cURL: 
    conda remove scipy curl
  • To confirm that a package has been removed: 
    conda list the package name

References

 

Install Miniconda on CentOS 7 / RedHat 7

This post introduces how to install Miniconda on CentOS 7 / RedHat 7. 

(Tested on CentOS 7 / RedHat 7, but it should work for Ubuntu OS as well. Note that you can install Miniconda / Anaconda onto your Linux OS even when you are not a sudo / root user.)

(For installing Miniconda on Mac, check out this post.)

(For commonly used conda commands check this post.)

(For the comparison among Anaconda, Miniconda, and Virtualenv with pip, check this post.)

Step 1: Open a Terminal window, type

$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

Step2:  Run the following bash script to install it

$ sh Miniconda3-latest-Linux-x86_64.sh

Step3: To make the changes take effect, close the terminal and open a new Terminal window. 

Step 4: Test conda

In the newly open Terminal window, type the following

$ conda -V

# If you see something like the following, it means Miniconda is successfully installed on your Linux OS.

conda 4.5.11

Step 5: Uninstalling Miniconda

To uninstall Python Anconda/Miniconda, we just simply remove the installation folder and remove the environment variables set in .bashrc file. For my installation, it will be just like this.

$ rm -rf /usr/local/miniconda/
$ rm -rf /usr/local/anaconda/

Then, you can edit the ~/.bashrc file and remove the following entries added for Anaconda/Miniconda directory from your PATH environment variable

$ export PATH=" /usr/local/anaconda/bin:$PATH" 
$ export PATH=" /usr/local/miniconda3/bin:$PATH"

 

 

For commonly used conda commands check this post.

For the comparison among Anaconda, Miniconda, and Virtualenv with pip, check this post.

References:

 

Install Miniconda on Mac

This post introduces how to install Miniconda on Mac.

(For installing Miniconda on Linux OS, check out this post.)

(For commonly used conda commands check this post.)

(For the comparison among Anaconda, Miniconda, and Virtualenv with pip, check this post.)

Step 1: download  Miniconda (bash installer) from

https://docs.conda.io/en/latest/miniconda.html 

see the highlighted in the pic below.

A file called Miniconda3-latest-MacOSX-x86_64.sh will be shown in your Downloads folder.

Step 2: Open a Terminal window.

(If you don’t know how to open a terminal window,  through lauchpad type in Terminal, you will see the application.)

In the terminal window, type in

$ cd Downloads

Step 3: run the bash “shell” script to install Miniconda

In the terminal window, type in the following.

$ bash Miniconda3-latest-MacOSX-x86_64.sh

Scroll through the license (press the Space bar or Enter to move through quickly), type ‘yes’ to approve the terms, and then accept all the installation defaults.

Step 4: Close the Terminal window, and open a new Terminal window.

in the newly opened Terminal window.

Type the following:

$ conda -V

If you see something like the following, it means you have successfully installed conda via miniconda on your Mac.

conda 4.5.11

Step 5: Uninstalling Miniconda

To uninstall Python Anconda/Miniconda, we just simply remove the installation folder and remove the environment variables set in the hidden file .bash_profile in your home directory. For my installation, it will be just like this.

$ rm -rf /users/my-user-name/miniconda/
$ rm -rf /users/my-user-name/anaconda/

Then, you can edit the .bash_profile file and remove the following entries added for Anaconda/Miniconda directory from your PATH environment variable.

# added by Miniconda3 installer
$ export PATH="/Users/my-user-name/miniconda3/bin:$PATH"
# added by Anaconda3 installer
$ export PATH="/Users/my-user-name/anaconda3/bin:$PATH"

If you do not know where the hidden .bash_profile is located and how to edit it, see below for detailed instructions.

(1) Open a new terminal and go to your home directory. You can do this by using the command below.

$ cd

(2) use the following command to list all files, including hidden files in your home directory.

$ ls -a 
# you should see there is a file called .bash_profile.

(3) Use the cat command to see the contents of the hidden file .bash_profile. Type the following command into your terminal.

$ cat .bash_profile

You will see something like the following (depends on what you installed, if you installed Miniconda3, you will only see the first two lines. If you installed Anaconda3, you will see the bottom two lines.

# added by Miniconda3 installer 
$ export PATH="/Users/my-user-name/miniconda3/bin:$PATH" 

# added by Anaconda3 installer 
$ export PATH="/Users/my-user-name/anaconda3/bin:$PATH"

(4) To remove installed Miniconda/Anaconda from your .bash_profile use the command below to edit the file using the nano editor.

$ nano .bash_profile

Remove the Miniconda /Anoconda path in your .bash_profile hidden file.

Then Type control + X to exit out of nano

Save changes by typing Y.

Close the terminal, and now Miniconda/Anaconda should be successfully uninstalled from your Mac.

 

(Tested on macOS Mojave. Note that you can install Miniconda onto your Mac even when you are not an admin user.)

For commonly used conda commands check this post.

For the comparison among Anaconda, Miniconda, and Virtualenv with pip, check this post.

 

 

 

 

 

 

Anaconda vs. Miniconda vs. Virtualenv

This post briefly introduces which to choose among Anaconda, Miniconda, and Virtualenv.

If you have used pip and virtualenv in the past, you can use conda to perform all of the same operations. Pip is a package manager, and virtualenv is an environment manager; and conda is both.

Specifically, conda is a packaging tool and installer that aims to do more than what pip does; it handles library dependencies outside of the Python packages as well as the Python packages themselves. Conda also creates a virtual environment, like virtualenv does.

Both Anaconda and Miniconda uses Conda as the package manager. The difference among Anaconda and Miniconda is that Miniconda only comes the package management system. So when you install it, there is just the management system and not coming with a bundle of pre-installed packages like Anaconda does. Once Conda is installed, you can then install whatever package you need from scratch along with any desired version of Python.

Choose Anaconda if you:

  • Are new to conda or Python
  • Prefer having Python and 720+ open source certified packages automatically installed at once
  • Have the time and disk space (a few minutes and 3 GB), and/or
  • Don’t want to install each of the packages you want to use individually.

Choose Miniconda if you:

  • Know what package(s) you need to install
  • Do not have time or disk space (about 3 GB) to install over 720+ packages (many of the packages are never used and could be easily installed when needed), and/or
  • Just want fast access to Python and the conda commands, and prefer to sorting out the other packages later.

Choose Virtualenv only when you have sudo access to the machine you are working on. It is much easier to setup conda rather than virtualenv for a regular (i.e., non sudo/root) user on a linux/Mac machine.

I use Miniconda myself (because it is much more light weight than Anaconda) when I need to setup python programming environment and when I do not have sudo privilege, and I use Virtualenv when I have sudo access on the machine.

(Thanks to  Dr. Brendt Wohlberg  for introducing Miniconda — Miniconda makes me switching from pip & virtualenv to conda.)

References

 

Setting up Python and Jupyter Notebook on Mac

This page provides the instructions for how to install and run IPython and  Jupyter Notebook in a virtualenv  on Mac.

Most probably your Mac has already come with Python installed (see step 1 and step 2 below to check whether Python and Python 3 is installed on your mac, because my Mac book air has both Python and Python3.6 installed, I will go ahead to step 3 to install virtualenv).  

Open a terminal on your mac and type in Python and Python3 to see whether Python and Python3 installed on your mac. See below for example.

Follow the steps below for detailed instructions.

Step 1: Check whether Python is installed on your Mac

To test whether your Mac has Python 2.x or 3.x, look at the instructions below.

Open a terminal  (in your launchpad, type in terminal, and then click and open it.)

Type in python, if you see similar thing as the pic shown below, it means python 2.7 was installed on your Mac

type exit() to exit from the Python 2.7 environment

Type python3 in your terminal, if you see similar warning as the pic shown below, it means you will need to install Python 3 (see the steps below for installation). If you see older version (< 3.5) of python 3 installed on your mac, follow step 2 to install Python 3.5

Step 2: Install Python 3.5

I recommend you download Python 3.5.4 from here, if you do not have Python 3.x pre-installed on your mac.

https://www.python.org/downloads/mac-osx/ 

Click the installer and install it following the wizard.

When you finish the installation, type python3 in your terminal, if you see similar stuff in the pic below, Python 3.5 is installed successfully and you are ready to proceed to step 3.

Step 3: Install virtualenv 

Note: the command text in blue is the exact command you should type into your terminal, the same through this whole instruction.

Lipings-MacBook-Pro:~ Liping$ pip3 install virtualenv

Step 4: Setup virtualenv environment

Issue the command below to set up a virtualenv environment that we will use later, using the vitualenvwe installed in step 3 above.

$ virtualenv --system-site-packages -p python3 ~/ipy-jupyter-venv3

Step 5: Install IPython

Before installing IPython and Jupyter Notebook, be sure to activate your python virtual environment first.

$ source ~/ipy-jupyter-venv3/bin/activate 
(ipy-jupyter-venv3)$  # Your prompt should change

Type the following command to install IPython

(ipy-jupyter-venv3) liping:~$ pip3 install ipython

Step 6: Install Jupyter Notebook

Use the following command to install Jupyter Notebook

(ipy-jupyter-venv3) liping:~$ pip3 install jupyter

Step 7: Test jupypter notebook installation

(ipy-jupyter-venv3) liping:~$ which python3

/Users/Liping/ipy-jupyter-venv3/bin/python3
#for python 3.x
(ipy-jupyter-venv3) liping:~$ which ipython3

/Users/Liping/ipy-jupyter-venv3/bin/ipython3
#for python 3.x
(ipy-jupyter-venv3) liping:~$ which jupyter-notebook

/Users/Liping/ipy-jupyter-venv3/bin/jupyter-notebook

Step 8: Add Kernel

The Jupyter Notebook and other frontends automatically ensure that the IPython kernel is available. However, if you want to use a kernel with a different version of Python, or in a virtualenv environment, you’ll need to install that manually. 

We are using virutalenv, so we need to install IPython kernel in the virtualenv we created in Step 4 above.

(ipy-jupyter-venv3) liping:~$  python3 -m ipykernel install --user --name myipy_jupter_env3 --display-name "ipy-jupyter-venv3"

Installed kernelspec myipy_jupter_env3 in /Users/Liping/Library/Jupyter/kernels/myipy_jupter_env3

 

Step 9: Before we are running our jupyter Notebook, let us create a folder from which we will start our jupyter notebook.

The following command will change our directory to Desktop (cd refers to change directory)

(ipy-jupyter-venv3) Lipings-MacBook-Pro:~ Liping$ cd Desktop 

the following command will help us create a folder named Geog597_ML_session under our Desktop folder

(ipy-jupyter-venv3) Lipings-MacBook-Pro:Desktop Liping$ mkdir test_jupyter

the following command will change our directory into the folder we just created.

(ipy-jupyter-venv3) Lipings-MacBook-Pro:Desktop Liping$ cd test_jupyter/

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$

Step 10: Run Jupyter Notebook

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$ jupyter-notebook

Then your default browser should automatically open a web page similar to the one shown below.

If the web page does not show up automatically, just type localhost:8888 into your browser, you should see the page shown above.

 

Step 11: Testing and Using Jupyter Notebook

By this point you should have Jupyter Notebook running, and you should be connected to it using a web browser. Jupyter Notebook is very powerful and has many features. Below I will outline a few of the basic features to get you started using the notebook. Automatically, Jupyter Notebook will show all of the files and folders in the directory it is run from (for our case, it is empty now, because we have not put anything in that folder Desktop/test_jupyter we just created in step 9 ).

To create a new notebook file, select New > ipy-jupyter-venv3 from the top right pull-down menu (Note: this is the so called kernel we installed in Step 8 above):

This will open a notebook. We can now run Python code in the cell or change the cell to markdown (markdown is for note, not for code). For example, change the first cell to accept Markdown by clicking Cell > Cell Type > Markdown from the top navigation bar, or by click the Markdown shown in the pic below. We can now write notes using Markdown,  for example, type the following into the cell after changing it to markdown:

# Testing

Hit Ctrl + Enter, you will see it the text changed into heading style.

          Then insert one cell by the menu shown below (insert Cell Below).

type in the following

a = 5
b = 10
print (a+b)

and then hit Ctrl + Enter.  You should see the following.

Step 12: Stop jupyter Notebook.

To stop the Jupyter Notebook process, press CTRL+C,

type Y, and hit ENTER to confirm. The following will be displayed:

Step 13: Exit virtualenv environment

Once you are done, remember to exit your virtualenv using the following command:

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$

deactivate

Note that your prompt changes back (see the pic below)

Step 14: Instructions for using Jupyter Notebook next time

Next time when you need to use you jupyther notebook, following the steps summarized below.

Lipings-MacBook-Pro:~ Liping$ source ~/ipy-jupyter-venv3/bin/activate  

(ipy-jupyter-venv3)$  # Your prompt should change

Change directory to where you would like to start your jupyter-notebook

(ipy-jupyter-venv3) Lipings-MacBook-Pro:~ Liping$ cd Desktop/test_jupyter

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$

Run jupyter notebook:

(ipy-jupyter-venv3) Lipings-MacBook-Pro:test_jupyter Liping$ jupyter-notebook

 

Congratulations, you have now set up ipython and jupyter notebook on your mac!

Cool Python Tricks

This post provides some cool python tricks.

(Stay tuned, as I may update the tricks while I plow in my deep learning garden)

With increase in popularity of python, more and more features are becoming available for python coding. Using this features makes writing the code in fewer lines and cleaner. In this article we will see 10 such python tricks which are very frequently used and most useful.

Reversing a List

We can simply reverse a given list by using a reverse() function. It handles both numeric and string data types present in the list.

Example

List = ["Shriya", "Lavina","Sampreeti" ]
List.reverse()
print(List)

Output

Running the above code gives us the following result −

['Sampreeti', 'Lavina', 'Shriya']

Print list elements in any order

If you need to print the values of a list in different orders, you can assign the list to a series of variables and programmatically decide the order in which you want to print the list.

Example

List = [1,2,3]
w, v, t = List
print(v, w, t )
print(t, v, w )

Output

Running the above code gives us the following result −

(2, 1, 3)
(3, 2, 1)

Using Generators Inside Functions

We can use generators directly inside a function to writer shorter and cleaner code. In the below example we find the sum using a generator directly as an argument to the sum function.

Example

sum(i for i in range(10) )

Output

Running the above code gives us the following result −

45

*Using the zip() function

When we need to join many iterator objects like lists to get a single list we can use the zip function. The result shows each item to be grouped with their respective items from the other lists.

Example

Year = (1999, 2003, 2011, 2017)
Month = ("Mar", "Jun", "Jan", "Dec")
Day = (11,21,13,5)
print zip(Year,Month,Day)

Output

Running the above code gives us the following result −

[(1999, 'Mar', 11), (2003, 'Jun', 21), (2011, 'Jan', 13), (2017, 'Dec', 5)]

Swap two numbers using a single line of code

Swapping of numbers usually requires storing of values in temporary variables. But with this python trick we can do that using one line of code and without using any temporary variables.

Example

x,y = 11, 34
print x
print y
x,y = y,x
print x
print y

Output

Running the above code gives us the following result −

11
34
34
11

Transpose a Matrix

Transposing a matrix involves converting columns into rows. In python we can achieve it by designing some loop structure to iterate through the elements in the matrix and change their places or we can use the following script involving zip function in conjunction with the * operator to unzip a list which becomes a transpose of the given matrix.

Example

x = [[31,17],
[40 ,51],
[13 ,12]]
print (zip(*x))

Output

Running the above code gives us the following result −

[(31, 40, 13), (17, 51, 12)]

*Print a string N Times

The usual approach in any programming language to print a string multiple times is to design a loop. But python has a simple trick involving a string and a number inside the print function.

Example

str ="Point";
print(str * 3);

Output

Running the above code gives us the following result −

PointPointPoint

*Reversing List Elements Using List Slicing

List slicing is a very powerful technique in python which can also be used to reverse the order of elements in a list.

Example

#Reversing Strings
list1 = ["a","b","c","d"]
print list1[::-1]

# Reversing Numbers
list2 = [1,3,6,4,2]
print list2[::-1]

Output

Running the above code gives us the following result −

['d', 'c', 'b', 'a']
[2, 4, 6, 3, 1]

Find the Factors of a Number

When we are need of the factors of a number, required for some calculation or analysis, we can design a small loop which will check the divisibility of that number with the iteration index.

Example

f = 32
print "The factors of",x,"are:"
for i in range(1, f + 1):
   if f % i == 0:
print(i)

Output

Running the above code gives us the following result −

The factors of 32 are:
1
2
4
8
16
32

*Checking the Usage of Memory

We can check the amount of memory consumed by each variable that we declare by using the getsizeof() function. As you can see below, different string lengths will consume different amount of memory.

Example

import sys
a, b, c,d = "abcde" ,"xy", 2, 15.06
print(sys.getsizeof(a))
print(sys.getsizeof(b))
print(sys.getsizeof(c))
print(sys.getsizeof(d))

Output

Running the above code gives us the following result −

38
35
24
24

 

 

References:

https://www.tutorialspoint.com/10-interesting-python-cool-tricks (Pradeep Elance Published on 08-Aug-2019 10:22:06)

https://www.tutorialspoint.com/questions/category/Python

Timing how long a python script runs

This post introduces several ways to find out how long a python script takes to complete its execution.

  • If you are using Linux or Mac OS, in your terminal
$ time ./your_script.py
  • Several ways to do the task by adding a few lines of code in your py script.
import time
startTime = time.time()

your_func() #python3: print ("It took", time.time() - startTime, "seconds.")

See the following for an example in python 3. 

import time
import functools

startTime = time.time()

print(functools.reduce(lambda x,y: x+y, [47,11,42,13]))

#python3:
print ("It took", time.time() - startTime, "seconds.")

Another way to do the same thing:

from datetime import datetime
startTime = datetime.now()

#do something

#Python 2: 
print datetime.now() - startTime 

#Python 3: 
print(datetime.now() - startTime)

One more way to do the same thing with a nicely formatted output.

import sys
import timeit

startTime = timeit.default_timer()

#do some nice things...

stopTime = timeit.default_timer()
totalRunningTime = stopTime - startTime

# output running time in a nice format.
mins, secs = divmod(totalRunningTime, 60)
hours, mins = divmod(mins, 60)

sys.stdout.write("Total running time: %d:%d:%d.\n" % (hours, mins, secs))

If you want to compare two blocks of code / functions quickly you can do the following:

import timeit

startTime = timeit.default_timer()
your_func1()
#python3
print(timeit.default_timer() - startTime)

startTime2 = timeit.default_timer()
your_func2()
#python3
print(timeit.default_timer() - starTime2)

MPI and OpenMP with Python

This post lists resources for using MPI with Python.

An application built with the hybrid model of parallel programming can run on a computer cluster using both OpenMP and Message Passing Interface (MPI), such that OpenMP is used for parallelism within a (multi-core) node while MPI is used for parallelism between nodes. There have also been efforts to run OpenMP on software distributed shared memory systems,[6] to translate OpenMP into MPI[7][8] and to extend OpenMP for non-shared memory systems.

cython.parallel is built on top of OpenMP (see Using Parallelism)

Please read Laurent Duchesne’s excellent step-by-step guide for parallelizing your Python code using multiple processors and MPI.

On our cluster, to run MPI Python programs, mpi4py has been compiled against OpenMPI 1.10.1 therefore we need to load that additional package:

module load python/3.4.3 mpi/openmpi/1.10.1-gcc

Create the the test MPI example file as described in Laurent’s guide above, using the same name mpi.py:

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

print("I am rank", rank, "of", size)

Create the SLURM submission script submit.sh:

#SBATCH -n 4 
mpirun python mpi.py

You should get output similar to:

I am rank 3 of 4
I am rank 0 of 4
I am rank 1 of 4
I am rank 2 of 4

Craig Finch has a more practical example for high throughput MPI on GitHub.

 

References:

 

Access PostgreSQL from Python using pscopg2

psycopg2 is a python module for PostgreSQL.

  • Basic module usage

http://initd.org/psycopg/docs/usage.html#basic-module-usage

  • with statement

Starting from version 2.5, psycopg2’s connections and cursors are context managers and can be used with the with statement:

http://initd.org/psycopg/docs/usage.html#with-statement

  • Server side cursors

http://initd.org/psycopg/docs/usage.html#server-side-cursors

When a database query is executed, the Psycopg cursor usually fetches all the records returned by the backend, transferring them to the client process. If the query returned an huge amount of data, a proportionally large amount of memory will be allocated by the client.

If the dataset is too large to be practically handled on the client side, it is possible to create a server side cursor. Using this kind of cursor it is possible to transfer to the client only a controlled amount of data, so that a large dataset can be examined without keeping it entirely in memory.

 

Posts Referenced: