Make a request to REST API using Python

This post introduces how to make a request to REST API using Python.

requests package is the commonly used one (its GitHub repo).

You can try out requests online here at codecademy, and here at runnable.com

Look at this post for a great tutorial using Requests with Python to make a request to REST API: Python API tutorial – An Introduction to using APIs (pdf) – a very good, comprehensive, and detailed tutorial.

To install Requests, simply:

$ pip install requests

See below for a simple example to make a request to REST API.

#Python 2.7
#RestfulClient.py

import requests
from requests.auth import HTTPDigestAuth
import json

# Replace with the correct URL
url = "http://api_url"

# It is a good practice not to hardcode the credentials. So ask the user to enter credentials at runtime
myResponse = requests.get(url,auth=HTTPDigestAuth(raw_input("username: "), raw_input("Password: ")), verify=True)
#print (myResponse.status_code)

# For successful API call, response code will be 200 (OK)
if(myResponse.ok):

    # Loading the response data into a dictionary variable
    # json.loads takes in only binary or string variables so using content to fetch binary content
    # Loads (Load String) takes a Json file and converts into python data structure (dictionary or list, depending on JSON)
    jData = json.loads(myResponse.content)
    #jData = json.loads(myResponse2.content, 'utf-8') #use this line if your data contains special characters

    print("The response contains {0} properties".format(len(jData)))
    print("\n")
    for key in jData:
        print key + " : " + jData[key]
else:
  # If response code is not ok (200), print the resulting http error code with description
    myResponse.raise_for_status()

 

======working with JSON data

For example, if data.json file looks like this:

{
 "maps":[
         {"id":"blabla","iscategorical":"0"},
         {"id":"blabla","iscategorical":"0"}
        ],
"masks":
         {"id":"mask-value"},
"om_points":"value",
"parameters":
         {"id":"blabla3"}
}

The python code should be something looks like this:

import json

with open('data.json') as data_file:    
    data = json.load(data_file)
print(data)

We can now  access single values in the json file — see below for some examples to get a sense of it:

data["maps"][0]["id"]  # will return 'blabla'
data["masks"]["id"]    # will return 'mask-value'
data["om_points"]      # will return 'value'

References:

Lambda, map, filter, and reduce functions in python 3

After migration to Python 3 from Python 2,  lambda operator, map() and filter()  functions are still part of core Python; only reduce() function had to go, and it was moved into the module functools

This post introduces how to use lambda, map, filter, and reduce functions in Python 3 (for python 2.7 version, check the references below.)

  • Lambda operator

Some people like it, others hate it and many are afraid of the lambda operator.

The lambda operator or lambda function is a way to create small anonymous functions (i.e., functions without a name). These functions are throw-away functions (i.e., they are just needed where they have been created).

Lambda functions are mainly used in combination with the functions filter(), map() and reduce(). The lambda feature was added to Python due to the demand from Lisp programmers. 

The general syntax of a lambda function is quite simple: 

lambda argument_list: expression 

The argument list consists of a comma separated list of arguments and the expression is an arithmetic expression using these arguments. You can assign the function to a variable, so you can  use it as a function. 

The following example of a lambda function returns the sum of its two arguments:

>>> sum = lambda x, y : x + y
>>> sum(3,4)
7

>>> 

The above example might look like a game for a mathematician — A formalization that turns a straight forward operation into an abstract  formalization.

The above has the same effect by using the following conventional function definition: 

>>> def sum(x,y):
...     return x + y
... 
>>> sum(3,4)
7
>>> 

But, when you learn how to use the map() function, you will see the apparent advantages of this lambda operation.

  • The map function 

The advantage of the lambda operator will be obvious when it is used in combination with the map() function. 

map() is a function which takes two arguments: 

r = map(func, seq)

The first argument func is the name of a function and the second a sequence (e.g. a list).

seqmap() applies the function func to all the elements of the sequence seq. Before Python3, map() used to return a list, where each element of the result list was the result of the function func applied on the corresponding element of the list or tuple “seq”. In Python 3, map() returns an iterator.

The following examples illustrate how map() function works:

>>> def fahrenheit(T):

...   return ((float(9)/5)*T + 32)

... 
# hit Return/Enter to exit to the >>> in your terminal.

>>> def celsius(T):

...   return (float(5)/9)*(T-32)

... 
# hit Return/Enter to exit to the >>> in your terminal.

>>> temperatures = (36.5, 37, 37.5, 38, 39)

>>> F = map(fahrenheit, temperatures)



>>> print(F)

<map object at 0x106d1c3c8>


>>> temperatures_in_Fahrenheit = list(F)

>>> print(temperatures_in_Fahrenheit) 
[97.7, 98.60000000000001, 99.5, 100.4, 102.2]

>>> C = map(celsius, map(fahrenheit, temperatures))

>>> print(C)

<map object at 0x106d1c438>


>>> temperatures_in_Celsius = list(C)


>>> print(temperatures_in_Celsius)
[36.5, 37.00000000000001, 37.5, 38.00000000000001, 39.0]

>>> 

In the example above we haven’t used lambda. When using lambda, we do not need to define and name the functions fahrenheit() and celsius(). You can see this in the following interactive session:

>>> C = [39.2, 36.5, 37.3, 38, 37.8]

>>> F = list(map(lambda x: (float(9)/5)*x + 32, C))

>>> print(F)

[102.56, 97.7, 99.14, 100.4, 100.03999999999999]

>>> C = list(map(lambda x: (float(5)/9)*(x-32), F))

>>> print(C)

[39.2, 36.5, 37.300000000000004, 38.00000000000001, 37.8]

>>> 

map() can be applied to more than one list.

The lists don’t have to have the same length.

map() will apply its lambda function to the elements of the argument lists (i.e., it first applies to the elements with the 0th index, then to the elements with the 1st index until the n-th index is reached). See the following for an illustration example:

>>> a = [1, 2, 3, 4]
>>> b = [17, 12, 11, 10]
>>> c = [-1, -4, 5, 9]
>>> list(map(lambda x, y : x+y, a, b))
[18, 14, 14, 14]
>>> list(map(lambda x, y, z : x+y+z, a, b, c))
[17, 10, 19, 23]
>>> list(map(lambda x, y, z : 2.5*x + 2*y - z, a, b, c))
[37.5, 33.0, 24.5, 21.0]
>>>

We can see in the example above that the parameter x gets its values from the list a, while y gets its values from b, and z from list c. 

If one list has less elements than the others, map() will stop when the shortest list has been completed the mapping:

>>> a = [1, 2, 3]
>>> b = [17, 12, 11, 10]
>>> c = [-1, -4, 5, 9]
>>> 
>>> list(map(lambda x, y, z : 2.5*x + 2*y - z, a, b, c))
[37.5, 33.0, 24.5]
>>>

 

  • The filter function 

filter(function, sequence) 

offers an elegant way to filter out all the elements of a sequence “sequence”, according to the return value of the function function (i.e., if the function returns True it will be kept in the returned iterator object of the filter function). 

In other words: The function filter(f,l) needs a function f as its first argument. f has to return a Boolean value (i.e. either True or False). This function will be applied to every element of the list l. Only if f returns True will the element be produced by the iterator — which is the return value of filter function. 

In the following example, we filter out first the odd and then the even elements of the sequence of the first 10 Fibonacci numbers: 

>>> fibonacci = [0,1,1,2,3,5,8,13,21,34]
>>> odd_numbers = list(filter(lambda x: x % 2, fibonacci))
>>> print(odd_numbers)
[1, 1, 3, 5, 13, 21]
>>> even_numbers = list(filter(lambda x: x % 2 == 0, fibonacci))
>>> print(even_numbers)
[0, 2, 8]

>>> 
  • An example of combining filter() and lambda functions
>>> filter(lambda x: x % 2 == 0, list(range(10,100)))

<filter object at 0x106d1c208>

#this will return all even number between 10 and 100.
>>> list(filter(lambda x: x % 2 == 0, list(range(10,100))))

[10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98]

>>> 

 

  • The reduce() function 

reduce() had been dropped from the core of Python when migrating to Python 3. It was moved into the module functools.

reduce(func, seq) 

continually applies the function func() to the sequence seq. It returns a single value. 

If seq = [ s1, s2, s3, … , sn ], calling reduce(func, seq) works like this:

  • the first two elements of seq will be applied to func, i.e. func(s1,s2). The list on which reduce() applied to looks like this now: [ func(s1, s2), s3, … , sn ]
  • Then,  func will be applied on the previous result and the third element of the list, that is, func(func(s1, s2),s3)
    The list now looks like this: [ func(func(s1, s2),s3), … , sn ]
  • repeat the steps until just one element is left and return this element as the result of reduce() function.

If n is equal to 4 the previous explanation can be illustrated like this: Reduce

The following  simple example illustrates how reduce() works. 

>>> import functools
>>> functools.reduce(lambda x,y: x+y, [47,11,42,13])
113
>>>

The following diagram shows the intermediate steps of the calculation: 

 

See below for some examples of using reduce() function.

 

#get maximum number from a list using reduce():

>>> from functools import reduce
>>> f = lambda a,b: a if (a > b) else b
>>> reduce(f, [47,11,42,102,13])
102
>>>
# Calculating the sum of the numbers from 1 to 100:
>>> from functools import reduce
>>> reduce(lambda x, y: x+y, range(1,101))
5050

It’s very straightforward to change the previous example to calculate the product (the factorial) from 1 to a number. We just need to change the “+” into “*”:

>>> reduce(lambda x, y: x*y, range(1,5))

24

>>> 

 

References:

 

Print multiple variables in Python3

This post introduces several ways to print multiple arguments in python 3.

  • Pass it as a tuple:
print("The cost for %s is %s" % (name, cost))
  • Pass it as a dictionary:
print("The cost for %(n)s is %(c)s" % {'n': name, 'c': cost})
  • Use the new-style string formatting:
print("the cost for {} is {}".format(name, cost))
  • Use the new-style string formatting with numbers (useful for reordering or printing the same one multiple times):
print("The cost for {0} is {1}".format(name, cost))
  • Use the new-style string formatting with explicit names:
print("The cost for {n} is {c}".format(n=name, c=cost))
  • Pass the values as parameters and print will do it:
print("The cost for", name, "is", cost)

If you don’t want spaces to be inserted automatically by print in the above example, change the sep parameter:

print("The cost for ", name, " is ", cost, sep='')
  • Use string concatenation
print("The cost for " + name + " is " + cost)

NOTE: If cost  is an int, then, you should convert it to str:

print("The cost for " + name + " is " + str(cost))
  • Note that %s mentioned above can be replace by %d or %f.

If cost is a number, then

print("The cost for %s is %d" % (name, cost))

If cost is a string, then

print("The cost for %s is %s" % (name, cost))

If cost is a number, then it’s %d, if it’s a string, then it’s %s, if cost is a float, then it’s %f

  • Use the new f-string formatting in Python 3.6:
print(f'The cost for {name} is {cost}')

Open MPI library

This post introduces the Open MPI library. (Its official website, its GitHub Repo)

Open MPI is a popular open source MPI implementation.

Open MPI is a Message Passing Interface (MPI) library project combining technologies and resources from several other projects (FT-MPI, LA-MPI, LAM/MPI, and PACX-MPI). It is used by many TOP500 supercomputers including Roadrunner, which was the world’s fastest supercomputer from June 2008 to November 2009,[1]and K computer, the fastest supercomputer from June 2011 to June 2012.[2][3]

The Open MPI developers selected these MPI implementations as excelling in one or more areas. Open MPI aims to use the best ideas and technologies from the individual projects and create one world-class open-source MPI implementation that excels in all areas. The Open MPI project specifies several top-level goals:

  • to create a free, open source software, peer-reviewed, production-quality complete MPI-3.0 implementation
  • to provide extremely high, competitive performance (low latency or high bandwidth)
  • to involve the high-performance computing community directly with external development and feedback (vendors, 3rd party researchers, users, etc.)
  • to provide a stable platform for 3rd-party research and commercial development
  • to help prevent the “forking problem” common to other MPI projects[4]
  • to support a wide variety of high-performance computing platforms and environments

 

 

 

 

Parallel Programming using MPI in Python

This post introduces Parallel Programming using MPI in Python.

The library is mpi4py (MPI and python extensions of MPI), see here for its code repo on bitbucket.

Laurent Duchesne provides an excellent step-by-step guide for parallelizing your Python code using multiple processors and MPI. Craig Finch has a more practical example for high throughput MPI on GitHub. See here for more mpi4py examples from Craig Finch.

An example of TensorFlow using MPI can be found here.

References:

 

OpenJDK or Oracle JDK? What is the main difference?

This post introduces what is OpenJDK and Oracle JDK and what is the difference and which one should we use on Ubuntu.

Both OpenJDK and Oracle JDK are created and maintained currently by Oracle only.

OpenJDK is the default version of Java that Ubuntu uses and is the easiest to install while Oracle Java 7/8 is Oracle’s own version of Java 7/8.

It entirely depends on the target platform on which you want to run JDK. Technical differences are a consequence of the goal of each one (OpenJDK is meant to be the reference implementation, open to the community, while Oracle is meant to be a commercial one)

They both have “almost” the same code of the classes in the Java API; but the code for the virtual machine itself is actually different, and when it comes to libraries, OpenJDK tends to use open libraries while Oracle tends to use closed ones.

OpenJDK was reported to work better for large number of users with small request count, while it become worse for small number of user with prolonged. This is an undocumented behaviour, and never seen anywhere other than experienced on some J2EE containers.

My conclusion:
I choose to install Oracle JDK, since there were complaints about using OpenJDK would meet bugs sometimes. (See this post if you decide to install Oracle Java 8 with PPA on Ubuntu.)

References:

Which Java package should I use: OpenJDK or Oracle JDK?

Performance OracleJDK or OpenJDK (pdf)

OpenJDK – Oracle is better? (pdf)

Is there any advantage of installing OpenJDK instead of Oracle Java Platform, Standard Edition on Ubuntu? (pdf)