96. Passing arguments to Python from the command line (or other programs)

Many Python programs have all the variables defined within the Python code (or the Python code reads input files). It my be useful at times, however, to be able to pass one or more arguments to Python when calling the program from the command line or another programme. This is simple in Python. Below is an example (saved as ‘mycode.py’) that takes two arguments and multiplies them together:

import sys

x,y = int(sys.argv[1]), int(sys.argv[2])

print (x * y)

Now we can call that from the command line, passing our variables:

python mycode.py 3 4

>12

This type of code may allow us to write generic Python code, and call it as part of an automated sequence from another piece of code which passes the required variables to Python.

92. Queues

Python queues allow objects of any kind to be added to and removed from queues.

Here are three basic types of queue:

  1. FIFO (First In First Out)
  2. LIFO (Last In First Out)
  3. Priority (each item is assigned a priority in the queue)

Note: Python queues are not iterable; we cannot look through objects in the queue. It may at times be useful to maintain a separate list of queued items.

FIFO (First In First Out) queue

Here we will add numbers to a queue, but any Python object may be added to a queue.

# FIFO (First In First Out) queue

import queue
print ('FIFO (First In First Out) Queue')

q = queue.Queue()

# Add items to queue (add numbers 0 to 4)
for item in range(5):
    q.put(item) 

# Retrieve items from queue using a loop. 
# See additional method below of continually retrieving
# until queue is empty.
    
for item in range(5):
    x = q.get() # put item in queue
    print (x, end = ' ')

OUT:
FIFO (First In First Out) Queue
0 1 2 3 4

LIFO (Last In First Out) queue

# LIFO (Last In First Out) queue

import queue
    
print ('\nLIFO (Last In First Out) Queue')

q = queue.LifoQueue()

for item in range(5):
    q.put(item) # put item in queue
    
# Alternative way of emptying queue

while not q.empty():
    x = q.get() # put item in queue
    print (x, end = ' ')

OUT:
LIFO (Last In First Out) Queue
4 3 2 1 0 

Priority

In a priority queue an object is passed to the queue in a tuple. The first item of the tuple is the priority (lower number is higher priority by default), and the second item is the object to be queued. By default items of equal priority are handled as FIFO.

# Priority queue
import queue
import random
print ('\nPriority Queue')

q = queue.PriorityQueue()

for i in range(5):
    priority = random.randint(1,100)
    name = 'Item ' + str(i)
    item = (priority, name) # A tuple is used to pass priority and item
    q.put(item)

while not q.empty():
    x = q.get()
    print (x, end = ' ')

OUT:

Priority Queue
(15, 'Item 2') (47, 'Item 1') (52, 'Item 0') (54, 'Item 3') (81, 'Item 4')

91. Speed up Python by 1,000 times or more using numba!

If you have installed Python via anaconda (https://www.anaconda.com/) then numba is already available to you. Otherwise numba may be installed using pip (pip install numba).

Functions written in pure Python or NumPy may be speeded up by using the numba library and using the decorator @jit before a function. This is especially useful for loops where Python will normally compile to machine code (the language the CPU understands) for each iteration of the loop. Using numba the loop is compiled into machine code just once the first time it is called.

Let’s look at an example:

from numba import jit
import numpy as np
import timeit

# Define a function normally without using numba

def test_without_numba():
for i in np.arange(1000):
x = i ** 0.5
x *= 0.5

# Define a function using numba jit. Using the argument nopython=True gives the
# fastest possible run time, but will error if numba cannot precomplile all the
# code. Using just @jit will allow the code to mix pre-compiled and normal code
# but will not be as fast as possible

@jit(nopython=True)
def test_with_numba():
for i in np.arange(1000):
x = i ** 0.5
x *= 0.5

# Run functions first time without timing (compilation included in first run)
test_without_numba()
test_with_numba()

# Time functions with timeit (100 repeats).
# Multiply by 1000 to give milliseconds

timed = timeit.timeit(stmt = test_without_numba, number=100) * 1000
print ('Milliseconds without numba: %.3f' %timed)

timed = timeit.timeit(stmt = test_with_numba, number=100) * 1000
print ('Milliseconds with numba: %.3f' %timed)

OUTPUT:
Milliseconds without numba: 183.771
Milliseconds with numba: 0.025

We have a 7,000 fold increase in speed!!

Note: not all code will be speeded up by numba. Pandas for example are not helped by numba, and using numba will actually slow panda code down a little (because it looks for what can be pre-complied which takes time). So always test numba to see which functions it can speed up (and consider breaking larger functions down into smaller ones so that blocks that can use numba may be separated out).

If the default decorator @jit is used, with no other arguments, numba will allow a mix of code that can be pre-compiled with code that can’t. For the fastest execution use @jit(nopython=True), but you may need to break your function down because this mode will error if parts of the function cannot be pre-compiled by numba.

 

84. Function decorators

Decorators (identified by @ in line above function definition) allow code to be run before and after the function (hence ‘decorating’ it). An example of use of a decorator is shown below when a decorator function is used to time two different functions. This removes the need for duplicating code in different functions, and also keeps the functions focussed on their primary objective. Continue reading “84. Function decorators”