121. More list comprehension examples

Example 1 – double the numbers

Standard loop approach:

foo = [1, 2, 3, 4]
bar = []

for x in foo:
    bar.append(x * 2)
    
print(bar)

Out:

[2, 4, 6, 8]

Using list comprehension:

foo = [1, 2, 3, 4]
bar = [x * 2 for x in foo]
print(bar)

Out:

[2, 4, 6, 8]

Example 2 – convert Celsius to Fahrenheit

This example calls a function from within the list comprehension.

Define the function:

def convert_celsius_to_fahrenheit(deg_celsius):
    """
    Convert degress celsius to fahrenheit
    Returns float value - temp in fahrenheit
    Keyword arguments:
        def_celcius -- temp in degrees celsius
    """
    return (9/5) * deg_celsius + 32

Standard loop approach:

#list of temps in degree celsius to convert to fahrenheit
celsius = [39.2, 36.5, 37.3, 41.0]

#standard for loop approach
fahrenheit = []
for x in celsius:
    fahrenheit.append(convert_celsius_to_fahrenheit(x))

print('Using standard for loop: {}'.format(fahrenheit))

Out:

Using standard for loop: [102.56, 97.7, 99.14, 105.8]

Using list comprehension

fahrenheit = [convert_celsius_to_fahrenheit(x) for x in celsius]
print('Using list comprehension: {}'.format(fahrenheit))

Out:

Using list comprehension: [102.56, 97.7, 99.14, 105.8]

Example 3 – convert the strings to different data types

This example also make use of the zip function. Zip allows you to iterate through two lists at the same time.

inputs = ["1", "3.142", "True", "spam"]
converters = [int, float, bool, str]

values_with_correct_data_types = [t(s) for (s, t) in zip(inputs, converters)]
print(values_with_correct_data_types)

Out:

[1, 3.142, True, 'spam']

Example 4 – Using if statements within a list comprehension

The example filters a list of file names to the python files only

unfiltered_files = ['test.py', 'names.csv', 'fun_module.py', 'prog.config']

# Standard loop form
python_files = []
# filter the files using a standard for loop 
for file in unfiltered_files:
    if file[-2:] == 'py':
        python_files.append(file)
        
print('using standard for loop: {}'.format(python_files))

#list comprehension
python_files = [file for file in unfiltered_files if file[-2:] == 'py']

print('using list comprehension {}'.format(python_files))

Out:

using standard for loop: ['test.py', 'fun_module.py']
using list comprehension ['test.py', 'fun_module.py']

Example 5 – List comprehension to create a list of lists

list_of_lists = []

# Standard loop form
for i in range(5):
    sub_list = []
    for j in range(3):
        sub_list.append(i * j)
    list_of_lists.append(sub_list)

print(list_of_lists)


# List comprehension
list_of_lists = [[i * j for j in range(3)] for i in range(5)]

print(list_of_lists)

Out:

[[0, 0, 0], [0, 1, 2], [0, 2, 4], [0, 3, 6], [0, 4, 8]]
[[0, 0, 0], [0, 1, 2], [0, 2, 4], [0, 3, 6], [0, 4, 8]]

Example 6: Iterate over all items in a list of lists

The code converts a list of lists to a list of items
We call this flattening the list.

list_of_lists = [[8, 2, 1], [9, 1, 2], [4, 5, 100]]

# Standard loop form
flat_list = []
for row in list_of_lists:
    for col in row:
        flat_list.append(col)

print(flat_list)

# List comprehension:
flat_list = [item for sublist in list_of_lists for item in sublist]
print(flat_list)

Out:

[8, 2, 1, 9, 1, 2, 4, 5, 100]
[8, 2, 1, 9, 1, 2, 4, 5, 100]

118: Python basics – saving python objects to disk with pickle

Sometimes we may wish to save Python objects to disc (for example if we have performed a lot of processing to get to a certain point). We can use Python’s pickle method to save and reload any Python object. Here we will save and reload a NumPy array, and then save and reload a collection of different objects.

Saving a single python object

Here we will use pickle to save a single object, a NumPy array.

import pickle 
import numpy as np

# Create array of random numbers:
my_array= np.random.rand(2,4)
print (my_array)

Out:

[[0.6383297  0.45250192 0.09882854 0.84896196]
 [0.97006917 0.29206495 0.92500062 0.52965801]]
# Save using pickle
filename = 'pickled_array.p'
with open(filename, 'wb') as filehandler:
    pickle.dump(my_array, filehandler)

Reload and print pickled array:

filename = 'pickled_array.p'
with open(filename, 'rb') as filehandler: 
    reloaded_array = pickle.load(filehandler)

print ('Reloaded array:')
print (reloaded_array)

Out:

Reloaded array:
[[0.6383297  0.45250192 0.09882854 0.84896196]
 [0.97006917 0.29206495 0.92500062 0.52965801]]

Using a tuple to save multiple objects

We can use pickle to save a collection of objects grouped together as a list, a dictionary, or a tuple. Here we will save a collection of objects as a tuple.

# Create an array, a list, and a dictionary
my_array = np.random.rand(2,4)
my_list =['A', 'B', 'C']
my_dictionary = {'name': 'Bob', 'Age': 42}
# Save all items in a tuple
items_to_save = (my_array, my_list, my_dictionary)
filename = 'pickled_tuple_of_objects.p'
with open(filename, 'wb') as filehandler:
    pickle.dump(items_to_save, filehandler)

Reload pickled tuple, unpack the objects, and print them.

filename = 'pickled_tuple_of_objects.p'
with open(filename, 'rb') as filehandler:
    reloaded_tuple = pickle.load(filehandler)


reloaded_array = reloaded_tuple[0]
reloaded_list = reloaded_tuple[1]
reloaded_dict = reloaded_tuple[2]

print ('Reloaded array:')
print (reloaded_array)
print ('\nReloaded list:')
print (reloaded_list)
print ('\n Reloaded dictionary')
print (reloaded_dict)

Out:

Reloaded array:
[[0.40193978 0.55173167 0.89411291 0.84625061]
 [0.86540981 0.27835353 0.43359222 0.31579122]]

Reloaded list:
['A', 'B', 'C']

 Reloaded dictionary
{'name': 'Bob', 'Age': 42}

98: Brief examples of applying lambda functions to lists, and filtering lists with list comprehensions, map and filter

These examples are intended as a reminder of how to use list comprehensions, map and filter. They are not intended to be an exhaustive tutorial.

Let’s start with a list of numbers, and we wish to:

1) Square all numbers
2) Find even numbers
3) Square all even numbers

That is where list comprehensions, map and filter come in. There is significant overlap between these methodologies, but here they are.

Define my list of numbers

my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Square numbers with list comprehension

answer = [x**2 for x in my_list]
print (answer)
Out:
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Map a named lambda function to square the numbers

We use a one-line lambda function here, but a full function with define and return could also be used.

sq = lambda x: x**2
answer = list(map(sq, my_list))
print (answer)
Out:
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Map a lambda function directly to square numbers

answer = list(map(lambda x: x**2, my_list))
print (answer)
Out:
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Use a lambda function in a list comprehension

sq = lambda x: x**2
answer = [sq(x) for x in my_list]
print (answer)
Out:
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Filter a list with list comprehension to find even numbers

For even numbers x%2 will equal zero:

answer = [x for x in my_list if x%2 == 0]
print (answer)
Out:
[2, 4, 6, 8, 10]

Filter a list with a named lambda function

Full functions call also be used. The function (full or lambda must return True or False)

is_even = lambda x: x%2 == 0
answer = list(filter(is_even, my_list))
print (answer)
Out:
[2, 4, 6, 8, 10]

Filter a list with a lambda function applied directly

answer = list(filter(lambda x: x%2 ==0, my_list))
print (answer)
[2, 4, 6, 8, 10]

Combine squaring and filtering with a list comprehension

List comprehensions may filter and apply a function in one line. To get the square of all even numbers in our list:

answer = [x**2 for x in my_list if x%2 == 0]
print (answer)
Out:
[4, 16, 36, 64, 100]

Or we may apply lambda (or full) functions in both the operation and the filter ina list comprehension.

sq = lambda x: x**2
is_even = lambda x: x%2 == 0
answer = [sq(x) for x in my_list if is_even(x)]
print (answer)
Out:
[4, 16, 36, 64, 100]

Combine squaring and filtering with map and filter

filter and map would need to be used in two statements to achieve the same end. Here we will use named lambda functions, but they could be applied directly as above.

sq = lambda x: x**2
is_even = lambda x: x%2 == 0

filtered = list(filter(is_even, my_list))
answer = list(map(sq, filtered))
print (answer)
Out:
[4, 16, 36, 64, 100]

 

 

92. Queues

Python queues allow objects of any kind to be added to and removed from queues.

Here are three basic types of queue:

  1. FIFO (First In First Out)
  2. LIFO (Last In First Out)
  3. Priority (each item is assigned a priority in the queue)

Note: Python queues are not iterable; we cannot look through objects in the queue. It may at times be useful to maintain a separate list of queued items.

FIFO (First In First Out) queue

Here we will add numbers to a queue, but any Python object may be added to a queue.

# FIFO (First In First Out) queue

import queue
print ('FIFO (First In First Out) Queue')

q = queue.Queue()

# Add items to queue (add numbers 0 to 4)
for item in range(5):
    q.put(item) 

# Retrieve items from queue using a loop. 
# See additional method below of continually retrieving
# until queue is empty.
    
for item in range(5):
    x = q.get() # put item in queue
    print (x, end = ' ')

OUT:
FIFO (First In First Out) Queue
0 1 2 3 4

LIFO (Last In First Out) queue

# LIFO (Last In First Out) queue

import queue
    
print ('\nLIFO (Last In First Out) Queue')

q = queue.LifoQueue()

for item in range(5):
    q.put(item) # put item in queue
    
# Alternative way of emptying queue

while not q.empty():
    x = q.get() # put item in queue
    print (x, end = ' ')

OUT:
LIFO (Last In First Out) Queue
4 3 2 1 0 

Priority

In a priority queue an object is passed to the queue in a tuple. The first item of the tuple is the priority (lower number is higher priority by default), and the second item is the object to be queued. By default items of equal priority are handled as FIFO.

# Priority queue
import queue
import random
print ('\nPriority Queue')

q = queue.PriorityQueue()

for i in range(5):
    priority = random.randint(1,100)
    name = 'Item ' + str(i)
    item = (priority, name) # A tuple is used to pass priority and item
    q.put(item)

while not q.empty():
    x = q.get()
    print (x, end = ' ')

OUT:

Priority Queue
(15, 'Item 2') (47, 'Item 1') (52, 'Item 0') (54, 'Item 3') (81, 'Item 4')

83. Automatically passing unpacked lists or tuples to a function (or why do you see * before lists and tuples)

Imagine we have a very simple function that adds three numebrs together:

In [1]:
def add_three_numbers (a, b, c):
    return a + b + c

We would normally pass separate numebrs to the function, e.g.

add_three_numbers (10, 20, 35)
65

But what if our numbers are in a list or a tuple. If we pass that as a list then we are passing only a single argument, and the function declares an error:

In [3]:
my_list = [10, 20, 35]

add_three_numbers (my_list)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-900947bff326> in <module>()
      1 my_list = [10, 20, 35]
      2 
----> 3 add_three_numbers (my_list)

TypeError: add_three_numbers() missing 2 required positional arguments: 'b' and 'c'

Python allows us to pass the list or tuple with the instruction to unpack it for input into the function. We instruct Python to unpack the list/tuple with an asterix before the list/tuple:

add_three_numbers (*my_list)
65

 

 

19. Accessing date and time, and timing code

Accessing date and time

It may sometimes be useful to access the current date and/or time. As an example, when writing code it may be useful to access the time at particular stages to monitor how long different parts of code are taking.

To access date and time we will use the datetime module (which is held in a package that is also, a little confusingly called datetime!): Continue reading “19. Accessing date and time, and timing code”

18. Python basics: Lambda functions (one line functions), and map/filter/reduce

Lambda functions are a Pythonic way of writing simple functions in a single line.

They may be applied to a list using the map statement.

The function filter offers a concise way to filter out all the elements of a list based on the results of a function.

The function reduce continually applies a function to a sequence until a single value is returned. Continue reading “18. Python basics: Lambda functions (one line functions), and map/filter/reduce”