Iterables and Iterators

Iterables and Iterators#

What you wil learn in this lesson:

Further understand the concept of iterables and how iteration works in Python.
Use common iteration methods such as for loops and the iter and next functions.
Implement list comprehension for concise iteration-based transformations.

Introduction#

In the previous lesson, we introduced loops, and we mentioned that in Python, loops can iterate through things like lists. Why is that? Because lists can behave as iterable objects. Let’s see this in more detail:

Iterables#

An iterable is an object capable of returning its elements one at a time. Examples of iterables include strings and any of the data structures studied so far: lists, tuples, dictionaries, sets and ranges.
They can are usually used in a for loop. Their general syntax is:

for var in iterable :
expression

When using an iterable, be mindful of the order. Remember that elements in lists, tuples, dictionaries, strings and range come out of them in the same order as they were put in, whereas for sets the ordering of the items is arbitrary.

Iterators#

An iterator is an object that keeps track of its position during iteration.
To define an iterator, you may use iter(). Iterators outputs a value each time it is used. Importantly, you can convert an iterable into an iterator using this function, i.e. iter(iterable)
The next() function allows you to manually retrieve the next element from the iterator.

Examples#

Lists#

iterating using for

tokens = ['living room','was','quite','large']

for tok in tokens:
    print(tok)

living room
was
quite
large

iterating using iter() and next()

tokens = ['living room','was','quite','large']

myit = iter(tokens)
print(next(myit)) 
print(next(myit))
print(next(myit)) 
print(next(myit)) 

living room
was
quite
large

Calling next() when the iterator has reached the end of the list produces an exception:

print(next(myit))

---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
Cell In[3], line 1
----> 1 print(next(myit))

StopIteration: 

Next, look at the type of the iterator, and the documentation

type(myit)

list_iterator

Iterating using for vs using iter and next for implicitly creates an iterator and executes next on each loop iteration.

Strings#

iterating using for

strn = 'datum'

for s in strn:
    print(s)

d
a
t
u
m

iterating using iter() and next()

st = iter(strn)

print(next(st))
print(next(st))
print(next(st))
print(next(st))
print(next(st))

d
a
t
u
m

Tuples#

iterating using for loop`

metrics = ('auc','recall','precision','support')

for met in metrics:
    print(met)

auc
recall
precision
support

iterating using iter() and next()

metrics = ('auc','recall','precision','support')

tup_metrics = iter(metrics)
print(next(tup_metrics))
print(next(tup_metrics))
print(next(tup_metrics))
print(next(tup_metrics))

auc
recall
precision
support

Dictionaries#

iterating using for-loop

courses = {'fall':['regression','python'], 'spring': ['ml','pyspark','nlp']}

Python’s default is to interate over the keys

# iterate over keys
for k in courses:
    print(k)

fall
spring

# iterate over keys, using keys() method
for k in courses.keys():
    print(k)

fall
spring

# iterate over values
for v in courses.values():
    print(v)

['regression', 'python']
['ml', 'pyspark', 'nlp']

# iterate over keys and values using `items()`
for k, v in courses.items():
    print("key  :", k)
    print("value:", v)
    print("-"*40)

key  : fall
value: ['regression', 'python']
----------------------------------------
key  : spring
value: ['ml', 'pyspark', 'nlp']
----------------------------------------

# Alternatively for the last one, iterate over keys and values using `key()`.
for k in courses.keys():
    print("key  :", k)
    print("value:", courses[k]) # index into the dict with the key
    print("-"*40)

key  : fall
value: ['regression', 'python']
----------------------------------------
key  : spring
value: ['ml', 'pyspark', 'nlp']
----------------------------------------

iterating using iter() and next()

courses_iter = iter(courses)

print(next(courses_iter))
print(next(courses_iter))

fall
spring

Sets#

iterating using for
note: set has no notion of order

princesses = {'belle','cinderella','rapunzel'}

for princess in princesses:
    print(princess)

cinderella
rapunzel
belle

iterating using iter() and next()

princesses = {'belle','cinderella','rapunzel'}

myset = iter(princesses) 
print(next(myset))
print(next(myset))
print(next(myset))

cinderella
rapunzel
belle

Ranges#

iterating using for

my_range = range(0, 10, 2)

for ii in my_range:
    print(ii)

iterating using iter() and next()

my_range = iter(my_range) 
print(next(my_range))
print(next(my_range))
print(next(my_range))
print(next(my_range))
print(next(my_range))

Nested Loops#

Iterations can be nested!

This works well with nested data structures, like dicts within dicts.

Be careful, though – these can get complicated.

for i, semester in enumerate(courses):
    print(f"{i+1}. {semester.upper()}:")
    for j, course in enumerate(courses[semester]):
        print(f"\t{i+1}.{j+1}. {course}")

FALL:
1. regression
2. python
SPRING:
1. ml
2. pyspark
3. nlp

Comprehension#

Comprehension provides a concise way to construct new iterables by iterating over existing ones. The most common type is list comprehension, but comprehension can also be used to create dictionaries, sets, and generators.

Introduction#

Let’s start with a for loop and list example:

vals = [1, 5, 6, 8, 12, 15]
is_odd = []

for val in vals:   
    if val % 2: # if remainder is one, val is odd
        is_odd.append(True)
    else:       # else it's not odd
        is_odd.append(False)

is_odd

[True, True, False, False, False, True]

The code loops over each value in the list, checks the condition, and appends to a new list.

The code works, but it’s lengthy compared to a list comprehension.

The approach takes extra time to write and understand.

Let’s improve this situation with a list comprehension!

is_odd = [val % 2 == 1 for val in vals]
is_odd

[True, True, False, False, False, True]

As you can see, this piece of code is much shorter, and if you understand the syntax, quicker to interpet.

Note the in-place use of an expression.

General Theory#

There are comprehensions for almost each list-like object:

List comprehensions
Dictionary comprehensions
Set comprehensions

Comprehensions are essentially very concise for loops. They are compact visually, but they also are more efficient than usual loops.

All comprehensions in Python share the same fundamental components, which can be summarized as:

[expression for item in iterable if condition]

Expression defines how each item in the resulting sequence will be transformed. This has the structure of the kind of comprehension. So, for example, dictionary comprehension expressions take the form key_expression:value_expression while lists and sets use value_expression.
Item is a variable representing each element from the iterable.
Iterable contains the series of elements you are looping over (e.g., a list, set, dictionary, string).
Condition allows you to exclude elements from the result if they do not meet a specified condition. This is optional.

The type of comprehension is indicated by the use of enclosing pairs:

List comprehensions [expression for item in iterable if condition]
Dictionary comprehensions {key_expression:value_expression for item in iterable if condition}
Set comprehensions {expression for item in iterable if condition}

Note

There is also another type of comprehension that we are not going to cover in this course, which is called generator comprehension. This looks like a tuple comprehension, i.e. (expression for item in iterable if condition). What it does it to create a generator, which lazily evaluates the expression and yields items one by one. In contrast to tuples, a generator does not store all the results in memory.

Examples#

Given list of words and a list of stop words, filter out the stop words (considered not important).

stop_words = ['a','am','an','i','the','of']
words      = ['i','am','not','a','fan','of','the','film']

clean_words = [wd for wd in words if wd not in stop_words]
clean_words

['not', 'fan', 'film']

# Compare with this
clean_words = []
for wd in words:
    if wd not in stop_words:
        clean_words.append(wd)
clean_words

['not', 'fan', 'film']

Given a list of measurements, retain elements with specific units

units = 'mmHg'
measures = {'20', '115mmHg', '5mg', '10 mg', '7.5dl', '120 mmHg'}

meas_mmhg = [meas for meas in measures if units in meas]

meas_mmhg

['120 mmHg', '115mmHg']

You can also filter on 2 conditions:

units1 = 'mmHg'
units2 = 'dl'
meas_mmhg_dl = [meas for meas in measures if units1 in meas or units2 in meas]

meas_mmhg_dl

['120 mmHg', '7.5dl', '115mmHg']

For clarity, you should read it like this:

[meas 
 for meas in measures 
 if units1 in meas 
 or units2 in meas]

['120 mmHg', '7.5dl', '115mmHg']

Example using Dictionary comprehensions

# various deep learning models and their depths
model_arch = {'cnn_1':'15 layers', 'cnn_2':'20 layers', 'rnn': '10 layers'}

# create a new dict containing only key-value pairs where the key contains 'cnn'

cnns = {key:model_arch[key] for key in model_arch.keys() if 'cnn' in key}
cnns

{'cnn_1': '15 layers', 'cnn_2': '20 layers'}

We build the key-value pairs using key:model_arch[key], where the key indexes into the dict model_arch

Practice exercises#

Exercise 19

1- Create a list of strings, where each string contains a mix of uppercase and lowercase letters. Write a for loop` to iterate over the strings such that, in each iteration, it prints the string in lower case.

2 - Using the previous list, use iter() and next() to iterate over the list, printing each string. In this case, the strings don’t need to be lowercased.

# Your answers start here

Exercise 20

Given a list of words, create a new list that contains only the words that start with a vowel. Do this with and without comprehension.

# Your answers start here

Iterables and Iterators

Contents

Iterables and Iterators#

Introduction#

Iterables#

Iterators#

Examples#

Lists#

Strings#

Tuples#

Dictionaries#

Sets#

Ranges#

Nested Loops#

Comprehension#

Introduction#

General Theory#

Examples#

Practice exercises#