Chapter 1. A Crash Course in Python

People are still crazy about Python after twenty-five years, which I find hard to believe.

Michael Palin

All new employees at DataSciencester are required to go through new employee orientation, the most interesting part of which is a crash course in Python.

This is not a comprehensive Python tutorial but instead is intended to highlight the parts of the language that will be most important to us (some of which are often not the focus of Python tutorials).

The Basics

The Zen of Python

Python has a somewhat Zen description of its design principles, which you can also find inside the Python interpreter itself by typing import this.

One of the most discussed of these is:

There should be one—and preferably only one—obvious way to do it.

Code written in accordance with this “obvious” way (which may not be obvious at all to a newcomer) is often described as “Pythonic.” Although this is not a book about Python, we will occasionally contrast Pythonic and non-Pythonic ways of accomplishing the same things, and we will generally favor Pythonic solutions to our problems.

Whitespace Formatting

Many languages use curly braces to delimit blocks of code. Python uses indentation:

for i in [1, 2, 3, 4, 5]:
    print i                     # first line in "for i" block
    for j in [1, 2, 3, 4, 5]:
        print j                 # first line in "for j" block
        print i + j             # last line in "for j" block
    print i                     # last line in "for i" block
print "done looping"

This makes Python code very readable, but it also means that you have to be very careful with your formatting. Whitespace is ignored inside parentheses and brackets, which can be helpful for long-winded computations:

long_winded_computation = (1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 +
                           13 + 14 + 15 + 16 + 17 + 18 + 19 + 20)

and for making code easier to read:

list_of_lists = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

easier_to_read_list_of_lists = [ [1, 2, 3],
                                 [4, 5, 6],
                                 [7, 8, 9] ]

You can also use a backslash to indicate that a statement continues onto the next line, although we’ll rarely do this:

two_plus_three = 2 + \
                 3

One consequence of whitespace formatting is that it can be hard to copy and paste code into the Python shell. For example, if you tried to paste the code:

for i in [1, 2, 3, 4, 5]:

    # notice the blank line
    print i

into the ordinary Python shell, you would get a:

IndentationError: expected an indented block

because the interpreter thinks the blank line signals the end of the for loop’s block.

IPython has a magic function %paste, which correctly pastes whatever is on your clipboard, whitespace and all. This alone is a good reason to use IPython.

Lists

Probably the most fundamental data structure in Python is the list. A list is simply an ordered collection. (It is similar to what in other languages might be called an array, but with some added functionality.)

integer_list = [1, 2, 3]
heterogeneous_list = ["string", 0.1, True]
list_of_lists = [ integer_list, heterogeneous_list, [] ]

list_length = len(integer_list)     # equals 3
list_sum    = sum(integer_list)     # equals 6

You can get or set the nth element of a list with square brackets:

x = range(10)   # is the list [0, 1, ..., 9]
zero = x[0]     # equals 0, lists are 0-indexed
one = x[1]      # equals 1
nine = x[-1]    # equals 9, 'Pythonic' for last element
eight = x[-2]   # equals 8, 'Pythonic' for next-to-last element
x[0] = -1       # now x is [-1, 1, 2, 3, ..., 9]

You can also use square brackets to “slice” lists:

first_three   = x[:3]               # [-1, 1, 2]
three_to_end = x[3:]                # [3, 4, ..., 9]
one_to_four = x[1:5]                # [1, 2, 3, 4]
last_three = x[-3:]                 # [7, 8, 9]
without_first_and_last = x[1:-1]    # [1, 2, ..., 8]
copy_of_x = x[:]                    # [-1, 1, 2, ..., 9]

Python has an in operator to check for list membership:

1 in [1, 2, 3]    # True
0 in [1, 2, 3]    # False

This check involves examining the elements of the list one at a time, which means that you probably shouldn’t use it unless you know your list is pretty small (or unless you don’t care how long the check takes).

It is easy to concatenate lists together:

x = [1, 2, 3]
x.extend([4, 5, 6])     # x is now [1,2,3,4,5,6]

If you don’t want to modify x you can use list addition:

x = [1, 2, 3]
y = x + [4, 5, 6]       # y is [1, 2, 3, 4, 5, 6]; x is unchanged

More frequently we will append to lists one item at a time:

x = [1, 2, 3]
x.append(0)      # x is now [1, 2, 3, 0]
y = x[-1]        # equals 0
z = len(x)       # equals 4

It is often convenient to unpack lists if you know how many elements they contain:

x, y = [1, 2]    # now x is 1, y is 2

although you will get a ValueError if you don’t have the same numbers of elements on both sides.

It’s common to use an underscore for a value you’re going to throw away:

_, y = [1, 2]    # now y == 2, didn't care about the first element

Dictionaries

Another fundamental data structure is a dictionary, which associates values with keys and allows you to quickly retrieve the value corresponding to a given key:

empty_dict = {}                         # Pythonic
empty_dict2 = dict()                    # less Pythonic
grades = { "Joel" : 80, "Tim" : 95 }    # dictionary literal

You can look up the value for a key using square brackets:

joels_grade = grades["Joel"]            # equals 80

But you’ll get a KeyError if you ask for a key that’s not in the dictionary:

try:
    kates_grade = grades["Kate"]
except KeyError:
    print "no grade for Kate!"

You can check for the existence of a key using in:

joel_has_grade = "Joel" in grades     # True
kate_has_grade = "Kate" in grades     # False

Dictionaries have a get method that returns a default value (instead of raising an exception) when you look up a key that’s not in the dictionary:

joels_grade = grades.get("Joel", 0)   # equals 80
kates_grade = grades.get("Kate", 0)   # equals 0
no_ones_grade = grades.get("No One")  # default default is None

You assign key-value pairs using the same square brackets:

grades["Tim"] = 99                    # replaces the old value
grades["Kate"] = 100                  # adds a third entry
num_students = len(grades)            # equals 3

We will frequently use dictionaries as a simple way to represent structured data:

tweet = {
    "user" : "joelgrus",
    "text" : "Data Science is Awesome",
    "retweet_count" : 100,
    "hashtags" : ["#data", "#science", "#datascience", "#awesome", "#yolo"]
}

Besides looking for specific keys we can look at all of them:

tweet_keys   = tweet.keys()     # list of keys
tweet_values = tweet.values()   # list of values
tweet_items  = tweet.items()    # list of (key, value) tuples

"user" in tweet_keys            # True, but uses a slow list in
"user" in tweet                 # more Pythonic, uses faster dict in
"joelgrus" in tweet_values      # True

Dictionary keys must be immutable; in particular, you cannot use lists as keys. If you need a multipart key, you should use a tuple or figure out a way to turn the key into a string.

defaultdict

Imagine that you’re trying to count the words in a document. An obvious approach is to create a dictionary in which the keys are words and the values are counts. As you check each word, you can increment its count if it’s already in the dictionary and add it to the dictionary if it’s not:

word_counts = {}
for word in document:
    if word in word_counts:
        word_counts[word] += 1
    else:
        word_counts[word] = 1

You could also use the “forgiveness is better than permission” approach and just handle the exception from trying to look up a missing key:

word_counts = {}
for word in document:
    try:
        word_counts[word] += 1
    except KeyError:
        word_counts[word] = 1

A third approach is to use get, which behaves gracefully for missing keys:

word_counts = {}
for word in document:
    previous_count = word_counts.get(word, 0)
    word_counts[word] = previous_count + 1

Every one of these is slightly unwieldy, which is why defaultdict is useful. A defaultdict is like a regular dictionary, except that when you try to look up a key it doesn’t contain, it first adds a value for it using a zero-argument function you provided when you created it. In order to use defaultdicts, you have to import them from collections:

from collections import defaultdict

word_counts = defaultdict(int)          # int() produces 0
for word in document:
    word_counts[word] += 1

They can also be useful with list or dict or even your own functions:

dd_list = defaultdict(list)             # list() produces an empty list
dd_list[2].append(1)                    # now dd_list contains {2: [1]}

dd_dict = defaultdict(dict)             # dict() produces an empty dict
dd_dict["Joel"]["City"] = "Seattle"     # { "Joel" : { "City" : Seattle"}}

dd_pair = defaultdict(lambda: [0, 0])
dd_pair[2][1] = 1                       # now dd_pair contains {2: [0,1]}

These will be useful when we’re using dictionaries to “collect” results by some key and don’t want to have to check every time to see if the key exists yet.

Truthiness

Booleans in Python work as in most other languages, except that they’re capitalized:

one_is_less_than_two = 1 < 2          # equals True
true_equals_false = True == False     # equals False

Python uses the value None to indicate a nonexistent value. It is similar to other languages’ null:

x = None
print x == None    # prints True, but is not Pythonic
print x is None    # prints True, and is Pythonic

Python lets you use any value where it expects a Boolean. The following are all “Falsy”:

  • False

  • None

  • [] (an empty list)

  • {} (an empty dict)

  • ""

  • set()

  • 0

  • 0.0

Pretty much anything else gets treated as True. This allows you to easily use if statements to test for empty lists or empty strings or empty dictionaries or so on. It also sometimes causes tricky bugs if you’re not expecting this behavior:

s = some_function_that_returns_a_string()
if s:
    first_char = s[0]
else:
    first_char = ""

A simpler way of doing the same is:

first_char = s and s[0]

since and returns its second value when the first is “truthy,” the first value when it’s not. Similarly, if x is either a number or possibly None:

safe_x = x or 0

is definitely a number.

Python has an all function, which takes a list and returns True precisely when every element is truthy, and an any function, which returns True when at least one element is truthy:

all([True, 1, { 3 }])   # True
all([True, 1, {}])      # False, {} is falsy
any([True, 1, {}])      # True, True is truthy
all([])                 # True, no falsy elements in the list
any([])                 # False, no truthy elements in the list

The Not-So-Basics

Here we’ll look at some more-advanced Python features that we’ll find useful for working with data.

Generators and Iterators

A problem with lists is that they can easily grow very big. range(1000000) creates an actual list of 1 million elements. If you only need to deal with them one at a time, this can be a huge source of inefficiency (or of running out of memory). If you potentially only need the first few values, then calculating them all is a waste.

A generator is something that you can iterate over (for us, usually using for) but whose values are produced only as needed (lazily).

One way to create generators is with functions and the yield operator:

def lazy_range(n):
    """a lazy version of range"""
    i = 0
    while i < n:
        yield i
        i += 1

The following loop will consume the yielded values one at a time until none are left:

for i in lazy_range(10):
    do_something_with(i)

(Python actually comes with a lazy_range function called xrange, and in Python 3, range itself is lazy.) This means you could even create an infinite sequence:

def natural_numbers():
    """returns 1, 2, 3, ..."""
    n = 1
    while True:
        yield n
        n += 1

although you probably shouldn’t iterate over it without using some kind of break logic.

The flip side of laziness is that you can only iterate through a generator once. If you need to iterate through something multiple times, you’ll need to either recreate the generator each time or use a list.

A second way to create generators is by using for comprehensions wrapped in parentheses:

lazy_evens_below_20 = (i for i in lazy_range(20) if i % 2 == 0)

Recall also that every dict has an items() method that returns a list of its key-value pairs. More frequently we’ll use the iteritems() method, which lazily yields the key-value pairs one at a time as we iterate over it.

Randomness

As we learn data science, we will frequently need to generate random numbers, which we can do with the random module:

import random

four_uniform_randoms = [random.random() for _ in range(4)]

#  [0.8444218515250481,      # random.random() produces numbers
#   0.7579544029403025,      # uniformly between 0 and 1
#   0.420571580830845,       # it's the random function we'll use
#   0.25891675029296335]     # most often

The random module actually produces pseudorandom (that is, deterministic) numbers based on an internal state that you can set with random.seed if you want to get reproducible results:

random.seed(10)         # set the seed to 10
print random.random()   # 0.57140259469
random.seed(10)         # reset the seed to 10
print random.random()   # 0.57140259469 again

We’ll sometimes use random.randrange, which takes either 1 or 2 arguments and returns an element chosen randomly from the corresponding range():

random.randrange(10)    # choose randomly from range(10) = [0, 1, ..., 9]
random.randrange(3, 6)  # choose randomly from range(3, 6) = [3, 4, 5]

There are a few more methods that we’ll sometimes find convenient. random.shuffle randomly reorders the elements of a list:

up_to_ten = range(10)
random.shuffle(up_to_ten)
print up_to_ten
# [2, 5, 1, 9, 7, 3, 8, 6, 4, 0]   (your results will probably be different)

If you need to randomly pick one element from a list you can use random.choice:

my_best_friend = random.choice(["Alice", "Bob", "Charlie"])     # "Bob" for me

And if you need to randomly choose a sample of elements without replacement (i.e., with no duplicates), you can use random.sample:

lottery_numbers = range(60)
winning_numbers = random.sample(lottery_numbers, 6)  # [16, 36, 10, 6, 25, 9]

To choose a sample of elements with replacement (i.e., allowing duplicates), you can just make multiple calls to random.choice:

four_with_replacement = [random.choice(range(10))
                         for _ in range(4)]
# [9, 4, 4, 2]

Object-Oriented Programming

Like many languages, Python allows you to define classes that encapsulate data and the functions that operate on them. We’ll use them sometimes to make our code cleaner and simpler. It’s probably simplest to explain them by constructing a heavily annotated example.

Imagine we didn’t have the built-in Python set. Then we might want to create our own Set class.

What behavior should our class have? Given an instance of Set, we’ll need to be able to add items to it, remove items from it, and check whether it contains a certain value. We’ll create all of these as member functions, which means we’ll access them with a dot after a Set object:

# by convention, we give classes PascalCase names
class Set:

    # these are the member functions
    # every one takes a first parameter "self" (another convention)
    # that refers to the particular Set object being used

    def __init__(self, values=None):
        """This is the constructor.
        It gets called when you create a new Set.
        You would use it like
        s1 = Set()          # empty set
        s2 = Set([1,2,2,3]) # initialize with values"""

        self.dict = {} # each instance of Set has its own dict property
                       # which is what we'll use to track memberships
        if values is not None:
            for value in values:
                self.add(value)

    def __repr__(self):
        """this is the string representation of a Set object
        if you type it at the Python prompt or pass it to str()"""
        return "Set: " + str(self.dict.keys())

    # we'll represent membership by being a key in self.dict with value True
    def add(self, value):
        self.dict[value] = True

    # value is in the Set if it's a key in the dictionary
    def contains(self, value):
        return value in self.dict

    def remove(self, value):
        del self.dict[value]

Which we could then use like:

s = Set([1,2,3])
s.add(4)
print s.contains(4)     # True
s.remove(3)
print s.contains(3)     # False

Functional Tools

When passing functions around, sometimes we’ll want to partially apply (or curry) functions to create new functions. As a simple example, imagine that we have a function of two variables:

def exp(base, power):
    return base ** power

and we want to use it to create a function of one variable two_to_the whose input is a power and whose output is the result of exp(2, power).

We can, of course, do this with def, but this can sometimes get unwieldy:

def two_to_the(power):
    return exp(2, power)

A different approach is to use functools.partial:

from functools import partial
two_to_the = partial(exp, 2)     # is now a function of one variable
print two_to_the(3)              # 8

You can also use partial to fill in later arguments if you specify their names:

square_of = partial(exp, power=2)
print square_of(3)                  # 9

It starts to get messy if you curry arguments in the middle of the function, so we’ll try to avoid doing that.

We will also occasionally use map, reduce, and filter, which provide functional alternatives to list comprehensions:

def double(x):
    return 2 * x

xs = [1, 2, 3, 4]
twice_xs = [double(x) for x in xs]        # [2, 4, 6, 8]
twice_xs = map(double, xs)                # same as above
list_doubler = partial(map, double)       # *function* that doubles a list
twice_xs = list_doubler(xs)               # again [2, 4, 6, 8]

You can use map with multiple-argument functions if you provide multiple lists:

def multiply(x, y): return x * y

products = map(multiply, [1, 2], [4, 5]) # [1 * 4, 2 * 5] = [4, 10]

Similarly, filter does the work of a list-comprehension if:

def is_even(x):
    """True if x is even, False if x is odd"""
    return x % 2 == 0

x_evens = [x for x in xs if is_even(x)]    # [2, 4]
x_evens = filter(is_even, xs)              # same as above
list_evener = partial(filter, is_even)     # *function* that filters a list
x_evens = list_evener(xs)                  # again [2, 4]

And reduce combines the first two elements of a list, then that result with the third, that result with the fourth, and so on, producing a single result:

x_product = reduce(multiply, xs)           # = 1 * 2 * 3 * 4 = 24
list_product = partial(reduce, multiply)   # *function* that reduces a list
x_product = list_product(xs)               # again = 24

args and kwargs

Let’s say we want to create a higher-order function that takes as input some function f and returns a new function that for any input returns twice the value of f:

def doubler(f):
    def g(x):
        return 2 * f(x)
    return g

This works in some cases:

def f1(x):
    return x + 1

g = doubler(f1)
print g(3)          # 8 (== ( 3 + 1) * 2)
print g(-1)         # 0 (== (-1 + 1) * 2)

However, it breaks down with functions that take more than a single argument:

def f2(x, y):
    return x + y

g = doubler(f2)
print g(1, 2)    # TypeError: g() takes exactly 1 argument (2 given)

What we need is a way to specify a function that takes arbitrary arguments. We can do this with argument unpacking and a little bit of magic:

def magic(*args, **kwargs):
    print "unnamed args:", args
    print "keyword args:", kwargs

magic(1, 2, key="word", key2="word2")

# prints
#  unnamed args: (1, 2)
#  keyword args: {'key2': 'word2', 'key': 'word'}

That is, when we define a function like this, args is a tuple of its unnamed arguments and kwargs is a dict of its named arguments. It works the other way too, if you want to use a list (or tuple) and dict to supply arguments to a function:

def other_way_magic(x, y, z):
    return x + y + z

x_y_list = [1, 2]
z_dict = { "z" : 3 }
print other_way_magic(*x_y_list, **z_dict)   # 6

You could do all sorts of strange tricks with this; we will only use it to produce higher-order functions whose inputs can accept arbitrary arguments:

def doubler_correct(f):
    """works no matter what kind of inputs f expects"""
    def g(*args, **kwargs):
        """whatever arguments g is supplied, pass them through to f"""
        return 2 * f(*args, **kwargs)
    return g

g = doubler_correct(f2)
print g(1, 2) # 6

Welcome to DataSciencester!

This concludes new-employee orientation. Oh, and also, try not to embezzle anything.

For Further Exploration