Generator Comprehensions / Using any() and all() in Python

Way back in 2006 (can you believe it’s been almost ten years?) Python 2.5 added two new awesome builtin language functions: any() and all(). Both of these take an iterable with an optional conditional using the standard list/generator syntax. The any() case returns true if any item in the iterable is true-ish and all() returns true if all items in the iterable are true-ish.

Here is a mistake I see over and over and over again (code example is obviously contrived here):

# creates a list and evaluates every user
if any([user.is_cool() for user in users if is_prime(user.id)]):
    unleash_awesome_machine()

This code is correct, in the sense that it works. But:

The correct way to write this code is only very slightly different:

# creates a list and does not evaluate every user
if any(user.is_cool() for user in users if is_prime(user.id)):
    unleash_awesome_machine()

See, the only thing I changed is I removed those two square brackets. That’s it.

The difference between the two is that the first uses the list comprehension syntax. This creates an actual Python list object which is passed to the any() function. Since this is a list comprehension every user in the users variable will be evaluated.

The “correct” way I’ve shown uses a generator comprehension instead. This creates a lazy generator object that evaluates from users one user a time. The function any() in that case can exit upon the first user detected that meets our criteria.

You can also create explicit generator objects in Python using the generator comprehension syntax. The syntax looks like this:

x = (user.is_cool() for user in users if is_prime(user.id))

This creates a generator object. You can’t call len() on it. Instead it has some weird methods like .next() defined. If you really want to know the details of how it works read the official Python wiki which explains things in plain English and also refers to the original PEP (which unfortunately does not seem to cover the generator comprehension syntax).

I’ll give an extremely easy to follow, simple example.

Consider the following python shell session:

>>> x = [1, 2, 3]
>>> type(x)
<type 'list'>
>>> len(x)
3
>>> x[1]
2

Its just a list. Easy as pie. Here’s the generator equivalent:

>>> x = (x for x in [1, 2, 3])
>>> type(x)
<type 'generator'>
>>> len(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'generator' has no len()
>>> x.next()
1
>>> x.next()
2
>>> x.next()
3
>>> x.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration>>> x = (x for x in [1, 2, 3])

Clearly something much weirder is going on. Generators are lazily evaluated. Now this case is dumb, because I’m lazily evaluating a non-lazy object (a list) to prove a point.

Here’s a more non-obvious and deleterious example (although it’s still contrived):

import re

CAPITAL_AT_START = re.compile('^[A-Z]')

# reads the ENTIRE FILE, creates a HUGE list object, and then has to GC it
def any_capitalized_1(filename):
    return any(line.match(CAPITAL_AT_START) for line in open(filename).readlines())


# lazily reads the file, creates less GC pressure
def any_capitalized_2(filename):
    return any(line.match(CAPITAL_AT_START) for line in open(filename))

This example is a little bit more subtle because both are actually using generator comprehensions, but the first is using a generator comprehension over an eagerly evaluated list object, whereas the second is using a generator comprehension over a lazily evaluated generator object.

Here’s my real take away. Any time you see any([...]) or all([...]) you should be very suspicious that the square brackets are necessary. You can almost always remove them and get code that runs faster, uses less memory, and saves a whole two bytes of disk space!