Asynchronous programming, a way to achieve a certain level of multi-tasking within a single thread. In Python 3.5 this is provided with the asyncio module and a new type of functions called coroutines.

To understand what are Python coroutines, we'll first try to understand what are:

  1. Coroutines from a general point of view
  2. Python iterators
  3. Python generators
  4. Python yield keyword
  5. Python generator send() method
  6. Python coroutines

I'm assuming we are all working with Python3.5+

Coroutines:

According to Wikipedia:

Coroutines are computer program components that generalize subroutines for nonpreemptive multitasking, by allowing multiple entry points for suspending and resuming execution at certain locations.

Simply said, outside of the Python language, coroutines are functions that can be paused during execution.

Iterator:

An iterable is an object capable of returning its members one at a time. An iterable can be used to feed a for loop. The built-in function iter takes an iterable object and returns an iterator. A call to the next method on the iterator gives us the next element if any, and raises a StopIteration exception otherwise:

>>> x = iter("iterable")
>>> next(x)
'i'
>>> next(x)
't'
>>> next(x)
'e'
>>> next(x)
'r'
>>> next(x)
'a'
>>> next(x)
'b'
>>> next(x)
'l'
>>> next(x)
'e'
>>> next(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>



Iterator objects are required to support two methods, the __iter__ method that returns the iterator object itself, used in for and in statements. And the __next__ method that returns the next value from the iterator. If there is no more items to return then the __next__ method should raise StopIteration exception:

class range_5:
    def __init__(self):
        self.i = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.i < 5:
            i = self.i
            self.i += 1
            return i
        else:
            raise StopIteration()
>>> y = range_5()
>>> y.next()
0
>>> y.next()
1
>>> y.next()
2
>>> y.next()
3
>>> y.next()
4
>>> y.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "test.py", line 15, in next
    raise StopIteration()
StopIteration
>>>

Python lists, tuples, dicts and sets are all examples of inbuilt iterators.

Generator:

If we browse the PEP 255, we can read somewhere:

A function that contains a yield statement is called a generator function. A generator function is an ordinary function object in all respects [...]

Anything that can be done with a generator can also be done with an iterator. However, generators are really nice to express certain ideas in a very clean and concise fashion. For example, this is a generator generating integers from zero to maxi excluded:

def range_m(maxi):
    index = 0
    while index < maxi:
        yield index
        index += 1


A generator function does not return any values. When we call it, we get a generator object:

>>> y = range_m(4)
>>> y
<generator object fib at 0x7fb93181c3b8>
>>>


Our generator object is indeed an iterator, as we can see, it implements the __iter__ and the __next__ methods:

>>> print(hasattr(range_m(4), '__iter__'))
True
>>> print(hasattr(range_m(4), '__next__'))
True


So yeswecan use a generator object to feed a for loop, as we did with an iterator.

What is the yield keyword?

We've just seen the yield keyword inside the definition of our range_m generator.

Again, somewhere inside the PEP 255, we can read:

If a yield statement is encountered, the state of the function is frozen, and the value of expression_list is returned to .next()'s caller. [...] the next time .next() is invoked, the function can proceed exactly as if the yield statement were just another external call. [...] One can think of yield as causing only a temporary interruption in the executions of a function.

Simply said, the yield instruction is put into a place where the generator returns an intermediate result to the caller and sleeps until the next invocation occurs. If the generator does not hit the yield statement anymore, it will raise a StopIteration exception.

Generators send() method

Another PEP, this time, the 342, introduced the send() method on generators.

However, if it were possible to pass values or exceptions *into* a generator at the point where it was suspended, a simple [...] scheduler [...] would let coroutines "call" each other without blocking .

This allowed one to pause a generator, but also to send a value back into the (already started) generator where it paused. Now, your generator can talk to you, but it can also listen yo you. Example:

def jump_range(up_to):
    index = 0
    while index < up_to:
        jump = yield index
        if jump is None:
            jump = 1
        index += jump
>>> y = jump_range(10)
>>> y
<generator object jump_range at 0x7f0ebda64570>
>>> next(y)
0
>>> next(y)
1
>>> y.send(3)
4
>>> next(y)
5
>>> y.send(-4)
1
>>> next(y)
2
>>> next(y)
3
>>> next(y)
4
>>> y.send(5)
9
>>> next(y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>


What is yield from?

The yield from syntax was added in the PEP 380. Being able to refactor a generator was the main reason:

[...] It should be possible to take a section of code containing one or more yield expressions, move it into a separate function [...], and call the new function using a yield from expression.

Simply said, yield from allow you to yield from an iterator. Consequence: you can now chain generators.

def yield_from_example0():
    # yielding from an iterable
    for c in "iterable":
        yield c

def yield_from_example1():
    yield from yield_from_example0()

def yield_from_example2():
    # yielding from a generator (which returns generator object, which is like an iterator)
    yield from yield_from_example1()
>>> y = yield_from_example2()
>>> next(y)
'i'
>>> next(y)
't'
>>> next(y)
'e'
>>> next(y)
'r'
>>> next(y)
'a'
>>> next(y)
'b'
>>> next(y)
'l'
>>> next(y)
'e'
>>> next(y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>


Where are we now:

  • yield, allowing a temporary interruption in the execution of a generator.
  • send, allowing us to send a value back, to resume into a paused generator.
  • yield from, allowing us to chain generators together.

So, we can say that coroutines can be implemented using generators. Indeed, you'll often heard the expression generator based coroutines.

Python coroutines:

PEP 0492

This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring.

Can't be more clear.

A Python coroutine, sometimes called native coroutine, is a function defined using async def. For example:

async def hello(name):
      return 'Hello ' + name


With generator based coroutines, yielding control was done with yield from keyword. With native coroutines a call to other coroutines is done using await keyword. For example:

async def main():
      s = await hello('Howdy')
      print(s)


Unlike a normal function, unlike a genrator based coroutine, a native coroutine can never run all on its own.

>>> hello("howdy")
<coroutine object hello at 0x7fd783742518>
>>>
>>> main()
__main__:1: RuntimeWarning: coroutine 'hello' was never awaited
<coroutine object main at 0x7fd783742570>
>>>

A coroutine always has to execute under the supervision of a manager (e.g., an event-loop, a kernel, etc.)

The event loop:

An event loop is a programming construct that can:

  • register coroutines to be executed
  • execute coroutines
  • delay or even cancel coroutines
  • handle all the events related to those coroutines operations

Whenever you want to actually execute a coroutine, you'll use an event loop.

>>> async def hello(name):
...       return 'Hello ' + name
...
>>> async def main():
...       s = await hello('Howdy')
...       print(s)
...
>>> import asyncio
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(main())
Hello Howdy
>>> loop.close()
>>>



According to the documentation:

asyncio provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives.

In a Future post, I'll try to explore some of those parts a little bit. But before that, I strongly recommend you the following links: