Using generator send() within a for loop

I discovered that my question would have had a one-line answer, using the extended "continue" statement proposed in the earlier version of PEP 342:

for n in bfs(g, start_node):
  continue process(n)

However, while PEP 342 was accepted, that particular feature was withdrawn after this June 2005 discussion between Raymond and Guido:

Raymond Hettinger said:

Let me go on record as a strong -1 for "continue EXPR". The for-loop is our most basic construct and is easily understood in its present form. The same can be said for "continue" and "break" which have the added advantage of a near zero learning curve for people migrating from other languages.

Any urge to complicate these basic statements should be seriously scrutinized and held to high standards of clarity, explainability, obviousness, usefulness, and necessity. IMO, it fails most of those tests.

I would not look forward to explaining "continue EXPR" in the tutorial and think it would stand out as an anti-feature.

[...] The correct argument against "continue EXPR" is that there are no use cases yet; if there were a good use case, the explanation would follow easily.

Guido

If python core developers have since changed their mind about the usefulness of extended "continue", perhaps this could be reintroduced into a future PEP. But, given a nearly identical use case as in this question was already discussed in the quoted thread, and wasn't found persuasive, it seems unlikely.

To simplify the client code, you could use an ordinary bsf() generator and check node.isleaf attribute in it:

 for node in bfs(g, start_node):
     node.isleaf = process(node) # don't follow if `process()` returns True

The disadvantage is that node is mutable. Or you have to pass a shared data structure that tracks leaf nodes: leaf[node] = process(node) where leaf dictionary is passed into bfs() earlier.

If you want to use .send() method explicitly; you have to handle StopIteration. See PEP 479 -- Change StopIteration handling inside generators. You could hide it in a helper function:

def traverse(tree_generator, visitor):
    try:
        node = next(tree_generator)
        while True:
             node = tree_generator.send(visitor(node))
    except StopIteration:
        pass

Example:

traverse(bfs(g, start_node), process)

I don't see this as a frequent use case, consider this as the original generator:

def original_gen():
    for x in range(10):
        should_break = yield x
        if should_break:
            break

If the value of should_break is always calculated based on some function call with x then why not just write the generator like this:

def processing_gen(check_f):
    for x in range(10):
        yield x
        should_break = check_f(x)
        if should_break:
            break

However I usually think of the code that processes the generated values as being written inside the loop (otherwise what is the point of having a loop at all?)

What it really seems you want to do is create a generator where calling the __next__ method really implies send(process(LAST_VALUE)) which can be implemented with a class:

class Followup_generator(): #feel free to use a better name
    def __init__(self,generator,following_function):
        self.gen = generator
        self.process_f = following_function
    def __iter__(self):
        return self
    def __next__(self):
        if hasattr(self,"last_value"):
            return self.send(self.process_f(self.last_value))
        else:
            self.last_value = next(self.gen)
            return self.last_value
    def send(self,arg):
        self.last_value = self.gen.send(arg)
        return self.last_value
    def __getattr__(self,attr):
        "forward other lookups to the generator (.throw etc.)"
        return getattr(self.gen, attr) 

# call signature is the exact same as @tobias_k's checking_generator
traversal = Followup_generator(bfs(g, start_node), process)
for n in traversal: 
    print(n)
    n = traversal.send(DATA) #you'd be able to send extra values to it

However this still doesn't see this as frequently used, I'd be perfectly fine with a while loop, although I'd put the .send call at the top:

traversal = bfs(g, start_node)
send_value = None
while True:
    n = traversal.send(send_value)
    #code for loop, ending in calculating the next send_value
    send_value = process(n)

And you might wrap that in a try: ... except StopIteration:pass although I find that simply waiting for an error to raise is better expressed with a context manager:

class Catch:
    def __init__(self,exc_type):
        if issubclass(exc_type,BaseException):
            self.catch_type = exc_type
        else:
            raise TypeError("can only catch Exceptions")
    def __enter__(self):
        return self
    def __exit__(self,exc_type,err, tb):
        if issubclass(exc_type, self.catch_type):
            self.err = err
            return True


with Catch(StopIteration):
    traversal = bfs(g, start_node)
    send_value = None
    while True:
        n = traversal.send(send_value)
        #code for loop, ending in calculating the next send_value
        send_value = process(n)

Probably this is the answer to the question from the thread's topic.

Take a look at the additional empty yields statements inside the traversal function and custom send function, that does the magical job.

# tested with Python 3.7

def traversal(n):
    for i in range(n):
        yield i, '%s[%s] %s' % (' ' * (4 - n), n, i)
        stop = yield
        if stop:
            yield  # here's the first part of the magic
        else:
            yield  # the same as above
            yield from traversal(int(n / 2))


def send(generator, value):
    next(generator)   # here's the second part of the magic
    generator.send(value)


g = traversal(4)

for i, (num, msg) in enumerate(g):
    print('>', i, msg)
    stop = num % 2 == 0
    send(g, stop)

I've written a small class SettableGenerator which uses a method to receive the value to be send and then forwards it to the actual generator when __next__ is invoked.

With this you can write:

gen = SettableGenerator(bfs(g, start_node))
for n in gen:
  gen.set(process(n))

Let's consider the following generator. It generates numbers from 0 to 9. For every generated number, it gets an input and stores it into ret:

def count_to_nine():
    # Output: numbers from 0 to 9
    # Input: converted numbers
    ret = []
    for i in range(10):
        # Yield a number, get something back
        val = (yield i)
        # Remember that "something"
        ret.append(val)
    return ret

You can, indeed, iterate it using next() + send(), but the best way is to iterate using send() alone:

g = count_to_nine()
value = None  # to make sure that the first send() gives a None
while True:
    value = g.send(value)  # send the previously generated value, get a new one
    value = f'#{value}'

Here's the result:

StopIteration: ['#0', '#1', '#2', '#3', '#4', '#5', '#6', '#7', '#8', '#9']

If you want that output, catch the StopIteration and get the result from it.

Cheers!