> Is it worth keeping generate_tokens as an alias for tokenize, just
> to avoid gratuitous 2-to-3 breakage?  Maybe not---I guess they're
> different beasts, in that one wants a string-valued iterator and the 
> other wants a bytes-valued iterator.

Exactly so - that was the primary rationale for renaming it. It
shouldn't "silently" return something else, but there should be
an explicit clue that you need to port actively.