Follow-up to #104 (PR #107).

Background

flattenDeepBinds breaks up deeply-nested do/>>= chains that would otherwise exceed Lua 5.1's parser nesting cap (chunk has too many syntax levels). It recognises bind/discard steps and lambda-lifts the continuation chain into flat $kont helpers. #107 widened recognition to cover discard statement lines and monad-transformer binds, with goldens for Maybe, Either, State, Reader, Writer, Except, and a StateT/Except stack.

Recognition is name-anchored: it matches Control.Bind.bind and Control.Bind.discard by name, and a Bind dictionary by its bind and Apply0 fields. That is precise but incomplete. Two properties make a different design possible.

First, the transform does not depend on recognition for correctness. Lambda-lifting only relocates continuations into let-bound helpers and forwards live variables by name. The head and action are kept verbatim, and no bind/k call is reordered, dropped, or duplicated. Flattening an expression that is not a bind is therefore still semantics-preserving. Recognition decides which expressions get restructured, never whether the result is correct.

Second, NestingCheck is a backstop. Anything that is not flattened and still nests too deeply is rejected at compile time with a clear error, never emitted as unloadable Lua.

The limitation

Because recognition is name-anchored, a chain shape it does not explicitly know about silently fails to flatten and falls through to a NestingTooDeep compile error instead of working:

• Applicative chains: ado/apply, and bindFlipped/=<<.
• Any future dictionary encoding that does not match the bind + Apply0 literal-record shape.
• A new monad whose bind head reduces through some form the normaliser does not handle. This is how the State transformer case was originally missed (see Flatten deeply-nested non-Effect/ST do-blocks #107).

The State case showed the failure is graceful (a compile error, not a miscompile) but real: a category of valid do blocks fails to compile where it could work.

Proposal

Replace name-anchored recognition with a structure-anchored breaker: flatten any sufficiently deep right-nested continuation chain f a (\x -> ...) (last argument a lambda), whether or not f is a known bind. This relies on the two properties above. False positives are semantically safe, and NestingCheck catches whatever still overflows.

It subsumes the name-gated cases and removes the fragile surface: hardcoded names, dictionary-field assumptions, and reduction-shape assumptions. This is the general Lua-AST-level nesting breaker the #107 scope referred to as separate work.

Tradeoffs and open questions

Golden churn. Any existing chain deeper than the threshold that is a right-nested lambda nest but not a bind would start flattening, changing its golden. The threshold (currently 50) bounds this, and in practice nests deeper than 50 are almost always binds. Needs a golden reset and a review of what actually moves.

Codegen quality. For genuinely non-bind deep nests, helper extraction may be less tidy than the original. Acceptable when the alternative is a compile error, but worth eyeballing the output.

Placement. This may fit better as a Lua-AST-level pass (truly monad-agnostic, and able to catch non-bind constructs like deep case trees or long <> chains) than as an IR pass. The IR-vs-Lua-AST choice is open.

Upvalue budget. The maxLiveSet bail and the per-segment upvalue cost still apply. A structure-anchored pass fires on more shapes, so the bail behaviour should be re-checked.

Acceptance

• A deep right-nested non-bind continuation chain, plus the currently-unsupported ado/bindFlipped cases, compiles to Lua that loads and runs under stock Lua 5.1.
• No regression in the existing monad goldens.
• NestingCheck retained as the backstop.

Non-goals

• Reworking NestingCheck itself.
• Improving codegen quality of the flattened output beyond "loads and runs".