Skip to content

Implement Proc#with_refinements#9486

Open
shugo wants to merge 12 commits into
jruby:ruby-4.1from
shugo:proc-with-refinements-ruby-4.1
Open

Implement Proc#with_refinements#9486
shugo wants to merge 12 commits into
jruby:ruby-4.1from
shugo:proc-with-refinements-ruby-4.1

Conversation

@shugo

@shugo shugo commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Implement Proc#with_refinements

Summary

Implements Proc#with_refinements(mod1, ...) for Ruby 4.1 compatibility (CRuby Feature #22097).

prc.with_refinements(mod, ...) returns a new Proc whose body runs with the given modules' refinements active. The receiver is left unchanged, the captured closure environment (binding / dynamic scope) is shared, and only the refinement scope differs.

module StringRefinement
  refine String do
    def shout; upcase + "!"; end
  end
end

prc     = ->(s) { s.shout }
refined = prc.with_refinements(StringRefinement)

refined.call("hi")   # => "HI!"
prc.call("hi")       # => NoMethodError (original is unaffected)

Semantics

  • Receiver Proc is not modified; a new Proc is returned.
  • The closure environment is shared with the original (same binding).
  • dup / clone of the returned proc preserve the refinements.
  • Errors:
    • no arguments → ArgumentError
    • non-Module argument → TypeError
    • procs not backed by a Ruby block (Symbol#to_proc, native, Method#to_proc / bmethod) → ArgumentError

Implementation

Whether a call resolves refinements is baked into each call instruction at IR build time: CallInstr.create reads scope.maybeUsingRefinements() and, when true, builds a RefinedCachingCallSite capturing the static scope. Swapping an overlay module at runtime is therefore not enough — the receiver proc's IR must be re-cloned as refinement-aware. This mirrors CRuby's approach of recursively deep-copying the block's iseq, and is the block-level analog of the existing Refinement#import_methods.

  • RubyProc#with_refinements validates arguments, rejects non-block / fromMethod procs, requests a refinement-aware clone of the closure IR, and wraps it in a new Proc that reuses a clone of the original binding (so the environment stays shared).
  • IRClosure#cloneForRefinements deep-clones the closure tree via cloneForInlining with a new SimpleCloneInfo(..., refinementsClone=true) flag, giving each clone its own duplicated StaticScope (enclosing scope re-pointed so the overlay is reachable at method-resolution time), then activates the requested refinements on the clone's overlay module. Nested closures are deep-cloned, and every call-bearing instruction is rebuilt as a refined call site bound to the clone's scope. Fixnum/Float fast-path call instructions downgrade to the generic refined call path.
  • Caching: a CRuby-style single-slot memo on the source IRClosure (volatile RefinementsCache, lazily allocated). Repeated with_refinements calls with the same modules (compared by identity and order) reuse the cached clone. Cost to code that never uses the feature is a single null reference field and zero added time; the path is lock-free.
  • JIT: a refinement-aware clone is grafted under the source proc's already-built enclosing scope, so the recursive prepareForCompilation driven from the nearest top scope short-circuits before reaching the clone, leaving its full IR unbuilt and causing JIT promotion to bail. MixedModeIRBlockBody#promoteToFullBuild now builds the closure's own full IR directly when it is still missing, so refined clones can be JIT-compiled. Ordinary closures are already prepared by the enclosing scope and are unaffected.

Tests

test/jruby/test_proc_with_refinements.rb covers: refinement applies, original unaffected, shared environment, multiple modules, nested blocks, dup/clone preservation, proc (not just lambda), all error cases, and a subprocess JIT regression test (low threshold, synchronous compilation) asserting the refinement still holds after compilation and the clone is not rejected with "JIT failed".

No regressions in the MRI suites test_refinement (105), test_proc (103), or test_method (76).

shugo and others added 12 commits June 10, 2026 10:52
Add Proc#with_refinements(mod1, ...), which returns a new Proc whose body
runs with the given modules' refinements active. The receiver is unchanged
and the captured closure environment is shared; only the refinement scope
differs.

In JRuby whether a call resolves refinements is decided at IR build time:
CallInstr.create reads scope.maybeUsingRefinements() and, when true, builds a
RefinedCachingCallSite that captures the static scope. So a refinement-aware
copy of the receiver proc's IR is required (the analog of CRuby recursively
deep-copying the block iseq); merely swapping an overlay at runtime is not
enough.

Implementation:
- RubyProc#with_refinements validates args (>=1, all Modules), rejects
  non-IR-block / method-backed procs, then deep-copies the block's IRClosure
  and wraps it in a new Block that shares a clone of the original binding.
- IRClosure#cloneForRefinements clones the closure tree via cloneForInlining
  with a new refinementsClone flag and activates the requested refinements on
  the top clone's overlay module.
- CloneInfo/SimpleCloneInfo gain isRefinementsClone(); when set each clone gets
  its own duplicated StaticScope (enclosing re-pointed to the parent clone) and
  maybeUsingRefinements, WrappedIRClosure deep-clones nested closures, and every
  call-bearing instruction's clone() rebuilds as a refined call site bound to
  the clone's scope. Fixnum/Float fast-path calls downgrade to generic refined
  calls. New IRScope/IRClosure copy-constructor variants accept a StaticScope
  since IRScope.staticScope is final.

When isRefinementsClone() is false (all other clone paths) behavior is
unchanged. Adds test/jruby/test_proc_with_refinements.rb.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deep-copying the receiver's IRClosure is heavy, so calling with_refinements
repeatedly with the same modules recomputed the clone every time. Add a
CRuby-style single-slot memo on the source IRClosure so the common
"cache and reuse" pattern reuses the clone.

IRClosure gains a lazily-set `volatile RefinementsCache` field (null until the
feature is first used on that closure, so closures that never use it pay only
for one null reference and zero time) holding the most recent
(modules -> clone) pair. refinementsClone(context, modules) returns the cached
clone when the modules match by identity and order (not RubyModule.hashCode,
which can dispatch to a user-defined hash), otherwise recomputes via
cloneForRefinements and overwrites the slot. A differing modules set overwrites
the single slot, matching CRuby. It is lock-free: a race only causes a
redundant clone.

RubyProc#with_refinements now passes a RubyModule[] and calls refinementsClone.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clarify that the proc must wrap a Ruby block, rather than referring to
an internal "refinements-aware proc" concept.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A refinement-aware clone is grafted under the source proc's already-built
enclosing scope, so the recursive prepareForCompilation driven from the
nearest top scope short-circuits before reaching the clone, leaving its
full IR unbuilt. JIT promotion then bailed with "no full IR" and the clone
ran interpreted forever.

Build the closure's own full IR directly when it is still missing at JIT
promotion time. Ordinary closures are already prepared by the enclosing
scope, so this only affects grafted refinement clones.

Add a subprocess regression test that drives both the original and the
clone past a low JIT threshold and asserts the refinement still holds and
the clone is not rejected with "JIT failed".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port the CRuby Proc#with_refinements follow-ups that change behavior
(bugs.ruby-lang.org #22097, CRuby commits 53f0a20 and d0e896b):

- Chaining: applying with_refinements to a proc that already has
  refinements now raises ArgumentError instead of silently replacing the
  refinement set.  Combining sets would raise ordering/precedence
  questions; pass all modules in a single call instead.  Rejecting also
  keeps the option open to give chaining a meaning later.

- define_method: defining a method from a with_refinements proc now
  raises ArgumentError.  A method is invoked against its method entry,
  not the proc's refinement scope, so the refinements would not take
  effect (in JRuby they happened to apply, but CRuby rejects this and we
  match for cross-implementation consistency).

Both rest on a new IRClosure.isRefinementsClone() flag, set on the
clone produced by cloneForRefinements.  The chain check lives in
RubyProc#with_refinements; the define_method check in a new
RubyModule#checkNotRefinementsProc called from both define_method forms
and Kernel#define_singleton_method.

The CRuby Ractor restriction is not needed (JRuby has no Ractor and its
single-slot clone cache is already thread-safe), and the cref-hash-copy
and once-cache-reset fixes are already satisfied by JRuby's per-clone
overlay scope and fresh IR clone; those are covered by new regression
tests (test_instance_and_module_eval, test_once_regexp).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The top clone of a with_refinements tree is grafted under the source
proc's real (shared, long-lived) lexical parent.  The IRScope
constructor registered it in that parent's lexicalChildren, and
cloneForInlining added it to the parent's nestedClosures.  Both lists
are never pruned, so every clone produced by a cache-missing call (a
distinct module set, or alternating sets) was retained forever -- an
unbounded memory leak that also pinned the activated refinement modules.
The nestedClosures mutation is unsynchronized, so concurrent
with_refinements calls (or a concurrent JIT-time iteration of that list)
could also corrupt it or throw ConcurrentModificationException.

The grafted clone only needs its lexicalParent *field* (set by the
constructor and used by getNearestTopLocalVariableScope at build time),
not a back-reference from the parent: it is reached at run time through
its own Block/BlockBody, its interpreter context is allocated during
cloning, and its full IR is built directly by promoteToFullBuild.  So
skip the parent's nestedClosures registration for the grafted top clone
(nested clones, whose parent is a clone we own, are still registered so
the clone's own build can find them via getClosures), and detach it from
the parent's lexicalChildren via a new synchronized removeChildScope.

Leaving the clone out of the parent's getClosures() also stops it from
being spuriously reprocessed when the parent's own FullInterpreterContext
is built.

Verified: parent closure/child lists stay flat across 500 distinct-module
clones; 8 threads x 300 concurrent cache-missing calls raise no errors;
feature suite (17 tests) and MRI test_proc/test_refinement/test_method
(284 tests) pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-slot cache hands the same clone IRClosure to every
with_refinements caller, so the clone is shared across threads.
IRClosure.getBlockBody() lazily creates the BlockBody without
synchronization, so two threads reaching the shared clone on a cache hit
could each create a BlockBody and split the JIT-promotion state between
two instances.

Rather than synchronize getBlockBody() -- which is on a hot path (one
call per block instantiation, for every closure) and would burden code
that never uses with_refinements -- build the clone's BlockBody eagerly
in refinementsClone(), before the clone is published through the volatile
cache slot. A thread that later reads the shared clone then sees a fully
built body via the publish's happens-before and never races to create a
second one. with_refinements requests the body on the very next line
anyway, so this adds no work, and getBlockBody() stays lock- and
volatile-free.

Verified: 16 threads x 2000 concurrent getBlockBody() calls on one shared
clone observe a single BlockBody instance; feature suite (17) and MRI
test_proc/test_refinement/test_method (284) pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The initial implementation OR-ed `ii.isRefinementsClone()` into the
`potentiallyRefined` argument of ~14 call-bearing instruction clone()
methods so that a Proc#with_refinements clone would rebuild each call as
a refined call site.  That pattern was scattered and fragile: any
call-bearing instruction whose clone() forgot the OR (e.g. MatchInstr,
ArrayDerefInstr) would silently lose refinements, with no compile-time
error.

It was also redundant.  CallBase's constructor already derives refinement
from the call's scope:

    boolean effectivelyRefined =
        potentiallyRefined || (scope != null && scope.maybeUsingRefinements());

cloneForInlining sets setIsMaybeUsingRefinements() on the clone's scope
before its instructions are cloned, and every cloned call instruction is
built with that scope, so the scope check alone already marks them
refined -- which is exactly how MatchInstr/ArrayDerefInstr (which never
had the OR) already worked.

So drop the per-clone OR and let the single CallBase choke point handle
all instruction types uniformly, including any that were missed or are
added later.  The Fixnum/Float fast-path clones keep their explicit
downgrade to a generic refined call: that changes instruction selection
(the primitive path bypasses the call site), not just the call-site
flavor, so it cannot be centralized into CallBase.

Verified: a probe covering plain/operator/===/[]=/block/varargs/nested/
super refined calls is unchanged before and after; feature suite (17)
and MRI test_proc/test_refinement/test_method (284) pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two cloning copy constructors differed only in their super() call:
one passed a caller-supplied StaticScope, the other relied on the
IRScope(IRScope, IRScope) copy constructor to reuse the source's scope.
Since that two-arg IRScope constructor itself just delegated with
s.staticScope, the no-StaticScope IRClosure constructor can delegate to
the StaticScope variant passing c.getStaticScope(), dropping the
duplicated closureId/name/isEND/signature initialization. Behavior
(including the staticScope.setFile side effect in the IRScope copy
constructor) is unchanged.

The two-arg IRScope copy constructor's only caller was that super()
call, so remove it as well and fold its comment into the surviving
copy constructor's doc.

Verified: refine probe unchanged, feature suite (17) and MRI
test_proc/test_refinement/test_method (284, which exercise the inlining
clone path under JIT) pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nements

Running a proc that called with_refinements on an inner proc from inside an
outer with_refinements proc crashed with a Java NullPointerException. Two
independent bugs combined to produce it.

1. Spurious chain rejection. cloneForInlining marked every closure in a
   with_refinements clone tree (the top clone AND all nested closures it
   lexically encloses) as a refinements clone, because the flag was
   propagated through CloneInfo to nested clones. A nested proc only inherits
   the enclosing refinements lexically; it is not itself "a proc that already
   has refinements". Calling with_refinements (or define_method) on it was
   therefore wrongly rejected. Split the flag in two: inRefinementsCloneTree
   stays on every clone for the internal grafting/registration check, while
   refinementsClone (the user-facing chain/define_method guard) is now set
   only on the top clone the user receives.

2. Null frame name in clone backtraces. A block created in compiled top-level
   code has a null frame method name (only the interpreter sets the top-level
   "<main>"). A with_refinements clone always runs interpreted, so it pushed
   that null as the backtrace frame name and then crashed building the trace
   whenever an exception was raised through the clone. Fall back to
   Interpreter.ROOT when the cloned binding's method name is null, matching
   how the same block prints when run interpreted directly.

Output now matches CRuby. Adds regression tests for both cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the with_refinements coverage from CRuby's test/ruby/test_proc.rb:
yield/C-call invocation paths, single-slot cache correctness across module
switches and distinct environments, rescue/ensure, keyword/optional args,
case/when, flip-flop, source_location/parameters/arity, operator refinements,
lambda/proc preservation, module precedence, super, recursion and GC survival.

Two CRuby behaviors that JRuby does not yet implement are recorded as pending
tests so they auto-flag when fixed: a Symbol#to_proc (&:shout) evaluated inside
a refined clone does not inherit the clone's refinements, and a literal def
inside the block does not capture the clone's refinement cref.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two CRuby behaviors were missing from Proc#with_refinements clones:

- A literal &:sym block-pass is baked at build time into a SymbolProc operand
  that caches a *non-refined* symbol proc (the builder only bakes it when the
  scope is not using refinements). Cloning copied that operand verbatim, so the
  symbol proc ignored the clone's refinements. SymbolProc.cloneForInlining now
  rewrites it to a plain Symbol operand for refinements clones, so the (now
  refined) call site converts it via Symbol#toRefinedProc with the clone's
  refinement scope -- exactly as the builder would have done under `using`.

- A `def` lexically inside the block was emitted referencing the shared,
  non-refined IRMethod, so the defined method did not capture the clone's
  refinements (CRuby captures the block's cref). IRMethod now supports a
  refinements clone (own StaticScope reaching the enclosing clone's overlay,
  maybe-using-refinements set so call sites are refined and
  captureParentRefinements runs at def time, instructions re-created), and the
  DefineInstance/ClassMethodInstr clones deep-copy the method for refinements
  clones. The shared original method is untouched.

The two previously-pending tests now assert the CRuby behavior (singleton and
instance defs), plus that the originals stay unrefined. Full feature suite
36/0, MRI test_proc/test_refinement/test_method 284/0, and both new scenarios
survive JIT compilation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant