Implement Proc#with_refinements#9486
Open
shugo wants to merge 12 commits into
Open
Conversation
Add Proc#with_refinements(mod1, ...), which returns a new Proc whose body runs with the given modules' refinements active. The receiver is unchanged and the captured closure environment is shared; only the refinement scope differs. In JRuby whether a call resolves refinements is decided at IR build time: CallInstr.create reads scope.maybeUsingRefinements() and, when true, builds a RefinedCachingCallSite that captures the static scope. So a refinement-aware copy of the receiver proc's IR is required (the analog of CRuby recursively deep-copying the block iseq); merely swapping an overlay at runtime is not enough. Implementation: - RubyProc#with_refinements validates args (>=1, all Modules), rejects non-IR-block / method-backed procs, then deep-copies the block's IRClosure and wraps it in a new Block that shares a clone of the original binding. - IRClosure#cloneForRefinements clones the closure tree via cloneForInlining with a new refinementsClone flag and activates the requested refinements on the top clone's overlay module. - CloneInfo/SimpleCloneInfo gain isRefinementsClone(); when set each clone gets its own duplicated StaticScope (enclosing re-pointed to the parent clone) and maybeUsingRefinements, WrappedIRClosure deep-clones nested closures, and every call-bearing instruction's clone() rebuilds as a refined call site bound to the clone's scope. Fixnum/Float fast-path calls downgrade to generic refined calls. New IRScope/IRClosure copy-constructor variants accept a StaticScope since IRScope.staticScope is final. When isRefinementsClone() is false (all other clone paths) behavior is unchanged. Adds test/jruby/test_proc_with_refinements.rb. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deep-copying the receiver's IRClosure is heavy, so calling with_refinements repeatedly with the same modules recomputed the clone every time. Add a CRuby-style single-slot memo on the source IRClosure so the common "cache and reuse" pattern reuses the clone. IRClosure gains a lazily-set `volatile RefinementsCache` field (null until the feature is first used on that closure, so closures that never use it pay only for one null reference and zero time) holding the most recent (modules -> clone) pair. refinementsClone(context, modules) returns the cached clone when the modules match by identity and order (not RubyModule.hashCode, which can dispatch to a user-defined hash), otherwise recomputes via cloneForRefinements and overwrites the slot. A differing modules set overwrites the single slot, matching CRuby. It is lock-free: a race only causes a redundant clone. RubyProc#with_refinements now passes a RubyModule[] and calls refinementsClone. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clarify that the proc must wrap a Ruby block, rather than referring to an internal "refinements-aware proc" concept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A refinement-aware clone is grafted under the source proc's already-built enclosing scope, so the recursive prepareForCompilation driven from the nearest top scope short-circuits before reaching the clone, leaving its full IR unbuilt. JIT promotion then bailed with "no full IR" and the clone ran interpreted forever. Build the closure's own full IR directly when it is still missing at JIT promotion time. Ordinary closures are already prepared by the enclosing scope, so this only affects grafted refinement clones. Add a subprocess regression test that drives both the original and the clone past a low JIT threshold and asserts the refinement still holds and the clone is not rejected with "JIT failed". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port the CRuby Proc#with_refinements follow-ups that change behavior (bugs.ruby-lang.org #22097, CRuby commits 53f0a20 and d0e896b): - Chaining: applying with_refinements to a proc that already has refinements now raises ArgumentError instead of silently replacing the refinement set. Combining sets would raise ordering/precedence questions; pass all modules in a single call instead. Rejecting also keeps the option open to give chaining a meaning later. - define_method: defining a method from a with_refinements proc now raises ArgumentError. A method is invoked against its method entry, not the proc's refinement scope, so the refinements would not take effect (in JRuby they happened to apply, but CRuby rejects this and we match for cross-implementation consistency). Both rest on a new IRClosure.isRefinementsClone() flag, set on the clone produced by cloneForRefinements. The chain check lives in RubyProc#with_refinements; the define_method check in a new RubyModule#checkNotRefinementsProc called from both define_method forms and Kernel#define_singleton_method. The CRuby Ractor restriction is not needed (JRuby has no Ractor and its single-slot clone cache is already thread-safe), and the cref-hash-copy and once-cache-reset fixes are already satisfied by JRuby's per-clone overlay scope and fresh IR clone; those are covered by new regression tests (test_instance_and_module_eval, test_once_regexp). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The top clone of a with_refinements tree is grafted under the source proc's real (shared, long-lived) lexical parent. The IRScope constructor registered it in that parent's lexicalChildren, and cloneForInlining added it to the parent's nestedClosures. Both lists are never pruned, so every clone produced by a cache-missing call (a distinct module set, or alternating sets) was retained forever -- an unbounded memory leak that also pinned the activated refinement modules. The nestedClosures mutation is unsynchronized, so concurrent with_refinements calls (or a concurrent JIT-time iteration of that list) could also corrupt it or throw ConcurrentModificationException. The grafted clone only needs its lexicalParent *field* (set by the constructor and used by getNearestTopLocalVariableScope at build time), not a back-reference from the parent: it is reached at run time through its own Block/BlockBody, its interpreter context is allocated during cloning, and its full IR is built directly by promoteToFullBuild. So skip the parent's nestedClosures registration for the grafted top clone (nested clones, whose parent is a clone we own, are still registered so the clone's own build can find them via getClosures), and detach it from the parent's lexicalChildren via a new synchronized removeChildScope. Leaving the clone out of the parent's getClosures() also stops it from being spuriously reprocessed when the parent's own FullInterpreterContext is built. Verified: parent closure/child lists stay flat across 500 distinct-module clones; 8 threads x 300 concurrent cache-missing calls raise no errors; feature suite (17 tests) and MRI test_proc/test_refinement/test_method (284 tests) pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-slot cache hands the same clone IRClosure to every with_refinements caller, so the clone is shared across threads. IRClosure.getBlockBody() lazily creates the BlockBody without synchronization, so two threads reaching the shared clone on a cache hit could each create a BlockBody and split the JIT-promotion state between two instances. Rather than synchronize getBlockBody() -- which is on a hot path (one call per block instantiation, for every closure) and would burden code that never uses with_refinements -- build the clone's BlockBody eagerly in refinementsClone(), before the clone is published through the volatile cache slot. A thread that later reads the shared clone then sees a fully built body via the publish's happens-before and never races to create a second one. with_refinements requests the body on the very next line anyway, so this adds no work, and getBlockBody() stays lock- and volatile-free. Verified: 16 threads x 2000 concurrent getBlockBody() calls on one shared clone observe a single BlockBody instance; feature suite (17) and MRI test_proc/test_refinement/test_method (284) pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The initial implementation OR-ed `ii.isRefinementsClone()` into the
`potentiallyRefined` argument of ~14 call-bearing instruction clone()
methods so that a Proc#with_refinements clone would rebuild each call as
a refined call site. That pattern was scattered and fragile: any
call-bearing instruction whose clone() forgot the OR (e.g. MatchInstr,
ArrayDerefInstr) would silently lose refinements, with no compile-time
error.
It was also redundant. CallBase's constructor already derives refinement
from the call's scope:
boolean effectivelyRefined =
potentiallyRefined || (scope != null && scope.maybeUsingRefinements());
cloneForInlining sets setIsMaybeUsingRefinements() on the clone's scope
before its instructions are cloned, and every cloned call instruction is
built with that scope, so the scope check alone already marks them
refined -- which is exactly how MatchInstr/ArrayDerefInstr (which never
had the OR) already worked.
So drop the per-clone OR and let the single CallBase choke point handle
all instruction types uniformly, including any that were missed or are
added later. The Fixnum/Float fast-path clones keep their explicit
downgrade to a generic refined call: that changes instruction selection
(the primitive path bypasses the call site), not just the call-site
flavor, so it cannot be centralized into CallBase.
Verified: a probe covering plain/operator/===/[]=/block/varargs/nested/
super refined calls is unchanged before and after; feature suite (17)
and MRI test_proc/test_refinement/test_method (284) pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two cloning copy constructors differed only in their super() call: one passed a caller-supplied StaticScope, the other relied on the IRScope(IRScope, IRScope) copy constructor to reuse the source's scope. Since that two-arg IRScope constructor itself just delegated with s.staticScope, the no-StaticScope IRClosure constructor can delegate to the StaticScope variant passing c.getStaticScope(), dropping the duplicated closureId/name/isEND/signature initialization. Behavior (including the staticScope.setFile side effect in the IRScope copy constructor) is unchanged. The two-arg IRScope copy constructor's only caller was that super() call, so remove it as well and fold its comment into the surviving copy constructor's doc. Verified: refine probe unchanged, feature suite (17) and MRI test_proc/test_refinement/test_method (284, which exercise the inlining clone path under JIT) pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nements Running a proc that called with_refinements on an inner proc from inside an outer with_refinements proc crashed with a Java NullPointerException. Two independent bugs combined to produce it. 1. Spurious chain rejection. cloneForInlining marked every closure in a with_refinements clone tree (the top clone AND all nested closures it lexically encloses) as a refinements clone, because the flag was propagated through CloneInfo to nested clones. A nested proc only inherits the enclosing refinements lexically; it is not itself "a proc that already has refinements". Calling with_refinements (or define_method) on it was therefore wrongly rejected. Split the flag in two: inRefinementsCloneTree stays on every clone for the internal grafting/registration check, while refinementsClone (the user-facing chain/define_method guard) is now set only on the top clone the user receives. 2. Null frame name in clone backtraces. A block created in compiled top-level code has a null frame method name (only the interpreter sets the top-level "<main>"). A with_refinements clone always runs interpreted, so it pushed that null as the backtrace frame name and then crashed building the trace whenever an exception was raised through the clone. Fall back to Interpreter.ROOT when the cloned binding's method name is null, matching how the same block prints when run interpreted directly. Output now matches CRuby. Adds regression tests for both cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the with_refinements coverage from CRuby's test/ruby/test_proc.rb: yield/C-call invocation paths, single-slot cache correctness across module switches and distinct environments, rescue/ensure, keyword/optional args, case/when, flip-flop, source_location/parameters/arity, operator refinements, lambda/proc preservation, module precedence, super, recursion and GC survival. Two CRuby behaviors that JRuby does not yet implement are recorded as pending tests so they auto-flag when fixed: a Symbol#to_proc (&:shout) evaluated inside a refined clone does not inherit the clone's refinements, and a literal def inside the block does not capture the clone's refinement cref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two CRuby behaviors were missing from Proc#with_refinements clones: - A literal &:sym block-pass is baked at build time into a SymbolProc operand that caches a *non-refined* symbol proc (the builder only bakes it when the scope is not using refinements). Cloning copied that operand verbatim, so the symbol proc ignored the clone's refinements. SymbolProc.cloneForInlining now rewrites it to a plain Symbol operand for refinements clones, so the (now refined) call site converts it via Symbol#toRefinedProc with the clone's refinement scope -- exactly as the builder would have done under `using`. - A `def` lexically inside the block was emitted referencing the shared, non-refined IRMethod, so the defined method did not capture the clone's refinements (CRuby captures the block's cref). IRMethod now supports a refinements clone (own StaticScope reaching the enclosing clone's overlay, maybe-using-refinements set so call sites are refined and captureParentRefinements runs at def time, instructions re-created), and the DefineInstance/ClassMethodInstr clones deep-copy the method for refinements clones. The shared original method is untouched. The two previously-pending tests now assert the CRuby behavior (singleton and instance defs), plus that the originals stay unrefined. Full feature suite 36/0, MRI test_proc/test_refinement/test_method 284/0, and both new scenarios survive JIT compilation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement
Proc#with_refinementsSummary
Implements
Proc#with_refinements(mod1, ...)for Ruby 4.1 compatibility (CRuby Feature #22097).prc.with_refinements(mod, ...)returns a newProcwhose body runs with the given modules' refinements active. The receiver is left unchanged, the captured closure environment (binding / dynamic scope) is shared, and only the refinement scope differs.Semantics
Procis not modified; a newProcis returned.dup/cloneof the returned proc preserve the refinements.ArgumentErrorModuleargument →TypeErrorSymbol#to_proc, native,Method#to_proc/ bmethod) →ArgumentErrorImplementation
Whether a call resolves refinements is baked into each call instruction at IR build time:
CallInstr.createreadsscope.maybeUsingRefinements()and, when true, builds aRefinedCachingCallSitecapturing the static scope. Swapping an overlay module at runtime is therefore not enough — the receiver proc's IR must be re-cloned as refinement-aware. This mirrors CRuby's approach of recursively deep-copying the block's iseq, and is the block-level analog of the existingRefinement#import_methods.RubyProc#with_refinementsvalidates arguments, rejects non-block /fromMethodprocs, requests a refinement-aware clone of the closure IR, and wraps it in a newProcthat reuses a clone of the original binding (so the environment stays shared).IRClosure#cloneForRefinementsdeep-clones the closure tree viacloneForInliningwith a newSimpleCloneInfo(..., refinementsClone=true)flag, giving each clone its own duplicatedStaticScope(enclosing scope re-pointed so the overlay is reachable at method-resolution time), then activates the requested refinements on the clone's overlay module. Nested closures are deep-cloned, and every call-bearing instruction is rebuilt as a refined call site bound to the clone's scope. Fixnum/Float fast-path call instructions downgrade to the generic refined call path.IRClosure(volatile RefinementsCache, lazily allocated). Repeatedwith_refinementscalls with the same modules (compared by identity and order) reuse the cached clone. Cost to code that never uses the feature is a single null reference field and zero added time; the path is lock-free.prepareForCompilationdriven from the nearest top scope short-circuits before reaching the clone, leaving its full IR unbuilt and causing JIT promotion to bail.MixedModeIRBlockBody#promoteToFullBuildnow builds the closure's own full IR directly when it is still missing, so refined clones can be JIT-compiled. Ordinary closures are already prepared by the enclosing scope and are unaffected.Tests
test/jruby/test_proc_with_refinements.rbcovers: refinement applies, original unaffected, shared environment, multiple modules, nested blocks,dup/clonepreservation,proc(not just lambda), all error cases, and a subprocess JIT regression test (low threshold, synchronous compilation) asserting the refinement still holds after compilation and the clone is not rejected with "JIT failed".No regressions in the MRI suites
test_refinement(105),test_proc(103), ortest_method(76).