…e pinning
This change resolves latency spikes, thread starvation, and CPU thrashing
encountered when virtual threads are enabled on GAE Standard (Java 21/25).
1. Eliminates Virtual Thread Pinning in Logging:
- Refactors com.google.appengine.setup.AppLogsWriter.
- Rewords flushAndWait() to be non-synchronized.
- Decouples the lock from the blocking gRPC logging flush future (.get() wait).
- Blocks on the future OUTSIDE of the synchronized block, allowing the Loom
runtime to safely unmount virtual threads from carrier threads during flushes.
2. Reliable Dynamic Carrier Parallelism Capping (gVisor Core Leak Mitigation):
- Refactors com.google.apphosting.runtime.JavaRuntimeMain.
- Configures the system property jdk.virtualThreadScheduler.parallelism
at the very beginning of the JVM bootstrap (main/load entry point).
- Architectural Root Cause: Under the GAE Standard gVisor sandboxed environment,
the guest OS does not automatically virtualize or cap cgroup CPU core visibility
to match GAE instance limits (F1/F2/F4). Instead, gVisor delegates physical
hardware resource visibility to the guest JVM, exposing the full underlying
host multiprocessor cores (often 64+ logical processors).
- When virtual threads are enabled under Project Loom, the JVM attempts to run
highly active CPU-bound ForkJoinPool carrier threads matching this host core count (64+).
Since GAE strictly throttles CPU quota externally on the host level (e.g. 0.5 CPU
for F1), this core mismatch triggers catastrophic OS scheduler thrashing
and CPU starvation inside the throttled sandbox container.
- Capping parallelism in JettyAdapter is too late as the default scheduler
might already be initialized. Bootstrapping it ensures the cap is respected
globally, protecting both the web container and user application business logic.
- Dynamically caps parallelism based on GAE_MEMORY_MB to match instance quotas:
* <= 512MB (F1 class, 0.5 CPU) -> 1 carrier thread
* <= 1024MB (F2 class, 1 CPU) -> 2 carrier threads
* > 1024MB (F4 class, 2+ CPU) -> 4 carrier threads
- Respects user explicit overrides configured via standard JVM arguments (e.g. JAVA_OPTS).
- Adds a detailed Javadoc comment to configureVirtualThreadParallelism() explaining
this gVisor sandbox core leakage mechanism and sizing guidelines.
3. Standard Jetty Virtual Threads Activation:
- Removes redundant late-bound parallelism property setting from
JettyServletEngineAdapter (both for Jetty 12 and 12.1).
- Retains standard delegation to VirtualThreads.getDefaultVirtualThreadsExecutor()
which is fully portable and compatible with upstream open-source Maven builds.
Impact on apps NOT using the virtual threads flag:
- Standard Multithreaded Apps: Positive impact. Removing lock-wait serialization
in AppLogsWriter reduces lock contention in the logging pipeline, improving
logging throughput under high concurrency.
- Hybrid Apps (using virtual threads in custom code only): Highly positive. The
early bootstrap cap automatically protects their business-logic virtual threads
from OS scheduler thrashing in throttled GAE containers, with zero developer
configuration required.
- Compile and Runtime Safety: 100% safe. The changes compile cleanly on JDK 17
Maven environments. On JDK 11/17 runtimes, virtual thread properties are safely
ignored, and Jetty bypasses virtual thread executors automatically.