[3.15] gh-115634: Fix ProcessPoolExecutor deadlock with max_tasks_per_child (GH-140900)#152926
Merged
Merged
Conversation
…child (pythonGH-140900) The idle worker semaphore counts task completions, not idle workers, so it can hold a stale token released by a worker that later exited upon reaching its max_tasks_per_child limit. The worker replacement path consumed such tokens and skipped spawning a replacement, deadlocking the remaining queued tasks once no workers were left. Replace dead workers based on len(self._processes) without consulting the semaphore. The submit() path is unchanged, preserving on-demand spawning and idle worker reuse. Replace the documentation note added in pythonGH-140897 with a versionchanged entry now that the bug is fixed. Based on a fix proposed by Tabrez Mohammed. (cherry picked from commit b706767d8fd7d21afc3f156fb9c173bc99855e0e) Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
Documentation build overview
295 files changed ·
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The idle worker semaphore counts task completions, not idle workers, so
it can hold a stale token released by a worker that later exited upon
reaching its max_tasks_per_child limit. The worker replacement path
consumed such tokens and skipped spawning a replacement, deadlocking
the remaining queued tasks once no workers were left.
Replace dead workers based on len(self._processes) without consulting
the semaphore. The submit() path is unchanged, preserving on-demand
spawning and idle worker reuse.
Replace the documentation note added in GH-140897 with a versionchanged
entry now that the bug is fixed.
Based on a fix proposed by Tabrez Mohammed.
(cherry picked from commit b706767)
Co-authored-by: Gregory P. Smith 68491+gpshead@users.noreply.github.com