Skip to content

gh-115634: document ProcessPoolExecutor max_tasks_per_child bug#140897

Merged
gpshead merged 1 commit into
python:mainfrom
gpshead:max-tasks-per-child-docs-warning
Dec 30, 2025
Merged

gh-115634: document ProcessPoolExecutor max_tasks_per_child bug#140897
gpshead merged 1 commit into
python:mainfrom
gpshead:max-tasks-per-child-docs-warning

Conversation

@gpshead

@gpshead gpshead commented Nov 2, 2025

Copy link
Copy Markdown
Member

This stems from the user comment in #115634 (comment) - this bug has existed since the feature was introduced in 3.11 and is a footgun that causes occasional hard to debug problems when people do try to use the documented feature. We should acknowledge it in the docs.

It won't be fixed in older out of bug fix support Python versions and we still don't have it fixed in the latest version. We can update the docs to note the versions with the bug in whatever versions we do ever manage to fix it in when that happens.

I'll backport this doc update to 3.11 & 3.12 if the release managers want it, but will tweak the wording in those backports to make it clear it will not be fixed in those releases.


📚 Documentation preview 📚: https://cpython-previews--140897.org.readthedocs.build/

@gpshead gpshead merged commit 04899b8 into python:main Dec 30, 2025
42 of 43 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in Docs PRs Dec 30, 2025
@miss-islington-app

Copy link
Copy Markdown

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Dec 30, 2025
…pythonGH-140897)

(cherry picked from commit 04899b8)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
@bedevere-app

bedevere-app Bot commented Dec 30, 2025

Copy link
Copy Markdown

GH-143302 is a backport of this pull request to the 3.14 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Dec 30, 2025
…pythonGH-140897)

(cherry picked from commit 04899b8)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Dec 30, 2025
@bedevere-app

bedevere-app Bot commented Dec 30, 2025

Copy link
Copy Markdown

GH-143303 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Dec 30, 2025
gpshead added a commit that referenced this pull request Dec 30, 2025
GH-140897) (#143302)

gh-115634: document ProcessPoolExecutor max_tasks_per_child bug (GH-140897)
(cherry picked from commit 04899b8)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
gpshead added a commit that referenced this pull request Dec 30, 2025
GH-140897) (#143303)

gh-115634: document ProcessPoolExecutor max_tasks_per_child bug (GH-140897)
(cherry picked from commit 04899b8)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
gpshead added a commit to gpshead/cpython that referenced this pull request Jul 3, 2026
…child

The idle worker semaphore counts task completions, not idle workers, so
it can hold a stale token released by a worker that later exited upon
reaching its max_tasks_per_child limit. The worker replacement path
consumed such tokens and skipped spawning a replacement, deadlocking
the remaining queued tasks once no workers were left.

Replace dead workers based on len(self._processes) without consulting
the semaphore. The submit() path is unchanged, preserving on-demand
spawning and idle worker reuse.

Replace the documentation note added in pythonGH-140897 with a versionchanged
entry now that the bug is fixed.

Based on a fix proposed by Tabrez Mohammed.
gpshead added a commit that referenced this pull request Jul 3, 2026
…H-140900)

The idle worker semaphore counts task completions, not idle workers, so
it can hold a stale token released by a worker that later exited upon
reaching its max_tasks_per_child limit. The worker replacement path
consumed such tokens and skipped spawning a replacement, deadlocking
the remaining queued tasks once no workers were left.

Replace dead workers based on len(self._processes) without consulting
the semaphore. The submit() path is unchanged, preserving on-demand
spawning and idle worker reuse.

Replace the documentation note added in GH-140897 with a versionchanged
entry now that the bug is fixed.

Based on a fix proposed by Tabrez Mohammed.
gpshead added a commit that referenced this pull request Jul 3, 2026
…_child (GH-140900) (#152927)

gh-115634: Fix ProcessPoolExecutor deadlock with max_tasks_per_child (GH-140900)

The idle worker semaphore counts task completions, not idle workers, so
it can hold a stale token released by a worker that later exited upon
reaching its max_tasks_per_child limit. The worker replacement path
consumed such tokens and skipped spawning a replacement, deadlocking
the remaining queued tasks once no workers were left.

Replace dead workers based on len(self._processes) without consulting
the semaphore. The submit() path is unchanged, preserving on-demand
spawning and idle worker reuse.

Replace the documentation note added in GH-140897 with a versionchanged
entry now that the bug is fixed.

Based on a fix proposed by Tabrez Mohammed.
(cherry picked from commit b706767)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
gpshead added a commit that referenced this pull request Jul 3, 2026
…_child (GH-140900) (#152926)

gh-115634: Fix ProcessPoolExecutor deadlock with max_tasks_per_child (GH-140900)

The idle worker semaphore counts task completions, not idle workers, so
it can hold a stale token released by a worker that later exited upon
reaching its max_tasks_per_child limit. The worker replacement path
consumed such tokens and skipped spawning a replacement, deadlocking
the remaining queued tasks once no workers were left.

Replace dead workers based on len(self._processes) without consulting
the semaphore. The submit() path is unchanged, preserving on-demand
spawning and idle worker reuse.

Replace the documentation note added in GH-140897 with a versionchanged
entry now that the bug is fixed.

Based on a fix proposed by Tabrez Mohammed.
(cherry picked from commit b706767)

Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
gpshead added a commit that referenced this pull request Jul 3, 2026
…_child (GH-140900) (#152928)

The idle worker semaphore counts task completions, not idle workers, so
it can hold a stale token released by a worker that later exited upon
reaching its max_tasks_per_child limit. The worker replacement path
consumed such tokens and skipped spawning a replacement, deadlocking
the remaining queued tasks once no workers were left.

Replace dead workers based on len(self._processes) without consulting
the semaphore. The submit() path is unchanged, preserving on-demand
spawning and idle worker reuse.

Replace the documentation note added in GH-140897 with a versionchanged
entry now that the bug is fixed.

Based on a fix proposed by Tabrez Mohammed.

(cherry picked from commit b706767)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation in the Doc dir skip news

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants