Skip to content

bpo-33078 - FIX queue size on pickling error#6119

Merged
pitrou merged 4 commits into
python:masterfrom
tomMoral:FIX_queue_size
Mar 21, 2018
Merged

bpo-33078 - FIX queue size on pickling error#6119
pitrou merged 4 commits into
python:masterfrom
tomMoral:FIX_queue_size

Conversation

@tomMoral

@tomMoral tomMoral commented Mar 15, 2018 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Contributor

The Queue._feed does not properly handle the size of the Queue on errors. This can lead to a situation where the Queue is considered as Full when it is empty. Here is a reproducing script:

import multiprocessing as mp

q = mp.Queue(1)


class FailPickle():
    def __reduce__(self):
        raise ValueError()

q.put(FailPickle())
print("Queue is full:", q.full())
q.put(0)
print(f"Got result: {q.get()}")

This PR fixes this behavior.

https://bugs.python.org/issue33078

@tomMoral tomMoral requested a review from rhettinger as a code owner March 15, 2018 06:42
pitrou
pitrou requested changes Mar 20, 2018
View reviewed changes

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! A few comments below.

Comment thread Lib/multiprocessing/queues.py
info('error in queue thread: %s', e)
return
else:
queue_sem.release()

@pitrou pitrou Mar 20, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment explaining why you need to adjust the semaphore's value?

Comment thread Lib/test/_test_multiprocessing.py
q.put(NotSerializable())
q.put(True)
# bpo-30595: use a timeout of 1 second for slow buildbots
self.assertTrue(q.get(timeout=1.0))

@pitrou pitrou Mar 20, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should you add a check for q.qsize() as well?

Comment thread Misc/NEWS.d/next/Library/2018-03-15-07-38-00.bpo-33078.RmjUF5.rst Outdated
@@ -0,0 +1,2 @@
Fix the size handeling in multiprocessing.Queue when a pickling error

@pitrou pitrou Mar 20, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: "handling".

@bedevere-bot

bedevere-bot commented Mar 20, 2018

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

pitrou
pitrou reviewed Mar 21, 2018
View reviewed changes
Comment thread Lib/test/_test_multiprocessing.py
q = self.Queue(maxsize=1)
q.put(NotSerializable())
q.put(True)
self.assertEqual(q.qsize(), 1)

@pitrou pitrou Mar 21, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this check correct? Depending on when the worker thread triggers, it might return 2, no?

@tomMoral tomMoral Mar 21, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is correct because maxsize=1 so the second q.put should return only once the failure has been triggered.

@pitrou pitrou Mar 21, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point, thank you.

@pitrou pitrou merged commit e2f33ad into python:master Mar 21, 2018
@miss-islington

miss-islington commented Mar 21, 2018

Copy link
Copy Markdown
Contributor

Thanks @tomMoral for the PR, and @pitrou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

@bedevere-bot

bedevere-bot commented Mar 21, 2018

Copy link
Copy Markdown

GH-6178 is a backport of this pull request to the 3.7 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 21, 2018
@tomMoral @miss-islington
bpo-33078 - Fix queue size on pickling error (pythonGH-6119)
0e919f9 
(cherry picked from commit e2f33ad)

Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
pitrou pushed a commit that referenced this pull request Mar 21, 2018
@miss-islington @tomMoral @pitrou
bpo-33078 - Fix queue size on pickling error (GH-6119) (GH-6178)
bb5b529 
(cherry picked from commit e2f33ad)

Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
@tomMoral tomMoral mentioned this pull request Mar 21, 2018
Merged
jo2y pushed a commit to jo2y/cpython that referenced this pull request Mar 23, 2018
@tomMoral @jo2y
bpo-33078 - Fix queue size on pickling error (pythonGH-6119)
7249ba0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pitrou pitrou pitrou approved these changes

@rhettinger rhettinger Awaiting requested review from rhettinger

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tomMoral @bedevere-bot @miss-islington @pitrou @the-knights-who-say-ni