bpo-30117: fix lib2to3 ParserIdempotency test#1242
Merged
Conversation
|
@appeltel, thanks for your PR! By analyzing the history of the files in this pull request, we identified @loewis, @benjaminp and @warsaw to be potential reviewers. |
Fix two spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test. 1. Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was intially utf-8-sig 2. If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using __future__ print_function
Force newline character to be used for linebreaks in generated file for lib2to3 parser idempotency tests. This prevents spurious diff failures in OS environments where other linebreak characters are used.
benjaminp
force-pushed
the
fix-parser-idem-test
branch
from
56b6674 to
cceff18
Compare
Contributor
|
Thanks @appeltel for the PR, and @benjaminp for merging it 🌮🎉.. I'm working now to backport this PR to: 3.6. |
Contributor
|
Sorry, @appeltel and @benjaminp, I could not cleanly backport this to |
benjaminp
pushed a commit
that referenced
this pull request
Jan 30, 2018
Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test.
Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.
If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function
For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.
For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change..
(cherry picked from commit 14e976e)
|
GH-5443 is a backport of this pull request to the 3.6 branch. |
benjaminp
added a commit
that referenced
this pull request
Jan 30, 2018
…H-5443) Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test. Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig. If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM. For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.. (cherry picked from commit 14e976e)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test.
Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.
If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function
For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.
For case (2), lib2to3 pre-detects the use of
from __future__ import print_functionor allows the user to force this interpretation with a-pflag, and then selects a different grammar with theprintstatement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.bpo-30117
https://bugs.python.org/issue30117