Stack trace shows different exception type than print

[micseydel]

Python's JSON module supports deserializing Javascript infinities (although the official JSON spec doesn't). Specifically, it supports "Infinity" and "-Infinity" in particular. I need to be able to deserialize "+Infinity" though. Here is my attempt at achieving this

import json

class PositiveInfinityJSONDecoder(json.JSONDecoder):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.original_scan_once = self.scan_once
        self.scan_once = self._scan_once

    def _scan_once(self, string, idx):
        print("Custom _scan_once entered; trying default first...")
        try:
            return self.original_scan_once(string, idx)
        except json.decoder.JSONDecodeError as e:
            print("Default failed, trying custom logic...")
            try:
                nextchar = string[idx]
            except IndexError:
                raise StopIteration(idx) from e
            if nextchar == '+' and string[idx:idx + 9] == '+Infinity':
                return self.parse_constant('Infinity'), idx + 9
            raise e
        except Exception as e:
            print(f"Got an unexpected exception from default (type: {type(e)}) - {e}")
            raise e

print(json.loads('+Infinity', cls=PositiveInfinityJSONDecoder))

The result of this is that (apparently) a StopIteration exception is thrown, although its traceback looks like a JSONDecodeError

Output:
$ python3 confusion.py 
Custom _scan_once entered; trying default first...
Got an unexpected exception from default (type: <class 'StopIteration'>) - 0
Traceback (most recent call last):
  File "confusion.py", line 28, in <module>
    print(json.loads('+Infinity', cls=PositiveInfinityJSONDecoder))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm baffled because the stack trace clearly shows a JSONDecodeError being raised, but at runtime when I try to catch it, it appears to have a different type. In case the class is doing something funky, I also tried catching a ValueError (its superclass) but that didn't work either.

What does work is catching a StopIteration exception, but that doesn't seem right. I'm afraid that there's something weird going on here and if I catch the wrong exception, there will be some surprise down the line. Does anyone know what's going on here?

[sheeba_weeba]

Henlo Fren,

My naem Sheba and I luvs da slithery codes please.

The JSONDecodeError in your codes is from JSONDecoder.raw_decode which calls self.scan_once.

Here is a good codes please:

import json

class PositiveInfinityJSONDecoder(json.JSONDecoder):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.original_scan_once = self.scan_once
        self.scan_once = self._scan_once

    def _scan_once(self, string, idx):
        print("Custom _scan_once entered; trying default first...")
        try:
            return self.original_scan_once(string, idx)
        except StopIteration as e:  # scan_once should throw a StopIteration
            print("Default failed, trying custom logic...")
            try:
                nextchar = string[idx]
            except IndexError:
                raise StopIteration(idx) from e
            if nextchar == '+' and string[idx:idx + 9] == '+Infinity':
                return self.parse_constant('Infinity'), idx + 9
            else:
                raise StopIteration(idx)
        except Exception as e:
            print(f"Got an unexpected exception from default (type: {type(e)}) - {e}")
            raise e

print(json.loads('Infinity', cls=PositiveInfinityJSONDecoder))
print()
print(json.loads('+Infinity', cls=PositiveInfinityJSONDecoder))
print()
print(json.loads('+BadData', cls=PositiveInfinityJSONDecoder))
print()

Output:
$ python3 confusion.py 
Custom _scan_once entered; trying default first...
inf

Custom _scan_once entered; trying default first...
Default failed, trying custom logic...
inf

Custom _scan_once entered; trying default first...
Default failed, trying custom logic...
Traceback (most recent call last):
  File "confusion.py", line 31, in <module>
    print(json.loads('+BadData', cls=PositiveInfinityJSONDecoder))
  File "/usr/lib/python3.6/json/__init__.py", line 367, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Sincerely,
Sheba Weeba Possum Woo

[micseydel]

Thanks for the reply but I feel like I must be missing something. I already mentioned that a StopIteration error appears to work, but that's the problem. As you can see in the link you provided, raw_decode should be throwing a JSONDecodeError, not a StopIteration error, and you can see that when I print a stacktrace the type of the error appears different than if I print the type of the error object.

[sheeba_weeba]

There are two exceptions being thrown:

1. json.loads('+Infinity') is called.
   
2. ...
   
3. raw_decode('+Infinity') is called.
   
4. raw_decode calls PositiveInfinityJSONDecoder._scan_once.
   
5. PositiveInfinityJSONDecoder._scan_once calls self.original_scan_once (a.k.a. JSONDecoder().scan_once).
   
6. self.original_scan_once raises a StopIteration as expected.
   
7. PositiveInfinityJSONDecoder._scan_once prints the StopIteration exception and reraises it.
   
8. raw_decode excepts StopIteration and raises JSONDecodeError.
   
9. JSONDecodeError is not caught, so it prints the stack trace for JSONDecodeError.
   

[scidam]

There are two version of make_scanner in json module, c_make_scanner and py_make_scanner. We need to force Python to use Python version (py_make_scanner).
The following isn't to be an elegant solution, but it straightforward and clear:

import json
from json.decoder import PosInf, _CONSTANTS
from json.scanner import NUMBER_RE

_CONSTANTS.update({'+Infinity': PosInf})  # Update known constants 

def py_make_scanner(context):  # there are c-version, we need to force python to use version only
    parse_object = context.parse_object
    parse_array = context.parse_array
    parse_string = context.parse_string
    match_number = NUMBER_RE.match
    strict = context.strict
    parse_float = context.parse_float
    parse_int = context.parse_int
    parse_constant = context.parse_constant
    object_hook = context.object_hook
    object_pairs_hook = context.object_pairs_hook
    memo = context.memo

    def _scan_once(string, idx):
        try:
            nextchar = string[idx]
        except IndexError:
            raise StopIteration(idx) from None

        if nextchar == '"':
            return parse_string(string, idx + 1, strict)
        elif nextchar == '{':
            return parse_object((string, idx + 1), strict,
                _scan_once, object_hook, object_pairs_hook, memo)
        elif nextchar == '[':
            return parse_array((string, idx + 1), _scan_once)
        elif nextchar == 'n' and string[idx:idx + 4] == 'null':
            return None, idx + 4
        elif nextchar == 't' and string[idx:idx + 4] == 'true':
            return True, idx + 4
        elif nextchar == 'f' and string[idx:idx + 5] == 'false':
            return False, idx + 5

        m = match_number(string, idx)
        if m is not None:
            integer, frac, exp = m.groups()
            if frac or exp:
                res = parse_float(integer + (frac or '') + (exp or ''))
            else:
                res = parse_int(integer)
            return res, m.end()
        elif nextchar == 'N' and string[idx:idx + 3] == 'NaN':
            return parse_constant('NaN'), idx + 3
        elif nextchar == 'I' and string[idx:idx + 8] == 'Infinity':
            return parse_constant('Infinity'), idx + 8
        elif nextchar == '-' and string[idx:idx + 9] == '-Infinity':
            return parse_constant('-Infinity'), idx + 9
        elif nextchar == '+' and string[idx:idx + 9] == '+Infinity':  #  These lines were added;
            return parse_constant('+Infinity'), idx + 9
        else:
            raise StopIteration(idx)

    def scan_once(string, idx):
        try:
            return _scan_once(string, idx)
        finally:
            memo.clear()

    return scan_once

 
class PositiveInfinityJSONDecoder(json.JSONDecoder):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.scan_once = py_make_scanner(self)  # Force to use python (not c) version of py_make_scnner
 

print(json.loads('+Infinity', cls=PositiveInfinityJSONDecoder))

[micseydel]

Thanks scidamn! I had meant to loop back around to that and forgot. There's one small improvement to what you had that I will be incorporating - if line 55 just uses "Infinity" instead of "+Infinity" then you can drop the _CONSTANTS update.