json.load causing IndexError: list index out of range

[WallyZ]

I'm trying to debug the ubuntu-mate-welcome python script which is refusing to load a valid json file.

I have validated the json file with jsonlint. However my experience with python is limited.

My environment is Ubuntu 18.04 aarch64 (arm64).

I have isolated the problem code in an example included here:

#! /usr/bin/python3
import sys
import json

print(sys.path)

json_path = ''

try:
    print('Reading index...')
    json_path = '/usr/share/ubuntu-mate-welcome/js/applications.json'
    with open(json_path) as data_file:
        print('Loading JSON.')
        index = json.load(data_file)
        print('LOADED JSON!!!!')
except Exception as e:
    index = None
    print('Software Index JSON is invalid or missing!')
    print(json_path)

The output is as follows:

Output:
['/home/odroid/Scripts', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']
Reading index...
Loading JSON.
Software Index JSON is invalid or missing!
/usr/share/ubuntu-mate-welcome/js/applications.json

However my example does not display the :

Error:
IndexError: list index out of range

So I am not certain I have faithfully reproduced the problem!

Output from jsonlint-php:

Output:
jsonlint-php /usr/share/ubuntu-mate-welcome/js/applications.json
Valid JSON (/usr/share/ubuntu-mate-welcome/js/applications.json)

I have looked in the python path but cannot identify the json module referred to in the import statement.

How can I be certain I have the json module installed?

[buran]

First step in this case is to remove the try/except. At the moment you catch all exceptions.
Then post full traceback you get as well as the json file itself.

How can I be certain I have the json module installed?

it's part of the python standard library, so you have it

[DeaD_EyE]

raise the exception again to see where the error occurs.
You could try https://github.com/codecobblers/dirtyjson

I downloaded the package and extracted the file: https://packages.ubuntu.com/xenial/all/u...te-welcome
I am able to load the json with Python 3.7.3

You can also upload your broken applications.json.

[WallyZ]

Ok after removing the try/except I now get a different picture:

Error:
Traceback (most recent call last):
  File "/home/odroid/Scripts/python_import_paths.py", line 14, in <module>
    index = json.load(data_file)
  File "/usr/lib/python3.6/json/__init__.py", line 296, in load
    return loads(fp.read(),
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 210343: ordinal not in range(128)

Does that mean there is a strange character in the json file?

---

Found some accented characters in the json file. The application Deja had accented e and a.

I changed them to normal a nad e and the json now loads. This is the standard applications.json for the ubuntu-mate-welcome package which loads OK on my Ubuntu 18.04 Mate Raspberry PI system.

So why does my code fail on one system and succeed on the other when both applications.json files are the same?

---

So is there some python setting I can set to make accented characters acceptable?

Why are accented characters acceptable on my Ubuntu 18.04 PI but unacceptable on my Ubuntu 18.04 Odroid?

[DeaD_EyE]

The file seems to be corrupt.

Usually json saves unicode chars with the prefix \u followed by a hex number.
If the character is encoded with utf8 (no \u notation), the normal json module is still able to load and decode it.
But if there for example quotes are missing, this can lead into this strange exception.

You could try the dirtyjson, load the string and dump it with the python-json module. Then
you can compare the original file together with the corrected fie (from json.dump or json.dumps).

[WallyZ]

Thanks for all the advise.

I did test the json file in both jsonlint and dirty json. Both tools stated the json was valid.

Here is snippet of the json file where the accented characters are found:

"deja-dup": {
"name": "Déjà Dup",
"img": "deja-dup",
"main-package": "deja-dup",
"launch-command": "deja-dup",

I just decide to replace the accented character in the end.

I thought it may have been a python 2 versus 3 issue but I tried both versions. The script doesn't appear to work at all in version 2 so I was suprised that the accented characters caused an issue in version 3.