Parsing JSON

[pyStund]

Hi,

I need to take some JSON values that are floats from an AWS DynamoDB and make them variables to use in my script but I keep on running into problems.
The code that works to retrieve them is:

 
import boto3, json
from boto3.dynamodb.conditions import Key, Attr

# Declare the empty variables ready
x1 = 0.0
y1 = 0.0
x2 = 0.0
y2 = 0.0
x3 = 0.0
y3 = 0.0

TABLE_NAME = "xx_data"

# Dynamo connection
dynamodb_client = boto3.client('dynamodb', region_name="xxx")
dynamodb = boto3.resource('dynamodb', region_name="xxx")
table = dynamodb.Table(TABLE_NAME)

response = table.query(
  KeyConditionExpression=Key('xx').eq('xx')
)
print(response['Items'])

This outputs:

xxx@xxxx ~ % /usr/bin/python3 /Volumes/xxxx/PyFiles/DynamoData.py
[{'device_data': {'y1': '1.663024479783291', 'x1': '0.6768443826584553', 'y2': '1.9438666581951096', 'x2': '1.39031925550876', 'y3': '2.179626982558035', 'x3': '2.0613910485295945'}, 'xx': 'xx'}]

I've tried to use

data = json.loads(response)
data["x1"] = x1

also trying to print just one value leads to the same which makes me think I need to try a different approach

print(data ['device_data'][0]['x1'])

But both give me errors about how JSON needs to be a string.

Please can you help point me to resources that are useful for explaining how to parse the output so I can use them in my downstream code.

[Pedroski55]

json.loads() needs a string.
json.load() needs a .json file.

Must be something wrong with your string!

Some advice I got here I believe:

# get the modules
import json
from pathlib import Path

"""
When you load the .json file with data = json.load(f) you have a dictionary of dictionaries
Then you can do all the things Python dictionaries can do.
"""

mypath = '/home/pedro/temp/clients_data_json.json'

# check if the json you want exists
chk_file = Path(mypath) 
if chk_file.is_file():
    print(mypath, 'exists, so open it ... \n\n')
    with open(mypath) as f:
        data = json.load(f)
else:
    print("File does not exist, so make data an empty dictionary!\n\n")
    data = {}  

[pyStund]

Hi @Pedroski55
Thank you for your reply.

JSON I specified as the format when I set up the CLI so it pushes the data out as a string.

CLI setup advisory here says:
"json – The output is formatted as a JSON string"

In the boto3 manual it points to me needing to use this to isolate x1 from my previous JSON string as a printed output

['device_data'][0]['x1']

Given that it will be string and your advice I used LOAD without the S:

data = json.load(response)

Then tried to print it using that boto3 guidance

print(data ['device_data'][0]['x1'])

But it returns an error saying: AttributeError: 'dict' object has no attribute 'read'

Thanks again

[ibreeden]

Hi @pyStund , welcome to the forum. I see you studied the manuals of AWS and boto3 quite well. I am sorry to say I didn't, I know nothing about AWS and boto3. But perhaps I can help by analyzing what you posted. I am not sure at all the output is a json string.

print(response['Items'])

This outputs:

Output:
[{'device_data': {'y1': '1.663024479783291', 'x1': '0.6768443826584553', 'y2': '1.9438666581951096', 'x2': '1.39031925550876', 'y3': '2.179626982558035', 'x3': '2.0613910485295945'}, 'xx': 'xx'}] 

From this we can learn response is not a string. It must be of type dict (or a subclass of dict).
So this must result in an error:

data = json.loads(response)

Error:
Traceback (most recent call last):
  File "/home/ibreeden/PycharmProjects/Forum/kanweg.py", line 9, in <module>
    data = json.loads(response["Items"])
  File "/usr/lib/python3.9/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list

You did not post the complete error (please always do this in future) but I quess you got something like this.

It still is possible response['Items'] is a string. You could test this with:

print(type(response))
print(type(response["Items"]))

Output:
<class 'dict'>
<class 'list'>

My guess is: it is a list. In this case there is no need for json. You can easily access your data directly.

print(response["Items"][0]["device_data"]["x1"])

Output:
0.6768443826584553

In the other case, when response['Items'] is a string, then you do need to parse it as json.

print(type(response))
print(type(response["Items"]))

data = json.loads(response["Items"])
print(data[0]["device_data"]["x1"])

Error:
<class 'dict'>
<class 'str'>
0.6768443826584553

Please let us know if this helps you.

[pyStund]

Hi @ibreeden

Thank you, it worked. Using

x1 = float(response["Items"][0]["device_data"]["x1"])
y1 = float(response["Items"][0]["device_data"]["y1"])

It assigns them back to floats and I can use them in the downstream code.

Great!!