Need help to correct my python function for fetching full data!

[PrateekG]

I am trying to fetch product data from an api using Python3.6.

By default this api returns 20 products and in a single request the api can return max 500 products if we use api's parameter Limit=500.

So for fetching all products we need to use one more parameter with Limit- Offset(Number of products to skip).

I have written following function to achieve this but in case of full data my function is not working well and it's giving me error like- Login failed, Signature mismatching.

def get_data(userid, api_key, action, pagination=True):
timeformat = datetime.datetime.now().replace(microsecond=0).isoformat() + '+08:00'
endpoint = 'https://example.com'
page_json = {}
# set required parameters for this api
parameters = {
  'UserID': userid,
  'Version': '1.0',
  'Action': action,
  'Format': 'JSON',
  'Timestamp': timeformat
}
if pagination:
    page = 0
    parameters['Limit'] = 500
    while True:
        parameters['Offset'] = 500 * page
        # set the required cryptographic signature
        concatenated = urllib.parse.urlencode(sorted(parameters.items()))
        parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest()
        page += 1
        try:
            response = requests.get(endpoint, params=parameters)
            page_json = response.json()
        except requests.exceptions.ConnectionError:
            print("Connection refused!")
            sleep(5)
else:
    try:
        concatenated = urllib.parse.urlencode(sorted(parameters.items()))
        # set the required cryptographic signature
        parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest()
        response = requests.get(endpoint, params=parameters)
        page_json = response.json()
    except requests.exceptions.ConnectionError:
        print("Connection refused!")
        sleep(5)

return page_json

It looks like I am not fitting my signature parameter line correctly in case of full data.

Can you please look into my function and help me to find out what I have written wrong and what it should be like?

[killerrex]

Hi,

The 2 things I see there are that when you put pagination = True you enter in an endless loop and that you overwrite the results of the previous query.

I have not way to try it, specially confirm that the json object returned is a list of results and there is not more wrappers... but try this:

def get_data(userid, api_key, action, limit=500):
timeformat = datetime.datetime.now().replace(microsecond=0).isoformat() + '+08:00'
endpoint = 'https://example.com'
page_json = {}
# set required parameters for this api
parameters = {
  'UserID': userid,
  'Version': '1.0',
  'Action': action,
  'Format': 'JSON',
  'Timestamp': timeformat
}
if limit > 0:
    parameters['Limit'] = limit
    page = 0
    while True:
        parameters['Offset'] = limit * page
        # set the required cryptographic signature
        concatenated = urllib.parse.urlencode(sorted(parameters.items()))
        parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest()
        try:
            response = requests.get(endpoint, params=parameters)
        except requests.exceptions.ConnectionError:
            print("Connection refused!")
            sleep(5)
            # As the page was not recovered, just try it again,
            # do not increase the page counter
            continue
        # Ok, one page recovered, append to the total result
        page += 1
        # Here I assume the answer is a dictionary of entries like:
        #  { 501: value, 502: value, ...}
        # you might need to look for the right element to accumulate.
        # i.e.: if response is something like:
        #   {'result': [value, value, value...]}
        # The following lines will not be valid
        delta = response.json()
        page_json.update(delta)
        # This might be the end, no more pages... but this criteria again depends
        # on the api you are using
        if len(delta) % limit == 0:
            # Here we enter only if the number of results are 0 or 500
            break
else:
    try:
        concatenated = urllib.parse.urlencode(sorted(parameters.items()))
        # set the required cryptographic signature
        parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest()
        response = requests.get(endpoint, params=parameters)
        page_json = response.json()
    except requests.exceptions.ConnectionError:
        print("Connection refused!")
        sleep(5)
 
return page_json

Notice that I changed your pagination flag with the limit number (so you can request 50 by 50, 100 by 100...) use limit=0 for the equivalent of your pagination=False.

[PrateekG]

Thanks killerrex!
I am able to find issue in my code.