Changing client.get() method type based on size of data...

[dl0dth]

How do I convert the following code to use the offset limit items and get a large item instead of doing a get_all type operation? The code I want to convert is below. There seems to be a way of paginating the download of the data but then creating the DataFrame at the end and just using that instead of going through the records thing you see below:

            client = Socrata(nmdx_url, token, username, password, timeout=90)

            item_count += 1

            # Process metadata
            ds_md = client.get_metadata(ds)
            print('=[ ' + str(item_count) + ' ]==================================================================================================================')
            print(f"ID: {ds_md['id']}")
            print(f"NAME: {ds_md['name']}")
            print(f"CATEGORY: {ds_md['category']}")
            print(f"DESCRIPTION: {ds_md['description']}")
            #print(f"rowsUpdatedAt: {ds_md['rowsUpdatedAt']} ({datetime.fromtimestamp(ds_md['rowsUpdatedAt'])})")

            # Fetch records from the table
            ds_md = client.get_metadata(ds)
            ds_cols = [colname['fieldName'] for colname in ds_md['columns']]
            ds_records = client.get(ds, where='', limit=10)
            #ds_records = client.get_all(ds, where='')
            #ds_records = client.get_all(ds)
            ds_df = pd.DataFrame.from_records(ds_records, columns=ds_cols)

How do you do this without the ds_records thing here and I resolve cols?

Something like:

            chunk = client.get(limit=chunk_size, offset=offset)

            if not chunk:  # Check for empty response
                break

            all_data.extend(chunk)
            offset += chunk_size

Again, I'm trying to do the client.get() where the get happens many times with limited data to keep the network traffic reasonable. How is this done?

Thanks

Yoriz write Jan-02-2025, 09:21 PM:
Please post all code, output and errors (in its entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

[dl0dth]

Basically, when I do the get with the "get all" type, the system crashes from time to time. With smaller downloads it doesn't seem to do that!