request.get to read large response

[pythonlearner1]

I am trying to use curl to read read large response.
I ran that from command line and redirected to text file and it is 30Mb file

using python I can put one record at a time but not everything that I want.
using following code which is simple.

    try:
        r = requests.get(url, auth=('username', 'password'), headers=headers, verify=False)
        r.raise_for_status()
    except requests.exceptions.RequestException as error:  
        print("Error:", error)
         exit()

    output = r.json()

[ibreeden]

But what is the output of your code? Does it show the error message or the json?

[pythonlearner1]

But what is the output of your code? Does it show the error message or the json?

if I pull one record it shows in json format.

if I do try to pull all the record it fails with error.
I can do same via curl command on Linux and redirect that to text file and I have 30mb file with all json data in it.

[buran]

Did you look at Streaming requests? Would it work in your case?
Or a package like json_stream?

[pythonlearner1]

Did you look at Streaming requests? Would it work in your case?
Or a package like json_stream?

Great, json_stream worked. now how do I encode it

with requests.get(url, auth=('username', 'password'),headers=headers, verify=False, stream=True) as response:
            data = json_stream.requests.load(response)
         print(type(data))

<class 'json_stream.base.TransientStreamingJSONObject'>

[ndc85430]

The point of streaming is that the server sends chunks of data at a time, as it's ready. So, you'll need to have read the whole thing before processing the data. The docs should tell you how to do that with this library.

[pythonlearner1]

I try two method describe in that document but didn't work that is why ask. searching around didn't find any example either.

here is what I try

# Option 1: supply json_stream.encoding.default as the default argument
print(json.dumps(data, default=default))

# Option 2: supply json_stream.encoding.JSONStreamEncoder as the cls argument
# This allows you to created your own subclass to further customise encoding
print(json.dumps(data, cls=JSONStreamEncoder))

both prints empty dictionary.

[pythonlearner1]

It is working with just request.get without json_stream. it was timing issue.

I have added timeout=200 and I got the whole result back.

Thank you everyone who help.