requests - ChunkedEncodingError with requests.get, but not when using Postman

Question

I have a URL that can only be accessed from within our VPC.

If I enter the URL via Postman, I am able to see the content (which is a Pdf) and I'm able to save the output to a file perfectly fine.

However, when trying to automate this using python requests with

import requests
r = requests.get(url, params=params)

I get an exception

ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

The other Stackoverflow questions haven't really helped much with this and this is consistently reproducible with requests

Answer 1

I think the reason for this error is server is not correctly encoding the chunk. some of the chunk sizes are not integer b'' due to which you are getting chunk encoding error.

Try below code

import request
response = requests.get(url, params=params, stream=True)
try:
    for data in response.iter_content(chunk_size=1024)
        print(data)
except ChunkEncodingError as ex:
    print(f"Invalid chunk encoding {str(ex)}")