Requests file upload multiple times to a URL fails the second time

Question

I was trying to upload the same files to two different URLs, which weirdly enough was not working. The second POST request never got the file. So, I tried to implement a minimal example to show what's going on:

import requests
files = [('file',open(os.path.join(os.getcwd(),"textFile.txt")))]
op1 = requests.post("https://httpbin.org/post",files=files)
op2 = requests.post("https://httpbin.org/post",files=files)
print(op1.json())
print("================")
print(op2.json())

So, I'd expect both the results to have the file file, with the text information. But what I got is:

{'args': {}, 'data': '', 'files': {'file': "I'm Mr. TextSeeks look at meeee!"}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '186', 'Content-Type': 'multipart/form-data; boundary=196646c7550f74ac2d0a130f90350f1b', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-5f0c87c3-3355ec90b9973128fe25d6d0'}, 'json': None, 'origin': 'XX.XX.XX.XX', 'url': 'https://httpbin.org/post'}
================
{'args': {}, 'data': '', 'files': {'file': ''}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '148', 'Content-Type': 'multipart/form-data; boundary=b9e13ed8891641a6b10faaf5cded597b', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-5f0c87c4-38a66f895aee787c28ce8910'}, 'json': None, 'origin': 'XX.XX.XX.XX', 'url': 'https://httpbin.org/post'}

So, as you can see, the second POST request doesn't have the file information. Am I doing something wrong here or is it a bug?

---

Weirdly enough, this works:

import requests
files = [('file',open(os.path.join(os.getcwd(),"textFile.txt")))]
op1 = requests.post("https://httpbin.org/post",files=files)
files = [('file',open(os.path.join(os.getcwd(),"textFile.txt")))]
op2 = requests.post("https://httpbin.org/post",files=files)

print(op1.json())
print("================")
print(op2.json())

Output:

{'args': {}, 'data': '', 'files': {'file': "I'm Mr. TextSeeks look at meeee!"}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '186', 'Content-Type': 'multipart/form-data; boundary=870c451b9854b5d7265b0f0681f9b4bb', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-5f0c8ad9-c0469aaa6d8a4d0023885a4d'}, 'json': None, 'origin': 'XX.XX.XX.XX', 'url': 'https://httpbin.org/post'}
================
{'args': {}, 'data': '', 'files': {'file': "I'm Mr. TextSeeks look at meeee!"}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '186', 'Content-Type': 'multipart/form-data; boundary=aadfc00c1cb0a6f1019eb2080f2a8461', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-5f0c8ada-1776e20418dda96c39a1bc48'}, 'json': None, 'origin': 'XX.XX.XX.XX', 'url': 'https://httpbin.org/post'}

So it makes one wonder that the file might get closed once used by requests, but:

import requests
files = [('file',open(os.path.join(os.getcwd(),"textFile.txt")))]
op1 = requests.post("https://httpbin.org/post",files=files)
#files = [('file',open(os.path.join(os.getcwd(),"textFile.txt")))]
print(files[0][1].closed) #<=============
op2 = requests.post("https://httpbin.org/post",files=files)

print(op1.json())
print("================")
print(op2.json())

Gives:

False
{'args': {}, 'data': '', 'files': {'file': "I'm Mr. TextSeeks look at meeee!"}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '186', 'Content-Type': 'multipart/form-data; boundary=a93b1c26216d7d905a5b684b0c32fe51', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-5f0c8b69-23e6606819acb268e57edf78'}, 'json': None, 'origin': 'XX.XX.XX.XX', 'url': 'https://httpbin.org/post'}
================
{'args': {}, 'data': '', 'files': {'file': ''}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '148', 'Content-Type': 'multipart/form-data; boundary=222f5304294b85527ae355bbd714a4e3', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-5f0c8b6c-5ded97a4da068830f0b38840'}, 'json': None, 'origin': 'XX.XX.XX.XX', 'url': 'https://httpbin.org/post'}

So I definitely don't know what is going on here.

Answer 1

I know this question is quite old, but I ran into the same problem and fixed it. Here's the explanation of the problem, and a solution.

What happens in your example

In your code sample, when you write:

import requests
files = [('file', open("textFile.txt"))]
op1 = requests.post("https://httpbin.org/post", files=files)
op2 = requests.post("https://httpbin.org/post", files=files)

You're actually creating some kind of io.IOBase (specifically a io.TextIOWrapper in this case). The first post request will read that file, moving the stream position for your file to the end. The second post request uses the exact same instance (you don't open the file again). The file stream is now pointing at the end of the file, there's nothing to read. You then send an empty file.

Your example can really be simplified to "I call fileobj.read() twice, the second call shows an empty content".

fobj = open("textFile.txt")
print(fobj.read())  # "I'm Mr. TextSeeks look at meeee!"
print(fobj.read())  # ""

This is why your version opening the file again did not run into any problem.

Possible solution

One possible solution is to reset the position of the bytes stream by using the seek method:

fobj = open("textFile.txt")
print(fobj.read())  # "I'm Mr. TextSeeks look at meeee!"
fobj.seek(0)
print(fobj.read())  # "I'm Mr. TextSeeks look at meeee!"

Applied to the initial code sample:

import requests
files = [('file',open("textFile.txt"))]
op1 = requests.post("https://httpbin.org/post", files=files)
files[0][1].seek(0)
op2 = requests.post("https://httpbin.org/post", files=files)

print(op1.json())  # {..., 'files': {'file': "I'm Mr. TextSeeks look at meeee!"}, ...}
print(op2.json())  # {..., 'files': {'file': "I'm Mr. TextSeeks look at meeee!"}, ...}

As a summary: requests only reads the file-object given as arguments. It does not reset the byte stream. If you use the same file-object twice, you need to reset it yourself.