We Keep Coding

Connection Error - uplaod file to s3 using presigned URL

I am trying to call an API to receive a presigned url using a gamma endpoint and upload an excel file to S3 bucket. Seeing the following error:
Traceback (most recent call last):
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 1049, in _send_output
self.send(chunk)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 971, in send
self.sock.sendall(data)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\requests\adapters.py", line 440, in send
resp = conn.urlopen(
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\util\retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\packages\six.py", line 769, in reraise
raise value.with_traceback(tb)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 1049, in _send_output
self.send(chunk)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\http\client.py", line 971, in send
self.sock.sendall(data)
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Users\AppData\Local\Programs\PythonCodingPack\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:/Users/Desktop/Life Jacket/file/Generate_presignedURL.py", line 53, in
http_response = requests.post(response['url'], data=response['fields'], files=files,headers=headers)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\requests\api.py", line 117, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\requests\sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\requests\sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "C:\Users\AppData\Roaming\Python\Python38\site-packages\requests\adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
PS C:\Users\Desktop\Life Jacket\file>
import constants
import boto3
from botocore.exceptions import ClientError
import requests
import pandas
OBJECT_NAME_TO_UPLOAD = r"C:\Users\Desktop\Life Jacket\file\PART_Data_4202022.xlsx"
s3_client = boto3.client(
's3',
#config= config(signature_version = "s3v4"), #do we need to add this???
aws_access_key_id=constants.access_key,
aws_secret_access_key=constants.secret_access_key
)
#Generate the presigned URL
try :
response = s3_client.generate_presigned_post(
Bucket='http://1234567890.execute-api.us-east-1.amazonaws.com/gamma/abcde/getS3PreSignedUrl',
Key = OBJECT_NAME_TO_UPLOAD,
Fields=None,
Conditions=None,
ExpiresIn = 3600
)
except ClientError as e:
if e.response['Error']['Code'] == 'EntityAlreadyExists':
print("User already exists")
else:
print("Unexpected error: %s" % e)
print(response)
#Upload file to S3 using presigned URL
try :
files = {'input_file': open(OBJECT_NAME_TO_UPLOAD, "rb")}
headers={
"Accept-Encoding": "*",
"Connection": "keep-alive"
}
http_response = requests.post(response['url'], data=response['fields'], files=files,headers=headers)
except ClientError as e:
if e.http_response['Error']['Code'] == 'EntityAlreadyExists':
print("User already exists")
else:
print("Unexpected error: %s" % e)
print(http_response)
print(f'File upload status code: {http_response.status_code}')
print(http_response)

Azure Instance with linux machine docker timing out on requests

I have a python script in a docker that runs and collects data from an api on a loop.
`for idx, items in enumerate(SearchList):
#get access token
access_token = GetPricingAccTok()
#request id to get results
reqid = RequestIdCompSearch(items, access_token)
#api to get results by doing a while loop appending results together till api returns 100% as completed percentage
Result = GetReqResults(reqid, access_token)
Pricings = GetPricingDict(Result)
filenamepricing = "pricinghistory/CorrectFormat/Pricings_Data_{idn}_{ts}.json".format(
idn = idx,
ts= dt.datetime.now().isoformat()
)
GetAzContainer().upload_blob(
name=filenamepricing,
data=json.dumps(obj= Pricings, indent=4),
blob_type='BlockBlob'
)
`
When i run and test this code locally on my windows machine it works well without an issue and completes the requests
but in the docker on azure container instance it then returns an error showing that the connection has timed out. Not sure what the difference is between the running it locally and in the docker that i then recieve this error.
I have tried addinng this code at the start
import socket
from urllib3.connection import HTTPConnection
HTTPConnection.default_socket_options =(HTTPConnection.default_socket_options + [
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) #Enables the feature
,(socket.SOL_TCP, socket.TCP_KEEPIDLE, 45) #Overrides the time when the stack willl start sending KeppAlives after no data received on a Persistent Connection
,(socket.SOL_TCP, socket.TCP_KEEPINTVL, 10) #Defines how often thoe KA will be sent between them
,(socket.SOL_TCP, socket.TCP_KEEPCNT, 60) #How many attemps will your code try if the server goes down before droping the connection.
]
)
but it still times out. showing the below error
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 353, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='service.xxx.com', port=443): Read timed out. (read timeout=None)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "src/script.py", line 298, in <module>
reqid = RequestIdCompSearch(items, access_token)
File "src/script.py", line 111, in RequestIdCompSearch
InitResp =rq.post( ReqUrl, headers= ReqHead, data=Jsondata)
File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 117, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='service.xxxx.com', port=443): Read timed out. (read timeout=None)

SOLVED Python POST (how to "execute" an url?)

SOLVED
I will start off with I've never dealt with HTTP requests, and I've tried a few examples that I saw here on the website, but none of them worked for my case.
I want to "activate" scripts that are on a hardware device that I am trying to control. The device has its own HTTP server and by using various URLs I can change its variables, such url looks like this:
http://ipofdevice/ScriptName?varName=varValue&varName2=varValue2&varNameN=varValueN
I am trying to figure out how can I "activate" such URLs with python and I've gotten as far as, that I have to use post requests (I think?) and here is what I've tried so far:
I have already checked a few other posts and tried sample codes from there, but none of them worked, one such post was this:
How to send POST request?
Here is are two sample codes that I have tried, but did not work:
from urllib.parse import urlencode
from urllib.request import Request, urlopen
url = "http://127.0.0.1/ScriptName"
post_fields = { 'testVar' : 'hello', 'secondVar' : 'fromPython' }
request = Request(url, urlencode(post_fields).encode())
json = urlopen(request).read().decode()
And the json line threw an exception with a very long Traceback
Traceback (most recent call last):
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 289, in _read_status
status = int(status)
ValueError: invalid literal for int() with base 10: '400,'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\urllib\request.py", line 525, in open
response = self._open(req, data)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\urllib\request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\urllib\request.py", line 502, in _call_chain
result = func(*args)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\urllib\request.py", line 1348, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\urllib\request.py", line 1323, in do_open
r = h.getresponse()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1322, in getresponse
response.begin()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 293, in _read_status
raise BadStatusLine(line)
http.client.BadStatusLine: HTTP/1.0 400, Invalid Request
This is the code from my second attempt:
import requests
base_url = "http://127.0.0.1/ScriptName"
post_fields = { 'testVar' : 'hello', 'secondVar' : 'fromPython' }
response = requests.post(final_url, data=post_fields)
With an even much longer traceback, which I will not post simply because it adds too much clutter and I dont think its needed, since obviously I am doing something fundamentally wrong.
I dont need anything fancy, just to be able to run a link like the one in the beginning of the thread, I am not even sure if POST request is what I need.
In javascript I was able to do it like this (but javascript lacks other things that I need):
var wnd = window.open("http://127.0.0.1/ScriptName?var=val");
wnd.close();
e.preventDefault();
EDIT: Adding the traceback for the second python attempt:
Traceback (most recent call last):
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1322, in getresponse
response.begin()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 293, in _read_status
raise BadStatusLine(line)
http.client.BadStatusLine: HTTP/1.0 400, Invalid Request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1322, in getresponse
response.begin()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 293, in _read_status
raise BadStatusLine(line)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BadStatusLine('HTTP/1.0 400, Invalid Request\r\n'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "C:\Users\Darkbound\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('HTTP/1.0 400, Invalid Request\r\n'))
EDIT 2: Solved, turns out that I actually needed a GET request as #MrBean Bremen suggested , all I had to do was:
import requests
requests.get(myurl)

Try this
import requests
tesVar = hello
secondVar = fromPython
baseurl = "http://127.0.0.1/ScriptName?{'tesVar'}&{'secondVar'}"
response = requests.get(baseurl)
print(response)

PythonAnywhere requests session raising exception 'connection aborted'

I am trying to launch my script to PA and while it works locally it does not work on PA and I cannot figure out why. I have the paid version of PA so I can access this site that is not whitelisted. Here is my script:
with requests.Session() as session:
# initates the login session
if os.path.exists(document_report + '.pdf'):
# initates the login session
post = session.post(POST_LOGIN_URL, data=payload, headers = dict(referrer = 'https://www.example.com/loginpage'))
#This URL is the page you actually want to pull down with requests.
REQUEST_URL = f'https://www.example.com/{stuff}/{document}'
#gets the request URL
r = session.get(REQUEST_URL)
# saves to directory
with open('./digests/1.pdf', 'wb') as j:
j.write(r.content)
j.close()
else:
post = session.post(POST_LOGIN_URL, data=payload, headers = dict(referrer = 'https://www.example.com/loginpage'))
#This URL is the page you actually want to pull down with requests.
REQUEST_URL = f'https://www.example.com/{stuff}/{document}'
#gets the request URL
r = session.get(REQUEST_URL)
# saves to directory
with open(document_report + '.pdf', 'wb') as j:
j.write(r.content)
j.close()
When I run the code, I get the following message:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 275, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 275, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/legalaide/emailProcessorV2.py", line 567, in <module>
post = session.post(POST_LOGIN_URL, data=payload, headers = dict(referrer = 'https://www.example.com/loginpage'))
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I tried it again locally, and it still works. Any insight into a solution would be amazing!

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

I'm having this nasty error:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.5/site-packages/requests/adapters.py", line 445, in send
timeout=timeout
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/script.py", line 44, in <module>
response_service_k = service_k_lib.service_kData(data, access_token, apifolder)
File "/home/ubuntu/script_lib.py", line 238, in service_kData
datarALL.extend(data.result())
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
return self.__get_result()
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ubuntu/script_lib.py", line 129, in getdata3
responsedata = requests.get(url, data=data, headers=hed, verify=False)
File "/home/ubuntu/.local/lib/python3.5/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/home/ubuntu/.local/lib/python3.5/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/home/ubuntu/.local/lib/python3.5/site-packages/requests/sessions.py", line 512, in request
resp = self.send(prep, **send_kwargs)
File "/home/ubuntu/.local/lib/python3.5/site-packages/requests/sessions.py", line 622, in send
r = adapter.send(request, **kwargs)
File "/home/ubuntu/.local/lib/python3.5/site-packages/requests/adapters.py", line 495, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
According to this answer the solution for my issue is to place :
time.sleep(0.01) strategically in my code. I get the idea of it but I'm not sure I know where the correct spot for it in my code:
This is the relevant code part from script.py :
result= []
with ThreadPoolExecutor(max_workers=num_of_pages) as executor:
futh = [(executor.submit(self.getdata3, page, hed, data, apifolder,additional)) for page in pages]
for data in as_completed(futh):
result.extend(data.result())
print ("Finished generateing data.")
return result
This is the relevant code part from getdata3 function :
def getdata3(...)
datarALL = []
responsedata = requests.get(url, data=data, headers=hed, verify=False)
if responsedata.status_code == 200: # 200 for successful call
responsedata = responsedata.text
jsondata = json.loads(responsedata)
if "results" in jsondata:
if jsondata["results"]:
datarALL.extend(jsondata["results"])
print ("{1} page {0} finished".format(page,str(datetime.now())))
return datarALL
Some info:
My code generates pages.
Create thread per page executing getdata3 (which makes GET request to API)
each thread return the result of a single page.
"merging" results.
My question:
Where to put the time.sleep(0.01) in order to avoid this error?

you might already figured this out. Any way, I had a similar problem while making post requests in a for loop, which run 400 times, each loop run with two requests and a last request after exiting the loop. I found out while making API requests the server (check_mk) was not ready, still compiling some hots and service data. So I placed the time.sleep() just before the each request and that fixed the problem. I don't know how often you make the requests but I'd try it this way:
def getdata3(...)
datarALL = []
time.sleep(0.01)
responsedata = requests.get(url, data=data, headers=hed, verify=False)
if responsedata.status_code == 200: # 200 for successful call
responsedata = responsedata.text
jsondata = json.loads(responsedata)
if "results" in jsondata:
if jsondata["results"]:
datarALL.extend(jsondata["results"])
print ("{1} page {0} finished".format(page,str(datetime.now())))
return datarALL

As suggested in this thread by the same user posting the problem, it might be due to your internet connection. I had the same problem and changed my internet connection to another network and the issue was solved for me. Then you can give it a try changing your internet network.