Python HTTP GET raise Exception after 300 seconds from Docker container - python

I'm trying to GET data from an external service using Python from a Docker container.
I need to reach two different endpoints, let's say:
endpointA = "https://myhost:1234/resA" # Requests take about 1 minute to complete
endpointB = "https://myhost:1234/resB" # Requests take about 8 minutes to complete
I tried to use two different libraries, Requests:
import requests
headers = {'Authorization': 'Basic XXXXXXXXXXXXXXXXXXXXXX', 'Content-Type': 'application/json'}
response = requests.get(endpointA, verify=False, headers=headers)
and Pycurl:
import pycurl
headers = ['Authorization: Basic XXXXXXXXXXXXXXXXXXXXXX', 'Content-Type: application/json']
buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.VERBOSE, True)
c.setopt(c.URL, endpointA)
c.setopt(c.WRITEDATA, buffer)
c.setopt(c.SSL_VERIFYPEER, False)
c.setopt(c.SSL_VERIFYHOST, False)
c.setopt(c.HTTPHEADER, headers)
c.perform()
c.close()
My code runs without problems from local (both endpointA and endpointB return).
When run inside a Docker container, requests to endpointA work but request to endpointB raise exceptions after 300 seconds:
in the case of requests:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
in the case of pycurl:
Traceback (most recent call last):
File "test.py", line 230, in get_data
c.perform()
pycurl.error: (52, 'Empty reply from server')
This only happens from Docker, as from my console everything works fine.
I tried both python:3.6 and python:3.8 Docker images.
I also tried to cURL both endpoints from inside the container and the server replies correctly.

You can verify whether docker is terminating the connection by using nc -l outside the container and nc in the container to connect to the first nc, then leaving them both alone for 10 minutes and then sending something to verify the connection is still up.
To troubleshoot pycurl, see http://pycurl.io/docs/latest/troubleshooting.html#transfer-related-issues.

Related

Azure Instance with linux machine docker timing out on requests

I have a python script in a docker that runs and collects data from an api on a loop.
`for idx, items in enumerate(SearchList):
#get access token
access_token = GetPricingAccTok()
#request id to get results
reqid = RequestIdCompSearch(items, access_token)
#api to get results by doing a while loop appending results together till api returns 100% as completed percentage
Result = GetReqResults(reqid, access_token)
Pricings = GetPricingDict(Result)
filenamepricing = "pricinghistory/CorrectFormat/Pricings_Data_{idn}_{ts}.json".format(
idn = idx,
ts= dt.datetime.now().isoformat()
)
GetAzContainer().upload_blob(
name=filenamepricing,
data=json.dumps(obj= Pricings, indent=4),
blob_type='BlockBlob'
)
`
When i run and test this code locally on my windows machine it works well without an issue and completes the requests
but in the docker on azure container instance it then returns an error showing that the connection has timed out. Not sure what the difference is between the running it locally and in the docker that i then recieve this error.
I have tried addinng this code at the start
import socket
from urllib3.connection import HTTPConnection
HTTPConnection.default_socket_options =(HTTPConnection.default_socket_options + [
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) #Enables the feature
,(socket.SOL_TCP, socket.TCP_KEEPIDLE, 45) #Overrides the time when the stack willl start sending KeppAlives after no data received on a Persistent Connection
,(socket.SOL_TCP, socket.TCP_KEEPINTVL, 10) #Defines how often thoe KA will be sent between them
,(socket.SOL_TCP, socket.TCP_KEEPCNT, 60) #How many attemps will your code try if the server goes down before droping the connection.
]
)
but it still times out. showing the below error
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 353, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='service.xxx.com', port=443): Read timed out. (read timeout=None)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "src/script.py", line 298, in <module>
reqid = RequestIdCompSearch(items, access_token)
File "src/script.py", line 111, in RequestIdCompSearch
InitResp =rq.post( ReqUrl, headers= ReqHead, data=Jsondata)
File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 117, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='service.xxxx.com', port=443): Read timed out. (read timeout=None)

ReadTimeoutError - Why is this occurring?

I encounter the following ReadTimeoutError, the request not responding and not giving out any output (in the state of runnning non-stop):
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\http\client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\socket.py", line 705, in readinto
return self._sock.recv_into(b)
TimeoutError: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 440, in send
resp = conn.urlopen(
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\packages\six.py", line 770, in reraise
raise value
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 451, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 340, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=4990): Read timed out. (read timeout=10)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\Downloads\getSupplierName-api\test.py", line 35, in <module>
response = requests.request("PUT", url, headers=headers, data=payload, timeout=10)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=4990): Read timed out. (read timeout=10)
The request code is as follows:
import requests
import json
import glob
url = "..."
headers = {
'Content-Type': 'application/json'
}
path = "..../*.pdf"
list_of_path = glob.glob(path)
print("Number of files:", len(list_of_path))
for i in list_of_path:
print(i)
payload = json.dumps({
"file_path": i,
"company_id": "abcde",
})
response = requests.request("PUT", url, headers=headers, data=payload, timeout=10)
print(response.json()['payload'])
Not sure why this is happening. The same issue happens with or without running in virtualenv. It was running fine when I ran it (many times) without any virtualenv, then I created the virtualenv on my machine and ran it again, it's working fine. But on the next day, the error occurs when I try to run it.
Appreciate any help, many thanks!
Looks to me that the service that was listening for at port 4990 is not listening anymore. I suppose you know what service this is, so I recommend to check if it's running.
After you start the service that listen at port 4990, there are two ways to confirm you the port is really open and listening:
If you are using linux, you can check what ports are listening in your machine with the following command:
sudo lsof -i -P -n | grep LISTEN
You can try to connect to the port 4990 using telnet:
telnet 127.0.0.1 4990
If the telnet works, the output should be something like this:
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

PythonAnywhere requests session raising exception 'connection aborted'

I am trying to launch my script to PA and while it works locally it does not work on PA and I cannot figure out why. I have the paid version of PA so I can access this site that is not whitelisted. Here is my script:
with requests.Session() as session:
# initates the login session
if os.path.exists(document_report + '.pdf'):
# initates the login session
post = session.post(POST_LOGIN_URL, data=payload, headers = dict(referrer = 'https://www.example.com/loginpage'))
#This URL is the page you actually want to pull down with requests.
REQUEST_URL = f'https://www.example.com/{stuff}/{document}'
#gets the request URL
r = session.get(REQUEST_URL)
# saves to directory
with open('./digests/1.pdf', 'wb') as j:
j.write(r.content)
j.close()
else:
post = session.post(POST_LOGIN_URL, data=payload, headers = dict(referrer = 'https://www.example.com/loginpage'))
#This URL is the page you actually want to pull down with requests.
REQUEST_URL = f'https://www.example.com/{stuff}/{document}'
#gets the request URL
r = session.get(REQUEST_URL)
# saves to directory
with open(document_report + '.pdf', 'wb') as j:
j.write(r.content)
j.close()
When I run the code, I get the following message:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 275, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 275, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/legalaide/emailProcessorV2.py", line 567, in <module>
post = session.post(POST_LOGIN_URL, data=payload, headers = dict(referrer = 'https://www.example.com/loginpage'))
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I tried it again locally, and it still works. Any insight into a solution would be amazing!

Python HTTP Server/Client: Remote end closed connection without response error

I made simple HTTP Server using BaseHTTPRequestHandler. The problem is, that when I want to post some data using requests from client, I get ConnectionError. I did simple request from requests lib documentation. Also interesting thing is, that HTTP Server will receive data from client and print it to console. I don't understand how its possible.
Client:
def post_data():
"""Client method"""
json_data = {
'sender': 'User',
'receiver': 'MY_SERVER',
'message': 'Hello server! Sending some data.'}
data_headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
data_payload = json.dumps(json_data)
try:
post = requests.post('http://localhost:8080/post', data=data_payload,
headers=data_headers)
print(post.status_code)
except ConnectionError as e:
print("CONNECTION ERROR: ")
print(e)
HTTP Server:
def do_POST(self):
"""Server method"""
self.send_response(200)
print("Receiving new data ...")
content_length = int(self.headers['Content-Length'])
post_data = self.rfile.read(content_length)
print(post_data)
Server result:
C:\Users\mypc\Projects\PythonFiles\httpserver>python server.py
Fri Jan 5 01:09:12 2018: HTTP Server started on port 8080.
127.0.0.1 - - [05/Jan/2018 01:09:21] "POST /post HTTP/1.1" 200 -
Receiving new data ...
b'{"sender": "User", "receiver": "MY_SERVER", "message": "Hello server! Sending some data."}'
Client result:
C:\Users\mypc\Projects\PythonFiles\httpserver>python client.py
CONNECTION ERROR:
('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
C:\Users\mypc\Projects\PythonFiles\httpserver>
Error without exception block:
Traceback (most recent call last):
File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
chunked=chunked)
File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "C:\Python36\lib\http\client.py", line 1331, in getresponse
response.begin()
File "C:\Python36\lib\http\client.py", line 297, in begin
version, status, reason = self._read_status()
File "C:\Python36\lib\http\client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python36\lib\site-packages\requests\adapters.py", line 440, in send
timeout=timeout
File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\Python36\lib\site-packages\urllib3\util\retry.py", line 357, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Python36\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
chunked=chunked)
File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "C:\Python36\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "C:\Python36\lib\http\client.py", line 1331, in getresponse
response.begin()
File "C:\Python36\lib\http\client.py", line 297, in begin
version, status, reason = self._read_status()
File "C:\Python36\lib\http\client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "client.py", line 137, in <module>
start_parser()
File "client.py", line 101, in start_parser
send_requests(args.get, args.post)
File "client.py", line 51, in send_requests
post_data()
File "client.py", line 129, in post_data
headers=data_headers)
File "C:\Python36\lib\site-packages\requests\api.py", line 112, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Python36\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python36\lib\site-packages\requests\sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python36\lib\site-packages\requests\sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "C:\Python36\lib\site-packages\requests\adapters.py", line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
It looks like the server is terminating the connection early without sending a full response. I've skimmed the docs and I think this is the problem (emphasis added):
send_response(code, message=None)
Adds a response header to the
headers buffer and logs the accepted request. The HTTP response line
is written to the internal buffer, followed by Server and Date
headers. The values for these two headers are picked up from the
version_string() and date_time_string() methods, respectively. If the
server does not intend to send any other headers using the
send_header() method, then send_response() should be followed by an
end_headers() call.
Changed in version 3.3: Headers are stored to an internal buffer and
end_headers() needs to be called explicitly.
So you probably just need to add the call to end_headers. If you were reading an old example (prior to Python 3.3) this wouldn't have been needed.
You can just put end_headers as Peter Gibson said
self.send_header('Content-Type', 'blabla' )
self.end_headers()

Python urllib3 - Crawling webpages failed at a specific webpage only from google cloud vm

I have built a crawler script that download webpages using urllib3 and python3. I ran the crawler in a Google Cloud Virtual Machine instance and tried it with many different urls. I found a url that i couldn't download on the VM instance but succeeded to download on my personal Linux.
The code that i've tried on my computer and worked:
>>> import urllib3
>>> pool = urllib3.PoolManager()
>>> response = pool.request('GET', 'http://www.azlyrics.com/lyrics/drake/dowhatyoudo.html')
>>> response.status
200
I have tried the exact same code on the VM instance using ssh and this exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 391, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/dist-packages/urllib3/request.py", line 66, in request
**urlopen_kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3/request.py", line 87, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3/poolmanager.py", line 244, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
release_conn=release_conn, **response_kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
release_conn=release_conn, **response_kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
release_conn=release_conn, **response_kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.5/dist-packages/urllib3/util/retry.py", line 303, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='www.azlyrics.com', port=80): Max retries exceeded with url: /lyrics/drake/dowhatyoudo.html (Caused by ProtocolError('Connection aborted.', RemoteDisconnected('Remote
end closed connection without response',)))
I have checked the VM dashboard and I didn't find any firewall or other restrictions that I may have mistakenly defined.
What could it possibly be?

Categories