We Keep Coding

Python Requests post times out despite timeout setting

I am using the Python Requests module (v. 2.19.1) with Python 3.4.3, calling a function on a remote server that generates a .csv file for download. In general, it works perfectly. There is one particular file that takes >6 minutes to complete, and no matter what I set the timeout parameter to, I get an error after exactly 5 minutes trying to generate that file.
import requests
s = requests.Session()
authPayload = {'UserName': 'myloginname','Password': 'password'}
loginURL = 'https://myremoteserver.com/login/authenticate'
login = s.post(loginURL, data=authPayload)
backupURL = 'https://myremoteserver.com/directory/jsp/Backup.jsp'
payload = {'command': fileCommand}
headers = {'Connection': 'keep-alive'}
post = s.post(backupURL, data=payload, headers=headers, timeout=None)
This times out after exactly 5 minutes with the error:
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 330, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 612, in urlopen
raise MaxRetryError(self, url, e)
urllib3.exceptions.MaxRetryError: > HTTPSConnectionPool(host='myremoteserver.com', port=443): Max retries exceeded with url: /directory/jsp/Backup.jsp (Caused by < class 'http.client.BadStatusLine'>: '')
If I set timeout to something much smaller, say, 5 seconds, I get a error that makes perfect sense:
urllib3.exceptions.ReadTimeoutError:
HTTPSConnectionPool(host='myremoteserver.com', port=443): Read
timed out. (read timeout=5)
If I run the process from a browser, it works fine, so it doesn't seem like it's the remote server closing the connection, or a firewall or something in-between closing the connection.

Posted at the request of the OP -- my comments on the original question pointed to a related SO problem
The clue to the problem lies in the http.client.BadStatusLine error.
Take a look at the following related SO Q & A that discusses the impact of proxy servers on HTTP requests and responses.

Error -3 while decompressing data: incorrect header check - urllib2

I am checking for url status with this code:
h = httplib2.Http()
hdr = {'User-Agent': 'Mozilla/5.0'}
resp = h.request("http://" + url, headers=hdr)
if int(resp[0]['status']) < 400:
return 'ok'
else:
return 'bad'
and getting
Error -3 while decompressing data: incorrect header check
the url i am checking is:
http://www.sueddeutsche.de/wirtschaft/deutschlands-innovationsangst-wir-neobiedermeier-1.2117528
the Exception Location is:
Exception Location: C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py in _decompressContent, line 403
try:
encoding = response.get('content-encoding', None)
if encoding in ['gzip', 'deflate']:
if encoding == 'gzip':
content = gzip.GzipFile(fileobj=StringIO.StringIO(new_content)).read()
if encoding == 'deflate':
content = zlib.decompress(content) ##<---- error line
response['content-length'] = str(len(content))
# Record the historical presence of the encoding in a way the won't interfere.
response['-content-encoding'] = response['content-encoding']
del response['content-encoding']
except IOError:
content = ""
http status is 200 which is ok for my case, but i am getting this error
I actually need only http status, why is it reading the whole content?

You may have any number of reasons why you choose httplib2, but it's far too easy to get the status code of an HTTP request using the python module requests. Install with the following command:
$ pip install requests
See an extremely simple example below.
In [1]: import requests as rq
In [2]: url = "http://www.sueddeutsche.de/wirtschaft/deutschlands-innovationsangst-wir-neobiedermeier-1.2117528"
In [3]: r = rq.get(url)
In [4]: r
Out[4]: <Response [200]>
Link
Unless you have a considerable constraint that needs httplib2 explicitly, this solves your problem.

This may be a bug (or just uncommon design decision) in httplib2. I don't get this problem with urllib2 or httplib in the 2.x stdlib, or urllib.request or http.client in the 3.x stdlib, or the third-party libraries requests, urllib3, or pycurl.
So, is there a reason you need to use this particular library?
If so:
I actually need only http status, why is it reading the whole content?
Well, most HTTP libraries are going to read and parse the whole content, or at least the headers, before returning control. That way they can respond to simple requests about the headers or chunked encoding or MIME envelope or whatever without any delay.
Also, many of them automate things like 100 continue, 302 redirect, various kinds of auth, etc., and there's no way they could do that if they didn't read ahead. In particular, according to the description for httplib2, handling these things automatically is one of the main reasons you should use it in the first place.
Also, the first TCP read is nearly always going to include the headers anyway, so why not read them?
This means that if the headers are invalid, you may get an exception immediately. They may still provide a way to get the status code (or the raw headers, or other information) anyway.
As a side note, if you only want the HTTP status, you should probably send a HEAD request rather than a GET. Unless you're writing and testing a server, you can almost always rely on the fact that, as the RFC says, the status and headers should be identical to what you'd get with GET. In fact, that would almost certainly solve things in this case—if there is no body to decompress, the fact that httplib2 has gotten confused into thinking the body is gzipped when it isn't won't matter anyway.

Proxy using Twython

I keep getting this error everytime I try running my code through proxy. I have gone through every single link available on how to get my code running behind proxy and am simply unable to get this done.
import twython
import requests
TWITTER_APP_KEY = 'key' #supply the appropriate value
TWITTER_APP_KEY_SECRET = 'key-secret'
TWITTER_ACCESS_TOKEN = 'token'
TWITTER_ACCESS_TOKEN_SECRET = 'secret'
t = twython.Twython(app_key=TWITTER_APP_KEY,
app_secret=TWITTER_APP_KEY_SECRET,
oauth_token=TWITTER_ACCESS_TOKEN,
oauth_token_secret=TWITTER_ACCESS_TOKEN_SECRET,
client_args = {'proxies': {'http': 'proxy.company.com:10080'}})
now if I do
t = twython.Twython(app_key=TWITTER_APP_KEY,
app_secret=TWITTER_APP_KEY_SECRET,
oauth_token=TWITTER_ACCESS_TOKEN,
oauth_token_secret=TWITTER_ACCESS_TOKEN_SECRET,
client_args = client_args)
print t.client_args
I get only a {}
and when I try running
t.update_status(status='See how easy this was?')
I get this problem :
Traceback (most recent call last):
File "<pyshell#40>", line 1, in <module>
t.update_status(status='See how easy this was?')
File "build\bdist.win32\egg\twython\endpoints.py", line 86, in update_status
return self.post('statuses/update', params=params)
File "build\bdist.win32\egg\twython\api.py", line 223, in post
return self.request(endpoint, 'POST', params=params, version=version)
File "build\bdist.win32\egg\twython\api.py", line 213, in request
content = self._request(url, method=method, params=params, api_call=url)
File "build\bdist.win32\egg\twython\api.py", line 134, in _request
response = func(url, **requests_args)
File "C:\Python27\lib\site-packages\requests-1.2.3-py2.7.egg\requests\sessions.py", line 377, in post
return self.request('POST', url, data=data, **kwargs)
File "C:\Python27\lib\site-packages\requests-1.2.3-py2.7.egg\requests\sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-1.2.3-py2.7.egg\requests\sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-1.2.3-py2.7.egg\requests\adapters.py", line 327, in send
raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='api.twitter.com', port=443): Max retries exceeded with url: /1.1/statuses/update.json (Caused by <class 'socket.gaierror'>: [Errno 11004] getaddrinfo failed)
I have searched everywhere. Tried everything that I possibly could. The only resources available were :
https://twython.readthedocs.org/en/latest/usage/advanced_usage.html#manipulate-the-request-headers-proxies-etc
https://groups.google.com/forum/#!topic/twython-talk/GLjjVRHqHng
https://github.com/fumieval/twython/commit/7caa68814631203cb63231918e42e54eee4d2273
https://groups.google.com/forum/#!topic/twython-talk/mXVL7XU4jWw
There were no topics I could find here (on Stack Overflow) either.
Please help. Hope someone replies. If you have already done this please help me with some code example.

Your code isn't using your proxy. The example shows, you specified a proxy for plain HTTP but your stackstrace shows a HTTPSConnectionPool. Your local machine probably can't resolve external domains.
Try setting your proxy like this:
client_args = {'proxies': {'https': 'http://proxy.company.com:10080'}}

In combination with #t-8ch's answer (which is that you must use a proxy as he has defined it), you should also realize that as of this moment, requests (the underlying library of Twython) does not support proxying over HTTPS. This is a problem with requests underlying library urllib3. It's a long running issue as far as I'm aware.
On top of that, reading a bit of Twython's source explains why t.client_args returns an empty dictionary. In short, if you were to instead print t.client.proxies, you'd see that indeed your proxies are being processed as they very well should be.
Finally, complaining about your workplace while on StackOverflow and linking to GitHub commits that have your GitHub username (and real name) associated with them in the comments is not the best idea. StackOverflow is indexed quite thoroughly by Google and there is little doubt that someone else might find this and associate it with you as easily as I have. On top of that, that commit has absolutely no effect on Twython's current behaviour. You're running down a rabbit hole with no end by chasing the author of that commit.

It looks like a domain name lookup failed. Assuming your configured DNS server can resolve Twitter's domain name (and surely it can), I would presume your DNS lookup for proxy.company.com failed. Try using a proxy by IP address instead of by hostname.

Post print dictionary/json returns error to client

I am sending post request in the body of some json data, to process on server and I want the results back to client(c++ app on phone) in the form of json data and hence parse on mobile.
I have the following code inside handler:
class ServerHandler(tornado.web.RequestHandler):
def post(self):
data = tornado.escape.json_decode(self.request.body)
id = data.get('id',None)
#process data from db (take a while) and pack in result which is dictinary
result = process_data(id)# returns dictionary from db= takes time
print 'END OF HANDLER'
print json.dumps(result)
#before this code below I have tried also
#return result
#return self.write(result)
#return self.write(json.dumps(result))
#return json.dumps(result)
self.set_header('Content-Type', 'application/json')
json_ = tornado.escape.json_encode(result)
self.write(json_)
self.finish()
#return json.dumps(result)
I always get printed 'END OF HANDLER' and valid dictinary/json below on console but when I read at client mobile I always get
<html><title>405: Method Not Allowed</title><body>405: Method Not Allowed</body></html>
Does anyone have any idea what is the bug ?
(I am using CIwGameHttpRequest for sending request and it works when file is static =>name.json but now same content is giving error in post request. )

The error (HTTP 405 Method Not Allowed) means that you have made a request to a valid URL, but you are using an HTTP verb (e.g. GET, POST, PUT, DELETE) that cannot be used with that URL.
Your web service code appears to handle the POST verb, as evidenced by the post method name, and also by the fact that incoming requests appear to have a request body. You haven't shown us your C++ client code, so all I can do is to speculate that it is making a GET request. Does your C++ code call Request->setPOST();? (I haven't worked with CIwGameHttpRequest before, but Googling for it I found this page from which I took that line of code.)
I've not worked with Tornado before, but I imagine that there is some mechanism somewhere that allows you to connect a URL to a RequestHandler. Given that you have a 405 Method Not Allowed error rather than 404 Not Found, it seems that however this is done you've done it correctly. You issue a GET request to Tornado for the URL, it determines that it should call your handler, and only when it tries to use your handler it realises that it can't handle GET requests, concludes that your handler (and hence its URL) doesn't support GETs and returns a 405 error.

how python http request and response works

I'm newbie for python, I'm having task so I need to scan wifi and send the data to the server, the below is the format which i have to send, this work fine when enter manually in browser url text box,
http://223.56.124.58:8080/ppod-web/ProcessRawData?data={"userId":"2220081127-14","timestamp":"2010-04-12 10:54:24","wifi":{"ssid":"guest","rssi":"80"}}
here is my code:
import httplib
import urllib
params = urllib.urlencode('{\"userId\":\"20081127-14\",\"timestamp\":\"2010-04-12 10:54:24\",\"wifi\":{\"ssid\":\"guest\",\"rssi\":\"80\"}}')
headers = {"Content-type":"application/x-www-form-urlencoded","Accept":"text/plain"}
conn = httplib.HTTPConnection("http://223.56.124.58:8080")
conn.request("POST","ppod-web/ProcessRawData?data=",params,headers)
response = conn.getresponse()
print response.status
print "-----"
print response.reason
data = response.read()
print data
conn.close()
thanks

Most likely, the issue with the script you posted in the question is you cannot directly do:
conn=httplib.HTTPConnection("http://223.56.124.58:8080/wireless")
The exception is triggered in getaddrinfo(), which calls the C function getaddrinfo() which returns EAI_NONAME:
The node or service is not known; or both node and service are NULL; or AI_NUMERICSERV was specified in hints.ai_flags and service was not a numeric port-number string."
There obviously is a problem with the parameters passed to getaddrinfo, and most likely you are trying to get information for the "223.56.124.58:8080/wireless" host. Ooops!
Indeed, you cannot directly connect to an URL address. As the documentation clearly states and shows, you connect to the server:
conn = httplib.HTTPConnection("223.56.124.58", 8080)
Then you can do:
conn.request("POST", "wireless", params, headers)
What about the script you are actually using?
conn.request("POST","http://202.45.139.58:8080/ppod-web",params,headers)
Even if the connection was correctly formed, that would have you POSTing to http://202.45.139.58:8080/http://202.45.139.58:8080/ppod-web. What you really want probably is:
conn = httplib.HTTPConnection("202.45.139.58", 8080)
conn.request("POST", "ppod-web", params, headers)
The error is shown for this line because most likely HTTPConnection is a lazy object and only attempts to actually connect to the server when you call request().
After you're done fixing the above, you'll need to fix params.
>>> urllib.urlencode({"wifi":{"ssid":"guest","rssi","80"}})
SyntaxError: invalid syntax
>>> urllib.urlencode({"wifi":{"ssid":"guest","rssi":"80"}})
'wifi=%7B%27rssi%27%3A+%2780%27%2C+%27ssid%27%3A+%27guest%27%7D'
To get what you think you want to get, you should do:
>>> urllib.urlencode({"data": {"wifi":{"ssid":"guest","rssi":"80"}}})
'data=%7B%27wifi%27%3A+%7B%27rssi%27%3A+%2780%27%2C+%27ssid%27%3A+%27guest%27%7D%7D'

Instead of:
conn = httplib.HTTPConnection("http://223.56.124.58:8080/wireless")
conn.request("POST", "data", params, headers)
try:
conn = httplib.HTTPConnection("223.56.124.58", port=8080)
conn.request("POST", "/wireless", params, headers)
Not sure if it will resolve all your problems, but at least your code will conform to the method/constructor signatures.

The traceback doesn't come from the same code you pasted.
On the error traceback there's a line:
conn.request("POST","http://202.45.139.58:8080/ppod-web",params,headers)
It is the line 9 of http.py however it is not on the code you pasted.
Please paste the actual code.