UPDATE: I managed to do a request with urllib2, but I'm still wondering what is happening here.
I would like to do a HTTPS request with Python.
This works fine with the requests module, but I don't want to use external dependencies, so I'd like to use the standard library.
httplib
When I follow this example I don't get a response. I get a timeout instead. I'm out of ideas as to what would cause this.
Code:
import requests
print requests.get('https://python.org')
from httplib import HTTPSConnection
conn = HTTPSConnection('www.python.org')
conn.request('GET', '/index.html')
print conn.getresponse()
Output:
<Response [200]>
Traceback (most recent call last):
File "test.py", line 6, in <module>
conn.request('GET', '/index.html')
File "C:\Python27\lib\httplib.py", line 1069, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 1109, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 1065, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 892, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 854, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1282, in connect
HTTPConnection.connect(self)
File "C:\Python27\lib\httplib.py", line 831, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 575, in create_connection
raise err
socket.error: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
urllib
This fails for a different (but possibly related) reason. Code:
import urllib
print urllib.urlopen("https://python.org")
Output:
Traceback (most recent call last):
File "test.py", line 10, in <module>
print urllib.urlopen("https://python.org")
File "C:\Python27\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 215, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 445, in open_https
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 1065, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 892, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 854, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1290, in connect
server_hostname=server_hostname)
File "C:\Python27\lib\ssl.py", line 369, in wrap_socket
_context=self)
File "C:\Python27\lib\ssl.py", line 599, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:727)
What is requests doing that makes it succeed where both of these libraries fail?
requests.get without timeout parameter mean no timeout at all.
httplib.HTTPSConnection accept parameter timeout in Python 2.6 and newer according to httplib docs. If your problem was caused by timeout, setting high enough timeout should help. Please try replacing:
conn = HTTPSConnection('www.python.org')
with:
conn = HTTPSConnection('www.python.org', timeout=300)
which will give 300 seconds (5 minutes) for processing.
Related
I am new to python and my script lasts about 3-4 hour and no error message is saved. What could be causing this problem?
Here is my code:
import time
import urllib.request
import threading
def load():
try:
content = str(urllib.request.urlopen("[URL]").read())
# do sth with content
threading.Timer(0.5, load).start()
except Exception as e:
file = open("Error.txt","w")
file.write(time.strftime("%H:%M:%S\n\n"))
file.write(e.message)
file.close()
threading.Timer(0.5, load).start()
def main(args):
load()
return 0
if __name__ == '__main__':
import sys
sys.exit(main(sys.argv))
And here is nohup.out file on ubuntu 14.04:
Exception in thread Thread-907:
Traceback (most recent call last):
File "/usr/lib/python3.4/urllib/request.py", line 1182, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/usr/lib/python3.4/http/client.py", line 1125, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.4/http/client.py", line 1163, in _send_request
self.endheaders(body)
File "/usr/lib/python3.4/http/client.py", line 1121, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 951, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 886, in send
self.connect()
File "/usr/lib/python3.4/http/client.py", line 863, in connect
self.timeout, self.source_address)
File "/usr/lib/python3.4/socket.py", line 512, in create_connection
raise err
File "/usr/lib/python3.4/socket.py", line 503, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable
Network is unreachable usually means you have a connectivity problem on your machine. Maybe it's an intermittent, short-lived problem, but you never try to detect it and recover from it.
Consider using something like retry decorator, or handle it manually.
(I also find your way of looping the fetch logic rather bizarre. Why a simple single-threaded loop did not work for you?)
I am using python 3.4 and my task is to download link from a website. But while fetching the data I got an error that:
I have tried it from both httplib2 and urllib, but in both cases it is showing the same error.
OUTPUT:
>>> import urllib.request
>>> request = urllib.request.Request(url)
>>> response = urllib.request.urlopen(request)
Traceback (most recent call last):
File "C:\Python34\lib\urllib\request.py", line 1183, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python34\lib\http\client.py", line 1137, in request
self._send_request(method, url, body, headers)
File "C:\Python34\lib\http\client.py", line 1182, in _send_request
self.endheaders(body)
File "C:\Python34\lib\http\client.py", line 1133, in endheaders
self._send_output(message_body)
File "C:\Python34\lib\http\client.py", line 963, in _send_output
self.send(msg)
File "C:\Python34\lib\http\client.py", line 898, in send
self.connect()`
File "C:\Python34\lib\http\client.py", line 871, in connect
self.timeout, self.source_address)`
File "C:\Python34\lib\socket.py", line 516, in create_connection
raise err
File "C:\Python34\lib\socket.py", line 507, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] A connection attempt failed because the
connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to respond
I am aware there is a patch for using the REST API with a proxy and it works for me. But the Streaming API uses a HTTPConnection which cannot be emulated by urllib and urllib2 (as far as I know). Is there any fix for this?
I tried using proxy with port, but it did not work.
In the streaming.py file, line 153.
if self.scheme == "http":
conn = httplib.HTTPConnection(self.api.proxy_host,self.api.proxy_port, timeout=self.timeout)
else:
conn = httplib.HTTPSConnection(self.api.proxy_host,self.api.proxy_port,timeout=self.timeout)
self.auth.apply_auth(url, 'POST', self.headers, self.parameters)
print conn.host
conn.connect()
conn.request('POST', self.scheme+'://'+self.host+self.url, self.body, headers=self.headers)
resp = conn.getresponse()
And, "self.scheme+'://'+self.host+ self.url" corresponds to - https://stream.twitter.com/1.1/statuses/filter.json?delimited=length
I get this error in return -
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2538, in run_code
exec code_obj in self.user_global_ns, self.user_ns
File "<ipython-input-3-4798d330f7cd>", line 1, in <module>
execfile('main.py')
File "main.py", line 130, in <module>
streamer.filter(track = ['AAP'])
File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 305, in filter
self._start(async)
File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 242, in _start
self._run()
File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 159, in _run
conn.connect()
File "/usr/lib/python2.7/httplib.py", line 1161, in connect
self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 143, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Tweepy currently does not support using a proxy. However, support for proxies should be coming in the next few days. I'll update this answer with more information.
Finally found a github patch that enabled streaming api via proxy. It works just fine!
https://github.com/shogo82148/tweepy/blob/64c6266018920e0e36c6d8d1600adb6caa0840de/tweepy/streaming.py
Tried the 3 following methods to control Tor:
using TorCtl/urllib2: Python script Exception with Tor
using socks/httplib: http://www.youtube.com/watch?v=KDsmVH7eJCs
using socks/urllib2: Python urllib over TOR?
Each of them fails w/ same error (tried to make it as clear as possible):
Traceback (most recent call last):
File "tor.py", line 26, in <module>
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py"
line 127, in urlopen
return _opener.open(url, data, timeout)
line 404, in open
response = self._open(req, data)
line 422, in _open
'_open', req)
line 382, in _call_chain
result = func(*args)
line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
line 1181, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py"
line 973, in request
self._send_request(method, url, body, headers)
line 1007, in _send_request
self.endheaders(body)
line 969, in endheaders
self._send_output(message_body)
line 829, in _send_output
self.send(msg)
line 791, in send
self.connect()
line 772, in connect
self.timeout, self.source_address)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py"
line 562, in create_connection
sock.connect(sa)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/SocksiPy_branch-1.01-py2.7.egg/socks.py"
line 392, in connect
self.__negotiatesocks5(destpair[0],destpair[1])
line 199, in __negotiatesocks5
self.sendall("\x05\x01\x00")
line 165, in sendall
socket.socket.sendall(self, bytes)
... last error repeating a lot of times and then ...
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/SocksiPy_branch-1.01-py2.7.egg/socks.py", line 163, in sendall
if 'encode' in dir(bytes):
RuntimeError: maximum recursion depth exceeded while calling a Python object
Does anyone understand where it comes from?
For me it actually failed with socksipy-branch installed with pip.
However, it worked ok after I downloaded socks.py directly from http://socksipy.sourceforge.net/ to my working directory.
It looks like you are using the SocksiPy library. I had the same problem and got it fixed by installing this library again directly from https://code.google.com/p/socksipy-branch/. The first time I installed it via pip.
Yowsup was running flawlessly yesterday...and today it's giving me an error.
I just download Yowsup from https://github.com/tgalal/yowsup and edit config.example file for change number and country code then fire below command in terminal
python ./yowsup-cli -c config.example -d --requestcode sms
To send request code. Please guess what could be problem?
erp#erp-desktop:~/Downloads/yowsup-master/src$ python ./yowsup-cli -c config.example -d --requestcode sms
{'Accept': 'text/json', 'User-Agent': 'WhatsApp/2.11.1 S40Version/14.26 Device/Nokia302'}
cc=91&in=9033308076&lc=US&lg=en&mcc=000&mnc=000&method=sms&id=f1b708bba17f1ce948dc979f4d7092bc&token=6ee7d82a64bb51c6f8742f30a245c7ef
Opening connection to v.whatsapp.net
Sending GET request to /v2/code?cc=91&in=9033308076&lc=US&lg=en&mcc=000&mnc=000&method=sms&id=f1b708bba17f1ce948dc979f4d7092bc&token=6ee7d82a64bb51c6f8742f30a245c7ef
Traceback (most recent call last):
File "./yowsup-cli", line 275, in <module>
result = wc.send()
File "/home/erp/Downloads/yowsup-master/src/Yowsup/Registration/v2/coderequest.py", line 60, in send
res = super(WACodeRequest, self).send(parser)
File "/home/erp/Downloads/yowsup-master/src/Yowsup/Common/Http/warequest.py", line 100, in send
return self.sendGetRequest(parser)
File "/home/erp/Downloads/yowsup-master/src/Yowsup/Common/Http/warequest.py", line 137, in sendGetRequest
self.response = WARequest.sendRequest(host, port, path, headers, params, "GET")
File "/home/erp/Downloads/yowsup-master/src/Yowsup/Common/Http/warequest.py", line 194, in sendRequest
conn.request(reqType, path, params, headers);
File "/usr/lib/python2.7/httplib.py", line 958, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 992, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 776, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 1161, in connect
self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 143, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
socket.error: [Errno 104] Connection reset by peer