I am new to python and my script lasts about 3-4 hour and no error message is saved. What could be causing this problem?
Here is my code:
import time
import urllib.request
import threading
def load():
try:
content = str(urllib.request.urlopen("[URL]").read())
# do sth with content
threading.Timer(0.5, load).start()
except Exception as e:
file = open("Error.txt","w")
file.write(time.strftime("%H:%M:%S\n\n"))
file.write(e.message)
file.close()
threading.Timer(0.5, load).start()
def main(args):
load()
return 0
if __name__ == '__main__':
import sys
sys.exit(main(sys.argv))
And here is nohup.out file on ubuntu 14.04:
Exception in thread Thread-907:
Traceback (most recent call last):
File "/usr/lib/python3.4/urllib/request.py", line 1182, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/usr/lib/python3.4/http/client.py", line 1125, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.4/http/client.py", line 1163, in _send_request
self.endheaders(body)
File "/usr/lib/python3.4/http/client.py", line 1121, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 951, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 886, in send
self.connect()
File "/usr/lib/python3.4/http/client.py", line 863, in connect
self.timeout, self.source_address)
File "/usr/lib/python3.4/socket.py", line 512, in create_connection
raise err
File "/usr/lib/python3.4/socket.py", line 503, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable
Network is unreachable usually means you have a connectivity problem on your machine. Maybe it's an intermittent, short-lived problem, but you never try to detect it and recover from it.
Consider using something like retry decorator, or handle it manually.
(I also find your way of looping the fetch logic rather bizarre. Why a simple single-threaded loop did not work for you?)
Related
UPDATE: I managed to do a request with urllib2, but I'm still wondering what is happening here.
I would like to do a HTTPS request with Python.
This works fine with the requests module, but I don't want to use external dependencies, so I'd like to use the standard library.
httplib
When I follow this example I don't get a response. I get a timeout instead. I'm out of ideas as to what would cause this.
Code:
import requests
print requests.get('https://python.org')
from httplib import HTTPSConnection
conn = HTTPSConnection('www.python.org')
conn.request('GET', '/index.html')
print conn.getresponse()
Output:
<Response [200]>
Traceback (most recent call last):
File "test.py", line 6, in <module>
conn.request('GET', '/index.html')
File "C:\Python27\lib\httplib.py", line 1069, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 1109, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 1065, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 892, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 854, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1282, in connect
HTTPConnection.connect(self)
File "C:\Python27\lib\httplib.py", line 831, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 575, in create_connection
raise err
socket.error: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
urllib
This fails for a different (but possibly related) reason. Code:
import urllib
print urllib.urlopen("https://python.org")
Output:
Traceback (most recent call last):
File "test.py", line 10, in <module>
print urllib.urlopen("https://python.org")
File "C:\Python27\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 215, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 445, in open_https
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 1065, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 892, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 854, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1290, in connect
server_hostname=server_hostname)
File "C:\Python27\lib\ssl.py", line 369, in wrap_socket
_context=self)
File "C:\Python27\lib\ssl.py", line 599, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:727)
What is requests doing that makes it succeed where both of these libraries fail?
requests.get without timeout parameter mean no timeout at all.
httplib.HTTPSConnection accept parameter timeout in Python 2.6 and newer according to httplib docs. If your problem was caused by timeout, setting high enough timeout should help. Please try replacing:
conn = HTTPSConnection('www.python.org')
with:
conn = HTTPSConnection('www.python.org', timeout=300)
which will give 300 seconds (5 minutes) for processing.
I am using python 3.4 and my task is to download link from a website. But while fetching the data I got an error that:
I have tried it from both httplib2 and urllib, but in both cases it is showing the same error.
OUTPUT:
>>> import urllib.request
>>> request = urllib.request.Request(url)
>>> response = urllib.request.urlopen(request)
Traceback (most recent call last):
File "C:\Python34\lib\urllib\request.py", line 1183, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python34\lib\http\client.py", line 1137, in request
self._send_request(method, url, body, headers)
File "C:\Python34\lib\http\client.py", line 1182, in _send_request
self.endheaders(body)
File "C:\Python34\lib\http\client.py", line 1133, in endheaders
self._send_output(message_body)
File "C:\Python34\lib\http\client.py", line 963, in _send_output
self.send(msg)
File "C:\Python34\lib\http\client.py", line 898, in send
self.connect()`
File "C:\Python34\lib\http\client.py", line 871, in connect
self.timeout, self.source_address)`
File "C:\Python34\lib\socket.py", line 516, in create_connection
raise err
File "C:\Python34\lib\socket.py", line 507, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] A connection attempt failed because the
connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to respond
I wrote a few lines of python script to get links from my small site. I tested it, used it and was working fine but suddenly it starts generating connection error. I tried to use it on another site, it still generate same problem. I'm new to python, though. Help please!!! Thanks in advance
Part of the code and error message is as follow:
>>>import urllib,sgmllib
>>> f=urllib.urlopen("http://www.example.com")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 345, in open_http
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 991, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 844, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 806, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 787, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 571, in create_connection
raise err
IOError: [Errno socket error] [Errno 10061] No connection could be made becausethe target machine actively refused it
>>>
Thank you Johan, Shashank, bmhkim for your contributions.
I just figured out the cause of the problem now and the solution. It was caused by the proxy setting of the host running the code. So, resetting the proxy solve it.
Tried the 3 following methods to control Tor:
using TorCtl/urllib2: Python script Exception with Tor
using socks/httplib: http://www.youtube.com/watch?v=KDsmVH7eJCs
using socks/urllib2: Python urllib over TOR?
Each of them fails w/ same error (tried to make it as clear as possible):
Traceback (most recent call last):
File "tor.py", line 26, in <module>
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py"
line 127, in urlopen
return _opener.open(url, data, timeout)
line 404, in open
response = self._open(req, data)
line 422, in _open
'_open', req)
line 382, in _call_chain
result = func(*args)
line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
line 1181, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py"
line 973, in request
self._send_request(method, url, body, headers)
line 1007, in _send_request
self.endheaders(body)
line 969, in endheaders
self._send_output(message_body)
line 829, in _send_output
self.send(msg)
line 791, in send
self.connect()
line 772, in connect
self.timeout, self.source_address)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py"
line 562, in create_connection
sock.connect(sa)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/SocksiPy_branch-1.01-py2.7.egg/socks.py"
line 392, in connect
self.__negotiatesocks5(destpair[0],destpair[1])
line 199, in __negotiatesocks5
self.sendall("\x05\x01\x00")
line 165, in sendall
socket.socket.sendall(self, bytes)
... last error repeating a lot of times and then ...
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/SocksiPy_branch-1.01-py2.7.egg/socks.py", line 163, in sendall
if 'encode' in dir(bytes):
RuntimeError: maximum recursion depth exceeded while calling a Python object
Does anyone understand where it comes from?
For me it actually failed with socksipy-branch installed with pip.
However, it worked ok after I downloaded socks.py directly from http://socksipy.sourceforge.net/ to my working directory.
It looks like you are using the SocksiPy library. I had the same problem and got it fixed by installing this library again directly from https://code.google.com/p/socksipy-branch/. The first time I installed it via pip.
I have a question on Python 2.7. The urlopen function sometimes fails to connect due to the bad internet. So I want to add a timer on it, so that when the connection times out, it close the previous connection and try to reconnect again. Here in my implementation, it will fall into a dead loop, to "Trying to reconnect" again and again... Could anyone help me with that? Thank you very much!
def getConnection(urlItem, keywords, address):
urlItem.close()
print "Trying to reconnect..."
findArticles(keywords, address)
def findArticles(keywords, address):
urlItem = urllib.urlopen(address)
t = Timer(20.0, getConnection(urlItem, keywords, address))
t.start()
htmlSource = urlItem.read()
t.cancel()
Here is the exception I got:
File "/usr/lib/python2.7/urllib.py", line 84, in urlopen
return opener.open(url)
File "/usr/lib/python2.7/urllib.py", line 205, in open
return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 342, in open_http
h.endheaders(data)
File "/usr/lib/python2.7/httplib.py", line 951, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 811, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 773, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 754, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno -2] Name or service not known
The old urllib library itself does not support setting a time out, but urllib2 does. You should look into using urllib2, that said...
You can import socket and set a default timeout on all newly created sockets. This might eb a roundabout way to do what you need to do.
http://docs.python.org/2.7/library/socket.html#socket.setdefaulttimeout