Error when deploying scrapy project on the scrapy cloud - python

I am using scrapy 0.20 on Python 2.7.
I want to deploy my scrapy project on scrapy cloud
I developed my scrapy project with simple spider.
navigate to my scrapy project folder.
typed scrapy deploy scrapyd -d koooraspider on cmd. Where koooraspider is my project's name, and scrapyd is my target.
I got the following error:
D:\Walid-Project\Tasks\koooraspider>scrapy deploy scrapyd -p koooraspider
Packing version 1395847344
Deploying to project "koooraspider" in http://dash.scrapinghub.com/api/scrapyd/a
ddversion.json
Traceback (most recent call last):
File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\Python27\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "C:\Python27\lib\site-packages\scrapy-0.20.2-py2.7.egg\scrapy\cmdline.py"
, line 168, in <module>
execute()
File "C:\Python27\lib\site-packages\scrapy-0.20.2-py2.7.egg\scrapy\cmdline.py"
, line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "C:\Python27\lib\site-packages\scrapy-0.20.2-py2.7.egg\scrapy\cmdline.py"
, line 89, in _run_print_help
func(*a, **kw)
File "C:\Python27\lib\site-packages\scrapy-0.20.2-py2.7.egg\scrapy\cmdline.py"
, line 150, in _run_command
cmd.run(args, opts)
File "C:\Python27\lib\site-packages\scrapy-0.20.2-py2.7.egg\scrapy\commands\de
ploy.py", line 105, in run
if not _upload_egg(target, egg, project, version):
File "C:\Python27\lib\site-packages\scrapy-0.20.2-py2.7.egg\scrapy\commands\de
ploy.py", line 194, in _upload_egg
return _http_post(req)
File "C:\Python27\lib\site-packages\scrapy-0.20.2-py2.7.egg\scrapy\commands\de
ploy.py", line 210, in _http_post
f = urllib2.urlopen(request)
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 391, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 409, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 369, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1173, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Python27\lib\urllib2.py", line 1142, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "C:\Python27\lib\httplib.py", line 946, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 987, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 940, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 801, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 373: ordina
l not in range(128)
scrapy.cfg
# Project: KoooraSpider
[deploy:scrapyd]
url = http://dash.scrapinghub.com/api/scrapyd/
username = MyUserName
password =
project = 2431
Note
When I did the previous steps, project.egg-info folder, build folder, and setup.py script have been generated.
Any help will be appreciated.

remove project.egg-info & build folders
and then try command scrapy deploy scrapyd -p 2431

Related

Raspberry pi - get-pip.py error

I'm working on a Raspberry pi 2 with piface. I'm trying to install openCV by following this steps : click here. When I try doing the step 6 (install pip), I got the following error :
pi#raspberrypi ~ $ sudo python get-pip.py
Exception:
Traceback (most recent call last):
File "/tmp/tmpxtbKBh/pip.zip/pip/basecommand.py", line 211, in main
status = self.run(options, args)
File "/tmp/tmpxtbKBh/pip.zip/pip/commands/install.py", line 305, in run
wb.build(autobuilding=True)
File "/tmp/tmpxtbKBh/pip.zip/pip/wheel.py", line 705, in build
self.requirement_set.prepare_files(self.finder)
File "/tmp/tmpxtbKBh/pip.zip/pip/req/req_set.py", line 334, in prepare_files
functools.partial(self._prepare_file, finder))
File "/tmp/tmpxtbKBh/pip.zip/pip/req/req_set.py", line 321, in _walk_req_to_install
more_reqs = handler(req_to_install)
File "/tmp/tmpxtbKBh/pip.zip/pip/req/req_set.py", line 409, in _prepare_file
req_to_install, finder)
File "/tmp/tmpxtbKBh/pip.zip/pip/req/req_set.py", line 365, in _check_skip_installed
finder.find_requirement(req_to_install, self.upgrade)
File "/tmp/tmpxtbKBh/pip.zip/pip/index.py", line 486, in find_requirement
all_versions = self._find_all_versions(req.name)
File "/tmp/tmpxtbKBh/pip.zip/pip/index.py", line 404, in _find_all_versions
index_locations = self._get_index_urls_locations(project_name)
File "/tmp/tmpxtbKBh/pip.zip/pip/index.py", line 378, in _get_index_urls_locations
page = self._get_page(main_index_url)
File "/tmp/tmpxtbKBh/pip.zip/pip/index.py", line 818, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/tmp/tmpxtbKBh/pip.zip/pip/index.py", line 928, in get_page
"Cache-Control": "max-age=600",
File "/tmp/tmpxtbKBh/pip.zip/pip/_vendor/requests/sessions.py", line 477, in get
return self.request('GET', url, **kwargs)
File "/tmp/tmpxtbKBh/pip.zip/pip/download.py", line 373, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/tmp/tmpxtbKBh/pip.zip/pip/_vendor/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/tmp/tmpxtbKBh/pip.zip/pip/_vendor/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/tmp/tmpxtbKBh/pip.zip/pip/_vendor/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/tmp/tmpxtbKBh/pip.zip/pip/_vendor/requests/adapters.py", line 370, in send
timeout=timeout
File "/tmp/tmpxtbKBh/pip.zip/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/tmp/tmpxtbKBh/pip.zip/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/tmp/tmpxtbKBh/pip.zip/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type Error)
I tried to do the step 7 but I got the barely the same message (install virtualenv) :
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 290, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1178, in prepare_files
url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 194, in find_requirement
page = self._get_page(main_index_url, req)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page
session=self.session,
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 694, in get_page
req, link, "connection error: %s" % exc, url,
TypeError: __str__ returned non-string (type Error)
Storing debug log for failure in /root/.pip/pip.log
I'm trying to solve this problem since 2 days, so if anyone can help me...
Please.
Don't hesitate if you need more information. I also need to apologize for my english, I'm a swiss student...

Google GMail API `installed` app shows IOError 13 from module SSL.py w/o sudo

I am playing around with GMAIL API to create an app to send auto emails from my server. Running my application as a simple user has the following results:
pankgeorg#snf-25181:~/tomotech/gmailer$ python mailer.py
Traceback (most recent call last):
File "mailer.py", line 36, in <module>
gmail_service = build('gmail', 'v1', http=http)
File "/usr/local/lib/python2.7/dist-packages/oauth2client-1.4.5-py2.7.egg/oauth2client/util.py", line 135, in positional_wrapper
File "/usr/local/lib/python2.7/dist-packages/google_api_python_client-1.3.1-py2.7.egg/googleapiclient/discovery.py", line 198, in build
File "/usr/local/lib/python2.7/dist-packages/oauth2client-1.4.5-py2.7.egg/oauth2client/util.py", line 135, in positional_wrapper
File "/usr/local/lib/python2.7/dist-packages/oauth2client-1.4.5-py2.7.egg/oauth2client/client.py", line 547, in new_request
File "/usr/local/lib/python2.7/dist-packages/httplib2-0.9-py2.7.egg/httplib2/__init__.py", line 1593, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/local/lib/python2.7/dist-packages/httplib2-0.9-py2.7.egg/httplib2/__init__.py", line 1335, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/local/lib/python2.7/dist-packages/httplib2-0.9-py2.7.egg/httplib2/__init__.py", line 1257, in _conn_request
conn.connect()
File "/usr/local/lib/python2.7/dist-packages/httplib2-0.9-py2.7.egg/httplib2/__init__.py", line 1021, in connect
self.disable_ssl_certificate_validation, self.ca_certs)
File "/usr/local/lib/python2.7/dist-packages/httplib2-0.9-py2.7.egg/httplib2/__init__.py", line 80, in _ssl_wrap_socket
cert_reqs=cert_reqs, ca_certs=ca_certs)
File "/usr/lib/python2.7/ssl.py", line 886, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 496, in __init__
self._context.load_verify_locations(ca_certs)
IOError: [Errno 13] Permission denied
On the other hand, running with sudo works perfectly
pankgeorg#snf-25181:~/tomotech/gmailer$ sudo python mailer.py
Message Id: 14ad0aea05e*****
To be completely honest, in order to authenicate using --noauth_local_webserver, I run the command with sudo, authenticated and chown the gmail.storage to myself again.
Also, I installed using easy_install because pip install was giving me the following error:
pankgeorg#snf-25181:~/tomotech/gmailer$ sudo pip install --upgrade google_api_python_client
Cleaning up...
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1096, in prepare_files
req_to_install, self.upgrade)
File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 194, in find_requirement
page = self._get_page(main_index_url, req)
File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page
session=self.session,
File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 670, in get_page
resp = session.get(url, headers={"Accept": "text/html"})
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 237, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 506, in send
history = [resp for resp in gen] if allow_redirects else []
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 168, in resolve_redirects
allow_redirects=False,
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/adapters.py", line 330, in send
timeout=timeout
File "/usr/local/lib/python2.7/dist- packages/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 480, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python2.7/httplib.py", line 1001, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 826, in send
self.sock.sendall(data)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/contrib/pyopenssl.py", line 323, in sendall
return self.connection.sendall(data)
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 969, in sendall
raise TypeError("buf must be a byte string")
TypeError: buf must be a byte string
Storing debug log for failure in /root/.pip/pip.logrom oath2client import tool
It is my understanding that the root of the problem is the same in both cases.
I also have to note that my laptop (on which I can authenticate normally, on webbrowser, instead of --noauth_local_webserver) it works just fine, even though the installation is done the same way (the problem with pip appears there too).
Thanks in advance and sorry for the long post!
Tutorials I used:
parse arg
code for sending mails
application body is pretty much the quickstart for gmail api.
The httplib2 installer sets incorrect permissions for its httplib2/cacerts.txt file. One solution is to simply make its files readable by anyone by running
chmod o+r -R /usr/local/lib/python2.7/dist-packages/httplib2-0.9-py2.7.egg
However, it may be better to uninstall the version installed by pip, and use your operating system's package manager instead, which probably has the correct permissions for all files. On Debian, this could be accomplished with
pip uninstall httplib2
apt-get install python-httplib2

Cannot use Google Native Client SDK on Windows 7

I am trying Google's Native Client SDK.
OS is Windows 7, I've already installed python 2.7.9 and setup the environment variable path accordingly.
I also downloaded nacl_sdk.zip from https://developer.chrome.com/native-client/sdk/download and extracted it.
However, as I run the command "naclsdk list" as it is described on the download page, I got the following messages:
C:\Temp\nacl_sdk>naclsdk list
Traceback (most recent call last):
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 759, in
sys.exit(main(sys.argv[1:]))
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 752, in main
InvokeCommand(args)
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 741, in InvokeCommand
command(options, args[1:], config)
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 510, in Update
manifest = LoadManifestFromURLs([options.manifest_url] + config.GetSources())
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 238, in LoadManifestFromURLs
url_stream = UrlOpen(url)
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 114, in UrlOpen
return url_opener.open(request)
File "C:\python27\lib\urllib2.py", line 431, in open
response = self._open(req, data)
File "C:\python27\lib\urllib2.py", line 449, in _open
'_open', req)
File "C:\python27\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\python27\lib\urllib2.py", line 1240, in https_open
context=self._context)
TypeError: do_open() got an unexpected keyword argument 'context'
Traceback (most recent call last):
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 759, in
sys.exit(main(sys.argv[1:]))
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 752, in main
InvokeCommand(args)
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 741, in InvokeCommand
command(options, args[1:], config)
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 444, in List
manifest = LoadManifestFromURLs([options.manifest_url] + config.GetSources())
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 238, in LoadManifestFromURLs
url_stream = UrlOpen(url)
File "C:\Temp\nacl_sdk\sdk_tools\sdk_update_main.py", line 114, in UrlOpen
return url_opener.open(request)
File "C:\python27\lib\urllib2.py", line 431, in open
response = self._open(req, data)
File "C:\python27\lib\urllib2.py", line 449, in _open
'_open', req)
File "C:\python27\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\python27\lib\urllib2.py", line 1240, in https_open
context=self._context)
TypeError: do_open() got an unexpected keyword argument 'context'
In fact, no matter what command I use, it doesn't seem to work at all.
Does anyone know the solution?
having the same issue.
Found this link: http://forums.udacity.com/questions/100247273/urllib2-may-be-a-bug-to-find-the-appropriate-method-overload
Solved by editing python27/lib/urllib2.py...
Hope it helps!

SocksiPy fails when connecting throug Tor

Tried the 3 following methods to control Tor:
using TorCtl/urllib2: Python script Exception with Tor
using socks/httplib: http://www.youtube.com/watch?v=KDsmVH7eJCs
using socks/urllib2: Python urllib over TOR?
Each of them fails w/ same error (tried to make it as clear as possible):
Traceback (most recent call last):
File "tor.py", line 26, in <module>
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py"
line 127, in urlopen
return _opener.open(url, data, timeout)
line 404, in open
response = self._open(req, data)
line 422, in _open
'_open', req)
line 382, in _call_chain
result = func(*args)
line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
line 1181, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py"
line 973, in request
self._send_request(method, url, body, headers)
line 1007, in _send_request
self.endheaders(body)
line 969, in endheaders
self._send_output(message_body)
line 829, in _send_output
self.send(msg)
line 791, in send
self.connect()
line 772, in connect
self.timeout, self.source_address)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py"
line 562, in create_connection
sock.connect(sa)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/SocksiPy_branch-1.01-py2.7.egg/socks.py"
line 392, in connect
self.__negotiatesocks5(destpair[0],destpair[1])
line 199, in __negotiatesocks5
self.sendall("\x05\x01\x00")
line 165, in sendall
socket.socket.sendall(self, bytes)
... last error repeating a lot of times and then ...
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/SocksiPy_branch-1.01-py2.7.egg/socks.py", line 163, in sendall
if 'encode' in dir(bytes):
RuntimeError: maximum recursion depth exceeded while calling a Python object
Does anyone understand where it comes from?
For me it actually failed with socksipy-branch installed with pip.
However, it worked ok after I downloaded socks.py directly from http://socksipy.sourceforge.net/ to my working directory.
It looks like you are using the SocksiPy library. I had the same problem and got it fixed by installing this library again directly from https://code.google.com/p/socksipy-branch/. The first time I installed it via pip.

Socket timeout when using Buildout

We're using Buildout as a way to build our packages here, and we're consistently getting really frustrating timeouts when trying to download eggs by running bin/buildout:
We have no distributions for python-dateutil that satisfies 'python-dateutil>=1.5'.
While:
Installing python_section.
Getting distribution for 'python-dateutil>=1.5'.
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/vagrant/eggs/zc.buildout-2.1.1-py2.7.egg/zc/buildout/buildout.py", line 1921, in main
getattr(buildout, command)(args)
File "/vagrant/eggs/zc.buildout-2.1.1-py2.7.egg/zc/buildout/buildout.py", line 602, in install
installed_files = self[part]._call(recipe.install)
File "/vagrant/eggs/zc.buildout-2.1.1-py2.7.egg/zc/buildout/buildout.py", line 1356, in _call
return f()
File "/vagrant/eggs/zc.recipe.egg-2.0.0-py2.7.egg/zc/recipe/egg/egg.py", line 126, in install
reqs, ws = self.working_set()
File "/vagrant/eggs/zc.recipe.egg-2.0.0-py2.7.egg/zc/recipe/egg/egg.py", line 84, in working_set
allow_hosts=self.allow_hosts)
File "/vagrant/eggs/zc.buildout-2.1.1-py2.7.egg/zc/buildout/easy_install.py", line 782, in install
return installer.install(specs, working_set)
File "/vagrant/eggs/zc.buildout-2.1.1-py2.7.egg/zc/buildout/easy_install.py", line 626, in install
for dist in self._get_dist(requirement, ws):
File "/vagrant/eggs/zc.buildout-2.1.1-py2.7.egg/zc/buildout/easy_install.py", line 448, in _get_dist
dist, avail = self._satisfied(requirement)
File "/vagrant/eggs/zc.buildout-2.1.1-py2.7.egg/zc/buildout/easy_install.py", line 204, in _satisfied
return None, self._obtain(req, source)
File "/vagrant/eggs/zc.buildout-2.1.1-py2.7.egg/zc/buildout/easy_install.py", line 372, in _obtain
if index.obtain(requirement) is None:
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 340, in obtain
self.prescan(); self.find_packages(requirement)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 325, in find_packages
self.scan_url(self.index_url + requirement.unsafe_name+'/')
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 668, in scan_url
self.process_url(url, True)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 223, in process_url
page = self.process_index(url, page)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 300, in process_index
self.scan_url(new_url)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 668, in scan_url
self.process_url(url, True)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 201, in process_url
f = self.open_url(url, "Download error on %s: %%s -- Some packages may not be found!" % url)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 610, in open_url
return open_with_auth(url)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 753, in _socket_timeout
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/setuptools/package_index.py", line 779, in open_with_auth
fp = urllib2.urlopen(request)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.7/socket.py", line 447, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
Is there a way to prevent these timeouts? why do they happen?
The index server failed to respond, the connection timed out. This happens when PyPI is down, for example. By default, the index server is http://pypi.python.org/simple, but you can set your own with the [buildout] option index.
We use an egg proxy (set up with collective.eggproxy) to create a local cache automatically, to avoid such problems. Our buildouts use that server as the index, and the proxy will download any missing packages from PyPI and cache them locally. So, for any packages that we have used in the past, we can continue to serve even when PyPI is (temporarily) down.

Categories