I'm trying to upload a file to Google via the Adwords API and I'm using the script provided by Google here:https://github.com/googleads/googleads-python-lib
I'm getting the following error after a few minutes. I've tried this on a smaller file and it works. Not sure why it is breaking here. Any ideas?
File "./g_leads.sh", line 71, in <module>
main(adwords_client, EMAILS)
File "./g_leads.sh", line 43, in main
response = user_list_service.mutateMembers([mutate_members_operation])
File "/opt/stateadm/.local/lib/python2.7/site-packages/googleads/common.py", line 296, in MakeSoapRequest
for arg in args])
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/client.py", line 521, in __call__
return client.invoke(args, kwargs)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/client.py", line 581, in invoke
result = self.send(soapenv)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/client.py", line 613, in send
reply = self.options.transport.send(request)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/transport/https.py", line 66, in send
return HttpTransport.send(self, request)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/transport/http.py", line 82, in send
fp = self.u2open(u2request)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/transport/http.py", line 132, in u2open
return url.open(u2request, timeout=tm)
File "/opt/python/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/opt/python/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/opt/python/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/opt/python/lib/python2.7/urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/opt/python/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 32] Broken pipe>
Errno 32 Broken Pipe means the remote server closed the socket. I would guess the transfer is taking too long because the API documentation shows that they would return an error if the file itself is too big.
Conversely, this may be a intermittent issue at Google per this old thread.
Related
I've been able to connect from a local script directly to the Google App Engine (GAE) ndb store on the server as described in this article.
But I'm trying to do the same thing to access my local dev server but am not able to. My dev API server runs at http://localhost:42020 but when I try to connect using the following command:
$ remote_api_shell.py -s http://localhost:42020
I get the following error:
Traceback (most recent call last):
File "/home/myuser/google_appengine/remote_api_shell.py", line 127, in <module>
run_file(__file__, globals())
File "/home/myuser/google_appengine/remote_api_shell.py", line 123, in run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/home/myuser/google_appengine/google/appengine/tools/remote_api_shell.py", line 150, in <module>
main(sys.argv)
File "/home/myuser/google_appengine/google/appengine/tools/remote_api_shell.py", line 146, in main
appengine_rpc.HttpRpcServer)
File "/home/myuser/google_appengine/google/appengine/tools/remote_api_shell.py", line 74, in remote_api_shell
rpc_server_factory=rpc_server_factory)
File "/home/myuser/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 874, in ConfigureRemoteApi
app_id = GetRemoteAppIdFromServer(server, path, rtok)
File "/home/myuser/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 569, in GetRemoteAppIdFromServer
response = server.Send(path, payload=None, **urlargs)
File "/home/myuser/google_appengine/google/appengine/tools/appengine_rpc.py", line 424, in Send
f = self.opener.open(req)
File "/usr/lib/python2.7/urllib2.py", line 401, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 419, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1211, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1181, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno -2] Name or service not known>
Is it not possible to connect locally?
Looks like I was doing it wrong. This answer solved my problem.
In short, I had to connect to the regular web server for it to work without the leading http. This asks for my email and password and once I enter that, it works!
$ remote_api_shell.py -s localhost:8080
I'm running many celery tasks (20,000) using gevent for the pool (also monkey patching all). Each of these tasks hit 3rd party services like adwords to pull data.
I keep having tasks fail because of underlying SSL errors. Below are the stack-traces from a few of the exceptions (in no particular order, these are failures from separate tasks). I also get WantWriteError and ZeroReturnError occasionally but the EOF error seems to come up the most.
These errors happen while using different client libraries like googleads (suds library for soap communication) as well as requests and elasticsearch. I'm guessing some of these libraries use urllib3 while others use urllib2 etc.
There has been a lot of info on the EOF issue and forcing TLSv1 but I can't seem to find a resolution that works.
I'm not sure if I'm running too many requests at once, if somethings blocking or what; any help would be greatly appreciated, I'm pulling my hair out over this one.
Traceback (most recent call last):
...
File "/srv/reporting/src/reporting/stats/adwords/client.py", line 58, in _awql_report
downloader = self._get_client(client_id).GetReportDownloader(version=self.REPORT_DOWNLOADER_VERSION)
File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 283, in GetReportDownloader
return ReportDownloader(self, version, server)
File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 400, in __init__
proxy=proxy_option, cache=self._adwords_client.cache).wsdl.schema
File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 115, in __init__
self.wsdl = reader.open(url)
File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 150, in open
d = self.fn(url, self.options)
File "/usr/local/lib/python2.7/dist-packages/suds/wsdl.py", line 136, in __init__
d = reader.open(url)
File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 74, in open
d = self.download(url)
File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 92, in download
fp = self.options.transport.open(Request(url))
File "/usr/local/lib/python2.7/dist-packages/suds/transport/https.py", line 62, in open
return HttpTransport.open(self, request)
File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 67, in open
return self.u2open(u2request)
File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 132, in u2open
return url.open(u2request, timeout=tm)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1216, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
Traceback (most recent call last):
...
File "/srv/reporting/src/reporting/stats/analytics/client.py", line 57, in get_access_token
response = requests.post('https://accounts.google.com/o/oauth2/token', data)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')
Traceback (most recent call last):
...
self.es.index(index=self.INDICE, doc_type=self.ROOT_CLASS.__name__, body=self.export(obj), id=obj.id)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 213, in index
_make_path(index, doc_type, id), params=params, body=body)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 284, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 44, in perform_request
response = self.session.request(method, url, data=body, timeout=timeout or self.timeout)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
timeout=timeout
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
httplib_response = conn.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline()
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 273, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 995, in recv
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
raise ZeroReturnError()
ZeroReturnError
So let's break this down by each traceback block. The first ends with:
File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
This is coming from urllib2. The fact that this receives an EOF makes me think that the server closed the connection while you were waiting for that "thread" to read from the socket again. You might want to use more time.sleep(0) to yield to gevent.
The second traceback comes from requests:
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')
The [Errno bad handshake] would make me tend to think this is a problem establishing the connection which could be caused by an unexpected EOF. Is that caused by using gevent? I'm uncertain.
The final traceback is definitely from requests as well but it also is coming out of PyOpenSSL and isn't being caught by urllib3 or requests.
File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
raise ZeroReturnError()
ZeroReturnError
I did some searching and found that "According to the pyOpenSSL docs ZeroReturnError means that the SSL connection has been closed cleanly." This says to me that the server again closed the connection because you took to long to read anything from the socket.
In short, I think you need to explicitly yield more often just to ensure that these socket problems don't arise. That's just a guess though, so take it with a grain of salt.
I have my selenium setup in Ubuntu and have written my own framework for Python.
I am running into following error whenever I run any test cases using the framework.
However I could run the same selenium cases successfully without using my framework.
Things I've tried, but dint work:
Reinstalled selenium however it dint help.
Restart the VM (I can connect to the network anyways)
Could anyone advice me on this error ?
ERROR: setUpClass (__main__.AuraAccountCreation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "../../../tiselenium/core/testcase.py", line 100, in setUpClass
self.driver = browser_mapping[driver](**args)
File "/usr/local/lib/python2.7/dist-packages/selenium-2.37.2-py2.7.egg/selenium /webdriver/remote/webdriver.py", line 71, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium-2.37.2-py2.7.egg/selenium/webdriver/remote/webdriver.py", line 113, in start_session
'desiredCapabilities': desired_capabilities,
File "/usr/local/lib/python2.7/dist-packages/selenium-2.37.2-py2.7.egg/selenium/webdriver/remote/webdriver.py", line 162, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium-2.37.2-py2.7.egg/selenium/webdriver/remote/remote_connection.py", line 355, in execute
return self._request(url, method=command_info[0], data=data)
File "/usr/local/lib/python2.7/dist-packages/selenium-2.37.2-py2.7.egg/selenium/webdriver/remote/remote_connection.py", line 402, in _request
response = opener.open(request)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>
I'm working my way through The Programming Historian 2, a self-tutorial in coding for historians focusing on HTML and Python. I am attempting to complete the lesson Working with Files and Web Pages but am stuck on the Opening URLs with Python unit. I am running the following program:
# open-webpage.py
import urllib2
url = 'http://www.oldbaileyonline.org/print.jsp?div=t17800628-33'
response = urllib2.urlopen(url)
webContent = response.read()
print webContent[0:300]
Every time I run the program Komodo Edit 7 returns the following error message:
Traceback (most recent call last):
File "open-webpage.py", line 7, in <module>
response = urllib2.urlopen(url)
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 418, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 378, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Python27\lib\urllib2.py", line 1180, in do_open
r = h.getresponse(buffering=True)
File "C:\Python27\lib\httplib.py", line 1030, in getresponse
response.begin()
File "C:\Python27\lib\httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "C:\Python27\lib\httplib.py", line 365, in _read_status
line = self.fp.readline()
File "C:\Python27\lib\socket.py", line 447, in readline
data = self._sock.recv(self._rbufsize)
socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host
I have attempted the program with a number of different urls, always with the same result. The guys at Komodo think the problem is to do with my university's firewall, because I access the internet through my university's proxy. The tech people here told me to change my default browser from RockMelt (chromium) to IE, because only IE is fully supported. I did so with no change and they have no other suggestions.
Can anyone suggest an alternate explanation for the error or a way to address the firewall problem? Thanks.
I have a problem while deploying application to GAE. I use ubuntu. When I type command to update application this error occurs:
2011-07-22 20:13:28,598 ERROR appcfg.py:2064 An unexpected error occurred. Aborting.
Traceback (most recent call last):
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 2005, in DoUpload
missing_files = self.Begin()
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 1674, in Begin
self.Send('/api/appversion/create', payload=self.config.ToYAML())
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 1632, in Send
return self.rpcserver.Send(url, payload=payload, **self.params)
File "/home/grzegorz/google_appengine/google/appengine/tools/appengine_rpc.py", line 365, in Send
f = self.opener.open(req)
File "/usr/lib/python2.7/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1193, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/home/grzegorz/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py", line 367, in do_open
raise url_error
URLError: <urlopen error [Errno 110] Connection timed out>
Traceback (most recent call last):
File "./google_appengine/appcfg.py", line 76, in <module>
run_file(__file__, globals())
File "./google_appengine/appcfg.py", line 72, in run_file
execfile(script_path, globals_)
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 3708, in <module>
main(sys.argv)
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 3699, in main
result = AppCfgApp(argv).Run()
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 2345, in Run
self.action(self)
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 3484, in __call__
return method()
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 2745, in Update
app_summary = self.UpdateVersion(rpcserver, self.basepath, appyaml)
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 2734, in UpdateVersion
lambda path: self.opener(os.path.join(basepath, path), 'rb'))
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 2005, in DoUpload
missing_files = self.Begin()
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 1674, in Begin
self.Send('/api/appversion/create', payload=self.config.ToYAML())
File "/home/grzegorz/google_appengine/google/appengine/tools/appcfg.py", line 1632, in Send
return self.rpcserver.Send(url, payload=payload, **self.params)
File "/home/grzegorz/google_appengine/google/appengine/tools/appengine_rpc.py", line 365, in Send
f = self.opener.open(req)
File "/usr/lib/python2.7/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1193, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/home/grzegorz/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py", line 367, in do_open
raise url_error
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>
I tried to figure out why this happens, especially that it didn't happen to me before.
Anything that I've done about it just didn't work. What's weird is that if I keep trying, then after about 20-30 attemps - it just magically deploys..
In my opinion there might be some Wifi problems. For example when I start xChat - it can't connect to server. I think that this problems might be connected.
Nevertheless I can't figure out anything, so I would be thankful for any help.