We Keep Coding

Google App Engine (python) - Connecting to local datastore via python script

I've been able to connect from a local script directly to the Google App Engine (GAE) ndb store on the server as described in this article.
But I'm trying to do the same thing to access my local dev server but am not able to. My dev API server runs at http://localhost:42020 but when I try to connect using the following command:
$ remote_api_shell.py -s http://localhost:42020
I get the following error:
Traceback (most recent call last):
File "/home/myuser/google_appengine/remote_api_shell.py", line 127, in <module>
run_file(__file__, globals())
File "/home/myuser/google_appengine/remote_api_shell.py", line 123, in run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/home/myuser/google_appengine/google/appengine/tools/remote_api_shell.py", line 150, in <module>
main(sys.argv)
File "/home/myuser/google_appengine/google/appengine/tools/remote_api_shell.py", line 146, in main
appengine_rpc.HttpRpcServer)
File "/home/myuser/google_appengine/google/appengine/tools/remote_api_shell.py", line 74, in remote_api_shell
rpc_server_factory=rpc_server_factory)
File "/home/myuser/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 874, in ConfigureRemoteApi
app_id = GetRemoteAppIdFromServer(server, path, rtok)
File "/home/myuser/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 569, in GetRemoteAppIdFromServer
response = server.Send(path, payload=None, **urlargs)
File "/home/myuser/google_appengine/google/appengine/tools/appengine_rpc.py", line 424, in Send
f = self.opener.open(req)
File "/usr/lib/python2.7/urllib2.py", line 401, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 419, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1211, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1181, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno -2] Name or service not known>
Is it not possible to connect locally?

Looks like I was doing it wrong. This answer solved my problem.
In short, I had to connect to the regular web server for it to work without the leading http. This asks for my email and password and once I enter that, it works!
$ remote_api_shell.py -s localhost:8080

GAE SDK 1.8.3 Python: InvalidCertificateException on deploy

I keep getting the below error every time I do a deploy after updating GAE to a new version (1.8.3 in this case).
I have read other questions about this and the going recommendation is to delete /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/cacerts/*cacerts.txt
Does anyone know of a better fix than deleting these files after every update?
Note: I have the Python SSL module installed
*** Running appcfg.py with the following flags:
--no_cookies --email=joshainglis#gmail.com --passin update
10:52 AM Host: appengine.google.com
10:52 AM Application: josha-search-py-demo; version: 1
Traceback (most recent call last):
File "google_appengine/appcfg.py", line 171, in <module>
run_file(__file__, globals())
File "google_appengine/appcfg.py", line 167, in run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4282, in <module>
main(sys.argv)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4273, in main
result = AppCfgApp(argv).Run()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 2409, in Run
self.action(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4003, in __call__
return method()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 2987, in Update
updatecheck.CheckForUpdates()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/sdk_update_checker.py", line 259, in CheckForUpdates
runtime=runtime))
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 393, in Send
f = self.opener.open(req)
File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py", line 383, in do_open
url_error.reason.args[1])
fancy_urllib.InvalidCertificateException: Host appengine.google.com returned an invalid certificate (_ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed):
To learn more, see http://code.google.com/appengine/kb/general.html#rpcssl
If deploy fails you might need to 'rollback' manually.
The "Make Symlinks..." menu option can help with command-line work.
*** appcfg.py has finished with exit code 1 ***

There is a copy of fancy_urllib inside fancy_urllib where the __init__.py is with InvalidCertificateException class definition.
Just copy .../fancy_urllib/fancy_urllib/__init__.py to .../fancy_urllib/__init__.py

Unable to connect to secure website using mechanize in Python

I'm trying to open a secure (https) website using mechanize library in Python. When I try to access the website, the server closes the connection and exception BadStatusLine is raised.
I have tried to modify the headers using the addheaders property, but no response.
import mechanize
br = mechanize.Browser()
print 'opening page ...'
resp = br.open('https://onlineservices.tin.nsdl.com/etaxnew/tdsnontds.jsp') #this one works fine
print 'ok'
print 'opening page 2 ...'
resp = br.open('https://incometaxindiaefiling.gov.in/portal/index.do') #exception raised
print 'ok'
Exception:
Traceback (most recent call last): File
pydev_imports.execfile(file, globals, locals) #execute the script File "Z:\pyTax\app_test.py", line 22, in
resp=br.open('https://incometaxindiaefiling.gov.in/portal/index.do')
File "build\bdist.win32\egg\mechanize_mechanize.py", line 203, in
open File "build\bdist.win32\egg\mechanize_mechanize.py", line 230,
in _mech_open File "build\bdist.win32\egg\mechanize_opener.py",
line 188, in open File "build\bdist.win32\egg\mechanize_http.py",
line 316, in http_request File
"build\bdist.win32\egg\mechanize_http.py", line 242, in read File
"build\bdist.win32\egg\mechanize_mechanize.py", line 203, in open
File "build\bdist.win32\egg\mechanize_mechanize.py", line 230, in
_mech_open File "build\bdist.win32\egg\mechanize_opener.py", line 193, in open File
"build\bdist.win32\egg\mechanize_urllib2_fork.py", line 344, in _open
File "build\bdist.win32\egg\mechanize_urllib2_fork.py", line 332, in
_call_chain File "build\bdist.win32\egg\mechanize_urllib2_fork.py", line 1170, in https_open File
"build\bdist.win32\egg\mechanize_urllib2_fork.py", line 1116, in
do_open File "D:\Python27\lib\httplib.py", line 1031, in getresponse
response.begin() File "D:\Python27\lib\httplib.py", line 407, in begin
version, status, reason = self._read_status() File "D:\Python27\lib\httplib.py", line 371, in _read_status
raise BadStatusLine(line) httplib.BadStatusLine: ''

httplib.BadStatusLineis s a subclass of HTTPException. Raised if a server responds with a HTTP status code that we don’t understand. That's whats causing your problem. I am not entirely sure about the fixup though, as your code works fine on my computer.

Google App Engine throws this InvalidURL("nonnumeric port: '%s'" % host[i+1:]) exception when I try to deploy. How do I fix it?

When I click the Deploy button for my project, it gives me the following error: File "C:\Python27\lib\httplib.py", line 721, in _set_hostport
raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
httplib.InvalidURL: nonnumeric port: 'port'.
I've successfully deployed this project before, so I tried reverting the files so that I would have the same files when it worked properly. No dice, it seems. That was just a jab in the dark, though -- the full trace shows this starting in Google App Engine's files, not mine.
Anyone know how I can deploy my project?
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 171, in <module>
run_file(__file__, globals())
File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 167, in run_file
execfile(script_path, globals_)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 4191, in <module>
main(sys.argv)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 4182, in main
result = AppCfgApp(argv).Run()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 2579, in Run
self.action(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 3927, in __call__
return method()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 3040, in Update
updatecheck.CheckForUpdates()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 596, in CheckForUpdates
runtime=self.config.runtime)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appengine_rpc.py", line 383, in Send
f = self.opener.open(req)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 418, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 378, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "C:\Program Files (x86)\Google\google_appengine\lib\fancy_urllib\fancy_urllib\__init__.py", line 370, in do_open
req)
File "C:\Python27\lib\urllib2.py", line 1146, in do_open
h = http_class(host, timeout=req.timeout) # will parse host:port
File "C:\Program Files (x86)\Google\google_appengine\lib\fancy_urllib\fancy_urllib\__init__.py", line 78, in __init__
httplib.HTTPSConnection.__init__(self, *args, **kwargs)
File "C:\Python27\lib\httplib.py", line 1149, in __init__
source_address)
File "C:\Python27\lib\httplib.py", line 693, in __init__
self._set_hostport(host, port)
File "C:\Python27\lib\httplib.py", line 721, in _set_hostport
raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
httplib.InvalidURL: nonnumeric port: 'port'
2012-10-22 16:55:29 (Process exited with code 1)
You can close this window now.

I'll preface this by saying I'm not familiar with the GUI, so feel free to completely disregard :).
The error seems to indicate that you are trying to specify a particular port for deployment, and furthermore, the value of that setting is the word 'port'. When using the commnand line with the dev server, port is an argument you can use to specify a particular port to use; my best guess is that this keyword parameter is still defined (as port) when you are trying to deploy. Is there a section where you can specify command-line flags? Is it possible that port is defined there when deploying?

I've resolved this by completely uninstalling and reinstalling both Python 2.7 and GAE. This complete wipe should include the deletion of any environment variables that have been set since your usage of GAE.
In my case, I think it was particularly the creation of environment proxy variables. A complete wipe of Python and GAE resolved this.

Problem with python urllib

I'm getting an error when ever I try to pull down a web page with urllib.urlopen. I've disabled windows firewall and my AV so its not that. I can access the pages in my browser. I even reinstalled python to rule out it being a broken urllib. Any help would be greatly appreciated.
>>> import urllib
>>> h = urllib.urlopen("http://www.google.com").read()
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
h = urllib.urlopen("http://www.google.com").read()
File "C:\Python26\lib\urllib.py", line 86, in urlopen
return opener.open(url)
File "C:\Python26\lib\urllib.py", line 205, in open
return getattr(self, name)(url)
File "C:\Python26\lib\urllib.py", line 344, in open_http
h.endheaders()
File "C:\Python26\lib\httplib.py", line 904, in endheaders
self._send_output()
File "C:\Python26\lib\httplib.py", line 776, in _send_output
self.send(msg)
File "C:\Python26\lib\httplib.py", line 735, in send
self.connect()
File "C:\Python26\lib\httplib.py", line 716, in connect
self.timeout)
File "C:\Python26\lib\socket.py", line 514, in create_connection
raise error, msg
IOError: [Errno socket error] [Errno 10061] No connection could be made because the target machine actively refused it
>>>

this could be the case:
Just found the problem I had set a
proxy through internet options, that
proxy went offline, and so did my
python shell.

urllib is working just fine.
Try using ethereal (or some similar network sniffer) on your box to determine if the denial coming from your machine or a machine beyond.