We Keep Coding

Python Geocoders Not Working

I am trying to access geolocation of addresses input by the User in my Django App.
The App was working fine with pygeocoder. Suddenly, the app has started giving problems.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ishaan/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pygeocoder.py", line 129, in geocode
return GeocoderResult(Geocoder.get_data(params=params))
File "/home/ishaan/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pygeocoder.py", line 204, in get_data
response = session.send(request.prepare())
File "/home/ishaan/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/home/ishaan/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/home/ishaan/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/home/ishaan/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/home/ishaan/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type Error)
I am unable to unserstand this problem, and have a close to no knowledge about SSL and its errors.
I dont know, how relevant it is, but the app was working until I installed Redis Server for another app in the same project. I have uninstalled it, but it is still not working.
Lastly, No other Geocoder is working - Every geocoder gives the same error. I tried to use GeoPy, geocoder 1.6.4
Thanks in advance

check Python requests with charles proxy for HTTPS

I want to debug some python requests using charles proxy.
I need to include the certificate for charles on the call, but is not working
import requests
endpoint_url = 'https://www.httpsnow.org/'
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
print "empexo"
print r
I have added the https address on Charles,
I get on Charles:
SSLHandshake: Remote host closed connection during handshake
and on python the log with error
empexo
Traceback (most recent call last):
File "/Users/iosdev/DopPy/GetCelebs.py", line 15, in <module>
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 49, in request
response = session.request(method=method, url=url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 336265225] _ssl.c:341: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Process finished with exit code 1

I found this thread while I was troubleshooting a similar issue. In the scenario I ran into the cert argument was being used to define the path to a ".crt" file when the verify argument should have been used instead.
The correct usage ended up looking like:
requests.get(endpoint_url, verify='/path/to/file.crt')
See Requests' documentation for more details: https://2.python-requests.org/en/v1.1.0/user/advanced/#ssl-cert-verification
As an aside, I find employing Request's ability to specify the path to a ".crt" via the REQUESTS_CA_BUNDLE environmental variable more effective when using Charles Proxy for local debugging.
Running something like the following in shell saves having to specify the path to Charles' ".crt" for every Requests call:
REQUESTS_CA_BUNDLE=/path/to/file.crt
export REQUESTS_CA_BUNDLE

Getting error when using praw to login to reddit

I've seen a couple of questions that already asked this but there were no responses, so I'll give it a try. When I use the following code:
import praw, time
r = praw.Reddit(user_agent="Bot experiment by redacted")
r.login('redacted', 'redacted')
I get a connection error that has the following traceback:
Traceback (most recent call last):
File "redacted", line 5, in <module>
r.login('redacted', 'redacted')
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/__init__.py", line 1263, in login
self.request_json(self.config['login'], data=data)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/decorators.py", line 161, in wrapped
return_value = function(reddit_session, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/__init__.py", line 519, in request_json
response = self._request(url, params, data)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/__init__.py", line 383, in _request
_raise_response_exceptions(response)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/internal.py", line 172, in _raise_response_exceptions
response.raise_for_status()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/models.py", line 831, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden
I have tried this from python 3.4 and 2.7, I've tried running from IDLE and from the terminal. I've tried leaving my username and password out and logging in when prompted. I've tried from my Mac in my hotel room and a Windows machine from work and I get the same error everytime. I've tried from my bot account that I just made and my normal account. Does anyone have any ideas?

The issue was that I had the word 'bot' in my user_agent string. After it was removed, there were no problems.

Paster does not work when installing DB

I have tried to use the below forum to item to fix the problem but it did not seam to work for me
https://stackoverflow.com/questions/21955234/ckan-install-paster-error
Amazingly I got the same issue when I tried to install CKAN on windows.
paster db init -c XXXX/development.ini not working for CKAN-command 'db' not know
This time I am trying to install CKAN on Ubuntu 12.04 (actually 12.04.5 as I couldn't get 12.0.4) as instructed in
http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html
I am having to install everything using a PROXY
I have added the password to the SQL Chemistry and the Development.ini does exist. This is my error (below)
Is this a proxy issue? I have used the chmod to change the access to the ini file as the other forum recommended. I also set the virtual path. The database base does exist as I check it.
:
(default)root#UbuntaDataServer:/usr/lib/ckan/default/src/ckan# paster db init -c /etc/ckan/default/development.ini
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/root/ckan/lib/default/src/ckan/ckan/lib/cli.py", line 156, in command
self._load_config()
File "/root/ckan/lib/default/src/ckan/ckan/lib/cli.py", line 98, in _load_config
load_environment(conf.global_conf, conf.local_conf)
File "/root/ckan/lib/default/src/ckan/ckan/config/environment.py", line 232, in load_environment
p.load_all(config)
File "/root/ckan/lib/default/src/ckan/ckan/plugins/core.py", line 124, in load_all
unload_all()
File "/root/ckan/lib/default/src/ckan/ckan/plugins/core.py", line 182, in unload_all
unload(*reversed(_PLUGINS))
File "/root/ckan/lib/default/src/ckan/ckan/plugins/core.py", line 210, in unload
plugins_update()
File "/root/ckan/lib/default/src/ckan/ckan/plugins/core.py", line 116, in plugins_update
environment.update_config()
File "/root/ckan/lib/default/src/ckan/ckan/config/environment.py", line 270, in update_config
search.check_solr_schema_version()
File "/root/ckan/lib/default/src/ckan/ckan/lib/search/__init__.py", line 291, in check_solr_schema_version
res = urllib2.urlopen(req)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 503: Service Unavailable

This part of the stacktrace:
File "/root/ckan/lib/default/src/ckan/ckan/lib/search/init.py", line 291, in check_solr_schema_version
res = urllib2.urlopen(req)
Suggests that there is a problem connecting to Solr. You should make sure solr is running, that you can connect to it, and that the setting in your .ini file for the location and port that solr is running on is correct.

This is not the complete answer. Maybe close.
This is what I see on http:||localhost|solr|
Solr Admin (ckan)
UbuntaDataServer:8983
cwd=/var/cache/jetty/tmp SolrHome=/usr/share/solr/
This is what is running on the URL. I assume this is close or correct?
Any more suggestions?

Using CKAN 2.2 I had the same problem with proxies that require authentication
If you are installing CKAN from sources, I suggest to move to 2.2.1 version (or newer).
In these versions I found no issues with auth proxy.
Anyway, if you're bound to a specific, older version of CKAN, you can manually add a proxy handler.
First of all, set your http_proxy env vars (both uppercase and lowercase)
Now you can edit the file ckan/ckan/lib/search/__init__.py and get your hands dirty.
We need to declare a handle_proxy() function:
import os
def handle_proxy():
proxy_settings = dict()
for k,v in os.environ.items():
if k.rfind('_proxy') > -1:
proxy_settings[k] = v
proxy_handler = urllib2.ProxyHandler(proxy_settings)
opener = urllib2.build_opener(proxy_handler)
urllib2.install_opener(opener)
Now we can call it in the check_solr_schema_version() function just before sending the request.
Replace
res = urllib2.urlopen(req)
with
handle_proxy()
res = urllib2.urlopen(req)
NOTE: this is a temporary workaround, just in case upgrading to newer versions (i currently use the 2.2.2 branch) does not fix the problem for you. I wouldn't suggest it for a production environment :)

I found another answer, if the above does not work, try:
Install this again:
sudo -E apt-get install python-pastescript
. /usr/lib/ckan/default/bin/activate
cd /usr/lib/ckan/default/src/ckan
paster make-config ckan /etc/ckan/default/development.ini
Change SOLR to your IP number and not localhost
paster db init -c /etc/ckan/default/development.ini
Hope that fixes your problem

What is causing urllib2.urlopen() to connect via proxy?

I'm trying to read a URL within our corporate network. Spesifically the server I'm contacting is in one office and the client PC is in another:
print(urlopen(r"http://london.mycompany/mydir/").read())
Whenever I run this function I get:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Python24\lib\urllib2.py", line 130, in urlopen
return _opener.open(url, data)
File "C:\Python24\lib\urllib2.py", line 364, in open
response = meth(req, response)
File "C:\Python24\lib\urllib2.py", line 471, in http_response
response = self.parent.error(
File "C:\Python24\lib\urllib2.py", line 402, in error
return self._call_chain(*args)
File "C:\Python24\lib\urllib2.py", line 337, in _call_chain
result = func(*args)
File "C:\Python24\lib\urllib2.py", line 480, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 407: Proxy Authentication Required
The odd thing is that there's no firewall between these two computers - for some reason url has decided to connect to the web-server via the proxy which we'd normally use to connect to content outside the company, and in this case that's failing because I've not authenticated it.
I'm pretty sure that the fault occurs within the client PC: I did a nslookup and a ping to the server to confirm that there's a connection between the two computers, however when I watch the transaction using TCPView for Windows I can see that the python.exe process is connecting to a completely different server (yes, the proxy!).
So what could be causing this? Note that the os.environ["http_proxy"] variable is NOT set - this variable is often used to make urllib connect via a proxy server. That's not the case here. Could there be something else which might have the same effect?
FYI, Running Python 2.4.4 on Windows XP 32bit in a very locked-down corporate environment.

It reads from the system settings. Use urllib.FancyURLOpener:
opener = urllib.FancyURLopener({})
f = opener.open("http://london.mycompany/mydir/")
f.read()