I want to use mwclient to edit an internal wiki behind HTTPS. This page suggests that you can pass in a host tuple like this:
site = mwclient.Site(('https','wiki.whatever.com'))
However I don't see how to authenticate with Apache. Do you need to set up a urllib2 opener handler with the authentication?
A solution that works with python-wikitools would alternatively be welcome, but also has a lack of https examples.
Update:
Perhaps it uses the same user ID and password as you're using to login to the wiki? I still get an error in that case though:
Traceback (most recent call last):
File "C:\ddc\DDC_Toolbox\python_root\apps\create_new_project\trunk\create_new_project.py", line 36, in <module>
site = mwclient.Site(('https',url))
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 92, in __init__
self.site_init()
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 100, in site_init
siprop = 'general|namespaces', uiprop = 'groups|rights')
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 165, in api
info = self.raw_api(action, **kwargs)
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 248, in raw_api
json_data = self.raw_call('api', data).read()
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 223, in raw_call
url, data = data, headers = headers)
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\http.py", line 226, in post
path, headers, data)
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\http.py", line 161, in post
return self.request('POST', host, path, headers, data)
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\http.py", line 152, in request
raise errors.HTTPStatusError, (res.status, res)
mwclient.errors.HTTPStatusError: (401, <httplib.HTTPResponse instance at 0x0241C0D0>)
Well, mwclient doesn't use urllib2, so installing an urllib2 AuthHandler won't do anything.
It's using httplib, and from a quick look at the source, there seems to be no way to add your own headers to the requests sent by the library, so adding a custom Authorization header won't work without modifying the library itself.
python-wikitools on the other hand uses urllib2, but instantiates it's own opener, and to be able to add an auth handler to that, you'd also need to modify the library-
So, unfortunately both apis don't allow you to use authentication out of the box.
I've been able to add HTTPS authentication to python-wikitools. I'll see about getting a patch uploaded to the maintainers of the package.
Related
I ask this question having run out of ideas - I assumed I was doing something stupid but even if I am I can't figure out what... I'm new to python having drifted away from coding 10 years ago but I was reasonably proficient once :-D I've written a python script to query a google CSE and search for relevant items, and the idea is that it will check a Podio materials list to see if they are already listed there. So at the moment all I want to do is authenticate with Podio and grab items so I can query them in my app.
I'm trying to connect to the Podio API using the prescribed method in the docs (and in questions here, and everywhere else I have looked)
At this point literally all this script is doing is this except for the credentials replaced with the actual strings, just in case I was missing some python variable madness.
import pypodio2
from pypodio2.api import OAuthClient
c = OAuthClient(
"<myappID>",
'<myAPIkey>',
'<myusername>',
'<mypassword>'
)
I'm using pythonanywhere as a sandbox and I am running the script from a bash console, I get this
16:26 ~/mysite $ python apitest.py
Traceback (most recent call last):
File "apitest.py", line 15, in <module>
'Splat_100'
File "/home/Trebuchet/.local/lib/python2.7/site-packages/pypodio2/api.py", line 13, in OAuthClient
api_key, api_secret, domain)
File "/home/Trebuchet/.local/lib/python2.7/site-packages/pypodio2/transport.py", line 41, in __init__
urlencode(body), headers=headers)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 2135, in request
cachekey,
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1796, in _request
conn, request_uri, method, body, headers
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1701, in _conn_request
conn.connect()
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1358, in connect
sock.connect((self.host, self.port) + sockaddr[:2])
File "/usr/local/lib/python2.7/dist-packages/httplib2/socks.py", line 496, in connect
self.__negotiatehttp(destpair[0], destpair[1])
File "/usr/local/lib/python2.7/dist-packages/httplib2/socks.py", line 457, in __negotiatehttp
raise HTTPError((statuscode, statusline[2]))
httplib2.socks.HTTPError: (403, 'Forbidden'
The password, key etc are correct. The username is correct. The clientID is correct. So I must be missing something that is obvious to people used to working with APIs. Mustn't I? Thanks :)
Free accounts on PythonAnywhere can only access http(s) sites that are on our whitelist. If the site you're using has a publicly documented API, send a link to it to PythonAnywhere support (support#pythonanywhere.com) and we'll consider add it to the whitelist.
So I'm running my google endpoint locally with dev_appserver.py.
I use the API explorer to test the application.
The code I'm using to create the Service, so I can call the API is the following:
from apiclient.discovery import build
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = build('speech', 'v1beta1', credentials=credentials)
I receive an SSL error (Invalid and/or missing SSL certificate), even though when I access the stated URL via browser it works fine (that is, the green padlock shows up).
I'm not sure what changed, but this was working fine not long ago.
I tried to disable SSL checking, but was unable to.
Full logs below:
INFO 2017-01-02 03:12:02,724 discovery.py:267] URL being requested: GET https://www.googleapis.com/discovery/v1/apis/speech/v1beta1/rest?userIp=0.2.0.3
ERROR 2017-01-02 03:12:03,022 wsgi.py:263]
Traceback (most recent call last):
File "/home/vini/opt/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/home/vini/opt/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/home/vini/opt/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/mnt/b117/home/vini/udacity/cerci-endpoint/api.py", line 28, in <module>
service = build('speech', 'v1beta1', credentials=credentials)
File "/mnt/b117/home/vini/udacity/cerci-endpoint/lib/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/mnt/b117/home/vini/udacity/cerci-endpoint/lib/googleapiclient/discovery.py", line 222, in build
cache)
File "/mnt/b117/home/vini/udacity/cerci-endpoint/lib/googleapiclient/discovery.py", line 269, in _retrieve_discovery_doc
resp, content = http.request(actual_url)
File "/mnt/b117/home/vini/udacity/cerci-endpoint/lib/httplib2/__init__.py", line 1609, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/mnt/b117/home/vini/udacity/cerci-endpoint/lib/httplib2/__init__.py", line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/mnt/b117/home/vini/udacity/cerci-endpoint/lib/httplib2/__init__.py", line 1307, in _conn_request
response = conn.getresponse()
File "/home/vini/opt/google-cloud-sdk/platform/google_appengine/google/appengine/dist27/gae_override/httplib.py", line 532, in getresponse
raise HTTPException(str(e))
HTTPException: Invalid and/or missing SSL certificate for URL: https://www.googleapis.com/discovery/v1/apis/speech/v1beta1/rest?userIp=0.2.0.3
Any ideas what could be causing this problem?
Do I have to "install" or update the SSL certificates used by python?
According to App Engine issue 13477 it seems that some of the certificates found in urlfetch_cacerts.txt that is included in the App Engine Python SDK / gcloud-sdk expired 2017-01-01.
As a temporary workaround, you can replace the contents of <your-cloud-sdk-path>/platform/google_appengine/lib/cacerts/urlfetch_cacerts.txt with https://curl.haxx.se/ca/cacert.pem
To build on the answer by #danielx for those on macOS, this is what worked for me. The path to the certificates for me was:
/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/cacerts/urlfetch_cacerts.txt
To update it, I used the following steps:
cd /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/cacerts
mv urlfetch_cacerts.txt urlfetch_cacerts.bup
curl -o urlfetch_cacerts.txt -k https://curl.haxx.se/ca/cacert.pem
If you don't have curl installed, you can manually download the certificates an move them to the folder above.
Don't forget to restart the App Engine dev server if it is already running.
Got this error on local dev environment as recently as Aug 2017. The fix is to update all urlfetch calls and force validation of the certs:
urlfetch.fetch(url=url, validate_certificate=True)
Did not have to touch the gcloud certs (MacOS). See Issuing an HTTPS request.
from amazon.api import AmazonAPI
AMAZON_ACCESS_KEY = "A******************A"
AMAZON_SECRET_KEY = "7***********************E"
AMAZON_ASSOC_TAG = "j*****-20"
amazon = AmazonAPI(AMAZON_ACCESS_KEY, AMAZON_SECRET_KEY, AMAZON_ASSOC_TAG, region='US')
print(amazon)
#product = amazon.lookup(ItemId='B002RL8FBQ')
When I run the code above it works fine and I get this output from the print function:
<amazon.api.AmazonAPI object at 0x7fb6e59f7b38>
So everything is working fine with my access key, secret key, and associate tag.
However, if I un-comment the last line #product = amazon.lookup(ItemId='B00EOE0WKQ') then I get this error traceback:
Traceback (most recent call last):
File "test.py", line 8, in <module>
product = amazon.lookup(ItemId='B00EOE0WKQ')
File "/home/darren/Python_projects/amazon_wp/myvenv/lib/python3.4/site-packages/amazon/api.py", line 173, in lookup
response = self.api.ItemLookup(ResponseGroup=ResponseGroup, **kwargs)
File "/home/darren/Python_projects/amazon_wp/myvenv/lib/python3.4/site-packages/bottlenose/api.py", line 251, in __call__
{'api_url': api_url, 'cache_url': cache_url})
File "/home/darren/Python_projects/amazon_wp/myvenv/lib/python3.4/site-packages/bottlenose/api.py", line 212, in _call_api
return urllib2.urlopen(api_request, timeout=self.Timeout)
File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 469, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 507, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
I have followed the instruction from the official github for this https://github.com/yoavaviram/python-amazon-simple-product-api and the code I am using you will see I have used from the "Usage" subheading on the github page, thus I am not sure what's going wrong.
For added info, I am using a virtual environment and to show that I have the correct packages installed here is my out put of pip freeze:
(myvenv) darren#my_comp:~/Python_projects/amazon_wp$ pip3 freeze
bottlenose==0.6.3
lxml==3.6.0
python-amazon-simple-product-api==2.1.0
python-dateutil==2.5.3
six==1.10.0
Also, I have tried several different asin numbers of valid products and I get the same error message.
I am using python 3.4 on ubuntu 14.04
I think, problem is with region. Please select valid value from here. Explanation can be that AWS can validate your credentials, but when it comes to "real" call it fails, since 'US' is not valid region...
You might need to authorize your account for API access. This step-by-step should walk you through it.
Edit:
I've installed all of the same versions and am using the same python code with my own keys and it works fine.
The only time I encountered that error was when I didn't specify the region (which you are clearly doing).
One thing I'd try is to add the following code into your script:
import logging
logging.basicConfig(level=logging.DEBUG)
which should display the following request url:
DEBUG:bottlenose.api:Amazon URL:
http://webservices.amazon.co.uk/onca/xml?AWSAccessKeyId=&AssociateTag=&ItemId=B00EOE0WKQ&Operation=ItemLookup&ResponseGroup=Large&S
ervice=AWSECommerceService&Timestamp=&Version=2013-08-01&Signature=
You can visit this in your browser and should see an XML document returned. If it fails it should hopefully give you a better error than what the pythonlib gives you.
For instance, if I visit https://associates-amazon.s3.amazonaws.com/scratchpad/index.html (never got this to work for me) but it provides a list of base URLs for the region.
I created my associate account on the .co.uk region, thus my requests will only be valid for http://webservices.amazon.co.uk, if I instead try to query http://webservices.amazon.com then I will see:
The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method. Consult
the service documentation for details.
If you've got an associate account on amazon.com, try it without specifying region as I believe that's default. Apart from the above, check that your VM has internet connectivity and if nothing else works try creating another access key and using that.
I want to debug some python requests using charles proxy.
I need to include the certificate for charles on the call, but is not working
import requests
endpoint_url = 'https://www.httpsnow.org/'
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
print "empexo"
print r
I have added the https address on Charles,
I get on Charles:
SSLHandshake: Remote host closed connection during handshake
and on python the log with error
empexo
Traceback (most recent call last):
File "/Users/iosdev/DopPy/GetCelebs.py", line 15, in <module>
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 49, in request
response = session.request(method=method, url=url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 336265225] _ssl.c:341: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Process finished with exit code 1
I found this thread while I was troubleshooting a similar issue. In the scenario I ran into the cert argument was being used to define the path to a ".crt" file when the verify argument should have been used instead.
The correct usage ended up looking like:
requests.get(endpoint_url, verify='/path/to/file.crt')
See Requests' documentation for more details: https://2.python-requests.org/en/v1.1.0/user/advanced/#ssl-cert-verification
As an aside, I find employing Request's ability to specify the path to a ".crt" via the REQUESTS_CA_BUNDLE environmental variable more effective when using Charles Proxy for local debugging.
Running something like the following in shell saves having to specify the path to Charles' ".crt" for every Requests call:
REQUESTS_CA_BUNDLE=/path/to/file.crt
export REQUESTS_CA_BUNDLE
I've used two different python oauth libraries with Django to authenticate with twitter. The setup is on apache with WSGI. When I restart the server everything works great for about 10 minutes and then the httplib seems to lock up (see the following error).
I'm running only 1 process and 1 thread of WSGI but that seems to make no difference.
I cannot figure out why it's locking up and giving this CannotSendRequest error. I've spent a lot of hours on this frustrating problem. Any hints/suggestions of what it could be would be greatly appreciated.
File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 92, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "mypath/auth/decorators.py", line 9, in decorated
return f(*args, **kwargs)
File "mypath/auth/views.py", line 30, in login
token = get_unauthorized_token()
File "/root/storm/eye/auth/utils.py", line 49, in get_unauthorized_token
return oauth.OAuthToken.from_string(oauth_response(req))
File "mypath/auth/utils.py", line 41, in oauth_response
connection().request(req.http_method, req.to_url())
File "/usr/lib/python2.5/httplib.py", line 866, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.5/httplib.py", line 883, in _send_request
self.putrequest(method, url, **skips)
File "/usr/lib/python2.5/httplib.py", line 770, in putrequest
raise CannotSendRequest()
CannotSendRequest
This exception is raised when you reuse httplib.HTTP object for new request while you havn't called its getresponse() method for previous request. Probably there was some other error before this one that left connection in broken state. The simplest reliable way to fix the problem is creating new connection for each request, not reusing it. Sure, it will be a bit slower, but I think it's not an issue having you are running application in single process and single thread.
Also check your Python version. I had a similar situation after updating to Py-2.7 from Py-2.6. In Py-2.6 everything worked without any problems. Py-2.7 httplib uses HTTP/1.1 by default which made the server did not send back the Connection: close option in the respond, therefore the connection handling was broken. In my case this worked with HTTP/1.0 though.
http.client.CannotSendRequest: Request-sent
while using http.client module HTTPConnection class ran into the error caus my host name was incorrect