How do I have python httplib accept untrusted certs? I created a snake oil/self signed cert on my webserver, and my python client fails to connect as I am using a untrusted cert.
I'd rather problematically fix this in my client code rather than have it trusted on my system.
import httplib
def main():
conn = httplib.HTTPSConnection("127.0.0.1:443")
conn.request("HEAD","/")
res = conn.getresponse()
print res.status, res.reason
data = res.read()
print len(data)
if __name__ == "__main__":
main()
Some of my scripts stopped working after updating my computer. Turns out, this was the problem: https://docs.python.org/2/library/httplib.html#httplib.HTTPSConnection
Changed in version 2.7.9: context was added.
This class now performs all the necessary certificate and hostname checks by default. To revert to the previous, unverified, behavior ssl._create_unverified_context() can be passed to the context parameter.
So if your version of Python is >= 2.7.9 (2.7.10 in my case), you'll likely run into this. To fix it, I updated my call:
httplib.HTTPSConnection(hostname, timeout=5, context=ssl._create_unverified_context())
This is likely the simplest change to retain the same behavior.
From inspecting the Python 2.7.14 source code, you may set an environment variable
PYTHONHTTPSVERIFY=0
and this will cause certificate verification to be disabled by default (this will apply to all requests from your program).
I believe this works from 2.7.12+ - but it does not apply to 3.x.
Ref. PEP 493: Verify HTTPS by default, but allow envvar to override that
Related
I’m using Python 3.6 and Fabric 2.4. I’m using Fabric to SSH into a server and run some commands. I need to set an environment variable for the commands being run on the remote server. The documentation indicates that something like this should work:
from fabric import task
#task(hosts=["servername"])
def do_things(c):
c.run("command_to_execute", env={"KEY": "VALUE"})
But that doesn’t work. Something like this should also be possible:
from fabric import task
#task(hosts=["servername"])
def do_things(c):
c.config.run.env = {"KEY": "VALUE"}
c.run("command_to_execute")
But that doesn’t work either. I feel like I’m missing something. Can anyone help?
I was able to do it by setting inline_ssh_env=True, and then explicitly setting the env variable, ex:
with Connection(host=hostname, user=username, inline_ssh_env=True) as c:
c.config.run.env = {"MY_VAR": "this worked"}
c.run('echo $MY_VAR')
As stated on the site of Fabric:
The root cause of this is typically because the SSH server runs non-interactive commands via a very limited shell call: /path/to/shell -c "command" (for example, OpenSSH). Most shells, when run this way, are not considered to be either interactive or login shells; and this then impacts which startup files get loaded.
You read more on this page link
So what you try to do won't work, and the solution is to pass the environment variable you want to set explicitly:
from fabric import task
#task(hosts=["servername"])
def do_things(c):
c.config.run.env = {"KEY": "VALUE"}
c.run('echo export %s >> ~/.bashrc ' % 'ENV_VAR=VALUE' )
c.run('source ~/.bashrc' )
c.run('echo $ENV_VAR') # to verify if it's set or not!
c.run("command_to_execute")
You can try that:
#task
def qa(ctx):
ctx.config.run.env['counter'] = 22
ctx.config.run.env['conn'] = Connection('qa_host')
#task
def sign(ctx):
print(ctx.config.run.env['counter'])
conn = ctx.config.run.env['conn']
conn.run('touch mike_was_here.txt')
And run:
fab2 qa sign
When creating the Connection object, try adding inline_ssh_env=True.
Quoting the documentation:
Whether to send environment variables “inline” as prefixes in front of command strings (export VARNAME=value && mycommand here), instead of trying to submit them through the SSH protocol itself (which is the default behavior). This is necessary if the remote server has a restricted AcceptEnv setting (which is the common default).
According to that part of the official doc, the connect_kwargs attribute of the Connection object is intended to replace the env dict. I use it, and it works as expected.
I am able to bind and query Active Directory via python-ldap without any issues except when it comes to adding or modifying attributes on AD. I can add the attribute but the encoding seems to be way off as all the text is garbled.
I've tried encoding my string with utf8 and a few others with no luck.
I've also tried binding with a Domain Admin account along with binding with the user account to which I will be changing an attribute, same result regardless.
Here is the method I use to update an attribute:
class LdapHelpers:
def __init__(self):
import ldap
# set globals
self.server = 'LDAP://dc.mycompany.com'
self.admin_dn = 'CN=Administrator,CN=users,DC=mycompany,DC=com'
self.admin_pass = 'coolpassword'
# init LDAP connection
#ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, 0)
ldap.set_option(ldap.OPT_REFERRALS, 0)
ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)
ldap.protocol_version = ldap.VERSION3
self.ldap = ldap.initialize(self.server)
def update_attribute(self, attrib, value):
try:
import ldap
conn = self.ldap
conn.simple_bind_s(self.admin_dn, self.admin_pass)
mod_attrs = [( ldap.MOD_REPLACE, "mobile", "6306564123")]
# I have tried other variations of the above
# mod_attrs = [( ldap.MOD_REPLACE, "mobile", "6306564123".encode('utf-8)]
conn.modify_s('CN=Mike Smith,OU=GoogleApps,DC=company,DC=com', mod_attrs)
print 'record updated'
except ldap.LDAPError as e:
return e.message
Doing a ldapsearch via terminal this is what the attribute looks like:
mobile:: MC8sAQAAAAAQNA==
This is what 'Hello World' looks like when I set mobile to it:
mobile:: 77+9ehsCAAAAABDvv70V
I've checked MSDN and it says that ldap attribute is just a Unicode string.
System: Ubuntu 15.10 64bit
Python: 2.7.10
python-ldap==2.4.21
As a side note I can search AD without any issues and parse/display returned user attributes, the issue only seems to be with creating or modifying attributes that this encoding issue comes in to play.
The '=' at the end is often an indicator that it is Base64 encoding. Python has a standard library for encoding/decoding base64 (The link is for Python 3, but Python 2 also has the library). LDAP does indeed use Base64 for something. See The LDAP Data Interchange Format (LDIF).
Take a look at the code from pyad to clarify what to do: https://pypi.python.org/pypi/pyad
It's Python-based.
Another example at already answered question: Use Python script to manage remote LDAP server
Ok I found out what was going on, I was using PyPy 4.0.1 as the interpreter and for some reason this was either causing issues with the python-ldap library and/or encoding for strings.
I switched back to Python 2.7.10 for the interpreter and now the very same modify commands up above work as expected using the python-ldap library. So definitely a word of caution if using PyPy and this particular library....
I want to build a client for my cloud endpoints in python like described in the Documentation.
I want to build the api from a Managed VM, so i get the path to the API by calling
modules.get_hostname(module="default")
This works fine for the devserver and i can create the complete path to the discovery endpoint, however on the live system this returns the url to a certain version like:
20150628t110011.default.guestbook.appspot.com
Thus the complete path to the API (default module) would be
https://20150628t110011.default.guestbook.appspot.com/_ah/api/discovery/v1/apis/guestbook/v1/rest?userIp=182.177.0.4"
But there is no discovery document, maybe due to the fact, that the certificate does not match a url that long and the https fails.
Is there a proper way to receive the base url to the default module? like so:
default.guestbook.appspot.com
because that would result in a working discovery endpoint:
https://default.guestbook.appspot.com/_ah/api/discovery/v1/apis/guestbook/v1/rest?userIp=182.177.0.4"
I would like to avoid doing string operations here, because on the local devserver this would not work as the module url resolves to something like localhost:1234.
You probably want to go through the GAE URl routing doc: https://cloud.google.com/appengine/docs/python/modules/routing#routing_via_url
Key points in there:
Google does not issue SSL certificates for double-wildcard domains
hosted at appspot.com, the certificate won't work for https://20150628t110011.default.guestbook.appspot.com
You can get the certificate to work using the -dot- delimiters; in particular the default version of the default
module can be accessed directly at guestbook.appspot.com
The problem gets even more complicated if your app has multiple modules and also if it's mapped to a custom domain.
While trying to address such complications I realized that modules.get_hostname() is nowadays no longer able to perform the original function that its name implies (I guess because of the multiple possible paths for accessing the same entity). Which probably explains why they won't attempt to fix the api to return a proper hostname: (see this Q&A)
But the info it can return (as applicable depending on the invocation arguments and the execution context) is IMHO extremely useful, allowing one to programatically obtain proper hostnames/URLs for all 3 possible app usage contextes: on the development server, on the .appspot.com domain and on custom domains mapping (including with hostname-based mapping):
<instance_id>.<module_version>.<module_name>.<app_name>.(appspot.com|<devserver_hostname>:<port#>)
This would be, for example, my approach for an app not interested in anything below the module name and using hostname-based custom domain dispatch routing - modules mapped to different hostnames):
def get_module_url(self, module_name='default'):
host_name = modules.get_hostname(module=module_name)
if os.environ.get('SERVER_SOFTWARE').startswith('Development'):
return 'http://' + host_name
app_name = app_identity.get_application_id()
split_name = self.request.host.split(':')[0].split('.')
if split_name[-2] == 'appspot':
new_host_name = app_name if module_name == 'default' else module_name + '-dot-' + app_name
else:
# custom hostname-based domain mapping, default module goes to `www`.mydomain.com
new_host_name = 'www' if module_name == 'default' else module_name
if app_name.endswith('-staging'):
# copy of the GAE app for staging purposes on mydomain.com
new_host_name += '-staging'
return '.'.join(['https://' + new_host_name] + split_name[1:])
As per this thread, unfortunately manual conversion is required to convert from the . hostname to -dot-.
I'm having some problems accessing http requests' bodies with the CherryPy framework.
I'm using CherryPy 3.2.0 on a x86_64 Arch Linux machine w/ Python3 and Aptana Web Studio IDE.
When I try to access a request's body via the usual cherrypy.request.body.read(), i get the error
File "/usr/lib/python3.2/site-packages/cherrypy/_cpreqbody.py", line 450, in read
return self.fp.read(size, fp_out)
TypeError: read() takes at most 2 positional arguments (3 given)
The code causing the error is:
import cherrypy
class Test:
def index(self):
print(cherrypy.request.body.read())
#print(cherrypy.request.body.readline()) <- this works!
return 'HelloWorld'
index.exposed = True
if __name__ == '__main__':
cherrypy.quickstart(Test())
However, using
cherrypy.request.body.readline() or cherrypy.request.body.readlines(n)
instead of
cherrypy.request.body.read()
i can skim through the request's body just fine. I tried googling for a solution but found none. Considering i'm a total python newbie, there must be something i am doing wrong, but what?
Thanks in advance for your precious help.
The body.read() method works properly only if the request body was processed, which happens only when request.process_request_body is True (it is by default) and when the request method is in request.method_with_bodies which is only PUT and POST by default, but not GET (which you probably used when requesting the page with a browser).
The following code causes Java to segfault:
import os.path
import neo4j
from paste import httpserver, fileapp
import tempfile
from webob.dec import wsgify
from webob import Response, Request
HOST = '127.0.0.1'
PORT = 8080
class DebugApp(object):
#wsgify
def __call__(self, req):
# db = neo4j.GraphDatabase(tempfile.mkdtemp())
db = neo4j.GraphDatabase(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'data'))
return Response(body='it worked')
def main():
app = DebugApp()
httpserver.serve(app, host=HOST, port=PORT)
if __name__ == '__main__':
main()
To reproduce, first save that code into a file (say, app.py), and then run python app.py. Then try http://localhost:8080 in your browser; you should see the Java crash handler.
The top of the Java stack trace looks like this:
Stack: [0xb42e7000,0xb4ae8000], sp=0xb4ae44f0, free space=8181k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [_jpype.so+0x26497] JPJavaEnv::NewObjectA(_jclass*, _jmethodID*, jvalue*)+0x37
C [_jpype.so+0x3c0e8] JPMethodOverload::invokeConstructor(_jclass*, std::vector<HostRef*, std::allocator<HostRef*> >&)+0x178
C [_jpype.so+0x3a417] JPMethod::invokeConstructor(std::vector<HostRef*, std::allocator<HostRef*> >&)+0x47
C [_jpype.so+0x1beba] JPClass::newInstance(std::vector<HostRef*, std::allocator<HostRef*> >&)+0x2a
C [_jpype.so+0x67b9c] PyJPClass::newClassInstance(_object*, _object*)+0xfc
C [python+0x96822] PyEval_EvalFrameEx+0x4332
C [python+0x991e7] PyEval_EvalCodeEx+0x127
I believe that's neo4j.GraphDatabase in Python triggering JPype to go looking for EmbeddedGraphDatabase in neo4j, under Java.
Running this code in an interactive Python session doesn't segfault:
>>> import webob
>>> import app
>>> debug_app = app.DebugApp()
>>> response = debug_app(webob.Request.blank('/'))
>>> response.body
'it worked'
Presumably that's because I'm avoiding Paste altogether in that example. Perhaps this has something to do with Paste's use of threads getting in the way of neo4j? I noted a somewhat similar problem in the neo4j forums: http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-CPython-Pylons-and-threading-td942435.html
...but that only occurs on shutdown.
The issue is not with Paste per se, but with the neo4j Python bindings, which use JPype. Paste creates threads to handle incoming requests; neo4j is supposed to be thread-safe, but JPype comes with this caveat from the documentation (1):
"For the most part, python threads based on OS level threads (i.e posix threads), will work without problem. The only thing to remember is to call jpype.attachThreadToJVM() in the thread body to make the JVM usable from that thread. For threads that you do not start yourself, you can call isThreadAttachedToJVM() to check."
I couldn't find the code that does this, but I think that some of the Java code in the neo4j bindings may call attachThreadToJVM at import time. If so, when a request is handed to a worker thread by paste, and that thread then goes to fetch data from neo4j, it is crossing thread boundaries, and the JVM attachment rule may not be satisfied.
You can avoid the crash by only running import neo4j from within a single thread. In the case above, this is the callable targeted by threading.Thread.
Unfortunately, this means that even though neo4j is thread-safe, it must be constrained to a single thread when used from Python. But that's not too disappointing, considering.
Update: the maintainers responded(2) and investigated the problem, and checked in a fix. I don't know which release of neo4j this was available in, and I can no longer find the commit to their github repo(3), so this stands for re-testing.