Tor's stem module (python) failing - python

Bear with me. This is my first post...
The Tor project has recently introduced Stem as a loadable python module. I've been playing around with it to see if it's a viable tool. My results have been mixed.
I try to enable a configuration for a hidden service within the controller (which is supposed to act as though it came directly from the torrc file. It always fails on me. Here's a quick example of what I try:
#!/usr/bin/env python
from stem.control import Controller
controller = Controller.from_port(port = 9051)
controller.authenticate()
controller.set_options({'HIDDENSERVICEDIR':'/tmp/hiddenservice/','HIDDENSERVICEPORT':'1234 127.0.0.1:1234'})
...which returns an error:
InvalidRequest Traceback (most recent call last)
/home/user/my/folder/<ipython-input-5-3921e9b46181> in <module>()
/usr/local/lib/python2.7/dist-packages/stem/control.pyc in set_options(self, params, reset)
1618 raise stem.InvalidRequest(response.code, response.message)
1619 elif response.code in ("513", "553"):
-> 1620 raise stem.InvalidRequest(response.code, response.message)
1621 else:
1622 raise stem.ProtocolError("Returned unexpected status code: %s" % response.code)
InvalidRequest: Unacceptable option value: Failed to configure rendezvous options. See logs
...and the following in /var/log/tor/log:
Aug 1 10:10:05.000 [warn] HiddenServicePort with no preceding HiddenServiceDir directive
Aug 1 10:10:05.000 [warn] Controller gave us config lines that didn't validate: Failed to configure rendezvous options. See logs for details.
I've tried this with Stem's "set_options" as seen above and in two separate commands with "set_conf". With "set_conf", I can set the HiddenServiceDir but it still fails the same when setting the port, making me think I have a fundamental misunderstanding of Tor.
I checked my circuits and it doesn't seem to matter if I have one with a hidden service rendezvous point; it keeps failing. I'd prefer to keep things pythonic, temporal and clean and not have a hacked up bash script that rewrites the torrc before restarting tor. (In a perfect world, I'd rather not write to a hidden service directory, but tor hasn't implemented that yet.)
I try to be as cross-platform as possible, but I'm running Linux with Tor 2.3.25...
So who has ideas of why Stem won't let me make a hidden service?

Thanks for pointing this out to me via our bug tracker. Answering this here. :)
The set_options() docs say...
The params can optionally be a list of key/value tuples, though the only reason this type of argument would be useful is for hidden service configuration (those options are order dependent).
The issue here is that Tor's hidden service options behave in a slightly different fashion from all the rest of its config options. Tor expects a 'HiddenServiceDir' followed by the properties associated with that hidden service (it's order dependent). This is because a single tor instance can provide multiple hidden services.
Please change your call from...
controller.set_options({'HIDDENSERVICEDIR':'/tmp/hiddenservice/','HIDDENSERVICEPORT':'1234 127.0.0.1:1234'})
... to be a list of tuples instead...
controller.set_options([('HiddenServiceDir', '/tmp/hiddenservice/'), ('HiddenServicePort', '1234 127.0.0.1:1234')])
Hope this helps! -Damian

Related

make any internet-accessing python code work (proxy + custom .crt)

The situation
If the following is not done, all outgoing HTTP or HTTPS requests made with python ends in a WinError 10054 Connection Reset, or a SSL bad handshake error.
set the HTTP_PROXY, HTTPS_PROXY environment variable, or their counterparts
What needs to be verified must be verified with a custom .crt file.
For example, assuming the .crt file is in place, both gets me a 200 OK:
import os
os.environ['HTTP_PROXY'] = #some_appropriate_address
os.environ['HTTPS_PROXY'] = #some appropriate_address
requests.get('http://www.google.com',verify="C:\the_file.crt") # 200 OK
requests.get('http://httpbin.org',verify=False) # 200 OK, but unsafe
requests.get('http://httpbin.org') # SSL bad handshake error
The Problem
There is this massive jumble of pre-written code (heavily utilizing urllib3 and requests and possibly other pieces of internet-accessing code) I have, and I have to make it work under the conditions outlined above.
Sure, I can write verify='C:\the_file.crt' for every requests.get(), but that can very quickly get hairy, right? And the code may also be using some other library (that is not requests). So I am looking for a global setting (environment variable etc.) I should alter, so that everything works well (return a 200 OK upon a GET request to a server, whether or not the code is written in requests-py).
Also, if there is no such way, I would like an explanation as to why.
What I tried (am trying)
Maybe editing the .condarc file (via conda --config) is a solution. I tried, to no avail: python gives me a "SSL verification failed" error. On the contrary, note that the code snippet above gave me a 200 OK. To my knowledge, this does not fit nicely with many situations that were previously discussed in Stack Overflow.
By the way, setting ssl_verify to false does not solve the problem either; I still get a bad handshake error for some reason.
I am using Win 10, Python 3.7.4 (Anaconda).
Update
I have edited the question to prevent future misunderstandings about the content of this question. A few answers below are a reiteration of what was written here from the start.
The current answers are not entirely satisfactory either, as they only seem to address the case where I am using requests or urllib3.
You should be able to get any python code that uses the requests module(which is inside urllib3) to work behind a proxy without modifying the python code itself by setting the following environment variables in Windows.
http_proxy http://[<user>:<pwd>#]<http_host>:<http_port>
https_proxy http://[<user>:<pwd>#]<https_host>:<https_port>
requests_ca_bundle <path_to_ca_bundle.crt>
curl_ca_bundle <path_to_ca_bundle.crt>
You can set environment variables by doing the following:
Press Windows-Key + R, enter sysdm.cpl ,3 (mind the space before the comma) and press Enter
Click the Environment variables button
In either of the fields (User variables or System variables), add the four variables
According to Doc in Requests:
https://requests.readthedocs.io/en/master/user/advanced/#proxies
you can use proxy in this way:
proxies = { 'http': 'http://10.10.1.10:3128', 'https': 'http://10.10.1.10:1080',}
requests.get('http://example.org', proxies=proxies)
Then depending on if you want to add .crt or .pem:
requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key'))
requests.get('https://kennethreitz.org', cert='/path/client.pem')
https://2.python-requests.org//en/v1.0.4/user/advanced/
You are trying to make https requests to an outer url and you need to provide the proper certificate files for verification. You are trying to make these configurations inside each component. But I would suggest that you make those configurations globally and system-wide so neither of the components need to provide certificates and deal with ssl-verification stuff.
I am awful at windows related networking configurations, but I would suggest you go check Proxifier and I am pretty sure you can configure a ssl proxy with proper certificates.

Why does the Google Calendar Quickstart for Python give me an OSError (WinError 10013) (access socket in a way forbidden)

I followed Google's Quickstart for Python, step-by-step. I followed each step exactly, often copying and pasting. I definitely have the Google Calendar API enabled. I've installed the Google Client Library with Pip. I've set up the sample code and the credentials.json in its own folder. So, why am I getting this error when I run it:
"OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions"
To figure this out, I've learned what a socket is. (It's literally the combination of an IP address and a single port). I've learned how to use netstat, though I don't know yet how this applies to what I'm doing. I've looked into using ShellExecuteEx based on an answer in this question, but I don't know how to use that with Python.
I've tried adding the script from the accepted answer to this question (which actually uses the ShellExecuteEx method though I don't notice this) into an admin.py file and import this admin.py script into quickstart.py. After updating the admin.py script to Python 3 syntax and running quickstart.py, Windows 8.1 asks me if I will allow access. I say yes, and it still gives me the OSError (WinError 10013) on accessing the socket in a forbidden way. The UAC is not the issue.
I suspect it's a port conflict, where something's already using the port that the script that Google's trying to use. But I'm worried that the port is decided by a black box function that I won't be able to change. The error itself doesn't say which port it's using, so I'll need to do more research.
It is a port issue.
Go to line 34 on the quickstart.py file (or where it says creds =
run_local_server()).
Go to the flow.py file in the
google_auth_oauthlib package with this function (in VS Code, click
run_local_server() and press F12 or right click and select "Go to
Definition").
You'll see line 369 (at the time of this writing) say self, host='localhost', port=8080,.
When I look at netstat, it actually says this port is in use, probably with an Apache server I never turned off.
Change the value in the flow.py file in the google_auth_oauthlib package to 8090, so 369 looks like self, host='localhost', port=8090,.
I ran the quickstart.py script again, and the window to authenticate my Google account popped up.
I selected my account, and it worked. No messing with the admin stuff.
I'm glad I was able to find it like this because I thought the port was selected in some black box manner, like it was decided from a server at Google.

Pepper API error during the example

I'm starting to play around with the new Pepper API for an important project (phasing out Java) and I'm having an issue with this example.
https://developer.chrome.com/native-client/devguide/devcycle/vs-addin
I've installed the plugin to VS, added the paths, started the python webserver yet when I debug it gives me a 404...
I'm starting the python webserver as per https://developer.chrome.com/native-client/sdk/examples
The issue being the HTML file it's looking for is in F:\nacl_sdk\vs_addin\examples\hello_world_gles\hello_world_gles and the localhost root is F:\nacl_sdk\pepper_42\getting_started
Has anyone else had this issue?
I also have plenty of intellisense errors:
Since I posted this I tried copying the example directory to the root directory being used by localhost. The page loads, however I'm not capable of running the plugin...
I think you're not supposed to be starting the Python web server, as per the vs addin documentation:
When you run one of the Native Client platforms Visual Studio builds
the corresponding type of Native Client module (either a .nexe or
.pexe), starts a web server to serve it up, and launches a copy of
Chrome that fetches the module from the server and runs it.
However, to be honest, I'm still unable to run this sample, even though I'm following this instruction. I'm seeing an "ERR_CONNECTION_REFUSED" result page. I'm using VS 2012 Express, and Chrome 43.
Update. I've finally managed to run the sample. First, I've installed VS 2012 Ultimate instead of Express (because Express doesn't support Add-Ins). Second, the latest VS addin seems to be unable to run the Python web-server, it passes the port paramater in a wrong format. You can see that if you read the output in the "Native Client Web Server Output" pane in VS. So what I did, is I modified the %NACL_SDK_ROOT%\tools\httpd.py, so that it doesn't attempt to parse the command line arguments :)
Here is the new main from my httpd.py:
def main(args):
server = LocalHTTPServer(os.path.abspath('.'), 5103)
# Serve until the client tells us to stop. When it does, it will give us an
# errorcode.
print 'Serving %s on %s...' % (options.serve_dir, server.GetURL(''))
return server.ServeForever()
HTH.

How do I disable sending traceback to remote hosts?

How can I keep Twisted from sending traceback to the browser when it has an error? It's been exposing file paths and code.
Just about every server framework that serves up something human readable will have an option to serve up tracebacks (for development) or not (for production).
In Twisted, it's called displayTracebacks, and it's documented as a member of twisted.web.server.Site. You configure it the same way(s) you configure anything else in Twisted. (The corresponding one-word option, e.g., for command-line flags, is notracebacks.)
If you're wondering why it defaults to True, you may want to read Glyph's rationale on the 2003 feature request that added this flag in the first place:
Nope, this should be turned on by default, since when you
are using defaults you are typically developing. System
administrators can make this a site-local default by adding
to sitecustomize.py or somesuch.
(As the 11-years-wiser version of Glyph points out in a comment, it's probably better to set it in the application than to use a site-local default. But the basic idea is the same. Set it to False at any level, and you'll get exactly the behavior you're after.)

How can a CGI server based on CGIHTTPRequestHandler require that a script start its response with headers that include a `content-type`?

Later note: the issues in the original posting below have been largely resolved.
Here's the background: For an introductory comp sci course, students develop html and server-side Python 2.7 scripts using a server provided by the instructors. That server is based on CGIHTTPRequestHandler, like the one at pointlessprogramming. When the students' html and scripts seem correct, they port those files to a remote, slow Apache server. Why support two servers? Well, the initial development using a local server has the benefit of reducing network issues and dependency on the remote, weak machine that is running Apache. Eventually porting to the Apache-running machine has the benefit of publishing their results for others to see.
For the local development to be most useful, the local server should closely resemble the Apache server. Currently there is an important difference: Apache requires that a script start its response with headers that include a content-type; if the script fails to provide such a header, Apache sends the client a 500 error ("Internal Server Error"), which too generic to help the students, who cannot use the server logs. CGIHTTPRequestHandler imposes no similar requirement. So it is common for a student to write header-free scripts that work with the local server, but get the baffling 500 error after copying files to the Apache server. It would be helpful to have a version of the local server that checks for a content-type header and gives a good error if there is none.
I seek advice about creating such a server. I am new to Python and to writing servers. Here are the issues that occur to me, but any helpful advice would be appreciated.
Is a content-type header required by the CGI standard? If so, other people might benefit from an answer to the main question here. Also, if so, I despair of finding a way to disable Apache's requirement. Maybe the relevant part of the CGI RFC is section 6.3.1 (CGI Response, Content-Type): "If an entity body is returned, the script MUST supply a Content-Type field in the response."
To make a local server that checks for the content-type header, perhaps I should sub-class CGIHTTPServer.CGIHTTPRequestHandler, to override run_cgi() with a version that issues an error for a missing header. I am looking at CGIHTTPServer.py __version__ = "0.4", which was installed with Python 2.7.3. But run_cgi() does a lot of processing, so it is a little unappealing to copy all its code, just to add a couple calls to a header-checking routine. Is there a better way?
If the answer to (2) is something like "No, overriding run_cgi() is recommended," I anticipate writing a version that invokes the desired script, then checks the script's output for headers before that output is sent to the client. There are apparently two places in the existing run_cgi() where the script is invoked:
3a. When run_cgi() is executed on a non-Unix system, the script is executed using Python's subprocess module. As a result, the standard output from the script will be available as an in-memory string, which I can presumably check for headers before the call to self.wfile.write. Does this sound right?
3b. But when run_cgi() is executed on a *nix system, the script is executed by a forked process. I think the child's stdout will write directly to self.wfile (I'm a little hazy on this), so I see no opportunity for the code in run_cgi() to check the output. Ugh. Any suggestions?
If analyzing the script's output is recommended, is email.parser the standard way to recognize whether there is a content-type header? Is another standard module recommended instead?
Is there a more appropriate forum for asking the main question ("How can a CGI server based on CGIHTTPRequestHandler require...")? It seems odd to ask if there is a better forum for asking programming questions than Stack Overflow, but I guess anything is possible.
Thanks for any help.

Categories