Tunneling httplib Through a Proxy - python

I am trying to figure out how to send data to a server through a proxy. I was hoping this would be possible through tor but being as tor uses SOCKS it apparently isn't possible with httplib (correct me if I am wrong)
This is what I have right now
import httplib
con = httplib.HTTPConnection("google.com")
con.set_tunnel(proxy, port)
con.send("Sent Stuff")
The problem is, it seems to freeze when the tunnel is set. Thanks for your help.


If you want to use http proxy, it should be like this:
import httplib
conn = httplib.HTTPConnection(proxyHost, proxyPort)
conn.request("POST", "http://www.google.com", params)
If you want to use SOCKS proxy, you can use SocksiPy as in this question: How can I use a SOCKS 4/5 proxy with urllib2?


Looks like the correct answer is:
http://bugs.python.org/issue11448#msg130413
import httplib
con = httplib.HTTPConnection(proxyHost, proxyPort)
con.set_tunnel("www.google.com", 80)
con.send("Sent Stuff")


As a follow-up to Khue Vu's answer, here's a complete example, the details of getting this working with a SOCKS proxy were more complex than expected.
First install PySocks with:
pip install PySocks
Then you need to manually set up your SOCKS proxy after instantiating your HTTPConnection and informing it that it's going to be using a proxy:
from http.client import HTTPConnection
from urllib.parse import urlparse, urlencode
import socks
url = urlparse("http://final.destination.example.com:8888/")
conn = HTTPConnection('127.0.0.1', 9000) # use socks proxy address
conn.set_tunnel(url.netloc, url.port) # remote host and port that you actually want to talk to
conn.sock = socks.socksocket() # manually set socket
conn.sock.set_proxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9000) # use socks proxy address
conn.sock.connect((url.netloc, url.port)) # remote host and port that you actually want to talk to
request_path = "%s?%s" % (url.path, url.query)
conn.request("POST", request_path, post_data)
Note that the imports above are python3.x

Related

How python handle DNS resolution with Tor Relay?

Supposing this working code :
import win_inet_pton
import socks
import socket
s = socks.socksocket()
s.set_proxy(socks.SOCKS5, "localhost", 9050)
s.connect(("xmh57jrzrnw6insl.onion",80))
s.send("GET / HTTP/1.1\r\n\r\n")
print 'sended'
data=s.recv(1024)
print data
Tor service is indeed running at port 9050.
In normal condition, python will perform DNS resolution through the SOCKS5 proxy, which is connected to the Tor relay. However, Tor do not handle UDP packet(It handles the resolution directly from the hostname in the TCP packet) so DNS resolution will fail.
How is it possible that this code work? (the equivalent code in java for exemple will fail as the DNS resolution can't be made).

IT is explained in this link: Python requests fails when tryign to connect to .onion site
You simply have to use socks5h instead of socks5

Python - Toggling a connection to my local SOCKS proxy

I am writing a crawler in Python that will run through Tor. I have Tor working and used code from this YouTube tutorial on how to route my Python requests to go through the Tor SOCKS proxy at 127.0.0.1:9050.
What I can't figure out is how to toggle this on/off within my script. Some requests I want to go through Tor and some I don't. Basically, I can't figure out the correct "close" or "shutdown" method in the socket objects I am using because I don't understand them.
Here's what happens now
import socket
import socks
import requests
def connect_to_socks():
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', 9050, True)
socket.socket = socks.socksocket
r = requests.get('http://wtfismyip.com/text')
print r.text #prints my ordinary IP address
connect_to_socks()
r = requests.get('http://wtfismyip.com/text')
print r.text #prints my Tor IP address
How do I turn off the socket routing to the SOCKS proxy so that it goes through my ordinary internet connection?
I'm hoping to use requests instead of urllib2 as it seems a lot easier but if I have to get into the guts of urllib2 or even httplib I will. But would prefer not to.

Figured it out by listening to this good YouTube tutorial.
Just need to call socket.setdefaultproxy() and it brings me back.

For Python 3 you can set back default socket by using this:
socks.setdefaultproxy(None)
socket.socket = socks.socksocket

How to get python gspread to use a SOCKS proxy server for connections?

How could I get my script that's using gspread to have the gspread connections to google's servers use a SOCKS proxy?

SocksiPy should work for this, as per the SO question: How can I use a SOCKS 4/5 proxy with urllib2?.
import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 8080)
socket.socket = socks.socksocket
import gspread
# do whatever
If this is not the desired result, you may have to create a custom instance of the bundled HTTPSession object.

Python - Using socket.gethostbyname through proxy

I'm using TOR to proxy connections but am having difficulty proxying DNS lookups via socket.gethostbyname("www.yahoo.com") -- I learned that it was not sending DNS traffic via proxy by sniffing traffic with wireshark. Here's a copy of the code I'm using
import StringIO
import socket
import socks # SocksiPy module
import stem.process
from stem.util import term
SOCKS_PORT = 7000
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT)
socket.socket = socks.socksocket
def getaddrinfo(*args):
return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]
socket.getaddrinfo = getaddrinfo
socket.gethostbyname("www.yahoo.com") <--- This line is not sending traffic via proxy
Any help is greatly appreciated!

You're calling gethostbyname in the socket module. It doesn't know anything about your SOCKS socket; it is simply interacting with your operating system's name resolution mechanisms. Setting socket.socket = socks.socksocket may affect network connections made through the socket module, but the module does not make direct connections to DNS servers to perform name resolution so replacing socket.socket has no impact on this behavior.
If you simply call the connect(...) method on a socks.socksocket object using a hostname, the proxy will perform name resolution via SOCKS:
s = socks.socksocket()
s.connect(('www.yahoo.com', 80))
If you actually want to perform raw DNS queries over your SOCKS connection, you'll need to find a Python DNS module to which you can provide your socksocket object.

If you resolve the DNS yourself with Socks5 you may leak information about your own computer. Instead try tunneling with Proxifier, then to Tor. Alternatively you can use SocksiPy's Socks4A extension. This will make sure information is not leaked.

How to make python Requests work via SOCKS proxy

I'm using the great Requests library in my Python script:
import requests
r = requests.get("http://example.com")
print(r.text)
I would like to use a SOCKS proxy, how can I do that? Requests seems to only support HTTP proxies.

The modern way:
pip install -U 'requests[socks]'
then
import requests
resp = requests.get('http://go.to',
proxies=dict(http='socks5://user:pass#host:port',
https='socks5://user:pass#host:port'))

In case someone has tried all of these older answers, and is still running into problems like:
requests.exceptions.ConnectionError:
SOCKSHTTPConnectionPool(host='myhost', port=80):
Max retries exceeded with url: /my/path
(Caused by NewConnectionError('<requests.packages.urllib3.contrib.socks.SOCKSConnection object at 0x106812bd0>:
Failed to establish a new connection:
[Errno 8] nodename nor servname provided, or not known',))
It may be because, by default, requests is configured to resolve DNS queries on the local side of the connection.
Try changing your proxy URL from socks5://proxyhost:1234 to socks5h://proxyhost:1234. Note the extra h (it stands for hostname resolution).
The PySocks package module default is to do remote resolution, and I'm not sure why requests made their integration this obscurely divergent, but here we are.

As of requests version 2.10.0, released on 2016-04-29, requests supports SOCKS.
It requires PySocks, which can be installed with pip install pysocks.
Example usage:
import requests
proxies = {'http': "socks5://myproxy:9191"}
requests.get('http://example.org', proxies=proxies)

You need install pysocks , my version is 1.0 and the code works for me:
import socket
import socks
import requests
ip='localhost' # change your proxy's ip
port = 0000 # change your proxy's port
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, ip, port)
socket.socket = socks.socksocket
url = u'http://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=inurl%E8%A2%8B'
print(requests.get(url).text)

As soon as python requests will be merged with SOCKS5 pull request it will do as simple as using proxies dictionary:
Update: PR was already merged.
#proxy
# SOCKS5 proxy for HTTP/HTTPS
proxies = {
'http' : "socks5://myproxy:9191",
'https' : "socks5://myproxy:9191"
}
#headers
headers = {
}
url='http://example.com/'
res = requests.get(url, headers=headers, proxies=proxies)
See SOCKS Proxy Support
Another options, in case that you cannot wait request to be ready, when you cannot use requesocks - like on GoogleAppEngine due to the lack of pwd built-in module, is to use PySocks that was mentioned above:
Grab the socks.py file from the repo and put a copy in your root folder;
Add import socks and import socket
At this point configure and bind the socket before using with urllib2 - in the following example:
import urllib2
import socket
import socks
socks.set_default_proxy(socks.SOCKS5, "myprivateproxy.example",port=9050)
socket.socket = socks.socksocket
res=urllib2.urlopen(url).read()

You can just run your script with https_proxy environment variable.
Install socks support if it necessary.
pip install PySocks
pip install pysocks5
Setup environment variable
export https_proxy=socks5://<hostname or ip>:<port>
Run your script. This example makes request using proxy and shows IP-address:
echo Your real IP
python -c 'import requests;print(requests.get("http://ipinfo.io/ip").text)'
echo IP with socks-proxy
python -c 'import requests;print(requests.get("https://ipinfo.io/ip").text)'

# SOCKS5 proxy for HTTP/HTTPS
proxiesDict = {
'http' : "socks5://1.2.3.4:1080",
'https' : "socks5://1.2.3.4:1080"
}
# SOCKS4 proxy for HTTP/HTTPS
proxiesDict = {
'http' : "socks4://1.2.3.4:1080",
'https' : "socks4://1.2.3.4:1080"
}
# HTTP proxy for HTTP/HTTPS
proxiesDict = {
'http' : "1.2.3.4:1080",
'https' : "1.2.3.4:1080"
}

I installed pysocks and monkey patched create_connection in urllib3, like this:
import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, "127.0.0.1", 1080)
def create_connection(address, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
source_address=None, socket_options=None):
"""Connect to *address* and return the socket object.
Convenience function. Connect to *address* (a 2-tuple ``(host,
port)``) and return the socket object. Passing the optional
*timeout* parameter will set the timeout on the socket instance
before attempting to connect. If no *timeout* is supplied, the
global default timeout setting returned by :func:`getdefaulttimeout`
is used. If *source_address* is set it must be a tuple of (host, port)
for the socket to bind as a source address before making the connection.
An host of '' or port 0 tells the OS to use the default.
"""
host, port = address
if host.startswith('['):
host = host.strip('[]')
err = None
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res
sock = None
try:
sock = socks.socksocket(af, socktype, proto)
# If provided, set socket level options before connecting.
# This is the only addition urllib3 makes to this function.
urllib3.util.connection._set_socket_options(sock, socket_options)
if timeout is not socket._GLOBAL_DEFAULT_TIMEOUT:
sock.settimeout(timeout)
if source_address:
sock.bind(source_address)
sock.connect(sa)
return sock
except socket.error as e:
err = e
if sock is not None:
sock.close()
sock = None
if err is not None:
raise err
raise socket.error("getaddrinfo returns an empty list")
# monkeypatch
urllib3.util.connection.create_connection = create_connection

I could do this on Linux.
$ pip3 install --user 'requests[socks]'
$ https_proxy=socks5://<hostname or ip>:<port> python3 -c \
> 'import requests;print(requests.get("https://httpbin.org/ip").text)'

Categories