I'm using the great Requests library in my Python script:
import requests
r = requests.get("http://example.com")
print(r.text)
I would like to use a SOCKS proxy, how can I do that? Requests seems to only support HTTP proxies.
The modern way:
pip install -U 'requests[socks]'
then
import requests
resp = requests.get('http://go.to',
proxies=dict(http='socks5://user:pass#host:port',
https='socks5://user:pass#host:port'))
In case someone has tried all of these older answers, and is still running into problems like:
requests.exceptions.ConnectionError:
SOCKSHTTPConnectionPool(host='myhost', port=80):
Max retries exceeded with url: /my/path
(Caused by NewConnectionError('<requests.packages.urllib3.contrib.socks.SOCKSConnection object at 0x106812bd0>:
Failed to establish a new connection:
[Errno 8] nodename nor servname provided, or not known',))
It may be because, by default, requests is configured to resolve DNS queries on the local side of the connection.
Try changing your proxy URL from socks5://proxyhost:1234 to socks5h://proxyhost:1234. Note the extra h (it stands for hostname resolution).
The PySocks package module default is to do remote resolution, and I'm not sure why requests made their integration this obscurely divergent, but here we are.
As of requests version 2.10.0, released on 2016-04-29, requests supports SOCKS.
It requires PySocks, which can be installed with pip install pysocks.
Example usage:
import requests
proxies = {'http': "socks5://myproxy:9191"}
requests.get('http://example.org', proxies=proxies)
You need install pysocks , my version is 1.0 and the code works for me:
import socket
import socks
import requests
ip='localhost' # change your proxy's ip
port = 0000 # change your proxy's port
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, ip, port)
socket.socket = socks.socksocket
url = u'http://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=inurl%E8%A2%8B'
print(requests.get(url).text)
As soon as python requests will be merged with SOCKS5 pull request it will do as simple as using proxies dictionary:
Update: PR was already merged.
#proxy
# SOCKS5 proxy for HTTP/HTTPS
proxies = {
'http' : "socks5://myproxy:9191",
'https' : "socks5://myproxy:9191"
}
#headers
headers = {
}
url='http://example.com/'
res = requests.get(url, headers=headers, proxies=proxies)
See SOCKS Proxy Support
Another options, in case that you cannot wait request to be ready, when you cannot use requesocks - like on GoogleAppEngine due to the lack of pwd built-in module, is to use PySocks that was mentioned above:
Grab the socks.py file from the repo and put a copy in your root folder;
Add import socks and import socket
At this point configure and bind the socket before using with urllib2 - in the following example:
import urllib2
import socket
import socks
socks.set_default_proxy(socks.SOCKS5, "myprivateproxy.example",port=9050)
socket.socket = socks.socksocket
res=urllib2.urlopen(url).read()
You can just run your script with https_proxy environment variable.
Install socks support if it necessary.
pip install PySocks
pip install pysocks5
Setup environment variable
export https_proxy=socks5://<hostname or ip>:<port>
Run your script. This example makes request using proxy and shows IP-address:
echo Your real IP
python -c 'import requests;print(requests.get("http://ipinfo.io/ip").text)'
echo IP with socks-proxy
python -c 'import requests;print(requests.get("https://ipinfo.io/ip").text)'
# SOCKS5 proxy for HTTP/HTTPS
proxiesDict = {
'http' : "socks5://1.2.3.4:1080",
'https' : "socks5://1.2.3.4:1080"
}
# SOCKS4 proxy for HTTP/HTTPS
proxiesDict = {
'http' : "socks4://1.2.3.4:1080",
'https' : "socks4://1.2.3.4:1080"
}
# HTTP proxy for HTTP/HTTPS
proxiesDict = {
'http' : "1.2.3.4:1080",
'https' : "1.2.3.4:1080"
}
I installed pysocks and monkey patched create_connection in urllib3, like this:
import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, "127.0.0.1", 1080)
def create_connection(address, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
source_address=None, socket_options=None):
"""Connect to *address* and return the socket object.
Convenience function. Connect to *address* (a 2-tuple ``(host,
port)``) and return the socket object. Passing the optional
*timeout* parameter will set the timeout on the socket instance
before attempting to connect. If no *timeout* is supplied, the
global default timeout setting returned by :func:`getdefaulttimeout`
is used. If *source_address* is set it must be a tuple of (host, port)
for the socket to bind as a source address before making the connection.
An host of '' or port 0 tells the OS to use the default.
"""
host, port = address
if host.startswith('['):
host = host.strip('[]')
err = None
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res
sock = None
try:
sock = socks.socksocket(af, socktype, proto)
# If provided, set socket level options before connecting.
# This is the only addition urllib3 makes to this function.
urllib3.util.connection._set_socket_options(sock, socket_options)
if timeout is not socket._GLOBAL_DEFAULT_TIMEOUT:
sock.settimeout(timeout)
if source_address:
sock.bind(source_address)
sock.connect(sa)
return sock
except socket.error as e:
err = e
if sock is not None:
sock.close()
sock = None
if err is not None:
raise err
raise socket.error("getaddrinfo returns an empty list")
# monkeypatch
urllib3.util.connection.create_connection = create_connection
I could do this on Linux.
$ pip3 install --user 'requests[socks]'
$ https_proxy=socks5://<hostname or ip>:<port> python3 -c \
> 'import requests;print(requests.get("https://httpbin.org/ip").text)'
Related
I'm trying to access the following domain nzxj65x32vh2fkhk.onion using requests.
I have tor running and I configured the session's object proxies correctly.
import requests
session = requests.session()
session.proxies = {'http': 'socks5://localhost:9050',
'https': 'socks5://localhost:9050'}
print(session.get('http://httpbin.org/ip').text) # prints {"origin": "67.205.146.164" }
print(requests.get('http://httpbin.org/ip').text) # prints {"origin": "5.102.254.76" }
However when I try to access the URL with the .onion domain I get the following error:
session.get('http://nzxj65x32vh2fkhk.onion/all')
ConnectionError: SOCKSHTTPConnectionPool(host='nzxj65x32vh2fkhk.onion', port=80): Max retries exceeded with url: /all (Caused by NewConnectionError('<requests.packages.urllib3.contrib.socks.SOCKSConnection object at 0x7f5e8c2dbbd0>: Failed to establish a new connection: [Errno -2] Name or service not known',))
I also tried to replace localhost with 127.0.0.1 as suggested in one of the answers. The result is the same unfortunately.
Performing the same request using urllib2 works just fine.
import socks, socket, urllib2
def create_connection(address, timeout=None, source_address=None):
sock = socks.socksocket()
sock.connect(address)
return sock
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', 9050)
socket.socket = socks.socksocket
socket.create_connection = create_connection
print(urllib2.urlopen('http://nzxj65x32vh2fkhk.onion/all').read()) # Prints the URL's contents
cURL also retrieves the contents of the page correctly.
I'm using Python 2.7.13, requests 2.13.0 & PySocks 1.6.7. Tor is running through a docker container with the following command:
sudo docker run -it -p 8118:8118 -p 9050:9050 -d dperson/torproxy
What am I doing wrong here? What do I need to do to make requests recognize the .onion URLs?
The solution is to use the socks5h protocol in order to enable remote DNS resolving in case the local DNS resolving process fails. See https://github.com/kennethreitz/requests/blob/e3f89bf23c53b98593e4248054661472aacac820/requests/packages/urllib3/contrib/socks.py#L158
The following code works as expected:
import requests
session = requests.session()
session.proxies = {'http': 'socks5h://localhost:9050',
'https': 'socks5h://localhost:9050'}
print(session.get('http://httpbin.org/ip').text) # prints {"origin": "67.205.146.164" }
print(requests.get('http://httpbin.org/ip').text) # prints {"origin": "5.102.254.76" }
print(session.get('http://nzxj65x32vh2fkhk.onion/all').text) # Prints the contents of the page
Is it possible to have the Python requests library resolve a consul domain name with a SRV record and utilize the correct IP address and port when making the request?
For example, given that I have serviceA running with the IP address 172.18.0.5 on port 8080 and this service is registered with consul. And given that DNS for the host is set to use consul to resolve queries. Can I make a request like:
requests.get('http://serviceA.service.consul')
and have it be equivalent to the request:
requests.get('http://172.18.0.5:8080')
I ended up writing a patch for requests that would do this using this answer. I had to make some changes due to updates to the requests library. This patch works with requests version 2.11.1.
I used the dnspython library to resolve the SRV records and it expects the IP address and port that consul is listening for DNS requests on to be available as the environment variable CONSUL_DNS_IP_PORT. To use the patch import the requests_use_srv_records function from whatever module the patch is in and then call it. It will only attempt to use consul SRV records for hosts that end with .service.consul, other hosts will be resolved regularly.
Here's the patch:
# Python Imports
import os
from socket import error as SocketError, timeout as SocketTimeout
# 3rd Party Imports
from dns import resolver
from requests.packages.urllib3.connection import HTTPConnection
from requests.packages.urllib3.exceptions import (NewConnectionError,
ConnectTimeoutError)
from requests.packages.urllib3.util import connection
def resolve_srv_record(host):
consul_dns_ip_port = os.environ.get('CONSUL_DNS_IP_PORT',
'172.17.0.1:53')
consul_dns_ip, consul_dns_port = consul_dns_ip_port.split(':')
res = resolver.Resolver()
res.port = consul_dns_port
res.nameservers = [consul_dns_ip]
ans = resolver.query(host, 'SRV')
return ans.response.additional[0].items[0].address, ans[0].port
def patched_new_conn(self):
if self.host.endswith('.service.consul'):
hostname, port = resolve_srv_record(self.host)
else:
hostname = self.host
port = self.port
extra_kw = {}
if self.source_address:
extra_kw['source_address'] = self.source_address
if self.socket_options:
extra_kw['socket_options'] = self.socket_options
try:
conn = connection.create_connection((hostname, port),
self.timeout,
**extra_kw)
except SocketTimeout as e:
raise ConnectTimeoutError(
self, "Connection to %s timed out. (connect timeout=%s)" %
(self.host, self.timeout))
except SocketError as e:
raise NewConnectionError(
self, "Failed to establish a new connection: %s" % e)
return conn
def requests_use_srv_records():
HTTPConnection._new_conn = patched_new_conn
No, you can't unless you rewrite requests.
SRV record is design to find a service.
In this case, you already indicate to use http. So client will only query A or AAAA record for serviceA.service.consul.
I want to make multiple GET requests using Tor to a webpage. I want to use a different ipaddress for each request.
import socks
import socket
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9150)
socket.socket = socks.socksocket
import requests
print (requests.get('http://icanhazip.com')).content
Using this, I made one request. How can I change the ipaddress to make another?
There are 2 aspects to your question -
Making requests using Tor
Renewing the connection as per requirement (in your case, after every request)
Part 1
The first one is easy to do with the latest (upwards of v2.10.0) requests library with an additional requirement of requests[socks] for using the socks proxy.
Installation -
pip install requests[socks]
Basic usage -
import requests
def get_tor_session():
session = requests.session()
# Tor uses the 9050 port as the default socks port
session.proxies = {'http': 'socks5://127.0.0.1:9050',
'https': 'socks5://127.0.0.1:9050'}
return session
# Make a request through the Tor connection
# IP visible through Tor
session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)
# Above should print an IP different than your public IP
# Following prints your normal public IP
print(requests.get("http://httpbin.org/ip").text)
Part 2
To renew the Tor IP, i.e. to have a fresh visible exit IP, you need to be able to connect to the Tor service through it's ControlPort and then send a NEWNYM signal.
Normal Tor installation does not enable the ControlPort by default. You'll have to edit your torrc file and uncomment the corresponding lines.
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:05834BCEDD478D1060F1D7E2CE98E9C13075E8D3061D702F63BCD674DE
Please note that the HashedControlPassword above is for the password "password". If you want to set a different password, replace the HashedControlPassword in the torrc by noting the output from tor --hash-password "<new_password>" where <new_password> is the password that you want to set.
................................................................................
Warning for Windows users: see post here.
There is an issue on windows where the setting for the controlport in the torrc file is ignored if tor was installed using the following command:
tor --service install
To resolve the issue, after editing your torrc file, type the following commands:
tor --service remove
tor --service install -options ControlPort 9051
................................................................................
Okay, so now that we have Tor configured properly, you will have to restart Tor if it is already running.
sudo service tor restart
Tor should now be up & running on the 9051 ControlPort through which we can send commands to it. I prefer to use the official stem library to control Tor.
Installation -
pip install stem
You may now renew the Tor IP by calling the following function.
Renew IP -
from stem import Signal
from stem.control import Controller
# signal TOR for a new connection
def renew_connection():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="password")
controller.signal(Signal.NEWNYM)
To verify that Tor has a new exit IP, just rerun the code from Part 1. For some reason unknown to me, you need to create a new session object in order to use the new IP.
session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)
Here is the code you want to use (download the stem package using pip install stem)
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password='your password set for tor controller port in torrc')
print("Success!")
controller.signal(Signal.NEWNYM)
print("New Tor connection processed")
Good luck and hopefully that works.
You can try pure-python tor protocol implementation Torpy. No need original Tor client or Stem dependency at all.
$ pip3 install torpy[requests]
...
$ python3.7
>>> from torpy.http.requests import TorRequests
>>> with TorRequests() as tor_requests:
... print("build circuit")
... with tor_requests.get_session() as sess:
... print(sess.get("http://httpbin.org/ip").json())
... print(sess.get("http://httpbin.org/ip").json())
... print("renew circuit")
... with tor_requests.get_session() as sess:
... print(sess.get("http://httpbin.org/ip").json())
... print(sess.get("http://httpbin.org/ip").json())
...
build circuit
{'origin': '23.129.64.190, 23.129.64.190'}
{'origin': '23.129.64.190, 23.129.64.190'}
renew circuit
{'origin': '198.98.50.112, 198.98.50.112'}
{'origin': '198.98.50.112, 198.98.50.112'}
So each time when you getting new session you get new identity (basically you get new circuit with new exit node). See more examples at readme file https://github.com/torpyorg/torpy
You can use torrequest library (shameless plug). It's available on PyPI.
from torrequest import TorRequest
with TorRequest() as tr:
response = tr.get('http://ipecho.net/plain')
print(response.text) # not your IP address
tr.reset_identity()
response = tr.get('http://ipecho.net/plain')
print(response.text) # another IP address, not yours
Requests supports proxies using the SOCKS protocol from version 2.10.0.
import requests
proxies = {
'http': 'socks5://localhost:9050',
'https': 'socks5://localhost:9050'
}
url = 'http://httpbin.org/ip'
print(requests.get(url, proxies=proxies).text)
This answer complete the one of Ashish Nitin Patil for windows
(feel free to update this answer)
Part 2
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:05834BCEDD478D1060F1D7E2CE98E9C13075E8D3061D702F63BCD674DE
The HashedControlPassword above is the password. If you want to set a different password in the console navigate to \Tor Browser\Browser\TorBrowser\Tor and type the following commands: tor.exe --hash-password password_XYZ | more). It will give you something like HashedControlPassword 16:54C092A8... This is your password. Now you can add it to the torrc file (Tor Browser\Browser\TorBrowser\Data\Tor\torrc).
You will need then to restart Tor:
tor --service remove
tor --service install -options ControlPort 9051
To check if that works type netstat -an you will now see that port 9051 is open.
Notice that tor --service install -... will create Tor Win32 Service. For some reason, it seems you have to stop the service to use the browser (run services.msc)
EDIT: you will find many pieces of information here (About port number & proxy, Tor, Privoxy, Auto switch user agent...).
This code works fine. Using Tor, it changes the IP address after each request.
import time, socks, socket
from urllib2 import urlopen
from stem import Signal
from stem.control import Controller
nbrOfIpAddresses=3
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password = 'my_pwd')
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
for i in range(0, nbrOfIpAddresses):
newIP=urlopen("http://icanhazip.com").read()
print("NewIP Address: %s" % newIP)
controller.signal(Signal.NEWNYM)
if controller.is_newnym_available() == False:
print("Waitting time for Tor to change IP: "+ str(controller.get_newnym_wait()) +" seconds")
time.sleep(controller.get_newnym_wait())
controller.close()
The requests in requesocks is super old, it doesn't have response.json() and many other stuff.
I would like to keep my code clean. However, requests currently doesn't have socks5 supported yet (for more detail, read this thread https://github.com/kennethreitz/requests/pull/478)
So I used Privoxy as a http proxy that connects Tor for now.
Install and configure Privoxy on Mac
brew install privoxy
vim /usr/local/etc/privoxy/config
# put this line in the config
forward-socks5 / localhost:9050 .
privoxy /usr/local/etc/privoxy/config
Install and configure Privoxy on Ubuntu
sudo apt-get install privoxy
sudo vim /etc/privoxy/config
# put this line in the config
forward-socks5 / localhost:9050 .
sudo /etc/init.d/privoxy restart
Now I can use Tor like a http proxy. Below is my python script.
import requests
proxies = {
'http': 'http://127.0.0.1:8118',
}
print requests.get('http://httpbin.org/ip', proxies=proxies).text
A good function to renew your IP. Windows example
def renew_tor_ip():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="aAjkaI19!!laksjd")
controller.signal(Signal.NEWNYM)
Example of usage
import requests
import time
from stem import Signal
from stem.control import Controller
def get_current_ip():
session = requests.session()
# TO Request URL with SOCKS over TOR
session.proxies = {}
session.proxies['http']='socks5h://localhost:9150'
session.proxies['https']='socks5h://localhost:9150'
try:
r = session.get('http://httpbin.org/ip')
except Exception as e:
print(str(e))
else:
return r.text
#16:8EE7AEE3F32EEEEB605C6AA6C47B47808CA6A81FA0D76546ADC05F0F15 to aAjkaI19!!laksjd
#cmd shell "C:\Users\Arthur\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe" --hash-password aAjkaI19!!laksjd | more
#Torcc config
#ControlPort 9051
#HashedControlPassword 16:8EE7AEE3F32EEEEB605C6AA6C47B47808CA6A81FA0D76546ADC05F0F15
def renew_tor_ip():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="aAjkaI19!!laksjd")
controller.signal(Signal.NEWNYM)
for i in range(5):
print(get_current_ip())
renew_tor_ip()
time.sleep(5)
I'm using python simple_salesforce module from this example: https://pypi.python.org/pypi/simple-salesforce. Specifically:
proxies = {
"http": "http://10.10.1.10:3128"
}
from simple_salesforce import Salesforce
sf = Salesforce(username='myemail#example.com.sandbox', password='password', security_token='token', sandbox=True, proxies=proxies)
Its failing with the below error.
requests.exceptions.ConnectionError: ('Connection aborted.', error(111, 'Connection refused'))
If I dont use proxy, it works fine. My requirement is to enable proxy.
Any suggestions?
Adding the following to the beginning of the program will solve this problem.
I was using urllib2 in python and that takes care of forwarding the request through proxy.
For the answer to my question:
If your hostname and port for proxy are xyz1-pqr01.abc.company.com and 3128 then
import os
os.environ['http_proxy'] = 'http://xyz1-pqr01.abc.company.com:3128'
os.environ['https_proxy'] = 'http://xyz1-pqr01.abc.company.com:3128'
I am trying to figure out how to send data to a server through a proxy. I was hoping this would be possible through tor but being as tor uses SOCKS it apparently isn't possible with httplib (correct me if I am wrong)
This is what I have right now
import httplib
con = httplib.HTTPConnection("google.com")
con.set_tunnel(proxy, port)
con.send("Sent Stuff")
The problem is, it seems to freeze when the tunnel is set. Thanks for your help.
If you want to use http proxy, it should be like this:
import httplib
conn = httplib.HTTPConnection(proxyHost, proxyPort)
conn.request("POST", "http://www.google.com", params)
If you want to use SOCKS proxy, you can use SocksiPy as in this question: How can I use a SOCKS 4/5 proxy with urllib2?
Looks like the correct answer is:
http://bugs.python.org/issue11448#msg130413
import httplib
con = httplib.HTTPConnection(proxyHost, proxyPort)
con.set_tunnel("www.google.com", 80)
con.send("Sent Stuff")
As a follow-up to Khue Vu's answer, here's a complete example, the details of getting this working with a SOCKS proxy were more complex than expected.
First install PySocks with:
pip install PySocks
Then you need to manually set up your SOCKS proxy after instantiating your HTTPConnection and informing it that it's going to be using a proxy:
from http.client import HTTPConnection
from urllib.parse import urlparse, urlencode
import socks
url = urlparse("http://final.destination.example.com:8888/")
conn = HTTPConnection('127.0.0.1', 9000) # use socks proxy address
conn.set_tunnel(url.netloc, url.port) # remote host and port that you actually want to talk to
conn.sock = socks.socksocket() # manually set socket
conn.sock.set_proxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9000) # use socks proxy address
conn.sock.connect((url.netloc, url.port)) # remote host and port that you actually want to talk to
request_path = "%s?%s" % (url.path, url.query)
conn.request("POST", request_path, post_data)
Note that the imports above are python3.x