Basically what I want to do is for me to be able to connect to a proxy while I am able to browse the internet
Here is the code I tried
import webbrowser
import socks, socket, requests
from multiprocessing import Process
def proxy():
while True:
socks.set_default_proxy(socks.SOCKS5, "184.32.91.92", 2901)
socket.socket = socks.socksocket
if __name__ == '__main__':
proxy_process = Process(target=proxy).start()
r = requests.get("http://icanhazip.com")
print(r.content) # stil gives me my actual IP address
webbrowser.open("http://icanhazip.com", new=2) # opening the webbrowser
So i tried to process the proxy to keep the connection alive but even when I open the browser it still gives me my actual IP
Looks like a timing issue. Your __main__ process probably reaches the line:
r = requests.get("http://icanhazip.com")
before proxy_process has done it's job.
How do I shutdown and close this particular connection?
import socket
import socks
from urllib.request import urlopen
socks.set_default_proxy(socks.SOCKS5, "localhost", 9150)
socket.socket = socks.socksocket
urlopen("http://www.sourceforge.net/")
I've tried socket.socket.shutdown(socket.SHUT_RDWR) but that doesnt work at all:
TypeError: descriptor 'shutdown' requires a '_socket.socket' object but received a 'int'
Hi there I've been trying to change my ip after some requests for scraping a website, but when I run this code which is the one i've found on some blogs, anything happens, I mean my ip is the same in the two outputs. Anyone have a clue of what I'm doing wrong?
from stem import Signal
from stem.control import Controller
import requests
def set_new_ip():
"""Change IP using TOR"""
with Controller.from_port(port=9051) as controller:
controller.authenticate(password='my_password')
controller.signal(Signal.NEWNYM)
local_proxy = '127.0.0.1:8118'
http_proxy = {'http://': local_proxy,
'https://': local_proxy}
current_ip = requests.get(url='http://icanhazip.com/',
proxies=http_proxy,
verify=False)
print(current_ip.text)
set_new_ip()
current_ip = requests.get(url='http://icanhazip.com/',
proxies=http_proxy,
verify=False)
print(current_ip.text)
Hey i've found a video and solve my question, here it is the video https://www.youtube.com/watch?v=qTc8Vqn57pM
and here it is my code
from stem import Signal
from stem.control import Controller
import requests
import socks, socket
import time
with Controller.from_port(port=9051) as controller:
controller.authenticate(password='my_password')
socks.setdefaultproxy(proxy_type=socks.PROXY_TYPE_SOCKS5, addr="127.0.0.1", port=9050)
socket.socket = socks.socksocket
while True:
current_ip = requests.get(url='http://icanhazip.com/')
print(current_ip.text)
controller.signal(Signal.NEWNYM)
time.sleep(controller.get_newnym_wait())
How to force the requests library to use a specific internet protocol version for a get request? Or can this be achieved better with another method in Python? I could but I do not want to use curl…
Example to clarify purpose:
import requests
r = requests.get('https://my-dyn-dns-service.domain/?hostname=my.domain',
auth = ('myUserName', 'my-password'))
I've found a minimalistic solution to force urrlib3 to use either ipv4 or ipv6. This method is used by urrlib3 for creating new connection both for Http and Https. You can specify in it any AF_FAMILY you want to use.
import socket
import requests.packages.urllib3.util.connection as urllib3_cn
def allowed_gai_family():
"""
https://github.com/shazow/urllib3/blob/master/urllib3/util/connection.py
"""
family = socket.AF_INET
if urllib3_cn.HAS_IPV6:
family = socket.AF_INET6 # force ipv6 only if it is available
return family
urllib3_cn.allowed_gai_family = allowed_gai_family
This is a hack, but you can monkey-patch getaddrinfo to filter to only IPv4 addresses:
# Monkey patch to force IPv4, since FB seems to hang on IPv6
import socket
old_getaddrinfo = socket.getaddrinfo
def new_getaddrinfo(*args, **kwargs):
responses = old_getaddrinfo(*args, **kwargs)
return [response
for response in responses
if response[0] == socket.AF_INET]
socket.getaddrinfo = new_getaddrinfo
You can use this hack to force requests to use IPv4:
requests.packages.urllib3.util.connection.HAS_IPV6 = False
(This was submitted as a comment by Nulano and I thought it deserved to be a proper answer).
I've written a runtime patch for requests+urllib3+socket that allows passing the required address family optionally and on a per-request basis.
Unlike other solutions there is no monkeypatching involved, rather you replace your imports of requests with the patched file and it present a request-compatible interface with all exposed classes subclassed and patched and all “simple API” function reimplemented. The only noticeable difference should be the fact that there is an extra family parameter exposed that you can use to restrict the address family used during name resolution to socket.AF_INET or socket.AF_INET6. A somewhat complicated (but mostly just LoC intensive) series of strategic method overrides is then used to pass this value all the way down to the bottom layers of urllib3 where it will be used in an alternate implementation of the socket.create_connection function call.
TL;DR usage looks like this:
import socket
from . import requests_wrapper as requests # Use this load the patch
# This will work (if IPv6 connectivity is available) …
requests.get("http://ip6only.me/", family=socket.AF_INET6)
# … but this won't
requests.get("http://ip6only.me/", family=socket.AF_INET)
# This one will fail as well
requests.get("http://127.0.0.1/", family=socket.AF_INET6)
# This one will work if you have IPv4 available
requests.get("http://ip6.me/", family=socket.AF_INET)
# This one will work on both IPv4 and IPv6 (the default)
requests.get("http://ip6.me/", family=socket.AF_UNSPEC)
Full link to the patch library (~350 LoC): https://gitlab.com/snippets/1900824
I took a similar approach to https://stackoverflow.com/a/33046939/5059062, but instead patched out the part in socket that makes DNS requests so it only does IPv6 or IPv4, for every request, which means this can be used in urllib just as effectively as in requests.
This might be bad if your program also uses unix pipes and other such things, so I urge caution with monkeypatching.
import requests
import socket
from unittest.mock import patch
import re
orig_getaddrinfo = socket.getaddrinfo
def getaddrinfoIPv6(host, port, family=0, type=0, proto=0, flags=0):
return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET6, type=type, proto=proto, flags=flags)
def getaddrinfoIPv4(host, port, family=0, type=0, proto=0, flags=0):
return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)
with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv6):
r = requests.get('http://ip6.me')
print('ipv6: '+re.search(r'\+3>(.*?)</',r.content.decode('utf-8')).group(1))
with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv4):
r = requests.get('http://ip6.me')
print('ipv4: '+re.search(r'\+3>(.*?)</',r.content.decode('utf-8')).group(1))
and without requests:
import urllib.request
import socket
from unittest.mock import patch
import re
orig_getaddrinfo = socket.getaddrinfo
def getaddrinfoIPv6(host, port, family=0, type=0, proto=0, flags=0):
return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET6, type=type, proto=proto, flags=flags)
def getaddrinfoIPv4(host, port, family=0, type=0, proto=0, flags=0):
return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)
with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv6):
r = urllib.request.urlopen('http://ip6.me')
print('ipv6: '+re.search(r'\+3>(.*?)</',r.read().decode('utf-8')).group(1))
with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv4):
r = urllib.request.urlopen('http://ip6.me')
print('ipv4: '+re.search(r'\+3>(.*?)</',r.read().decode('utf-8')).group(1))
Tested in 3.5.2
This is totally untested and will probably require some tweaks, but combining answers from Using Python “requests” with existing socket connection and how to force python httplib library to use only A requests, it looks like you should be able to create an IPv6 only socket and then have requests use that for its connection pool with something like:
try:
from http.client import HTTPConnection
except ImportError:
from httplib import HTTPConnection
class MyHTTPConnection(HTTPConnection):
def connect(self):
print("This actually called called")
self.sock = socket.socket(socket.AF_INET6)
self.sock.connect((self.host, self.port,0,0))
if self._tunnel_host:
self._tunnel()
requests.packages.urllib3.connectionpool.HTTPConnection = MyHTTPConnection
After reading the previous answer, I had to modify the code to force IPv4 instead of IPv6. Notice that I used socket.AF_INET instead of socket.AF_INET6, and self.sock.connect() has 2-item tuple argument.
I also needed to override the HTTPSConnection which is much different than HTTPConnection since requests wraps the httplib.HTTPSConnection to verify the certificate if the ssl module is available.
import socket
import ssl
try:
from http.client import HTTPConnection
except ImportError:
from httplib import HTTPConnection
from requests.packages.urllib3.connection import VerifiedHTTPSConnection
# HTTP
class MyHTTPConnection(HTTPConnection):
def connect(self):
self.sock = socket.socket(socket.AF_INET)
self.sock.connect((self.host, self.port))
if self._tunnel_host:
self._tunnel()
requests.packages.urllib3.connectionpool.HTTPConnection = MyHTTPConnection
requests.packages.urllib3.connectionpool.HTTPConnectionPool.ConnectionCls = MyHTTPConnection
# HTTPS
class MyHTTPSConnection(VerifiedHTTPSConnection):
def connect(self):
self.sock = socket.socket(socket.AF_INET)
self.sock.connect((self.host, self.port))
if self._tunnel_host:
self._tunnel()
self.sock = ssl.wrap_socket(self.sock, self.key_file, self.cert_file)
requests.packages.urllib3.connectionpool.HTTPSConnection = MyHTTPSConnection
requests.packages.urllib3.connectionpool.VerifiedHTTPSConnection = MyHTTPSConnection
requests.packages.urllib3.connectionpool.HTTPSConnectionPool.ConnectionCls = MyHTTPSConnection
I have the following script:
import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
import urllib2
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
which uses tor and SocksiPy
Now I want to change tor identity with each request, for example:
for i in range(0, 10):
#somehow change tor identity
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
How can I do this?
Tor wrote a new TOR control library in Python, stem. It can be found on PyPI. They provide some nice tutorials how to work with it, one of them explains how to change your identity:
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
Make sure your config is correct.
Today, I have searched a lot about this question, and finally managed to answer myself. But before I need to say that pirvoxy and tor should be configured correctly. First script, then a little bit about configuration:
import urllib2
from TorCtl import TorCtl
proxy_support = urllib2.ProxyHandler({"http" : "127.0.0.1:8118"})
opener = urllib2.build_opener(proxy_support)
def newId():
conn = TorCtl.connect(controlAddr="127.0.0.1", controlPort=9051, passphrase="your_password")
conn.send_signal("NEWNYM")
for i in range(0, 10):
print "case "+str(i+1)
newId()
proxy_support = urllib2.ProxyHandler({"http" : "127.0.0.1:8118"})
urllib2.install_opener(opener)
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
Above script gets new IP and checks it from ifconfig.me web site. About configuration:
We need Privoxy. to use TOR with HTTP connections, privoxy should work with tor. We can do it by adding thi to /etc/privoxy/config file:
forward-socks5 / localhost:9050 . #dot is important at the end
then we configure ControlPort in /etc/tor/torrc file. We need just uncomment this line:
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:872860B76453A77D60CA2BB8C1A7042072093276A3D701AD684053EC4C
then we just restart tor:
/etc/init.d/tor restart
Another simple solution, no external libraries required, works for both IPv4 and IPv6:
import socket
try:
tor_c = socket.create_connection((TOR_CTRL_HOST, TOR_CTRL_PORT))
tor_c.send('AUTHENTICATE "{}"\r\nSIGNAL NEWNYM\r\n'.format(TOR_CTRL_PWD))
response = tor_c.recv(1024)
if response != '250 OK\r\n250 OK\r\n':
sys.stderr.write('Unexpected response from Tor control port: {}\n'.format(response))
except Exception, e:
sys.stderr.write('Error connecting to Tor control port: {}\n'.format(repr(e)))
This is a video where im using STEM, SockSipy, Tor 100% working :)
#!/usr/bin/python
import socks
import socket
import time
from stem.control import Controller
from stem import Signal
import urllib2
import sys
def info():
print "[*] Welcome to Chart-Cheat Script"
print "[*] This script works with running TOR only"
print "[*] usage is chartcheat.py domain"
print "[*] argument domain must be in format www.example.com"
print "[*] Example: chartcheat.py www.example.com"
return
if len(sys.argv)==2:
info();
counter = 0
url = str(sys.argv[1]);
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
#visiting url in infinite loop
while True:
urllib2.urlopen("http://"+url)
counter=counter+1
print "Page " + url + " visited = " + str(counter)
#wait till next identity will be available
controller.signal(Signal.NEWNYM)
time.sleep(controller.get_newnym_wait())
else:
info();
In case you are running python3, urllib package in python3 will be the same as urllib2 package in python2.
You can enable tor control server by uncommenting few lines in
/etc/tor/torrc
And use stem library to send NEWNYM signal to change circuit.
controller.signal(Signal.NEWNYM)
You can read tutorial here.
you can write something like this :
def renew_connection():
with Controller.from_port(port=9051) as controller:
controller.authenticate(password='password')
controller.signal(Signal.NEWNYM)
controller.close()
def request_tor(url, headers):
renew_connection()
session = requests.session()
session.proxies = {}
session.proxies['http'] = 'socks5h://localhost:9050'
print((session.get(url)).text)
The following could work:
for i in range(0, 10):
#somehow change tor identity
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050+i)
socket.socket = socks.socksocket
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
You basically set set the proxy prior to making each connection. I am asuming that you have different proxies for different IPs since you have not stated how you intend to change the IP