We Keep Coding

Django Failed to establish a new connection <urllib3.connection.HTTPSConnection>

I use Django 2.2 and I have an api that is connected to two Esx servers.
a = 10.131.171.80
b = 10.131.171.90
everything works perfectly, except that during my test when I add a third fake server, it tells me that the server does not respond because connection time out. So the error is handled and I don't have a yellow page from Django.
Except when enter fourth server this time private ip like 172.16.15.15 I have an error Max retries exceeded NewConnectionError(<urllib3.connection.HTTPSConnection'>
But when I use a public IP like 1.1.1.1, I don't get any error.
What I would like is to handle the error better when someone enters a private IP.
what's weird is that on local env localhost everything works even with private ip but when I am in my test server it doesn't work.
My full traceback
Here is my api.py file that connects to Esx servers.
import requests
import os
import json
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
os.environ['REQUESTS_CA_BUNDLE'] = os.path.join('/etc/ssl/certs/')
req = requests.Session()
req.verify = False
class VmwareApi:
def __init__(self):
self.ip = ""
self.user = ""
self.password = ""
self.arg1 = ""
self.ses = ""
self.params = ""
def session(self):
try:
a = req.post('https://' + self.ip + '/api/session',
auth=(self.user, self.password),
timeout=1, verify=False)
self.ses = str(a.json())
except requests.exceptions.Timeout:
return 'ConnectTimeoutError'
return req
def param_loader(self):
if self.params:
self.params = json.loads(self.params)
def vapirequestget(self):
try:
VmwareApi.param_loader(self)
myreq = req.get('https://' + self.ip + self.arg1,
params=self.params, verify=False,
headers={"vmware-api-session-id": self.ses},
timeout=1)
return myreq
except requests.exceptions.Timeout:
return 'ConnectTimeoutError'

How to make a proxy server for python requests

I have seen code like this that shows how to use a proxy for python requests.
import requests
proxies = {
'http': 'http://localhost:7777',
'https': 'http://localhost:7777',
}
requests.get('http://example.org', proxies=proxies)
requests.get('https://example.org', proxies=proxies)
But I am wondering how can we make a very simple proxy server in Python that would be able to respond to the GET request?

You can find many examples how to do it - even in questions on Stackoverflow.
Some of them use standard module socket (but it doesn't look simply).
Other use standard module http but they show code for Python 2 which was using different names.
Version for Python 3
import http.server
import socketserver
import urllib.request
class MyProxy(http.server.SimpleHTTPRequestHandler):
def do_GET(self):
print(self.path)
url = self.path
self.send_response(200)
self.end_headers()
self.copyfile(urllib.request.urlopen(url), self.wfile)
# --- main ---
PORT = 7777
httpd = None
try:
socketserver.TCPServer.allow_reuse_address = True # solution for `OSError: [Errno 98] Address already in use`
httpd = socketserver.TCPServer(('', PORT), MyProxy)
print(f"Proxy at: http://localhost:{PORT}")
httpd.serve_forever()
except KeyboardInterrupt:
print("Pressed Ctrl+C")
finally:
if httpd:
httpd.shutdown()
#httpd.socket.close()
Test using page httpbin.org
import requests
proxies = {
'http': 'http://localhost:7777',
'https': 'http://localhost:7777',
}
response = requests.get('http://httpbin.org/get', proxies=proxies)
print(response.text)
response = requests.get('http://httpbin.org/get?arg1=hello&arg2=world', proxies=proxies)
print(response.text)
But it works only for HTTP.
For HTTPS it may need to use ssl.socket from module ssl.
And it works only with GET.
For POST, PUT, DELETE, etc. it would need do_POST, do_PUT, do_DELETE, etc. with different code.
EDIT:
def do_POST(self):
url = self.path
# - post data -
content_length = int(self.headers.get('Content-Length', 0)) # <--- size of data
if content_length:
content = self.rfile.read(content_length) # <--- data itself
else:
content = None
req = urllib.request.Request(url, method="POST", data=content)
output = urllib.request.urlopen(req)
# ---
self.send_response(200)
self.end_headers()
self.copyfile(output, self.wfile)
But if you need local proxy only to test your code then you could use
Python module/program: mitmproxy (Man-In-The-Middle-Proxy)
not-python, not-free (but work 30 days for free), with nice GUI: Charles Proxy
More complex OWASP ZAP, Burp Suite (community edition)

changing ip in iteration with tor python

I want to change my IP everytime I run through loop. I am trying to achieve it with TOR. I have seen few posts with similar question, but solution given there is not working. So far my code looks like:
import socks
#import socket
import requests
import time
for i in range(1,3):
socks.setdefaultproxy(proxy_type=socks.PROXY_TYPE_SOCKS5, addr="127.0.0.1", port=9050)
try:
print (requests.get("http://icanhazip.com").text)
except Exception as e:
time.sleep(30)
print (type(e))
print (e)
I need different IP every time, instead of same IP.
edit : I have tried using approach given on How to change Tor identity in Python?. My limitation is not to use any external libraries. also solution provided by Nedim is without external library.
so far I have tried following to get different IP from mentioned link:
import socket
import sys
import os
try:
tor_c = socket.create_connection(("127.0.0.1", 9051 ))
secret = os.urandom(32) # pass this to authenticate
hash = tor_c.s2k_gen(secret) # pass this to Tor on startup.
tor_c.send('AUTHENTICATE "{}"\r\nSIGNAL NEWNYM\r\n'.format(hash))
response = tor_c.recv(1024)
if response != '250 OK\r\n250 OK\r\n':
sys.stderr.write('Unexpected response from Tor control port: {}\n'.format(response))
except Exception as e:
sys.stderr.write('Error connecting to Tor control port: {}\n'.format(repr(e)))
but it is throwing following error:
Error connecting to Tor control port: ConnectionRefusedError(10061, 'No connection could be made because the target machine actively refused it', None, 10061, None)

def renew_connection():
with Controller.from_port(port=9051) as controller:
controller.authenticate(password='password')
controller.signal(Signal.NEWNYM)
controller.close()
def request_tor(url, headers):
print((requests.get(url,proxies={'http': 'socks5h://localhost:9050'}, headers=headers)).text)
r = requests.get(url)
print('direct IP:', r.text)
if __name__ == "__main__":
url = 'http://icanhazip.com'
headers = { 'User-Agent': UserAgent().random }
for i in range(5):
request_tor(url,headers)
renew_connection()
time.sleep(5)

Data from a Python script to URL as JSON

I've spent a lot of time on this but still can't seem to get it to work. The task is - I have to send system stats to a URL and the script is supposed to pull it, convert the namedtuple of each cpu stats of a machine and then send them all in 1 single POST request as JSON. The connection must close once the data has been sent.
For the '1 single POST request' functionality, I added the latter function (senddata_to_server) in the script. Without it (with the connection details simply listed there without a function) , when I ran it on Mac/Windows/Linux, it used to return all the namedtuples 1 by 1 and then a '200 OK' and then go on printing 'Connection refused' forever. Now when I run it, it just hangs there without returning anything.
(I have asked this question earlier ( HTTP Post request with Python JSON ) but I need to have the 'params' inside the loop and the connection details outside it.
import psutil
import socket
import time
import sample
import json
import httplib
import urllib
serverHost = sample.host
port = sample.port
thisClient = socket.gethostname()
currentTime = int(time.time())
s = socket.socket()
s.connect((serverHost, port))
cpuStats = psutil.cpu_times_percent(percpu=True)
def loop_thru_cpus():
while True:
global cpuStats
cpuStats = "/n".join([json.dumps(stats._asdict()) for stats in cpuStats])
try:
command = 'put cpu.usr ' + str(currentTime) + " " + str(cpuStats[0]) + "host ="+thisClient+ "/n"
s.sendall(command)
command = 'put cpu.nice ' + str(currentTime) + " " + str(cpuStats[1]) + "host ="+ thisClient+ "/n"
s.sendall(command)
command = 'put cpu.sys ' + str(currentTime) + " " + str(cpuStats[2]) + "host ="+ thisClient+ "/n"
s.sendall(command)
command = 'put cpu.idle ' + str(currentTime) + " " + str(cpuStats[3]) + "host ="+ thisClient+ "/n"
s.sendall(command)
params = urllib.urlencode({'cpuStats': cpuStats, 'deviceKey': 1234, 'timeStamp': str(currentTime)})
return params
print cpuStats
except IndexError:
continue
except socket.error:
print "Connection refused"
continue
finally:
s.close()
def senddata_to_server():
x = loop_thru_cpus()
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
conn = httplib.HTTPConnection(serverHost, port)
conn.request = ("POST", "", x.params, headers)
response = conn.response()
print response.status, response. reason
print x.cpuStats
conn.close()
loop_thru_cpus()
senddata_to_server()

Given the task and the code/logic here, what am I doing wrong?
I can't quite tell what your task is, but here are some things you may be doing wrong:
You are connecting to the server directly (via socket.connect()) and through the framework (via HTTPlib.connect())
You misspelled newline: it should be '\n', not '/n'
You have a while loop that can only execute once (because you return in the middle of it).
You have a print statement after your return statement
You are sending malformed put commands to the web server
You call loop_thru_cpus() twice
You set the content-type incorrectly to application/json -- you aren't sending well-formed json.
You aren't sending a url to HTTPlib.HTTPConnection.request() (may be allowed in practice, disallowed in the documentation)
You aren't invoking conn.request() correctly -- get rid of =
In the documentation it says to call conn.getresponse(), not conn.response()
Here is a program that hopefully does what you ask for:
import psutil
import socket
import time
import json
import httplib
import urllib
# httpbin provides an echo service at http://httpbin.org/post
serverHost = 'httpbin.org'
port = 80
url = 'http://httpbin.org/post'
# My psutil only has cpu_times, not cpu_times_percent
cpuStats = psutil.cpu_times(percpu=True)
# Convert each namedTuple to a json string
cpuStats = [json.dumps(stats._asdict()) for stats in cpuStats]
# Convert each json string to the form required by the assignment
cpuStats = [urllib.urlencode({'cpuStats':stats, 'deviceKey':1234}) for stats in cpuStats]
# Join stats together, one per line
cpuStats = '\n'.join(cpuStats)
# Send the data ...
# connect
conn = httplib.HTTPConnection(serverHost, port)
# Send the data
conn.request("POST", url, cpuStats)
# Check the response, should be 200
response = conn.getresponse()
print response.status, response.reason
# httpbin.org provides an echo service -- what did we send?
print response.read()
conn.close()

How to change Tor identity in Python?

I have the following script:
import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
import urllib2
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
which uses tor and SocksiPy
Now I want to change tor identity with each request, for example:
for i in range(0, 10):
#somehow change tor identity
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
How can I do this?

Tor wrote a new TOR control library in Python, stem. It can be found on PyPI. They provide some nice tutorials how to work with it, one of them explains how to change your identity:
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
Make sure your config is correct.

Today, I have searched a lot about this question, and finally managed to answer myself. But before I need to say that pirvoxy and tor should be configured correctly. First script, then a little bit about configuration:
import urllib2
from TorCtl import TorCtl
proxy_support = urllib2.ProxyHandler({"http" : "127.0.0.1:8118"})
opener = urllib2.build_opener(proxy_support)
def newId():
conn = TorCtl.connect(controlAddr="127.0.0.1", controlPort=9051, passphrase="your_password")
conn.send_signal("NEWNYM")
for i in range(0, 10):
print "case "+str(i+1)
newId()
proxy_support = urllib2.ProxyHandler({"http" : "127.0.0.1:8118"})
urllib2.install_opener(opener)
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
Above script gets new IP and checks it from ifconfig.me web site. About configuration:
We need Privoxy. to use TOR with HTTP connections, privoxy should work with tor. We can do it by adding thi to /etc/privoxy/config file:
forward-socks5 / localhost:9050 . #dot is important at the end
then we configure ControlPort in /etc/tor/torrc file. We need just uncomment this line:
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:872860B76453A77D60CA2BB8C1A7042072093276A3D701AD684053EC4C
then we just restart tor:
/etc/init.d/tor restart

Another simple solution, no external libraries required, works for both IPv4 and IPv6:
import socket
try:
tor_c = socket.create_connection((TOR_CTRL_HOST, TOR_CTRL_PORT))
tor_c.send('AUTHENTICATE "{}"\r\nSIGNAL NEWNYM\r\n'.format(TOR_CTRL_PWD))
response = tor_c.recv(1024)
if response != '250 OK\r\n250 OK\r\n':
sys.stderr.write('Unexpected response from Tor control port: {}\n'.format(response))
except Exception, e:
sys.stderr.write('Error connecting to Tor control port: {}\n'.format(repr(e)))

This is a video where im using STEM, SockSipy, Tor 100% working :)
#!/usr/bin/python
import socks
import socket
import time
from stem.control import Controller
from stem import Signal
import urllib2
import sys
def info():
print "[*] Welcome to Chart-Cheat Script"
print "[*] This script works with running TOR only"
print "[*] usage is chartcheat.py domain"
print "[*] argument domain must be in format www.example.com"
print "[*] Example: chartcheat.py www.example.com"
return
if len(sys.argv)==2:
info();
counter = 0
url = str(sys.argv[1]);
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
#visiting url in infinite loop
while True:
urllib2.urlopen("http://"+url)
counter=counter+1
print "Page " + url + " visited = " + str(counter)
#wait till next identity will be available
controller.signal(Signal.NEWNYM)
time.sleep(controller.get_newnym_wait())
else:
info();
In case you are running python3, urllib package in python3 will be the same as urllib2 package in python2.

You can enable tor control server by uncommenting few lines in
/etc/tor/torrc
And use stem library to send NEWNYM signal to change circuit.
controller.signal(Signal.NEWNYM)
You can read tutorial here.

you can write something like this :
def renew_connection():
with Controller.from_port(port=9051) as controller:
controller.authenticate(password='password')
controller.signal(Signal.NEWNYM)
controller.close()
def request_tor(url, headers):
renew_connection()
session = requests.session()
session.proxies = {}
session.proxies['http'] = 'socks5h://localhost:9050'
print((session.get(url)).text)

The following could work:
for i in range(0, 10):
#somehow change tor identity
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050+i)
socket.socket = socks.socksocket
print(urllib2.urlopen("http://www.ifconfig.me/ip").read())
You basically set set the proxy prior to making each connection. I am asuming that you have different proxies for different IPs since you have not stated how you intend to change the IP