Thank you for reading.
I was able to make a request with a socket connected to a proxy. But, return a 407. That is, there is an error in authentication in the proxy.
I put a Headers ['proxy-yauthorization'], but it doesn't work. It is sent with Basic + "User: Pass" based64.
def http_proxy_connect(address, proxy = None, auth = None, headers = {}):
import socket
import base64
def valid_address(addr):
""" Verify that an IP/port tuple is valid """
return isinstance(addr, (list, tuple)) and len(addr) == 2 and isinstance(addr[0], str) and isinstance(addr[1], (int, int))
if not valid_address(address):
raise ValueError('Invalid target address')
if proxy == None:
s = socket.socket()
s.connect(address)
return s, 0, {}
if not valid_address(proxy):
raise ValueError('Invalid proxy address')
headers = {
'host': address[0]
}
headers['proxy-authorization'] = 'Basic ' + auth
headers['Proxy-Authorization'] = 'Basic ' + auth
headers['Proxy-Authenticate'] = 'Basic'
s = socket.socket()
s.connect(proxy)
fp = s.makefile('wr')
fp.write('CONNECT %s:%d HTTP/1.0\r\n' % address)
fp.write('\r\n'.join('%s: %s' % (k, v) for (k, v) in headers.items()) + '\r\n\r\n')
print('\r\n'.join('%s: %s' % (k, v) for (k, v) in headers.items()) + '\r\n\r\n')
fp.flush()
statusline = fp.readline().rstrip('\r\n')
if statusline.count(' ') < 2:
fp.close()
s.close()
raise IOError('Bad response')
version, status, statusmsg = statusline.split(' ', 2)
if not version in ('HTTP/1.0', 'HTTP/1.1'):
fp.close()
s.close()
raise IOError('Unsupported HTTP version')
try:
status = int(status)
except ValueError:
fp.close()
s.close()
raise IOError('Bad response')
response_headers = {}
while True:
tl = ''
l = fp.readline().rstrip('\r\n')
if l == '':
break
if not ':' in l:
continue
k, v = l.split(':', 1)
response_headers[k.strip().lower()] = v.strip()
fp.close()
return (s, status, response_headers)
print(http_proxy_connect(('xxxxx',80), ('xxxxx',50000), 'user:pass in base64!'));
I am currently sending in Headers ['proxy-yauthorization'], "Basic User: Pass" in base64.
I'm on a network that requires me to connect through an authenticated HTTP proxy to access anything outside the network. What I need to do is basically to make a socket (or equivalent) to connect to the Internet, but sending all the data through the proxy instead of trying to send it directly. Any ideas as to how this could be done?
As I didn't find any actual modules or other code that I could use, I ended up writing my own function that connects through the proxy:
def http_proxy_connect(address, proxy = None, auth = None, headers = {}):
"""
Establish a socket connection through an HTTP proxy.
Arguments:
address (required) = The address of the target
proxy (def: None) = The address of the proxy server
auth (def: None) = A tuple of the username and password used for authentication
headers (def: {}) = A set of headers that will be sent to the proxy
Returns:
A 3-tuple of the format:
(socket, status_code, headers)
Where `socket' is the socket object, `status_code` is the HTTP status code that the server
returned and `headers` is a dict of headers that the server returned.
"""
import socket
import base64
def valid_address(addr):
""" Verify that an IP/port tuple is valid """
return isinstance(addr, (list, tuple)) and len(addr) == 2 and isinstance(addr[0], str) and isinstance(addr[1], (int, long))
if not valid_address(address):
raise ValueError('Invalid target address')
if proxy == None:
s = socket.socket()
s.connect(address)
return s, 0, {}
if not valid_address(proxy):
raise ValueError('Invalid proxy address')
headers = {
'host': address[0]
}
if auth != None:
if isinstance(auth, str):
headers['proxy-authorization'] = auth
elif auth and isinstance(auth, (tuple, list)):
if len(auth) == 1:
raise ValueError('Invalid authentication specification')
t = auth[0]
args = auth[1:]
if t.lower() == 'basic' and len(args) == 2:
headers['proxy-authorization'] = 'Basic ' + base64.b64encode('%s:%s' % args)
else:
raise ValueError('Invalid authentication specification')
else:
raise ValueError('Invalid authentication specification')
s = socket.socket()
s.connect(proxy)
fp = s.makefile('r+')
fp.write('CONNECT %s:%d HTTP/1.0\r\n' % address)
fp.write('\r\n'.join('%s: %s' % (k, v) for (k, v) in headers.items()) + '\r\n\r\n')
fp.flush()
statusline = fp.readline().rstrip('\r\n')
if statusline.count(' ') < 2:
fp.close()
s.close()
raise IOError('Bad response')
version, status, statusmsg = statusline.split(' ', 2)
if not version in ('HTTP/1.0', 'HTTP/1.1'):
fp.close()
s.close()
raise IOError('Unsupported HTTP version')
try:
status = int(status)
except ValueError:
fp.close()
s.close()
raise IOError('Bad response')
response_headers = {}
while True:
tl = ''
l = fp.readline().rstrip('\r\n')
if l == '':
break
if not ':' in l:
continue
k, v = l.split(':', 1)
response_headers[k.strip().lower()] = v.strip()
fp.close()
return (s, status, response_headers)
I'm not sure if this will be of much help, but have you taken a look to pycurl? This may help you to connect to the Proxy providing a username/password authentication system (see this and this)
I have the following situation:
WWW <- ServerBehindNAT(C) -> (S)MyApp(S) <- (C)BrowserUser
C == Client connection.
S == Server connection.
ServerBehindNAT represents a node that connects via SOCKS to MyApp on my server.
BrowserUser represents a client browser with SOCKS proxy set as IP of MyApp.
MyApp represents this script which accepts incoming connection from BrowserUser, accepts incoming connection from ServerBehindNAT, and sends requests from BrowserUser through to ServerBehindNAT.
WWW represents the external Internet/www.
I have created a multi-threaded SOCKS server which listens on 2 ports. What I'm having trouble designing is the part where it forwards requests from BrowserUser through to ServerBehindNAT:
(S)<--(S)
The first code I had:
import logging
import select
import socket
import struct
from socketserver import ThreadingMixIn, TCPServer, StreamRequestHandler, BaseRequestHandler
import threading
import mysql.connector
logging.basicConfig(level=logging.DEBUG)
SOCKS_VERSION = 5
def sql_connect():
""" database connection"""
try:
sql = mysql.connector.connect(host='localhost',
database='prox_main232',
user='prox_main551',
password='H20gN!vsi#rJ')
if sql.is_connected():
print('connected to sql')
return sql
except Exception as e:
print(e)
return False
def sql_auth(username, password):
sql = sql_connect()
cursor = sql.cursor()
query = "select `id` from partners where username = %s and password = %s"
params = (username, password)
try:
cursor.execute(query, params)
except Exception as e:
print("Error: ", e)
class ThreadingTCPServer(ThreadingMixIn, TCPServer):
pass
class SocksProxy(StreamRequestHandler):
username = 'username'
password = 'password'
def handle(self):
logging.info('Accepting connection from %s:%s' % self.client_address)
# greeting header
# read and unpack 2 bytes from a client
header = self.connection.recv(2)
version, nmethods = struct.unpack("!BB", header)
# socks 5
assert version == SOCKS_VERSION
assert nmethods > 0
# get available methods
methods = self.get_available_methods(nmethods)
# accept only USERNAME/PASSWORD auth
if 2 not in set(methods):
# close connection
self.server.close_request(self.request)
return
# send welcome message
self.connection.sendall(struct.pack("!BB", SOCKS_VERSION, 2))
if not self.verify_credentials():
return
# request
version, cmd, _, address_type = struct.unpack("!BBBB", self.connection.recv(4))
assert version == SOCKS_VERSION
if address_type == 1: # IPv4
address = socket.inet_ntoa(self.connection.recv(4))
elif address_type == 3: # Domain name
domain_length = ord(self.connection.recv(1)[0])
address = self.connection.recv(domain_length)
port = struct.unpack('!H', self.connection.recv(2))[0]
# reply
try:
if cmd == 1: # CONNECT
remote = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote.connect((address, port))
bind_address = remote.getsockname()
logging.info('Connected to %s %s' % (address, port))
else:
self.server.close_request(self.request)
addr = struct.unpack("!I", socket.inet_aton(bind_address[0]))[0]
port = bind_address[1]
reply = struct.pack("!BBBBIH", SOCKS_VERSION, 0, 0, address_type,
addr, port)
except Exception as err:
logging.error(err)
# return connection refused error
reply = self.generate_failed_reply(address_type, 5)
self.connection.sendall(reply)
# establish data exchange
if reply[1] == 0 and cmd == 1:
self.exchange_loop(self.connection, remote)
self.server.close_request(self.request)
def get_available_methods(self, n):
methods = []
for i in range(n):
methods.append(ord(self.connection.recv(1)))
return methods
def verify_credentials(self):
version = ord(self.connection.recv(1))
assert version == 1
username_len = ord(self.connection.recv(1))
username = self.connection.recv(username_len).decode('utf-8')
password_len = ord(self.connection.recv(1))
password = self.connection.recv(password_len).decode('utf-8')
if username == self.username and password == self.password:
# success, status = 0
response = struct.pack("!BB", version, 0)
self.connection.sendall(response)
return True
# failure, status != 0
response = struct.pack("!BB", version, 0xFF)
self.connection.sendall(response)
self.server.close_request(self.request)
return False
def generate_failed_reply(self, address_type, error_number):
return struct.pack("!BBBBIH", SOCKS_VERSION, error_number, 0, address_type, 0, 0)
def exchange_loop(self, client, remote):
while True:
# wait until client or remote is available for read
r, w, e = select.select([client, remote], [], [])
if client in r:
data = client.recv(4096)
if remote.send(data) <= 0:
break
if remote in r:
data = remote.recv(4096)
if client.send(data) <= 0:
break
if __name__ == '__main__':
####sql_connect() # connect to database
print("Accept partner connections...")
#with ThreadingTCPServer(('127.0.0.1', 9013), SocksProxy) as server:
# server.serve_forever()
partner_server = ThreadingTCPServer(('127.0.0.1', 9013), SocksProxy)
partner_thread = threading.Thread(target=partner_server.serve_forever)
print("Accept app connections...")
app_server = ThreadingTCPServer(('127.0.0.1', 9014), SocksProxy)
app_thread = threading.Thread(target=app_server.serve_forever)
for t in partner_thread, app_thread:
t.start()
for t in partner_thread, app_thread:
t.join()
To conceptualize accepting inbound connection on both ports and sending request of one through the other (Making it a type of reverse proxy) is difficult to know the next step. I got the idea to communicate between two SOCKS server classes. I attempted that here:
import logging
import select
import socket
import struct
from socketserver import ThreadingMixIn, TCPServer, StreamRequestHandler, BaseRequestHandler
import threading
logging.basicConfig(level=logging.DEBUG)
SOCKS_VERSION = 5
class ThreadingTCPServer(ThreadingMixIn, TCPServer):
pass
class NatServer(StreamRequestHandler):
username = 'username'
password = 'password'
def __init__(self):
NatServer.client = None
NatServer.remote = None
def handle(self):
logging.info('Accepting app connection from %s:%s' % self.client_address)
# greeting header
# read and unpack 2 bytes from a client
header = self.connection.recv(2)
version, nmethods = struct.unpack("!BB", header)
# socks 5
assert version == SOCKS_VERSION
assert nmethods > 0
# get available methods
methods = self.get_available_methods(nmethods)
# accept only USERNAME/PASSWORD auth
if 2 not in set(methods):
# close connection
self.server.close_request(self.request)
return
# send welcome message
self.connection.sendall(struct.pack("!BB", SOCKS_VERSION, 2))
if not self.verify_credentials():
return
# request
version, cmd, _, address_type = struct.unpack("!BBBB", self.connection.recv(4))
assert version == SOCKS_VERSION
if address_type == 1: # IPv4
address = socket.inet_ntoa(self.connection.recv(4))
elif address_type == 3: # Domain name
domain_length = ord(self.connection.recv(1)[0])
address = self.connection.recv(domain_length)
port = struct.unpack('!H', self.connection.recv(2))[0]
# reply
try:
if cmd == 1: # CONNECT
remote = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote.connect((address, port))
bind_address = remote.getsockname()
logging.info('Connected to %s %s' % (address, port))
else:
self.server.close_request(self.request)
addr = struct.unpack("!I", socket.inet_aton(bind_address[0]))[0]
port = bind_address[1]
reply = struct.pack("!BBBBIH", SOCKS_VERSION, 0, 0, address_type,
addr, port)
except Exception as err:
logging.error(err)
# return connection refused error
reply = self.generate_failed_reply(address_type, 5)
self.connection.sendall(reply)
# establish data exchange
if reply[1] == 0 and cmd == 1:
self.exchange_loop(self.connection, remote)
self.server.close_request(self.request)
def get_available_methods(self, n):
methods = []
for i in range(n):
methods.append(ord(self.connection.recv(1)))
return methods
def verify_credentials(self):
version = ord(self.connection.recv(1))
assert version == 1
username_len = ord(self.connection.recv(1))
username = self.connection.recv(username_len).decode('utf-8')
password_len = ord(self.connection.recv(1))
password = self.connection.recv(password_len).decode('utf-8')
if username == self.username and password == self.password:
# success, status = 0
response = struct.pack("!BB", version, 0)
self.connection.sendall(response)
return True
# failure, status != 0
response = struct.pack("!BB", version, 0xFF)
self.connection.sendall(response)
self.server.close_request(self.request)
return False
def generate_failed_reply(self, address_type, error_number):
return struct.pack("!BBBBIH", SOCKS_VERSION, error_number, 0, address_type, 0, 0)
def exchange_loop(self, client, remote):
self.bridge_sockets(client, remote)
while True:
# wait until client or remote is available for read
r, w, e = select.select([client, remote], [], [])
if client in r:
data = client.recv(4096)
if remote.send(data) <= 0:
break
if remote in r:
data = remote.recv(4096)
if client.send(data) <= 0:
break
def bridge_sockets(self, client, remote):
NatServer.client = client
NatServer.remote = remote
browser_server = ThreadingTCPServer(('127.0.0.1', 9013), BrowserServer)
browser_thread = threading.Thread(target=partner_server.serve_forever)
''', kwargs={'app_client': client,
'app_remote': remote}'''
browser_thread.start()
#partner_thread.join()
class BrowserServer(StreamRequestHandler):
username: "username"
password: "password"
def __init__(self): # , app_client, app_remote
self.app_client = NatServer.client
self.app_remote = NatServer.remote
def handle(self):
logging.info('Accepting partner connection from %s:%s' % self.client_address)
# greeting header
# read and unpack 2 bytes from a client
header = self.connection.recv(2)
version, nmethods = struct.unpack("!BB", header)
# socks 5
assert version == SOCKS_VERSION
assert nmethods > 0
# get available methods
methods = self.get_available_methods(nmethods)
# accept only USERNAME/PASSWORD auth
if 2 not in set(methods):
# close connection
self.server.close_request(self.request)
return
# send welcome message
self.connection.sendall(struct.pack("!BB", SOCKS_VERSION, 2))
if not self.verify_credentials():
return
# client request
version, cmd, _, address_type = struct.unpack("!BBBB", self.connection.recv(4))
assert version == SOCKS_VERSION
if address_type == 1: # IPv4
address = socket.inet_ntoa(self.connection.recv(4))
elif address_type == 3: # Domain name
domain_length = ord(self.connection.recv(1)[0])
address = self.connection.recv(domain_length)
port = struct.unpack('!H', self.connection.recv(2))[0]
# reply
try:
if cmd == 1: # CONNECT
remote = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote.connect((address, port))
bind_address = remote.getsockname()
logging.info('Connected to %s %s' % (address, port))
else:
self.server.close_request(self.request)
addr = struct.unpack("!I", socket.inet_aton(bind_address[0]))[0]
port = bind_address[1]
reply = struct.pack("!BBBBIH", SOCKS_VERSION, 0, 0, address_type,
addr, port)
except Exception as err:
logging.error(err)
# return connection refused error
reply = self.generate_failed_reply(address_type, 5)
self.connection.sendall(reply)
# establish data exchange
if reply[1] == 0 and cmd == 1:
self.exchange_loop(self.connection, remote)
self.server.close_request(self.request)
def get_available_methods(self, n):
methods = []
for i in range(n):
methods.append(ord(self.connection.recv(1)))
return methods
def verify_credentials(self):
version = ord(self.connection.recv(1))
assert version == 1
username_len = ord(self.connection.recv(1))
username = self.connection.recv(username_len).decode('utf-8')
password_len = ord(self.connection.recv(1))
password = self.connection.recv(password_len).decode('utf-8')
if username == self.username and password == self.password:
# success, status = 0
response = struct.pack("!BB", version, 0)
self.connection.sendall(response)
return True
# failure, status != 0
response = struct.pack("!BB", version, 0xFF)
self.connection.sendall(response)
self.server.close_request(self.request)
return False
def generate_failed_reply(self, address_type, error_number):
return struct.pack("!BBBBIH", SOCKS_VERSION, error_number, 0, address_type, 0, 0)
def exchange_loop(self, client, remote):
# app_<-partner_client
app_client = self.app_client
app_remote = self.app_remote
while True:
# wait until client or remote is available for read
#r, w, e = select.select([client, remote], [], [])
r, w, e = select.select([client, app_client], [], [])
if app_client in r:
data = app_client.recv(4096)
if client.send(data) <= 0:
break
if client in r:
data = client.recv(4096)
if app_client.send(data) <= 0:
break
if __name__ == '__main__':
nat_server = ThreadingTCPServer(('127.0.0.1', 9014), NatServer)
nat_thread = threading.Thread(target=app_server.serve_forever)
nat_thread.start()
nat_thread.join()
I'm not sure if that's the right idea but it gave me the error when I connected to BrowserServer port via a web browser:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 36632)
Traceback (most recent call last):
File "/home/user/.pyenv/versions/3.7.3/lib/python3.7/socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "/home/user/.pyenv/versions/3.7.3/lib/python3.7/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
TypeError: __init__() takes 1 positional argument but 4 were given
Just so that everyone is clear on what I'm trying to accomplish: MyApp will run on a server and open 2 ports. ServerBehindNAT will be running client code which will maintain a connection to MyApp. BrowserUser will use MyApp server IP as a SOCKS server in his browser. When BrowserUser visits a site, the request will go through MyApp to ServerBehindNAT, who will process the request and send responses back through MyApp.
I overrided the method emit of python logging httphandler to adapt it to my needs, and I noticed the line
h.getresponse() #can't do anything with the result
Why is this line necessary?
I noticed that removing this line has no effect when using unsecure logging, but makes the logs fail when using secure connection.
def emit(self, record):
"""
Emit a record.
Send the record to the Web server as a percent-encoded dictionary
"""
try:
import http.client, urllib.parse
host = self.host
if self.secure:
h = http.client.HTTPSConnection(host, context=self.context)
else:
h = http.client.HTTPConnection(host)
url = self.url
data = urllib.parse.urlencode(self.mapLogRecord(record))
if self.method == "GET":
if (url.find('?') >= 0):
sep = '&'
else:
sep = '?'
url = url + "%c%s" % (sep, data)
h.putrequest(self.method, url)
# support multiple hosts on one IP address...
# need to strip optional :port from host, if present
i = host.find(":")
if i >= 0:
host = host[:i]
# See issue #30904: putrequest call above already adds this header
# on Python 3.x.
# h.putheader("Host", host)
if self.method == "POST":
h.putheader("Content-type",
"application/x-www-form-urlencoded")
h.putheader("Content-length", str(len(data)))
if self.credentials:
import base64
s = ('%s:%s' % self.credentials).encode('utf-8')
s = 'Basic ' + base64.b64encode(s).strip().decode('ascii')
h.putheader('Authorization', s)
h.endheaders()
if self.method == "POST":
h.send(data.encode('utf-8'))
h.getresponse() #can't do anything with the result
except Exception:
self.handleError(record)
The getresponse() call guarantees that the request is actually sent to the server by getting the response to the request.
Im coding a python script that connects to a remote server, and parses the returned response. For some odd reason, 9 out of 10 times, Once the header is read, the script continues and returns before getting the body of the response. Im no expert at python, but im certain that my code is correct on the python side of things. Here is my code:
class miniclient:
"Client support class for simple Internet protocols."
def __init__(self, host, port):
"Connect to an Internet server."
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.settimeout(30)
try:
self.sock.connect((host, port))
self.file = self.sock.makefile("rb")
except socket.error, e:
#if e[0] == 111:
# print "Connection refused by server %s on port %d" % (host,port)
raise
def writeline(self, line):
"Send a line to the server."
try:
# Updated to sendall to resolve partial data transfer errors
self.sock.sendall(line + CRLF) # unbuffered write
except socket.error, e:
if e[0] == 32 : #broken pipe
self.sock.close() # mutual close
self.sock = None
raise e
except socket.timeout:
self.sock.close() # mutual close
self.sock = None
raise
def readline(self):
"Read a line from the server. Strip trailing CR and/or LF."
s = self.file.readline()
if not s:
raise EOFError
if s[-2:] == CRLF:
s = s[:-2]
elif s[-1:] in CRLF:
s = s[:-1]
return s
def read(self, maxbytes = None):
"Read data from server."
if maxbytes is None:
return self.file.read()
else:
return self.file.read(maxbytes)
def shutdown(self):
if self.sock:
self.sock.shutdown(1)
def close(self):
if self.sock:
self.sock.close()
self.sock = None
I use the ReadLine() method to read through the headers until i reach the empty line (Delimiter between headers and body). From there, my objects just call the "Read()" method to read the body. As stated before, 9 of 10 times, read returns nothing, or just partial data.
Example use:
try:
http = miniclient(host, port)
except Exception, e:
if e[0] == 111:
print "Connection refused by server %s on port %d" % (host,port)
raise
http.writeline("GET %s HTTP/1.1" % str(document))
http.writeline("Host: %s" % host)
http.writeline("Connection: close") # do not keep-alive
http.writeline("")
http.shutdown() # be nice, tell the http server we're done sending the request
# Determine Status
statusCode = 0
status = string.split(http.readline())
if status[0] != "HTTP/1.1":
print "MiniClient: Unknown status response (%s)" % str(status[0])
try:
statusCode = string.atoi(status[1])
except ValueError:
print "MiniClient: Non-numeric status code (%s)" % str(status[1])
#Extract Headers
headers = []
while 1:
line = http.readline()
if not line:
break
headers.append(line)
http.close() # all done
#Check we got a valid HTTP response
if statusCode == 200:
return http.read()
else:
return "E\nH\terr\nD\tHTTP Error %s \"%s\"\n$\tERR\t$" % (str(statusCode), str(status[2]))
You call http.close() before you call http.read(). Delay the call to http.close() until after you have read all of the data.