We Keep Coding

Get incoming URL from listening socket/HTTPServer

Good day Stackoverflow,
This morning I've ran into a problem of which I can't seem to find a working answer. I'm trying to get the full URL (what shows up in the address bar) via either a HTTPServer or simple socket-ing that I get from a server that redirects me to localhost (Which has nothing behind it (no webserver, no pages, nothing), except the listening code below.) with the token and scope variables (as seen in the URL in question below). My desired result would be these variables to be saved so I can work with them:
http://localhost/#token=aai789as&scope=book%3Aedit+chat%3Aedit
I have tried the following with some progress but not the desired result:
from http.server import SimpleHTTPRequestHandler, HTTPServer
from urllib.parse import parse_qs
class MyHandler(SimpleHTTPRequestHandler):
def do_GET(self):
qs = {}
path = self.path
if '?' in path:
path, tmp = path.split('?', 1)
qs = urlparse.parse_qs(tmp)
print(self.path)
print (path, qs)
def log_request(self, code=None, size=None):
print('Request')
def log_message(self, format, *args):
print('Message')
if __name__ == "__main__":
try:
server = HTTPServer(('localhost', 80), MyHandler)
print('Started http server')
server.serve_forever()
except KeyboardInterrupt:
print('^C received, shutting down server')
server.socket.close()
The above snippet loads, but doesn't actually print anything of use. In fact, it prints just blank statements. But it does detect a connection being made.
So does this:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("localhost", 80))
s.listen(1)
conn, addr = s.accept()
d = conn.recv(4096)
conn.close()
print(d)
But this DOES return more than blank statements, yet it's hardly enough to get the variables from the URL:
b'\x07\n'
b'GET / HTTP/1.1\r\nHost: localhost\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en,en-US;q=0.7,nl;q=0.3\r\nAccept-Encoding: gzip, deflate\r\nDNT: 1\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\n\r\n'
I don't know what I'm supposed to be doing and since I don't know what exactly I'm looking for; searching through the documentation has taken up the better half of my day. As such I turn to the ever helpful Stackoverflow in the hopes of finding better knowledge than I possess.
Thank you for your time,
- Brent

http://localhost/#token=aai789as&scope=book%3Aedit+chat%3Aedit
This kind of URL is only transferred in part to the server. The # and everything after is only known to the browser and can be accessed as location.hash from withing Javascript. It will not be transferred to the server, i.e. all the server will see is http://localhost/.
b'GET / HTTP/1.1\r\nHost: localhost\r\n ...'
This part provides everything from the URL which the server will know. The localhost in the Host header specifies the hostname and the GET / specifies the path / - which makes together 'http://' + 'localhost' + '/' i.e. http://localhost/.
For more information see Is the anchor part of a URL being sent to a web server?.

How to download image from HTTP server [Python/sockets]

I want to download an example image from a HTTP server using methods defined in HTTP protocol (and socket's, of course).
I tried to implement it, but it seems that my code does not download the whole image, no matter if I have the while loop or not.
An example image is here: https://httpbin.org/image/png.
My code downloads only part of the image, and I do not know how to fix it. I do not want use any libraries, such as urllib, I want to use just the sockets.
Any ideas?
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('httpbin.org', 80))
s.sendall('GET /image/png HTTP/1.1\r\nHOST: httpbin.org\r\n\r\n')
reply = ""
while True:
data = s.recv(2048)
if not data: break
reply += data
# get image size
size = -1
tmp = reply.split('\r\n')
for line in tmp:
if "Content-Length:" in line:
size = int(line.split()[1])
break
headers = reply.split('\r\n\r\n')[0]
image = reply.split('\r\n\r\n')[1]
# save image
f = open('image.png', 'wb')
f.write(image)
f.close()

You are doing a HTTP/1.1 request. This HTTP version implicitly behaves like Connection: keep-alive was set. This means that the server might not close the TCP connection immediately after sending the response as you expect in your code but might keep the connection open to wait for more HTTP requests.
When replacing the version with HTTP/1.0 instead the server closes the connection after the request is done and the image is complete because HTTP/1.0 implies Connection: close.
Apart from that: HTTP is way more complex than you might think. Please don't just design your code after some example messages you've seen somewhere but actually read and follow the standards if you really want to implement HTTP yourself.

import socket
import select
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('httpbin.org', 80))
s.sendall(b'GET /image/png HTTP/1.1\r\nHOST: httpbin.org\r\n\r\n')
reply = b''
while select.select([s], [], [], 3)[0]:
data = s.recv(2048)
if not data: break
reply += data
headers = reply.split(b'\r\n\r\n')[0]
image = reply[len(headers)+4:]
# save image
f = open('image.png', 'wb')
f.write(image)
f.close()
Note this example is not perfect. The elegant way should be checking Content-Length header and recv exact length of data. (Instead of hard coding 3 seconds as timeout.) And if the server can use chunked encoding, it becomes even more complicated.)
--
The example is in python 3

when i made a 3 handshake with ubuntu in VMware return package R

#!/usr/bin/python
from scapy.all import *
def findWeb():
a = sr1(IP(dst="8.8.8.8")/UDP()/DNS(qd=DNSQR(qname="www.google.com")),verbose=0)
return a[DNSRR].rdata
def sendPacket(dst,src):
ip = IP(dst = dst)
SYN = TCP(sport=1500, dport=80, flags='S')
SYNACK = sr1(ip/SYN)
my_ack = SYNACK.seq + 1
ACK = TCP(sport=1050, dport=80, flags='A', ack=my_ack)
send(ip/ACK)
payload = "stuff"
PUSH = TCP(sport=1050, dport=80, flags='PA', seq=11, ack=my_ack)
send(ip/PUSH/payload)
http = sr1(ip/TCP()/'GET /index.html HTTP/1.0 \n\n',verbose=0)
print http.show()
src = '10.0.0.24'
dst = findWeb()
sendPacket(dst,src)
I'm trying to do HTTP packets with SCAPY
I am using UBUNTU on VMwaer
The problem is that every time I send messages I have RESET
How do we fix it?
Thanks
sniff package image

Few things I notice wrong.
1. You have your sequence number set statically (seq=11) which is wrong. Sequence numbers are always randomly generated and they must be used as per RFC793. So the sequence should be = SYNACK[TCP].ack
You set your source port as 1500 during SYN packet, but then you use it as 1050 (typo?)
You don't need extra payload/PUSH.
Also, have a look at these threads:
How to create HTTP GET request Scapy?
Python-Scapy or the like-How can I create an HTTP GET request at the packet level

Python proxy -> dansguardian: How to send original source ip?

I have a python proxy for DNS. When I get a DNS request I need to pass an http request to dansguardian on behalf of the original source, let it to decide what happens to the request, get the result and redirect client to elsewhere based on the response from dansguardian.
The network skeleton is like this:
Client -> DNS Proxy -> DG -> Privoxy -> Web.
Client requests A, DNS Proxy intercepts, asks DG on behalf of the client, get's answer: 1. If DG filtered it, proxy send a local ip address instead of actual IP for A question. 2. If DG didn't filter, DNS proxy let's the client's net to flow naturally.
Here is the sample python code that I've tried:
data,addr = sock.recvfrom(1024)
OriginalDNSPacket = data
# I get OriginalDNSPacket from a socket
# to which iptables redirected all port 53 packets
UDPanswer = sendQues(OriginalDNSPacket, '8.8.8.8')
proxies = {'http': 'http://127.0.0.1:8080'} # DG Port
s = requests.Session()
d = DNSRecord.parse(UDPanswer)
print d
ques_domain = str(d.questions[0].get_qname())[:-1]
ques_tld = tldextract.extract(ques_domain)
ques_tld = "{}.{}".format(ques_tld.domain, ques_tld.suffix)
print ques_tld
for rr in d.rr:
try:
s.mount("http://"+ques_tld, SourceAddressAdapter(addr[0])) # This was a silly try, I know.
s.proxies.update(proxies)
response = s.get("http://"+ques_tld)
print dir(response.content)
print response.content
if "Access Denied" in response.content:
d.rr = []
d.add_answer(*RR.fromZone(ques_domain + " A " + SERVER_IP))
d.add_answer(*RR.fromZone(ques_domain + " AAAA fe80::a00:27ff:fe4a:c8ec"))
print d
socket.sendto(d.pack(), addr)
return
else:
socket.sendto(UDPanswer, addr)
return
except Exception, e:
print e
pass
The question is how can I send the request to DG, and fool it, like, the req comes from a client?

In dansguardian.conf, usexforwardedfor is needed to enabled.
So the conf now looks like this:
...
# if on it adds an X-Forwarded-For: <clientip> to the HTTP request
# header. This may help solve some problem sites that need to know the
# source ip. on | off
forwardedfor = on
# if on it uses the X-Forwarded-For: <clientip> to determine the client
# IP. This is for when you have squid between the clients and DansGuardian.
# Warning - headers are easily spoofed. on | off
usexforwardedfor = on
...
And on proxy server I just needed to add the following, which I tried before but because of the DG conf it didn't work:
response = s.get("http://"+ques_tld, headers={'X-Forwarded-For': addr[0]})
It worked like a charm.
Thanks #boardrider.

Change TCP Payload with nfqueue/scapy

Hello I am using nfqueue and scapy and I my goal is to recieve packets at my NFQUEUE, change the payload and resend them.
I can change fields like the TTL without any kind of problem, but when it comes to change the payload, I am encoutering problems.
When I change the payload, I sniff the packet with wireshark and apparently I send the packet with the payload modified, but the server doesn't answer.
This is my code:
#!/usr/bin/env python
import nfqueue
from scapy.all import *
def callback(payload):
data = payload.get_data()
pkt = IP(data)
pkt[TCP].payload = str(pkt[TCP].payload).replace("ABC","GET")
pkt[IP].ttl = 40
print 'Data: '+ str(pkt[TCP].payload)
print 'TTL: ' + str(pkt[IP].ttl)
del pkt[IP].chksum
payload.set_verdict_modified(nfqueue.NF_ACCEPT, str(pkt), len(pkt))
def main():
q = nfqueue.queue()
q.open()
q.bind(socket.AF_INET)
q.set_callback(callback)
q.create_queue(0)
try:
q.try_run() # Main loop
except KeyboardInterrupt:
q.unbind(socket.AF_INET)
q.close()
main()
I have set this rule for outgoing traffic to port 80: iptables -I OUTPUT -s 192.168.1.10 -p tcp --dport 80 -j NFQUEUE
And, to test it, for example I open telnet to google port 80, do a GET / HTTP/1.1 and this is what I see:
TTL: 40
DATA: GET / HTTP/1.1
Now, if I do ABC / HTTP/1.1 I receive no answer! My telnet just get stuck.
I have also tried on HTTP websites browers to browse something, check on wireshark how my TTL is really changing to 40, then, browse the string "ABC" and my browser again get stuck.
I sniff the request changed to GET but I receive no answer.
Thank is kind of giving me a headache and I would really appreciate if someone with more experience could lead me to the right way. Thank you in advance.

I added the line for recalculate the TCP checksum, that was usefull.
That only works if I change payload I don't alter the lenght of it, otherwise, I would need to change the field length of the IP Header, and answering myself, and maybe other people that is looking for this answer, I achieve that just by doing:
payload_before = len(pkt[TCP].payload)
pkt[TCP].payload = str(pkt[TCP].payload).replace("Heading","Other string")
payload_after = len(pkt[TCP].payload)
payload_dif = payload_after - payload_before
pkt[IP].len = pkt[IP].len + payload_dif
I know that I have to change more fields, because sometimes, if you add enough payload for needing to fragment into a new packet, you have to change more fields.
Currently I don't know how to achieve this efficiently but little by little. Hope someone find my solution for altering the payload useful.

In the second case, you are tampering the TCP layer as well as the IP layer.
You're letting Scapy fix the IP checksum, but not the TCP one. Change del pkt[IP].chksum to del pkt[IP].chksum pkt[TCP].chksum in your code.