I try to build a simple web server using Python.
I try to send a minimum response to mozilla web browser as a client. But, the client browser keep spinning. Code is below:
import socket
mysocket = socket.socket(2,1)
mysocket.bind(('',80))
mysocket.listen(5)
cli2,addr2 = mysocket.accept()
print('Client connected')
status = b'HTTP/1.1 200 OK\r\n'
connection_type=b'Connection: close\r\n'
content_type = b'Content-Type: text/html\r\n'
server = b'Server: Python-Server/5.2\r\n\r\n'
f = open('c:/users/totz/documents/index.html','r')
data = f.read()
data_b = data.encode()
content_html_length_calculation = len(data) * 8
content_length_header = 'Content-Length: ' + str(content_html_length_calculation) + '\r\n'
content_length_header_b = content_length_header.encode()
sending_data = status + connection_type + content_length_header_b + content_type + server
cli2.send(sending_data)
print('Data sent')
mysocket.close()
Why the client keep spinning, even Wireshark has told me that this web server has sent this response correctly to the client?
content_html_length_calculation = len(data) * 8
It looks like that you assume that the content-length is given in bits since you multiple the length of the data with 8. Only, the content-length is given in bytes. Since your claimed content-length is far bigger than the actual data the browser is still waiting for more data.
Apart from that your server does not read the request from the client which might cause additional problems (like reports about "Connection reset" if you close the client socket cli).
Related
Im trying to make a web proxy in python which is able to get the texts from the main server but not the images. The url http://gaia.cs.umass.edu/wireshark-labs/HTTP-wireshark-file1.html contains a line of text i am able to view in the browser and the url http://images.mid-day.com/images/2017/feb/15-salman-khan.jpg contains a image which i am not able to display in the browser. Im using Google Chrome. Below is my code. (I have hard-coded the hostname of the the image url for this post). Can anyone help me fix the problem.
from socket import *
client= socket(AF_INET, SOCK_STREAM)
proxy_port = 8880
client.bind(("", proxy_port ))
client.listen(10)
while 1:
client_connection, client_address = CLIENT.accept()
request = client_connection.recv(102400).decode()
if request.startswith("GET"):
try:
print(request)
web = socket(AF_INET, SOCK_STREAM)
web.connect(("images.mid-day.com", 80))
web.send(request.encode())
reply = web.recv(102400).decode()
print(reply)
client_connection.send(reply.encode())
web.close()
except:
print("illegal req")
client.close()
This is my get request from the browser:
You have only read 102400 bytes from the upstream server, but the image response is (at least) 567702 bytes. You should read until upstream shutdown connection, besides use sendall() to make sure all data has been sent:
reply = b''
while True:
data = web.recv(4096)
if not data:
break
reply += data
client_connection.sendall(reply)
I need to create HTTP GET request and save the data response.
I tried to use this:
syn = IP(dst=URL) / TCP(dport=80, flags='S')
syn_ack = sr1(syn)
getStr = 'GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n'
request = IP(dst='www.google.com') / TCP(dport=80, sport=syn_ack[TCP].dport,
seq=syn_ack[TCP].ack, ack=syn_ack[TCP].seq + 1, flags='A') / getStr
reply = sr1(request)
print reply.show()
But when I print reply I don't see any data response.
In addition, when I checked in 'Wireshark' I got SYN, SYN/ACK but I didn't get an ACK.
Image:
Edit:
I try to do that now:
# Import scapy
from scapy.all import *
# Print info header
print "[*] ACK-GET example -- Thijs 'Thice' Bosschert, 06-06-2011"
# Prepare GET statement
get='GET / HTTP/1.0\n\n'
# Set up target IP
ip=IP(dst="www.google.com")
# Generate random source port number
port=RandNum(1024,65535)
# Create SYN packet
SYN=ip/TCP(sport=port, dport=80, flags="S", seq=42)
# Send SYN and receive SYN,ACK
print "\n[*] Sending SYN packet"
SYNACK=sr1(SYN)
# Create ACK with GET request
ACK=ip/TCP(sport=SYNACK.dport, dport=80, flags="A", seq=SYNACK.ack, ack=SYNACK.seq + 1) / get
# SEND our ACK-GET request
print "\n[*] Sending ACK-GET packet"
reply,error=sr(ACK)
# print reply from server
print "\n[*] Reply from server:"
print reply.show()
print '\n[*] Done!'
but its print me in reply from server;
0000 IP / TCP 192.168.44.130:23181 > 216.58.208.164:http A / Raw ==>
IP / TCP 216.58.208.164:http > 192.168.44.130:23181 A / Padding None
And I need Line-based text data: text/html.
You are sending a RST segment in response to the SYN-ACK because your kernel has no knowledge of the SYN you sent via Scapy (see here). This could be solved with an iptable rule:
iptables -A OUTPUT -p tcp --tcp-flags RST RST -s <your ip> -j DROP
Because you are ending the connection with that RST segment, when you send your HTTP request, the endpoint answers with a RST too because connection is not established and so you are using show() on a RST segment with no data, that is why you do not see anything.
You are sending a SYN and correctly receiving a SYN_ACK. At this point, you should generate and send an ACK based on the SYN_ACK that you've received, and THEN finally transmit the HTTP GET request. It seems that you are somewhat confused about the TCP 3-way handshake mechanism. In short, you are not supposed to 'get' an ACK, you are supposed to generate and send this yourself.
After setting the rule in your iptables as has been suggested above, you could do the following :
from scapy.all import *
seq = 12345
sport = 1040
dport = 80
ip_packet = IP(dst='192.168.56.107')
syn_packet = TCP(sport=sport, dport=dport, flags='S', seq=seq)
packet = ip_packet/syn_packet
synack_response = sr1(packet)
next_seq = seq + 1
my_ack = synack_response.seq + 1
ack_packet = TCP(sport=sport, dport=dport, flags='A', seq=next_seq, ack=my_ack)
send(ip_packet/ack_packet)
payload_packet = TCP(sport=sport, dport=dport, flags='A', seq=next_seq, ack=my_ack)
payload = "GET / HTTP/1.0\r\nHOST: 192.168.56.107\r\n\r\n"
reply, error = sr(ip_packet/payload_packet/payload, multi=1, timeout=1)
for r in reply:
r[0].show2()
r[1].show2()
Hope this helps. Basically, the first response you get back does not really hold the HTTP response data. I tested the script against an INETSIM simulated HTTP server and in that case (at least) the first packet (after the 3-way TCP handshake) that the server responded with was a series of NULL (0x00) bytes. Hence using multi somehow did the stuff in my case.
I'm trying to send a stream of images across http using a "persistent" connection. I have keep-alive in the header and so does the client (firefox) but I dont know how to send the second request or how to wrap it.
I've read the rfc. I've also wiresharked the connection while looking at a streaming service just like the one I'm trying to duplicate. I dont see any http messages going back and forth as the data is streamed (mjpg streamer)
So My question is this, what is required to send back to client in a http persistent connection ? If I just send fully formed http responses or just jpg data the client disconnects. If I wait for the client to send more data before sending the second image the client never sends anything.
import socket
import sys
import time
import binascii
from thread import *
HOST = '' # Symbolic name meaning all available interfaces
PORT = 9998 # Arbitrary non-privileged port
in_file = open("picture.jpg", "rb") # opening for [r]eading as [b]inary
testJpg = in_file.read() # if you only wanted to read 512 bytes, do .read(512)
in_file.close()
in_file = open("picture2.jpg", "rb") # opening for [r]eading as [b]inary
testJpg2 = in_file.read() # if you only wanted to read 512 bytes, do .read(512)
in_file.close()
OddNumber = 0
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print 'Socket created'
#Bind socket to local host and port
try:
s.bind((HOST, PORT))
except socket.error , msg:
print 'Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1]
sys.exit()
print 'Socket bind complete'
#Start listening on socket
s.listen(10)
print 'Socket now listening'
#Function for handling connections. This will be used to create threads
def clientthread(conn):
global OddNumber, testJpg, testJpg2
OddNumber = 0
# get data from the client (assume its a http get request)
data = conn.recv(1024)
print("\r\n Got1: "+str(len(data)))
#time.sleep(.05)
while OddNumber < 30:
# create a new reply
reply = "HTTP/1.0 200 OK\r\n"
reply += "Server: MJPG-Streamer/0.2\r\n"
reply += "Cache-Control: no-store, no-cache, must-revalidate, pre-check=0, post-check=0, max-age=0\r\n"
reply += "Pragma: no-cache\r\n"
reply += "Content-type: image/jpeg\r\n"
# add in the content len
if OddNumber % 2 == 0:
reply += "Content-Length: "+str(len(testJpg))+"\r\n"
else:
reply += "Content-Length: "+str(len(testJpg2))+"\r\n"
# add the keep alive portion
reply += "Connection: Keep-Alive\r\n"
# I think we need this
reply += "\r\n"
# I use this toggle to switch between different images so if it does "stream" i'll be able to see it
# flipping these images
if OddNumber % 2 == 0:
reply += testJpg
else:
reply += testJpg2
# after the image
reply += "\r\n"
conn.sendall(reply)
OddNumber += 1
print("\r\n OddNum: "+str(OddNumber))
data = conn.recv(1024)
print("\r\n Got1: "+str(len(data)))
print(str(data))
#time.sleep(.05)
#came out of loop
conn.close()
#now keep talking with the client
while 1:
#wait to accept a connection - blocking call
conn, addr = s.accept()
print 'Connected with ' + addr[0] + ':' + str(addr[1])
#start new thread takes 1st argument as a function name to be run, second is the tuple of arguments to the function.
start_new_thread(clientthread ,(conn,))
s.close()
Apart from that I'm wondering if the persistent connection will even work for me. I'm developing a embedded device that is hosting the images, similar to a ipcamera. But I need to be able to send different URL's back to the device while it is sending me images. Is this possible on the same port ?
Someone requested I show whats in my http response. It is what you see in the python script. Its a valid http response and it works fine if I set
Connection: Keep-Alive\r\n"
to
Connection: close\r\n"
I guess my question is, what do you have to do in a persistent connection. Will the client send more "get" requests or is the server expected to stream some data ? At least firefox only sends one "get" request and then I send the response with a image but then what has to happen next to keep it going ?
Thanks for all the help so far.
I'm afraid that's because you use the same socket object in both client and server. Can you separate your code into two files? One file for server and the other for client (means your clientthread function).
In HTTP, when client get an HTTP response with 'keep-alive', it will hold TCP connection no matter what kinds of data you send.
In python 3.3, I want to get response headers from a youtube webpage. Using HTTP 1.0, the code below works fine:
import socket
PATH='/watch?v=GVIjOr98B7Q'
HOST='www.youtube.com'
buffer = bytes('HEAD %s HTTP/1.0\r\nHost: %s\r\n\r\n' %(PATH, HOST),'ascii')
PORT=80
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.sendall(buffer)
td=b''
while 1:
data = s.recv(1024)
td+=data
if not data:
break
However, if I replace HTTP 1.0 with HTTP 1.1:
buffer = bytes('HEAD %s HTTP/1.1\r\nHost: %s\r\n\r\n' %(PATH, HOST),'ascii')
and any other lines remain the same. It will stop at the while loop for a really long time (It is not looping but waiting for the end signal). Why does this happen?
The HTTP 1.1 keeps connections open unless you pass along the header Connection: close. See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
If you want the connection to close immediately either use HTTP 1.0 or send the header Connection: close
Following is the code which listens on a port for HTTP requests and sends the request packet to the server running on port 80, gets the response and sends the data back to the client. Now, everything is executing fine but the following line of code :
data = req_soc.recv(1024)
is taking too much time to execute and I have observed that, it takes long time to execute when it is going to/has received the last packet. I have also tried the same code using select.select() but the results are the same. Since I want to handle the data (raw) that is coming from the client and the actual HTTP server, I have no other choice than using sockets.
import socket
import thread
def handle_client(client):
data = client.recv(512)
request = ''
request += data
print data
print '-'*20
spl = data.split("\r\n")
print spl[0]
print spl[1]
if len(request):
req_soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
req_soc.connect(('localhost', 80))
req_soc.send(request)
response = ''
data = req_soc.recv(1024)
while data:
response += data
print 1
data = req_soc.recv(1024)
req_soc.close()
print response
if len(response):
client.send(response)
client.close()
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('localhost', 4422))
server.listen(5)
print("Server is running...\n")
MSGLEN = 1024
while 1:
client, address = server.accept()
thread.start_new_thread(handle_client, (client, ))
Clients can do multiple commands (eg: GET) within one connection. You cannot wait for the client to send all the commands because based on what you return it could request more (eg: images of a web page). You have to parse the parts (commands) of request, find the boundary, forward that request to the server and write back the answer to the client. All this in a way that doesn't block on reading the client.
I'm not sure what's the best way to do this in python, but if you spend 5 minutes of googling you'll find a perfect HTTP proxy library.