I'm using python version 2.7.9 and i try to send png file.
But something strange happens..i using sockets and sends a post request(or kind of).
I send the request to the server from the client,then i prints the length of the request received on the server, for example, the length is:1051.
Then I do a regex to take the png file data, and then prints the length, and the length is 2632, that he larger than the response?!
I think the problem is that it's actually write the content, but not the right of representation, I tried different things but they did not work, so I ask here how to solve this problem.
Server source code:
import socket
import re
server = socket.socket()
server.bind(('0.0.0.0',8080))
while True:
server.listen(2)
(client, client_addr) = server.accept()
print 'IP :',client_addr
res = client.recv(0xfffffff)
print len(res)
#get file name
file_name = res.split('&')[0]
file_name = str(file_name.split('=')[1])
print repr(res)
#get the data of the file
raw_img = str(re.findall("&photo_data=(.*)" ,res ,re.DOTALL))
print "File name:" + file_name
print "Size:" + str(len(raw_img))
with open(file_name, 'wb') as f:
f.write(raw_img)
print "Done"
Client source code:
import socket
client = socket.socket()
client.connect(('127.0.0.1',8080))
raw_data = open('test.png', 'rb').read()
save_file_name = raw_input("Enter the file name:")
print len(raw_data)
output = 'POST /upload HTTP/1.1\r\n'
output += 'Content-Length:' + str(len(raw_data)) + str(len(save_file_name)) + '\r\n\r\n'
output += 'file_name=' + save_file_name + '&'
output += 'photo_data=' + raw_data
print len(output)
client.send(output)
client.close()
First, you should use while True to receive the full data:
res = ''
while True:
data = client.recv(1024)
if not data:
break
res += data
print len(res)
Then, re.findall actually returns an array, not a string. So you should do this:
r = re.findall("&photo_data=(.*)" ,res ,re.DOTALL)
raw_img = str(r[0])
Now it works fine.
Why doesn't the code before work? Let's say we have a list:
r = ['\x45']
The data in raw_img part is basically like this. If we brutely convert this list to a str, we have:
print len(str[r])) # ['E'], 5
Actually, what we need is r[0]:
print len(str[r[0])) # 1
That's why the size of the file became larger.
Related
I have a pcapng file that contains a little bit of traffic. One of the packets I am trying to print out is containing JSON data. If I open the packet up in Wireshark, I am able to see the values in the JSON. But when using scapy to read the file and print I don't see it.
from scapy.all import IP, sniff
from scapy.layers import http
def process_tcp_packet(packet):
if packet.haslayer(http.HTTPRequest):
http_layer = packet.getlayer(http.HTTPRequest)
ip_layer = packet.getlayer(IP)
#print('\n{0[src]} just requested a {1[Method]} {1[Host]}{1[Path]}'.format(ip_layer.fields, http_layer.fields))
#print(ip_layer.fields)
#print(http_layer.fields)
#packet.show()
print('Packet: ' + str(packet))
print('\n\n')
# Start sniffing the network.
sniff(offline='test.pcapng', prn=process_tcp_packet, count=2)
Here is the JSON content Wireshark is showing me:
And this is the output I am getting for that packet using the code above..
Packet: b'\x18\x0fv\xef0\x8a\xc4\x98\\\xe7=\x18\x08\x00E\x00\x01&&S#\x00#\x06}\n\xc0\xa8\x89\x94#\xa7(\x91\x9b\xd0\x00P\x16-/\x9e\xb1\xa1\xe8V\x80\x18\x01K\x97\xaf\x00\x00\x01\x01\x08\n\x00\x00\t\xd5\xfb\xc3b\x89POST /v1/identify HTTP/1.1\r\nHost: api.segment.io\r\nUser-Agent: Roku/DVP-9.10 (489.10E04121A)\r\nAccept: application/json\r\nAuthorization: Basic: NHJmY3AzUEJmTUhPVlJsWVZZNTZKRDZ0N1JuMUNoaVY=\r\nContent-Type: application/json\r\nContent-Length: 704\r\n\r\n'
I was reading on how to print the entire content of the packet and thats where I came across both packet.show() and print(packet) however both of them are still missing the JSON data.
I want to get the JSON data because I want to be able to manually parse it. I don't like how Wireshark has all the JSON nested into arrows that I have to drop down to see.
This is the output of show:
And I am using the latest version of scapy.
It's an old question, but for future people who search for an answer, here is how I did it:
packet_dict = {}
for line in packet.show2(dump=True).split('\n'):
if '###' in line:
layer = line.strip('#[] ')
packet_dict[layer] = {}
elif '=' in line:
key, val = line.split('=', 1)
packet_dict[layer][key.strip()] = val.strip()
print(json.dumps(packet_dict))
If it can be useful to someone, starting from Yechiel's code I made some improvements:
Key values are returned in the correct format instead of all as a string
Sublayers are parsed
def pkt2dict(pkt):
packet_dict = {}
for line in pkt.show2(dump=True).split('\n'):
if '###' in line:
if '|###' in line:
sublayer = line.strip('|#[] ')
packet_dict[layer][sublayer] = {}
else:
layer = line.strip('#[] ')
packet_dict[layer] = {}
elif '=' in line:
if '|' in line and 'sublayer' in locals():
key, val = line.strip('| ').split('=', 1)
packet_dict[layer][sublayer][key.strip()] = val.strip('\' ')
else:
key, val = line.split('=', 1)
val = val.strip('\' ')
if(val):
try:
packet_dict[layer][key.strip()] = eval(val)
except:
packet_dict[layer][key.strip()] = val
else:
log.debug("pkt2dict packet not decoded: " + line)
return packet_dict
To check if it works on all types of layers returned by scapy.
So I have to send a specific message to a server (which already exists).
This request follows this format:
DO dirname number_of_files [file_size file_name file_data]*
so for example it can be something like this:
DO dir 2 1421 house.png [binary data I assume?] 1239 info.txt [more binary data?]
As you can see there can be as many files as you want.
So what I was doing is creating a string where I append all the information and at the end send it all together, but I'm stuck at the data part.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(ip, port)
request = "DO " + directory + " " + str(number_of_files)
for file_number in range(0,number_of_files):
data = open("./" + directory + "/" + files_info[file_number*2], 'rb').read()
client_request += " " + " ".join(files_info[file_number*2 : 1 + file_number*2]) + " " + data
s.send((request + "\n").encode())
This does not working because I'm trying to append the data which is binary to a string (TypeError: must be str, not bytes). I tried using .decode() before appending but I get an utf-8 decoder error (UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte). I tried with utf-16 and ascii and it didn't work either.
Should I be handling this in another way? I have done more requests like this but none of them included data from files, only numbers/strings so it was never a problem before.
I am trying to follow this format because I have all the code right now done like this (and these steps I showed don't all happen in the same functions) so changing the way I send the messages is not optimal
Here is my simple working code to get something close to what you want:
import socket
ip = '192.168.10.137'
port = 4043
directory = 'C:/123/'
#with open('1.dat', 'wb') as file:
# file.write(b'\x00\x01\x1a\xa1')
#with open('2.dat', 'wb') as file:
# file.write(b'\x00\x01\x00\x00')
file_info = ['1.dat', '2.dat']
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((ip, port))
sock.send(
'DO {} {}'.format(directory, len(file_info)).encode('UTF-8'))
for filename in file_info:
with open(filename, mode='rb') as file:
data = file.read()
sock.send(
' {} '.format(len(data)).encode('UTF-8'))
sock.send(
'{} '.format(filename).encode('UTF-8'))
sock.send(data)
Data received by server (bytes):
b'DO C:/123/ 2 4 1.dat \x00\x01\x1a\xa1 4 2.dat \x00\x01\x00\x00'
Client side:
def send_file_to_hashed(data, tcpsock):
time.sleep(1)
f = data
flag = 0
i=0
tcpsock.send(hashlib.sha256(f.read()).hexdigest())
f.seek(0)
time.sleep(1)
l = f.read(BUFFER_SIZE-64)
while True:
while (l):
tcpsock.send(hashlib.sha256(l).hexdigest() + l)
time.sleep(1)
hashok = tcpsock.recv(6)
if hashok == "HASHOK":
l = f.read(BUFFER_SIZE-64)
flag = 1
if hashok == "BROKEN":
flag = 0
if not l:
time.sleep(1)
tcpsock.send("DONE")
break
return (tcpsock,flag)
def upload(filename):
flag = 0
while(flag == 0):
with open(os.getcwd()+'\\data\\'+ filename +'.csv', 'rU') as UL:
tuplol = send_file_to_hashed(UL ,send_to_sock(filename +".csv",send_to("upload",TCP_IP,TCP_PORT)))
(sock,flagn) = tuplol
flag = flagn
time.sleep(2)
sock.close()
Server Side:
elif(message == "upload"):
message = rec_OK(self.sock)
fis = os.getcwd()+'/data/'+ time.strftime("%H:%M_%d_%m_%Y") + "_" + message
f = open(fis , 'w')
latest = open(os.getcwd()+'/data/' + message , 'w')
time.sleep(1)
filehash = rec_OK(self.sock)
print("filehash:" + filehash)
while True:
time.sleep(1)
rawdata = self.sock.recv(BUFFER_SIZE)
log.write("rawdata :" + rawdata + "\n")
data = rawdata[64:]
dhash = rawdata[:64]
log.write("chash: " + dhash + "\n")
log.write("shash: " + hashlib.sha256(data).hexdigest() + "\n")
if dhash == hashlib.sha256(data).hexdigest():
f.write(data)
latest.write(data)
self.sock.send("HASHOK")
log.write("HASHOK\n" )
print"HASHOK"
else:
self.sock.send("HASHNO")
print "HASHNO"
log.write("HASHNO\n")
if rawdata == "DONE":
f.close()
f = open(fis , 'r')
if (hashlib.sha256(f.read()).hexdigest() == filehash):
print "ULDONE"
log.write("ULDONE")
f.close()
latest.close()
break
else:
self.sock.send("BROKEN")
print hashlib.sha256(f.read()).hexdigest()
log.write("BROKEN")
print filehash
print "BROKEN UL"
f.close()
So the data upload is working fine in all tests that i ran from my computer, even worked fine while uploading data over my mobile connection and still sometimes people say it takes a long time and they kill it after a few minutes. the data is there on their computers but not on the server. I don't know what is happening please help!
First of all: this is unrelated to sha.
Streaming over the network is unpredictable. This line
rawdata = self.sock.recv(BUFFER_SIZE)
doesn't guarantee that you read BUFFER_SIZE bytes. You may have read only 1 byte in the worst case scenario. Therefore your server side is completely broken because of the assumption that rawdata contains whole message. It is even worse. If the client sends command and hash fast you may get e.g. rawdata == 'DONEa2daf78c44(...) which is a mixed output.
The "hanging" part just follows from that. Trace your code and see what happens when the server receives partial/broken messages ( I already did that in my imagination :P ).
Streaming over the network is almost never as easy as calling sock.send on one side and sock.recv on the other side. You need some buffering/framing protocol. For example you can implement this simple protocol: always interpret first two bytes as the size of incoming message, like this:
client (pseudocode)
# convert len of msg into two-byte array
# I am assuming the max size of msg is 65536
buf = bytearray([len(msg) & 255, len(msg) >> 8])
sock.sendall(buf)
sock.sendall(msg)
server (pseudocode)
size = to_int(sock.recv(1))
size += to_int(sock.recv(1)) << 8
# You need two calls to recv since recv(2) can return 1 byte.
# (well, you can try recv(2) with `if` here to avoid additional
# syscall, not sure if worth it)
buffer = bytearray()
while size > 0:
tmp = sock.recv(size)
buffer += tmp
size -= len(tmp)
Now you have properly read data in buffer variable which you can work with.
WARNING: the pseudocode for the server is simplified. For example you need to check for empty recv() result everywhere (including where size is calculated). This is the case when the client disconnects.
So unfortunately there's a lot of work in front of you. You have to rewrite whole sending and receving code.
I am trying to send a file over a tcp socket in a peer to peer chat system coded in python. The receiving socket seems not to know that there is no more file to receive. The only way I can get the receiving socket to not anticipate the data that isn't coming is by closing the sending socket (using socket.shutdown(socket.SHUT_WR)). However, closing the sending socket is not an option because I need that socket to send other messages. I fisrt tried allocating a new port for file sending/receiving and failed. Now, I have tried creating an end of file "signal" but it does not get recognized on the receiving end as a message separate from the tcp segment, so I am stuck.
The sending code looks like this:
def sendFile(self,filePath):
try:
f = open(filePath, 'rb')
print 'file opened'
for soc in self.allClients.keys():
try:
f = open(filePath, 'rb')
except:
print "File does not exist"
print 'Sending File: ' + filePath
l = f.read(1024)
while (l):
print 'Sending...'
soc.send(l)
l = f.read(1024)
soc.send('end')
f.close()
print 'File sent'
except:
print "File does not exist"
the receiving code looks like this:
def receiveFile(self, ext, clientsoc):
f = open('receivedFile' + ext,'wb')
print "Receiving File..."
l = clientsoc.recv(1024)
while(l):
print "Receiving..."
if (l is not 'end'):
f.write(l)
print l + '\n'
l = clientsoc.recv(1024)
else:
break
f.close()
print "Received Fileeeeeooooo"
Even more strange, this code works when I am using it outside of my peer programme. Any help would be greatly appreciated, I have been struggling with this for two days now.
First of all, you must not compare strings with is or is not:
>>> a = "he"
>>> a + "llo" is "hello"
False
Second, TCP is a streaming protocol. recv is getting up to 1024 bytes, but could be less, and if you send two pieces from your server, they can be merged into one recv. So l is not likely to be "end" but "last bytes of the file's end". And if you check for "end", "end" is not allowed to be in the file. Best solution is to send the length of the file first, and then send and recv length bytes.
PS: you probably want to use sendall at the server side.
Sending the file size first worked!!
New code looks like this.
Sending code:
def sendFileOrImage(self,path):
name, ext = os.path.splitext(path)
filesize = str(os.path.getsize(path))
message = filesize + ',' + ext
print 'message'
for client in self.allClients.keys():
client.send(message)
self.sendFile(path)
def sendFile(self,filePath):
try:
f = open(filePath, 'rb')
except:
print "File does not exist"
print 'Sending File: ' + filePathty
for soc in self.allClients.keys():
l = f.read(1024)
while (l):
print 'Sending...'
soc.send(l)
l = f.read(1024)
f.close()
print 'File sent'
Receiving code:
def receiveFile(self,ext, filesize, clientsoc):
total = 0
f = open('receivedFile' + ext,'wb')
print "Receiving File..."
l = clientsoc.recv(1024)
total = len(l)
while(l):
print "Receiving..."
f.write(l)
if (str(total) != filesize):
print 'trying to receive'
l = clientsoc.recv(1024)
total = total + len(l)
else:
break
f.close()
print "Received Fileeeeeooooo"
Thanks everyone for the help!
Your receiving code works but still there is a problem you need to fix real quick. If your filesize is under 1024, your socket will keep listening for ever and in order to fix this problem you need to change your code like below.
def receiveFile(self,ext, filesize, clientsoc):
total = 0
f = open('receivedFile' + ext,'wb')
print("Receiving File...")
l = clientsoc.recv(1024)
total = len(l)
while(l):
print("Receiving...")
f.write(l)
if (str(total) != filesize):
print 'trying to receive'
l = clientsoc.recv(1024)
total = total + len(l)
if(int(total)>= int(filesize)):
break
f.close()
print("Received Fileeeeeoooo")
I am trying to receive an image in python to use it in my program.
Here is the sever code:
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(("127.0.0.1", 5005))
server_socket.listen(5)
data = ' '
client_socket, address = server_socket.accept()
print "Conencted to - ",address,"\n"
while (1):
data = client_socket.recv(1024)
print "The following data was received - ",data
print "Opening file - ",data
img = open(data,'r')
while True:
strng = img.readline(512)
if not strng:
break
client_socket.send(strng)
img.close()
print "Data sent successfully"
exit()
#data = 'viewnior '+data
#os.system(data)
And here is the client code:
import socket,os
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(("127.0.0.1", 5005))
size = 1024
while(1):
print "Enter file name of the image with extentsion (example: filename.jpg,filename.png or if a video file then filename.mpg etc) - "
fname = raw_input()
client_socket.send(fname)
#fname = 'documents/'+fname
fp = open(fname,'w')
while True:
strng = client_socket.recv(512)
if not strng:
break
fp.write(strng)
fp.close()
print "Data Received successfully"
exit()
#data = 'viewnior '+fname
#os.system(data)
The received should now be read to be able to use it. I am opening it like this:
input_image = Image.open('data').convert('L').resize((100, 100))
but when I run both codes in cmd the output is:
The following data was received - + path Opening file - + path
Then nothing happens although the image should be used and the final output should be shown.
Anyone can help?
I don't know if this is your (only) problem, but when working with binary files, you should pass the b flag to the built-in function open:
img = open(data, 'rb')