I wrote a simple WebSocket client. I used the code I found on SO, here: How can I send and receive WebSocket messages on the server side?.
I'm using Python 2.7 and my server is echo.websocket.org on 80 TCP port. Basically, I think that I have a problem with receiving messages. (Or maybe the sending is wrong too?)
At least I am sure that the handshake is all ok, since I receive a good handshake response:
HTTP/1.1 101 Web Socket Protocol Handshake
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: content-type
Access-Control-Allow-Headers: authorization
Access-Control-Allow-Headers: x-websocket-extensions
Access-Control-Allow-Headers: x-websocket-version
Access-Control-Allow-Headers: x-websocket-protocol
Access-Control-Allow-Origin: http://example.com
Connection: Upgrade
Date: Tue, 02 May 2017 21:54:31 GMT
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Server: Kaazing Gateway
Upgrade: websocket
And my code:
#!/usr/bin/env python
import socket
def encode_text_msg_websocket(data):
bytesFormatted = []
bytesFormatted.append(129)
bytesRaw = data.encode()
bytesLength = len(bytesRaw)
if bytesLength <= 125:
bytesFormatted.append(bytesLength)
elif 126 <= bytesLength <= 65535:
bytesFormatted.append(126)
bytesFormatted.append((bytesLength >> 8) & 255)
bytesFormatted.append(bytesLength & 255)
else:
bytesFormatted.append(127)
bytesFormatted.append((bytesLength >> 56) & 255)
bytesFormatted.append((bytesLength >> 48) & 255)
bytesFormatted.append((bytesLength >> 40) & 255)
bytesFormatted.append((bytesLength >> 32) & 255)
bytesFormatted.append((bytesLength >> 24) & 255)
bytesFormatted.append((bytesLength >> 16) & 255)
bytesFormatted.append((bytesLength >> 8) & 255)
bytesFormatted.append(bytesLength & 255)
bytesFormatted = bytes(bytesFormatted)
bytesFormatted = bytesFormatted + bytesRaw
return bytesFormatted
def dencode_text_msg_websocket(stringStreamIn):
byteArray = [ord(character) for character in stringStreamIn]
datalength = byteArray[1] & 127
indexFirstMask = 2
if datalength == 126:
indexFirstMask = 4
elif datalength == 127:
indexFirstMask = 10
masks = [m for m in byteArray[indexFirstMask: indexFirstMask + 4]]
indexFirstDataByte = indexFirstMask + 4
decodedChars = []
i = indexFirstDataByte
j = 0
while i < len(byteArray):
decodedChars.append(chr(byteArray[i] ^ masks[j % 4]))
i += 1
j += 1
return ''.join(decodedChars)
# connect
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((socket.gethostbyname('echo.websocket.org'), 80))
# handshake
handshake = 'GET / HTTP/1.1\r\nHost: echo.websocket.org\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Key: gfhjgfhjfj\r\nOrigin: http://example.com\r\nSec-WebSocket-Protocol: echo\r\n' \
'Sec-WebSocket-Version: 13\r\n\r\n'
sock.send(handshake)
print sock.recv(1024)
# send test msg
msg = encode_text_msg_websocket('hello world!')
sock.sendall(msg)
# receive it back
response = dencode_text_msg_websocket(sock.recv(1024))
print '--%s--' % response
sock.close()
What is wrong here? It gets complicated after the handshake.
The dencode_text_msg_websocket method returns an empty string but it should return the same string I send to the server, which is hello world!.
I DO NOT WANT to use libraries (I know how to use them). The question is about achieving the same thing WITHOUT libraries, using only sockets.
I only want to send a message to echo.websocket.org server and receive a response, that's all. I do not want to modify the headers, just build the headers like they're used by this server. I checked how they should look like using Wireshark, and tried to build the same packets with Python.
For tests below, I used my browser:
Not masked data, from server to client:
Masked data, from client to server:
I have hacked your code into something that at least sends a reply and receives an answer, by changing the encoding to use chr() to insert byte strings rather than decimals to the header. The decoding I have left alone but the other answer here has a solution for that.
The real guts of this is detailed here https://www.rfc-editor.org/rfc/rfc6455.txt
which details exactly what it is that you have to do
#!/usr/bin/env python
import socket
def encode_text_msg_websocket(data):
bytesFormatted = []
bytesFormatted.append(chr(129))
bytesRaw = data.encode()
bytesLength = len(bytesRaw)
if bytesLength <= 125:
bytesFormatted.append(chr(bytesLength))
elif 126 <= bytesLength <= 65535:
bytesFormatted.append(chr(126))
bytesFormatted.append((chr(bytesLength >> 8)) & 255)
bytesFormatted.append(chr(bytesLength) & 255)
else:
bytesFormatted.append(chr(127))
bytesFormatted.append(chr((bytesLength >> 56)) & 255)
bytesFormatted.append(chr((bytesLength >> 48)) & 255)
bytesFormatted.append(chr((bytesLength >> 40)) & 255)
bytesFormatted.append(chr((bytesLength >> 32)) & 255)
bytesFormatted.append(chr((bytesLength >> 24)) & 255)
bytesFormatted.append(chr((bytesLength >> 16)) & 255)
bytesFormatted.append(chr((bytesLength >> 8)) & 255)
bytesFormatted.append(chr(bytesLength) & 255)
send_str = ""
for i in bytesFormatted:
send_str+=i
send_str += bytesRaw
return send_str
# connect
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(5.0)
try:
sock.connect((socket.gethostbyname('ws.websocket.org'), 80))
except:
print "Connection failed"
handshake = '\
GET /echo HTTP/1.1\r\n\
Host: echo.websocket.org\r\n\
Upgrade: websocket\r\n\
Connection: Upgrade\r\n\
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==\r\n\
Origin: http://example.com\r\n\
WebSocket-Protocol: echo\r\n\
Sec-WebSocket-Version: 13\r\n\r\n\
'
sock.send(bytes(handshake))
data = sock.recv(1024).decode('UTF-8')
print data
# send test msg
msg = encode_text_msg_websocket('Now is the winter of our discontent, made glorious Summer by this son of York')
print "Sent: ",repr(msg)
sock.sendall(bytes(msg))
# receive it back
response = sock.recv(1024)
#decode not sorted so ignore the first 2 bytes
print "\nReceived: ", response[2:].decode()
sock.close()
Result:
HTTP/1.1 101 Web Socket Protocol Handshake
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: content-type
Access-Control-Allow-Headers: authorization
Access-Control-Allow-Headers: x-websocket-extensions
Access-Control-Allow-Headers: x-websocket-version
Access-Control-Allow-Headers: x-websocket-protocol
Access-Control-Allow-Origin: http://example.com
Connection: Upgrade
Date: Mon, 08 May 2017 15:08:33 GMT
Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Server: Kaazing Gateway
Upgrade: websocket
Sent: '\x81MNow is the winter of our discontent, made glorious Summer by this son of York'
Received: Now is the winter of our discontent, made glorious Summer by this son of York
I should note here, that this is going to be a pig to code without pulling in some extra libraries, as #gushitong has done.
Accoding to https://www.rfc-editor.org/rfc/rfc6455#section-5.1:
You should mask the client frames. (And the server frames is not masked at all.)
a client MUST mask all frames that it
sends to the server (see Section 5.3 for further details). (Note
that masking is done whether or not the WebSocket Protocol is running
over TLS.) The server MUST close the connection upon receiving a
frame that is not masked. In this case, a server MAY send a Close
frame with a status code of 1002 (protocol error) as defined in
Section 7.4.1. A server MUST NOT mask any frames that it sends to
the client. A client MUST close a connection if it detects a masked
frame。
This is a working version:
import os
import array
import six
import socket
import struct
OPCODE_TEXT = 0x1
try:
# If wsaccel is available we use compiled routines to mask data.
from wsaccel.xormask import XorMaskerSimple
def _mask(_m, _d):
return XorMaskerSimple(_m).process(_d)
except ImportError:
# wsaccel is not available, we rely on python implementations.
def _mask(_m, _d):
for i in range(len(_d)):
_d[i] ^= _m[i % 4]
if six.PY3:
return _d.tobytes()
else:
return _d.tostring()
def get_masked(data):
mask_key = os.urandom(4)
if data is None:
data = ""
bin_mask_key = mask_key
if isinstance(mask_key, six.text_type):
bin_mask_key = six.b(mask_key)
if isinstance(data, six.text_type):
data = six.b(data)
_m = array.array("B", bin_mask_key)
_d = array.array("B", data)
s = _mask(_m, _d)
if isinstance(mask_key, six.text_type):
mask_key = mask_key.encode('utf-8')
return mask_key + s
def ws_encode(data="", opcode=OPCODE_TEXT, mask=1):
if opcode == OPCODE_TEXT and isinstance(data, six.text_type):
data = data.encode('utf-8')
length = len(data)
fin, rsv1, rsv2, rsv3, opcode = 1, 0, 0, 0, opcode
frame_header = chr(fin << 7 | rsv1 << 6 | rsv2 << 5 | rsv3 << 4 | opcode)
if length < 0x7e:
frame_header += chr(mask << 7 | length)
frame_header = six.b(frame_header)
elif length < 1 << 16:
frame_header += chr(mask << 7 | 0x7e)
frame_header = six.b(frame_header)
frame_header += struct.pack("!H", length)
else:
frame_header += chr(mask << 7 | 0x7f)
frame_header = six.b(frame_header)
frame_header += struct.pack("!Q", length)
if not mask:
return frame_header + data
return frame_header + get_masked(data)
def ws_decode(data):
"""
ws frame decode.
:param data:
:return:
"""
_data = [ord(character) for character in data]
length = _data[1] & 127
index = 2
if length < 126:
index = 2
if length == 126:
index = 4
elif length == 127:
index = 10
return array.array('B', _data[index:]).tostring()
# connect
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((socket.gethostbyname('echo.websocket.org'), 80))
# handshake
handshake = 'GET / HTTP/1.1\r\nHost: echo.websocket.org\r\nUpgrade: websocket\r\nConnection: ' \
'Upgrade\r\nSec-WebSocket-Key: gfhjgfhjfj\r\nOrigin: http://example.com\r\nSec-WebSocket-Protocol: ' \
'echo\r\n' \
'Sec-WebSocket-Version: 13\r\n\r\n'
sock.send(handshake)
print(sock.recv(1024))
sock.sendall(ws_encode(data='Hello, China!', opcode=OPCODE_TEXT))
# receive it back
response = ws_decode(sock.recv(1024))
print('--%s--' % response)
sock.close()
Related
I'm trying to create an app using python, to watch together movies, but I constantly getting errors with :
[Errno 10040] A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram was smaller than the datagram itself
Server.py
BUFFOR_SIZE_DATA = 65536
# SERVER SETTINGS
server_socket_main = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket_main.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, BUFFOR_SIZE_DATA)
print(Fore.RED + 'UDP Server')
host_name = socket.gethostname()
print('HOST NAME:',host_name)
host_ip_adress = ''
print('HOST IP:',host_ip_adress)
host_port = 1337
socket_adress_main = (host_ip_adress, host_port)
print('PORT:',host_port)
server_socket_main.bind(socket_adress_main)
print(Fore.GREEN + 'Server is listening > > >')
print(Fore.WHITE + ' ')
print(Fore.WHITE + 'Connected devices: ')
# VIDEO U WANT TO WATCH TOGETHER
server_video_main = cv2.VideoCapture('movies/exmpl.mp4')
# MAIN LOOP
while True:
msg, client_addres_obligatory = server_socket_main.recvfrom(BUFFOR_SIZE_DATA)
print('Connected from ', client_addres_obligatory)
WIDTH = 1024
while(server_video_main.isOpened()):
_,frame = server_video_main.read()
frame = imutils.resize(frame, width=WIDTH)
encoded, buffer = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 80])
message = base64.b64encode(buffer)
server_socket_main.sendto(message, client_addres_obligatory)
cv2.imshow('HOST | Currently hosting', frame)
key = cv2.waitKey(1) & 0xFF
if key == ord('c'):
server_socket_main.close()
break
Client.py
client_socket_main.sendto(welcomemessage.encode(), (client_ip_adress, client_port))
while True:
packet,_ = client_socket_main.recvfrom(BUFFOR_SIZE_DATA)
data = base64.b85decode(packet,' /')
npdata = np.fromstring(data, dtype=np.unit8)
frame = cv2.imdecode(npdata, 1)
cv2.imshow('Currently watching ', frame)
key = cv2.waitKey(1) & 0xFF
if key == ord('c'):
client_socket_main.close()
break
data = base64.b85decode(packet,' /')
TypeError: b85decode() takes 1 positional argument but 2 were given
Thanks in advance!
The 64KiB limit is determined not by UDP but by IP. Here's the IPv4 header from the original RFC:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live | Protocol | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
See that "Total Length" field? It's 16 bits, so its maximum value is 0xffff, which is 64Ki. That even includes headers, so the actual maximum size is smaller.
Additionally, the typical MTU on the internet is 1500 bytes, sometimes less (again, including headers). Sending packets larger than the MTU is a recipe for trouble because it means that IP fragmentation will take place, which is not desirable.
You need to break up the data into smaller chunks or use TCP which takes care of that for you.
I made a game in python with pygame and socket, my game works perfectly in local server, but when i want to try it with my friend it give me these errors :
("ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
and
Traceback (most recent call last
File "C:/Users/Zahraa Rached/Desktop/Poké-aim/client.py", line 54, in <module>
game = n.send("get")
File "C:\Users\Zahraa Rached\Desktop\Poké-aim\network.py", line 25, in send
return pickle.loads(self.client.recv(4096))EOFError: Ran out of input
My port are frowarded and i arleady played a game that i have coded on different pc (with my friends)
so the problem should not be my server host
(i tried a lot of different things like disabling my firewall and my antivirus)
network class:
class Network:
def __init__(self):
self.client=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.server="my public ip"
self.port=50
self.addr=(self.server,self.port)
self.p=self.connect()
def getP(self):
return self.p
def connect(self):
try:
self.client.connect(self.addr)
return self.client.recv(4096).decode()
except:
pass
def send(self,data):
try:
self.client.send(str.encode(data))
return pickle.loads(self.client.recv(4096))
except socket.error as e:
print(e)
server
import socket
from _thread import*
import pickle
from game import*
g=Game()
clientnb=0
server="192.168.1.43"
port=50
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
try:
s.bind((server,port))
except socket.error as e:
print(e)
s.listen()
print("J'attend une connexion, Dresseur de Pokémon")
def thread(conn,p):
global clientnb
global g
print(clientnb)
g.pos[p]=(0,0)
conn.send(str.encode(str(p)))
r=False
reply=""
while True:
if g.ready == clientnb and clientnb > 1:
g.start=True
else:
g.start=False
try:
data=conn.recv(4096).decode()
if data=="ready":
r=True
g.ready+=1
elif data=="clik":
print("x")
print(g.ball)
g.changepos()
g.changeball()
print(g.ball)
elif data=="reset":
g.reset()
elif data=="win":
g.end=True
elif data!="get":
co=eval(data)
g.pos[p]=co
elif not data:
print("oh")
if r:
g.ready-=1
clientnb-=1
del g.pos[p]
break
reply = g
conn.sendall(pickle.dumps(reply))
except:
print("ohh")
if r:
g.ready -= 1
clientnb -= 1
g.reset()
break
conn.close()
while True:
conn,addr=s.accept()
print(addr)
p = clientnb
start_new_thread(thread,(conn,p))
clientnb+=1
client:
import pygame
import pickle
import sys
from network import Network
import time
import socket
pygame.font.init()
pygame.mixer.pre_init(44100, -16, 2, 512)
pygame.mixer.init()
pygame.init()
run=True
timevar=False
pygame.mixer.music.load("music.mp3")
pygame.mixer.music.set_volume(0.1)
myscore=0
w=1280
h=720
screen= pygame.display.set_mode((w,h))
pygame.display.set_caption("Poké-Aim")
n=Network()
fond_color=(0,0,0)
color1 = (255, 0, 0)
click=pygame.mixer.Sound("click.ogg")
ball=pygame.transform.scale(pygame.image.load("ball.png").convert_alpha(),(71,62))
master=pygame.transform.scale(pygame.image.load("master.png").convert_alpha(),(35,31))
bigmaster=pygame.transform.scale(pygame.image.load("master.png").convert_alpha(),(71,62))
cursor=pygame.transform.scale(pygame.image.load("cursor.png").convert_alpha(),(48,48))
other_cursor=pygame.transform.scale(pygame.image.load("black_cursor.png").convert_alpha(),(48,48))
playing=False
loosing=False
winning=False
def find_key(v):
for k, val in pos.items():
if v == val:
return k
clock=pygame.time.Clock()
ready=False
start=False
p=int(n.getP())
print(p)
pygame.mouse.set_visible(False)
while run:
clock.tick(60)
screen.fill(fond_color)
try:
game = n.send("get")
except socket.error as e:
print(e)
run=False
print("Connexion perdue")
break
if game!=None:
start=game.start
name=game.name
bol=game.ball
pos=game.pos
end=game.end
if game.ready==0:
ready=False
if not start:
if not ready:
color1=(255,0,0)
font = pygame.font.Font("Pokemon Classic.ttf", 20)
sc=font.render("= 1 pts", 1,(255,255,255) )
sc2=font.render("= 3pts", 1,(255,255,255) )
screen.blit(sc,(w/2,150))
screen.blit(sc2, (w / 2, 215))
screen.blit(ball,((w/2-80),140))
screen.blit(bigmaster, ((w / 2 - 80), 200))
text = font.render("Clique sur l'écran pour te mettre prêt !", 1,color1 )
screen.blit(text, ((w - text.get_width()) / 2, (h - text.get_height()) / 2))
elif start:
screen.fill((255,244,10))
if not playing:
pygame.mixer.music.play()
playing=True
for event in pygame.event.get():
if event.type == pygame.QUIT:
pygame.quit()
sys.exit()
if event.type == pygame.MOUSEBUTTONUP:
if event.button == 1:
while not ready and not timevar:
color1 = (0,255, 0)
screen.blit(text, ((w - text.get_width()) / 2, (h - text.get_height()) / 2))
n.send("ready")
click.play()
ready=True
if start:
if name=="P":
if (bol[0]) <= (pygame.mouse.get_pos()[0] ) <= (bol[0] + 71):
if pygame.mouse.get_pos()[1]>= (bol[1]) and pygame.mouse.get_pos()[1] <= (bol[1] + 62):
click.play()
n.send("clik")
click.play()
myscore+=1
else:
if (bol[0]) <= (pygame.mouse.get_pos()[0] ) <= (bol[0] + 35):
if (pygame.mouse.get_pos()[1]) >= (bol[1]) and (pygame.mouse.get_pos()[1] )<= (bol[1] + 31):
click.play()
n.send("clik")
myscore+=3
try:
n.client.send(str.encode(str(pygame.mouse.get_pos())))
except :
pass
if start:
if name=="P":
screen.blit(ball, (bol[0], bol[1]))
else:
screen.blit(master, (bol[0], bol[1]))
for e in pos.values():
if find_key(e) != p:
if find_key(e) != p:
screen.blit(other_cursor, (e[0]-24,e[1]-24))
score = font.render(str(myscore), 1, (0,0,0))
screen.blit(score, (5,0))
if myscore >= 1:
pygame.mixer.music.stop()
screen.fill((0, 0, 0))
wintext = font.render("T'as gagné sale bg !", 1, color1)
screen.blit(wintext, ((w - wintext.get_width()) / 2, (h - wintext.get_height()) / 2))
pygame.display.update()
n.send("win")
if not timevar:
chrono=time.time()
timevar=True
if round(time.time()-chrono)>4:
n.send("reset")
color1 = (255, 0, 0)
start = False
ready = False
playing = False
myscore = 0
timevar=False
end = False
elif end:
pygame.mixer.music.stop()
screen.fill((0, 0, 0))
loosetext = font.render("T'as perdu t'es trop nul sérieux !", 1, (255,0,0))
screen.blit(loosetext, ((w - loosetext.get_width()) / 2, (h - loosetext.get_height()) / 2))
pygame.display.update()
if not timevar:
chrono = time.time()
timevar = True
if round(time.time() - chrono) > 4:
color1 = (255, 0, 0)
start = False
ready = False
playing = False
myscore = 0
timevar = False
end = False
screen.blit(cursor,(pygame.mouse.get_pos()[0]-24,pygame.mouse.get_pos()[1]-24))
pygame.display.update()
game_class:
import random
h=720
w=1280
class Game:
def __init__(self):
self.name="P"
self.pos= {}
self.ball=[w/2,h/2]
self.ready=0
self.start=False
self.end=False
def changepos(self):
if self.name=="P":
self.ball[0]=random.randrange(0+71,w-71)
self.ball[1]=random.randrange(0,h-62)
else:
self.ball[0] = random.randrange(0 + 35, w- 35)
self.ball[1] = random.randrange(0, h - 31)
def changeball(self):
self.name=random.choice("MPPPPPPPPPP")
def reset(self):
self.name = "P"
self.ball = [w / 2, h / 2]
self.ready=0
self.start = False
self.end = False
A quick read of the manual about socket.recv() tells us that self.client.recv(4096) uses that 4096 buf_size parameter as:
The maximum amount of data to be received at once is specified by [...]
So, it's the maximum, not the minimum. Your .recv() will never receive more than 4096 bytes. That's all.
But the code is assuming it will always recv() enough bytes:
return pickle.loads( self.client.recv(4096) ) # gambling a crash on Rx size
On your current network set-up, something is causing smaller packets to arrive at the recv(), and not a full "pickle-worth". This is always an issue with network communications, it may be temporary, it may be permanent. Maybe a damp connection is causing packet loss, maybe maybe maybe. It was working before because the network/computers behaved differently.
So what do you do? The solution is check to see if any bytes have arrived. If bytes have arrived, add them to an "incoming-buffer". If there's enough bytes in the buffer, unpack it.
Pickling a data structure for sending over the network is probably an OK method, but I would first transmit the data-size of the pickled object, so the receiving client can know whether enough bytes have arrived to unpack it. Sure you can try..catch the un-pickling, but that's grossly inefficient.
I would use struct.pack to send the pickle-buffer size as a 4-byte integer (in network-byte order). Receive those 4 bytes first, then wait for the further N bytes to arrive.
EDIT: More details on struct, etc.
import struct
import pickle
def sendData( server_sock, object ):
result = True
data = pickle.dumps( object )
data_size = struct.pack( '!I', len( data ) ) # good for 2^32 bytes
try:
server_sock.sendall( data_size ) # send the size of the pickled obj
server_sock.sendall( data ) # send the obj
except:
sys.stderr.write( "Error sending data to server\n" )
result = False
return result
And then to receive the data, read the 4-byte size field and then the pickle data. Something like the below should work (note: this is not tested code, I'm just answering off the top of my head)
def recvData( client_sock ):
""" Try to receive a picked data-structure from the socket.
Returns True and an unpicked object, or False and None """
result = False
packet = None
expected = -1
# First we read a 4-byte pickle size, and following that is /N/ bytes
# of pickled structure data.
# Read the 4-byte packet size first
buffer = bytes()
while ( len( buffer ) < 4 ):
try:
some_data = client_sock.recv( 4 - len( buffer ) )
if ( some_data ):
buffer.append( some_data )
if ( len( buffer ) == 4 ):
expected = struct.unpack( '!I', buffer )
except:
# TODO: work out which sort of exceptions are OK
break
# If we received a buffer size, try to receive the buffer
if ( expected > 0 ):
buffer = bytes()
while ( len( buffer ) < expected ):
try:
some_data = client_sock.recv( expected - len( buffer ) )
if ( some_data ):
buffer.append( some_data )
# Have we received the full data-set yet?
if ( len( buffer ) == expected ):
packet = pickle.loads( buffer )
result = True # success
except:
# TODO: work out which sort of exceptions are OK
break
return result, packet
To be honest if you're just sending fixed amounts all the time (like a pair of x/y co-ordinates), it might be easier to just pack and unpack everything, using fixed-size datagrams. Maybe that's not "pythonic" enough though ;)
Anyway, what makes good network code is handling all the little rough edges. Sometimes (for a hundred different reasons), data does not arrive in nicely sized blocks, or your network blocks for a second or so. Simply handling this with a buffer, and non-blocking code solves 99% of the ills.
Really the recv() above should be non-blocking code, but this is left as an exercise for the reader. I like to use select() for this, but that's a bit old-fashioned.
I want to use tcpsocket to send nv21 images between smartphone and server and convert it to rgb.I use Qt for android make the smartphone application and use python in server. When i use QByteArray to send the data, the picture is (1920*1080) so the size of data is 3110400. However i only receive 335620 bytes at server.
I want to know how can i send bypes correctly?
//QT client
void came::processFrame(const QVideoFrame& frame)
{
QVideoFrame f(frame);
QImage img;
f.map(QAbstractVideoBuffer::ReadOnly);
output.append((char*)f.bits(),f.mappedBytes());
tcpSocket->write(output,output.size());
}
//Python server
def recvfromTcpSocket(sock, blockSize=4096, accept_addr=None):
''' a function to lisen TCP socket,
and rece bytes till buffer has no more. '''
d = ''
while True:
print ("Got tcp connection: " + str(tcpServerAddr))
if accept_addr is None:
break
elif accept_addr == tcpServerAddr[0]:
break
else :
sock.close()
continue
while True:
block = sock.recv(blockSize)
d += block
if len(block) < blockSize and len(block)!=0:
print ("TCP recv done, all size: " + str(len(d)))
break
return d
if __name__ == '__main__':
data = recvfromTcpSocket(tcpServerSocket)
bin_y = data[0:rows * cols]
num_y = np.fromstring(bin_y, np.uint8)
img_y = np.reshape(num_y, (rows, cols))
bin_u = data[rows * cols::2]
num_u = np.fromstring(bin_u, np.uint8)
img_u = np.reshape(num_u, (rows / 2, cols / 2))
bin_v = data[rows * cols+1::2]
num_v = np.fromstring(bin_v, np.uint8)
img_v = np.reshape(num_v, (rows / 2, cols / 2))
enlarge_u = cv2.resize(img_u, dsize=(cols, rows), interpolation=cv2.INTER_CUBIC)
enlarge_v = cv2.resize(img_v, dsize=(cols, rows), interpolation=cv2.INTER_CUBIC)
dst = cv2.merge([img_y, enlarge_u, enlarge_v])
bgr = cv2.cvtColor(dst, cv2.COLOR_YUV2BGR)
Consider this simple python dns spoofer:
#!/usr/bin/python3
import socket
from scapy.all import *
port = 53
def handle(payload):
data = payload[0]
src_ip = payload[1][0]
src_port = payload[1][1]
print("-" * 10 + " Incoming " + "-" * 10)
a = DNS(data)
a.show2()
output = IP(dst=src_ip, chksum = 0)/ UDP(sport=53, dport=src_port) / DNS(id=a.id, qr=1, opcode=a.opcode, aa=1, tc=0, rd=1, ra=1, z=0, ad=0, cd=0, rcode=0, qdcount=1, ancount=1, nscount=0, arcount=0,qd=DNSQR(qname=a[DNS].qd.qname, qtype=a[DNS].qd.qtype, qclass=a[DNS].qd.qclass), an=DNSRR(rrname='cnn.com', rdata='192.168.1.100'), )
print("-" * 10 + " Output " + "-" * 10)
output.show2()
send(output)
if __name__ == "__main__":
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # create a socket object
s.bind(("", port))
while True:
payload = s.recvfrom(1024)
handle(payload)
I would run it from sudo (so I can bind to udp port 53):
$ sudo python3 -i ./dns_python.py
However my requests to it are timing out:
$ host cnn.com 127.0.0.1
;; connection timed out; no servers could be reached
I need help to figure out how to spoof successfully:
Some more details:
$ sudo python3 -i ./dns_python.py
---------- Incoming ----------
###[ DNS ]###
id = 59652
qr = 0
opcode = QUERY
aa = 0
tc = 0
rd = 1
ra = 0
z = 0
ad = 0
cd = 0
rcode = ok
qdcount = 1
ancount = 0
nscount = 0
arcount = 0
\qd \
|###[ DNS Question Record ]###
| qname = 'cnn.com.'
| qtype = A
| qclass = IN
an = None
ns = None
ar = None
---------- Output ----------
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 76
id = 1
flags =
frag = 0
ttl = 64
proto = udp
chksum = 0x0
src = 127.0.0.1
dst = 127.0.0.1
\options \
###[ UDP ]###
sport = domain
dport = 36774
len = 56
chksum = 0xb87f
###[ DNS ]###
id = 59652
qr = 1
opcode = QUERY
aa = 1
tc = 0
rd = 1
ra = 1
z = 0
ad = 0
cd = 0
rcode = ok
qdcount = 1
ancount = 1
nscount = 0
arcount = 0
\qd \
|###[ DNS Question Record ]###
| qname = 'cnn.com.'
| qtype = A
| qclass = IN
\an \
|###[ DNS Resource Record ]###
| rrname = 'cnn.com.'
| type = A
| rclass = IN
| ttl = 0
| rdlen = 4
| rdata = '192.168.1.100'
ns = None
ar = None
.
Sent 1 packets.
I see packets when I watch from a ngrep -d lo '' src or dst port 53 window. Any help would much appreciated!
Scapy does not work with 127.0.0.1 or on the loopback interface
http://scapy.readthedocs.io/en/latest/troubleshooting.html
The loopback interface is a very special interface. Packets going through it are not really assembled and disassembled. The kernel routes the packet to its destination while it is still stored an internal structure. What you see with tcpdump -i lo is only a fake to make you think everything is normal. The kernel is not aware of what Scapy is doing behind his back, so what you see on the loopback interface is also a fake. Except this one did not come from a local structure. Thus the kernel will never receive it.
In order to speak to local applications, you need to build your packets one layer upper, using a PF_INET/SOCK_RAW socket instead of a PF_PACKET/SOCK_RAW (or its equivalent on other systems than Linux):
Whenever I send a SYN packet using my program, I get no reply.I know the server is working because I can connect to it using the normal socket connect() method but when I try using RAW sockets to do it I get no reply, not even an RST.
Here is my packet according to Wireshark
Transmission Control Protocol, Src Port: 5173 (5173), Dst Port: 5005 n (5005), Seq: 0, Len: 0
Source Port: 5173
Destination Port: 5005
[Stream index: 15]
[TCP Segment Len: 0]
Sequence number: 0 (relative sequence number)
Acknowledgment number: 0
Header Length: 40 bytes
Flags: 0x002 (SYN)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...0 .... = Acknowledgment: Not set
.... .... 0... = Push: Not set
.... .... .0.. = Reset: Not set
.... .... ..1. = Syn: Set
.... .... ...0 = Fin: Not set
[TCP Flags: **********S*]
Window size value: 53270
[Calculated window size: 53270]
Checksum: 0x9f18 [incorrect, should be 0x90ae (maybe caused by "TCP checksum offload"?)]
Urgent pointer: 0
Options: (20 bytes), Maximum segment size, SACK permitted, Timestamps, No-Operation (NOP), Window scale
Maximum segment size: 65495 bytes
Kind: Maximum Segment Size (2)
Length: 4
MSS Value: 65495
TCP SACK Permitted Option: True
Kind: SACK Permitted (4)
Length: 2
Timestamps: TSval 378701, TSecr 0
Kind: Time Stamp Option (8)
Length: 10
Timestamp value: 378701
Timestamp echo reply: 0
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
Window scale: 7 (multiply by 128)
[SEQ/ACK analysis]
And here is my Python code
#!/usr/bin/python
import socket
from struct import *
import random
s = socket.socket()
host = "127.0.0.1"
destination = "127.0.0.1"
CLRF = '\r\n'
#socket.gethostname()
print destination
port = 5173
#s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#s.connect((host, 5005))
try:
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_RAW)
except socket.error , msg:
print 'Socket could not be created. Error Code : ' + str(msg[0]) + ' Message ' + msg[1]
sys.exit()
ipSource = '192.168.0.106'
#IP header
ipIHL = 5 # Header Length
ipVersion = 4 # ipv4/v6
ipTOS = 0 # type of service
ipTotalLen = 0 ## Kernel will fill correct length apparently
ipPackID = random.randint(1,1000)
#Flags
ipReserved = 0
ipNoFrag = 1
ipMoreFrags = 0
ipFragOffset = 0 #Fragment offset
ipTTL = 64
ipProtocol = socket.IPPROTO_TCP
ipChecksum = 0 # Magic kernel filling in at work again
ipSource = socket.inet_aton (host)
ipDest = socket.inet_aton (destination)
#Packing IP flags
ipFlags = ipMoreFrags + (ipNoFrag << 1) + (ipReserved << 2)
ipFragOffset = (ipFlags << 13) + ipFragOffset
ipIHLVersion = (ipVersion << 4) + ipIHL
headerIP = pack('!BBHHHBBH4s4s',ipIHLVersion, ipTOS, ipTotalLen, ipPackID, ipFragOffset, ipTTL, ipProtocol, ipChecksum, ipSource, ipDest)
#Checksum function
def carry_around_add(a, b):
c = a + b
return (c & 0xffff) + (c >> 16)
def checksum(msg):
s = 0
for i in range(0, len(msg), 2):
w = ord(msg[i]) + (ord(msg[i+1]) << 8)
s = carry_around_add(s, w)
return ~s & 0xffff
#TCP Header
tcpSourcePort = port #Source Port
tcpDestPort = 5005 #Destination Port
tcpSeqNum = 0 #Packet sequence
tcpAckNum = 0 #Ackknowledge Number
tcpOffset = 10 #Size of tcp header 20 bytes
#tcpReserved = 0
#tcpECN = 0
#Control Flags
tcpURG = 0
tcpACK = 0
tcpPSH = 0
tcpRST = 0
tcpSYN = 1
tcpFIN = 0
tcpWindow = socket.htons (5840) #Dunno how this works
tcpChecksum = 0
tcpUrgentPointer = 0
#TCP Options
tcpMaxSegmentSize = (2 << 24) + (4 << 16) + 65495 # Kind + Length + Max Segment Size
tcpSACKPermitted = (4 << 8) + 2#Kind + Length
#Split TCP TImestamps into 2 because too large
tcpTimestampPartOne = (8 << 8) + (10) #Kind + Length
tcpTimestampPartTwo = (378701 << 32) + 0 #Timestamp Value + Timestamp echo reply
tcpNoOp = (0 << 7) + (0 << 5) + 1 #Copy on fragmentation + Class + Number
tcpWindowScale = (3 << 16)+ (3 << 8) + 7 #Kind + Length(Bytes) +Shift CountS
#Combine both due to length issues
tcpNoOpAndWindowScale = (tcpNoOp << 24) + tcpWindowScale
tcpOffsetResult = (tcpOffset << 4) + 0 #Shift 4 bytes to left
#Putting together all the TCP Control Flags
tcpFlags = tcpFIN + (tcpSYN << 1) + (tcpRST << 2) + (tcpPSH << 3) + (tcpACK << 4) + (tcpURG << 5)
#Packing the pseudo TCP header
headerTCP = pack('!HHLLBBHHHLHHQL', tcpSourcePort, tcpDestPort, tcpSeqNum, tcpAckNum, tcpOffsetResult, tcpFlags, tcpWindow, tcpChecksum, tcpUrgentPointer, tcpMaxSegmentSize, tcpSACKPermitted, tcpTimestampPartOne, tcpTimestampPartTwo, tcpNoOpAndWindowScale)
#headerTCP = pack('!HHLLBBHHH', tcpSourcePort, tcpDestPort, tcpSeqNum, tcpAckNum, tcpOffsetResult, tcpFlags, tcpWindow, tcpChecksum, tcpUrgentPointer)
#data = 'GET ./asd HTTP/1.1'
data = ''
#Checksum Calculation
#Pseudo Header Fields
sourceAddr = socket.inet_aton(host)
destAddr = socket.inet_aton(destination)
placeholder = 0
protocol = socket.IPPROTO_TCP
tcpLen = len(headerTCP) + len(data)
psh = pack('!4s4sBBH', sourceAddr, destAddr, placeholder, protocol, tcpLen);
psh = psh + headerTCP + data;
#Calc checksum
tcpChecksumReal = (checksum(psh) << 1)
print(tcpChecksumReal)
#Pack actual tcp header with checksum
headerTCP = pack('!HHLLBBH', tcpSourcePort, tcpDestPort, tcpSeqNum, tcpAckNum, tcpOffsetResult, tcpFlags, tcpWindow) + pack('!H', 40728) + pack ('!H', tcpUrgentPointer) + pack('!LHHQL', tcpMaxSegmentSize, tcpSACKPermitted, tcpTimestampPartOne, tcpTimestampPartTwo, tcpNoOpAndWindowScale)
#Build full packet / ip with tcp with data
packet = headerIP + headerTCP + data
#print [hex(ord(c)) for c in packet]
s.sendto(packet, (destination,0))
Any help would be appreciated, thanks in advance.
Credits to #KenCheung for the answer
Turns out it was the checksum, the checksum from the headers I used as reference also were incorrect but the network card was offloading them.