I'm making a LAN multiplayer arena shooter using pygame and sockets and am having troubling transferring pickled object data from server to client. I have 2 objects, player and projectile (the bullets). I don't know how to send multiple objects at once, so i decided to put the 2 objects in a list and pickle them. But when unpickling, I can't index the list as I keep getting the 'EOFError: Ran out of input'
So I want to unpickle the list that I receive and separate out the 2 objects in that list. But python won't let me index the list after I unpickled them. Any help would be much appreciated.
Here's my code:
#instantiating instances for Player and Projectile classes
players=[Player(450,400,"sprite1.png",1),Player(500,400,"sprite2.png",2)]
bullets=[Projectile(50,50,5,"right","projectile.png",0)]
def threaded_client(conn, player):
player_data=(players[player])
bullet_data=(bullets[player])
alldata=[player_data,bullet_data] #putting the 2 objects in a list.
conn.send(pickle.dumps(alldata)) #pickling list
reply = ""
while True:
try:
alldata = pickle.loads(conn.recv(2048))
players[player] = alldata[0]
...
self.client.connect(self.addr)
alldata=pickle.loads(self.client.recv(2048)) #unpickling the list
return alldata[0] #trying to return the first object
You need to make arrangements to ensure that you have the entire object before you unpickle. You're doing a conn.recv(XXX) but that does not mean you actually received all XXX bytes. On success, it means you got somewhere between 1 and XXX bytes (inclusive). If it's a small buffer, you often get the entire thing in one chunk but you should never count on that.
Generally, you'll want to send the byte count in a fixed-size binary format (typically using the struct module), then after retrieving the byte count, keep receiving until you got all the expected bytes or you get an error (i.e. your peer disconnected).
Something like this on the sending side:
import struct
pickled_bytes = pickle.dumps(thing_youre_sending)
p_size = len(pickled_bytes) # Size of pickled buffer
p_size_buf = struct.pack("!I", p_size) # Packed binary size (4 byte field)
conn.sendall(p_size_buf) # Send length (Note sendall!)
conn.sendall(pickled_bytes) # Send actual pickled object
On the receiving side, you'll do something like this:
import struct
...
def recv_all(conn, rlen):
""" Function to receive all bytes """
recvd = 0
buf = b''
while recvd < rlen:
rbuf = conn.recv(rlen - recvd)
if not rbuf:
# Client disconnected. Handle error in whatever way makes sense
raise ClientDisconnected()
recvd += len(rbuf)
buf += rbuf
...
p_size_buf = recv_all(conn, 4) # Receive entire binary length field
p_size = struct.unpack("!I", p_size_buf)[0] # (Unpack returns an array)
pickled_bytes = recv_all(conn, p_size) # Receive actual pickled object
thing_you_sent = pickle.loads(pickled_bytes)
Related
First off, I'm trying to do something like screenshare, I'm confronting an error. I tried to send the image row by row than I figured out that it could work only in LAN since the connection speed was far better. Then I transmitted exclusively each pixel and constructed the array on my own which would be pretty slow. Here is the code:
Server side:
def getImg(socket,y,x):
pixels = []
print(x*y*3)
for a in range(x*y*3):
pixels.append(struct.unpack('i', socket.recv(4))[0])
if a%(x*y)==0:
pass
return np.array(pixels).reshape(y,x,3)
Client side:
def sendSS(img):
y, x, color = img.shape
print(y*x*3)
for a in range(y):
for b in range(x):
for m in img[a][b]:
socket.send(struct.pack('i', m))
The error is this:
(<class 'cv2.error'>, error("OpenCV(4.5.1) c:\\users\\appveyor\\appdata\\local\\temp\\1\\pip-req-build-wvn_it83\\opencv\\modules\\imgproc\\src\\color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function '__cdecl cv::impl::`anonymous-namespace'::CvtHelper<struct cv::impl::`anonymous namespace'::Set<3,4,-1>,struct cv::impl::A0xeee51b91::Set<3,4,-1>,struct cv::impl::A0xeee51b91::Set<0,2,5>,2>::CvtHelper(const class cv::_InputArray &,const class cv::_OutputArray &,int)'\n> Unsupported depth of input image:\n> 'VDepth::contains(depth)'\n> where\n> 'depth' is 4 (CV_32S)\n"), <traceback object at 0x0000023221307740>)
The way I constructed the numpy array works fine:
import pyautogui
import cv2
import numpy as np
import pickle
import struct
from pprint import pprint as pp
ss = np.array(pyautogui.screenshot())
y,x,color = ss.shape
pixels = []
for a in range(y):
for b in range(x):
for m in ss[a][b]:
pixels.append(struct.pack('i',m))
pixels2 = []
for val in pixels:
pixels2.append(struct.unpack('i',val)[0])
pixels2 = np.array(pixels2).reshape((1080,1920,3))
cv2.imwrite('name.png',pixels2)
How can I transmit faster and solve this problem?
edit:
Here is the part of the code where the error occurs, it says the depth is 4 but I'm sure the dimensions of the image are 1080,1920,3 because I can print it.
image = getImg(aSocket,y,x)
SS = cv2.cvtColor(image,cv2.COLOR_RGB2BGR) # error occurs in this line
aSocket.send('image is gotten'.encode('utf-8'))
print('image is gotten')
cv2.imwrite(f'{ip[0]}\{count}.png',SS)
You need to pass the new shape as tuple instead for reshape, eg. arr.reshape((y,x,3)). You might also get another error if OpenCV expects different data type (8 bits per channel, unsigned) than what it got from your program (32 bit signed integer), see documentation for cvtColor(). You can specify the data type when you create the array np.array(..., dtype=np.uint8).
Regarding performance, in Python for loops over large amounts of data can be slow and often you can make your code significantly faster if you can avoid the loop and use library routine instead. Also, you should read and write larger chunks with send() and recv(). If you are not familiar with socket programming, you should at minimum check out the python documentation Socket Programming HOWTO. You eliminate the loops over the image data by directly interpreting the data as bytes. The socket code follows the examples in linked documentation page.
def send_im(sock, im):
totalsent = 0
im_bytes = im.tobytes()
while totalsent < len(im_bytes):
sent = sock.send(im_bytes[totalsent:])
if sent == 0:
raise RuntimeError("socket error")
totalsent += sent
Receiver needs to know the size of expected input.
def recv_im(sock, dtype, shape):
# expected number of bytes
size = np.dtype(float).itemsize * np.prod(shape)
im_bytes = bytearray(size) # received bytes
bytes_recvd = 0
chunk_size = 2048
while bytes_recvd < size:
chunk = sock.recv(min(size-bytes_recvd, chunk_size))
if chunk == b'':
raise RuntimeError("socket connection broken")
im_bytes[bytes_recvd:(bytes_recvd+chunk_size)] = memoryview(chunk)
bytes_recvd += len(chunk)
# intrepret bytes in correct data type and shape and return image
return np.frombuffer(im_bytes, dtype=dtype).reshape(shape)
In general, you should always use a serialization library instead of directly sending raw bytes over network, as those libraries usually handle various important details such as byte order, message sizes, data types, conversions, and so on.
I am working on a server engine in Python, for my game made in GameMaker Studio 2. I'm currently having some issues with making and sending a packet.
I've successfully managed to establish a connection and send the first packet, but I can't find a solution for sending data in a sequence of which if the first byte in the packed struct is equal to a value, then unpack other data into a given sequence.
Example:
types = 'hhh' #(message_id, x, y) example
message_id = 0
x = 200
y = 200
buffer = pack(types, 0,x, y)
On the server side:
data = conn.recv(BUFFER_SIZE)
mid = unpack('h', data)[0]
if not data: break
if mid == 0:
sequnce = 'hhh'
x = unpack(sequnce, data)[1]
y = unpack(sequnce, data)[2]
It looks like your subsequent decoding is going to vary based on the message ID?
If so, you will likely want to use unpack_from which allows you to pull only the first member from the data (as written now, your initial unpack call will generate an exception because the buffer you're handing it is not the right size). You can then have code that varies the unpacking format string based on the message ID. That code could look something like this:
from struct import pack, unpack, unpack_from
while True:
data = conn.recv(BUFFER_SIZE)
# End of file, bail out of loop
if not data: break
mid = unpack_from('!h', data)[0]
if mid == 0:
# Message ID 0
types = '!hhh'
_, x, y = unpack(types, data)
# Process message type 0
...
elif mid == 1:
types = '!hIIq'
_, v, w, z = unpack(types, data)
# Process message type 1
...
elif mid == 2:
...
Note that we're unpacking the message ID again in each case along with the ID-specific parameters. You could avoid that if you like by using the optional offset argument to unpack_from:
x, y = unpack_from('!hh', data, offset=2)
One other note of explanation: If you are sending messages between two different machines, you should consider the "endianness" (byte ordering). Not all machines are "little-endian" like x86. Accordingly it's conventional to send integers and other structured numerics in a certain defined byte order - traditionally that has been "network byte order" (which is big-endian) but either is okay as long as you're consistent. You can easily do that by prepending each format string with '!' or '<' as shown above (you'll need to do that for every format string on both sides).
Finally, the above code probably works fine for a simple "toy" application but as your program increases in scope and complexity, you should be aware that there is no guarantee that your single recv call actually receives all the bytes that were sent and no other bytes (such as bytes from a subsequently sent buffer). In other words, it's often necessary to add a buffering layer, or otherwise ensure that you have received and are operating on exactly the number of bytes you intended.
Could you unpack whole data to list, and then check its elements in the loop? What is the reason to unpack it 3 times? I guess, you could unpack it once, and then work with that list - check its length first, if not empty -> check first element -> if equal to special one, continue on list parsing. Did you try like that?
I am building a python project that receives bytes from a serial port. The bytes are responses to commands sent (also via serial port). The responses have no identifying marks, i.e. from the bytes alone, I don't know which command response this is. The decoder would of course need to know in advance which command this is a response to.
I would like to have the incoming byte sequence represented as a nested object, indicating the frame, header, payload, decoded payload, etc. I would much prefer to push 1 byte at a time to the decoder and have it call a callback once it has received enough bytes for a full object (or errorCallback if there are errors or timeout).
The actual frame has a start byte and an end byte. It has a header with a few bytes (some id, command status (basically ok/fail)), one is a data length byte. This is followed by the data which is followed by a checksum (single byte). The data is the response to the command.
The response is predictable in that the previous bytes decide what the coming bytes mean.
Example response:
aa:00:0c:00:01:00:00:d3:8d:d4:5c:50:01:04:e0:6e:bb
Broken down:
aa: start frame
00: id
0c: data length (incl status): 12 bytes
00: command status (byte 1)
01: 1 data frame (byte 2)
00:00: flags of first data frame (byte 3-4)
d3:8d:d4:5c:50:01:04:e0: first data (aa and bb could be in it) (byte 5-12)
6e: checksum (includes all bytes except start, checksum, end bytes)
bb: end frame
This being serial port communication, bytes may be lost (and extra produced) and I expect to use timeout to handle resets (no responses are expected without first a command being sent).
I really would like an object oriented approach where the decoder would produce an object that when serialized, would produce the same byte sequence again. I am using python 2.7, but really any object oriented language would do (as long as I could convert it to python).
I am just not sure how to structure the decoder to make it neat looking. I am looking for a full solution, just something that would get me going in the right direction (right direction being somewhat subjective here).
I don't completely understand what you want to do but if you want to receive fixed-length responses from some device and make them attributes of some object, would something like this be okay?:
START_FRAME = 0xAA
END_FRAME = 0xBB
TIMEOUT = 2
class Response:
def __init__(self, response):
if (len(response) - 6) % 11 == 0 and response[0] == START_FRAME and response[-1] == END_FRAME: # verify that its a valid response
self.header = {} # build header
self.header['id'] = response[1]
self.header['length'] = response[2]
self.header['status'] = response[3]
self.r_checksum = response[-2] # get checksum from response
self.checksum = self.header['id'] ^ self.header['length'] ^ self.header['status'] # verify the checksum
self.payload = response[4:-2] # get raw payload slice
self.decode(self.payload) # parse payload
if self.r_checksum == self.checksum: # final check
self.parsed = True
else:
self.parsed = False
else: # if response didnt follow the pattern
self.parsed = False
def decode(self, packet): # decode payload
self.frm_count = packet[0] # get number of data frames
self.checksum ^= self.frm_count
self.decoded = [] # hold decoded payload
frames = packet[1:]
for c in range(self.frm_count): # iterate through data frames
flags = frames[(c*10):(c*10 + 2)]
for f in flags:
self.checksum ^= f
data = frames[(c*10 + 2):(c+1)*10]
for d in data:
self.checksum ^= d
self.decoded.append({'frame': c+1, 'flags': flags, 'data':data})
def serialize(): # reconstruct response
res = bytearray()
res.append(START_FRAME)
res.extend([self.header['id'], self.header['length'], self.header['status']])
res.extend(self.payload)
res.extend([self.checksum, END_FRAME])
return res
response = bytearray()
ser = serial.Serial('COM3', 9600) # timeout is 2 seconds
last_read = time.clock()
while time.clock() - last_read < TIMEOUT:
while ser.inWaiting() > 0:
response.append(ser.read())
last_read = time.clock()
decoded_res = Response(response)
if decoded_res.parsed:
# do stuff
else:
print('Invalid response!')
This code assumes there may be more than one data frame, with the data frames immediately preceded by a byte indicating the number of data frames.
Parsing a packet is fast compared to the time taken for serial comms (even at 115200 baud). The whole thing is roughly O(n), i think.
I have revised my question in response to being put on hold. Hopefully this will better match SO standards.
The purpose of this program is to build and send UDP packets which use Alternating Bit Protocol as a simple resending mechanism. I have confirmed already that the packets can be sent and received correctly. The issue is with the ABP bit and its flipping.
The problem facing me now is that despite trying multiple different methods, I cannot flip the ABP bit used for confirming that a packet or ack received is the correct numbered one. I start out by sending a packet with ABP bit=0, and in response, the receiving process should see this and send back an ack with ABP bit=0. Upon receiving that, the sender program flips its ABP bit to 1 and sends a new packet with this ABP bit value. The receiver will get that, send a matching ack with ABP bit=1, and the sender will receive, flip its bit back to 0, and continue the cycle until the program has finished sending information.
Code below, sorry for length, but it is complete and ready to run. The program takes four command line arguments, here is the command I have been using:
% python ftpc.py 164.107.112.71 4000 8000 manygettysburgs.txt
where ftpc.py is the name of the sender program, 164.107.112.71 is an IP address, 4000 and 8000 are port numbers, and manygettysburgs.txt is a text file I have been sending. It should not make a difference if a different .txt is used, but for full accuracy use a file with a length of between 8000 and 9000 characters.
import sys
import socket
import struct
import os
import select
def flipBit(val): #flip ABP bit from 0 to 1 and vice versa
foo = 1 - val
return foo
def buildPacketHeader(IP, Port, Flag, ABP):
#pack IP for transport
#split into four 1-byte values
SplitIP = IP.split('.')
#create a 4-byte struct to pack IP, and pack it in remoteIP
GB = struct.Struct("4B")
remoteIP = GB.pack(int(SplitIP[0]),int(SplitIP[1]),int(SplitIP[2]),int(SplitIP[3]))
#remoteIP is now a 4-byte string of packed IP values
#pack Port for transport
#create a 2-byte struct to pack port, and pack it in remotePort
GBC = struct.Struct("H")
remotePort = GBC.pack(int(Port)) #needs another byte
#remotePort is now a 2-byte string
#add flag
flag = bytearray(1)
flag = str(Flag)
#add ABP
ABP = flipBit(ABP)
abp = str(ABP)
#create header and join the four parts together
Header = ''.join(remoteIP)
Header += remotePort
Header += flag
Header += abp
return Header
#assign arguments to local values
IP = sys.argv[1]
PORT = sys.argv[2]
TPORT = sys.argv[3]
localfile = sys.argv[4]
#declare the socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
#create destination information arrays
remoteIP = bytearray(4)
remotePort = bytearray(2)
#create flag array
flag = bytearray(1)
#create ABP bit
bit = 1
print bit
#send file size packet
#get size of file from os command
filesize = bytearray(4)
#save as 4-byte string
filesize = str(os.stat(localfile).st_size)
#package the filesize string
filesizestr = buildPacketHeader(IP,PORT,1,bit) #build header
print bit
filesizestr += filesize #complete the packet
s.sendto(filesizestr, ('127.0.0.1', int(TPORT))) #send packet
# end of send file name packet
#begin listening for responses
read, write, err = select.select([s], [], [], 1) #timeout set to 1 seconds
if len(read) > 0:
#data received
data = read[0].recv(1200)
if data[7] != bit:
print "failed ack"
#resend packet
else:
print "Timeout."
#resend packet
#send next data packet
#get filename as string (from arg4 diredtly)
filename = bytearray(20)
#save as 20-byte string
filename = localfile
#package the filename string
filenamestr = buildPacketHeader(IP,PORT,2,bit) #build header
print bit
filenamestr += filename #complete the packet
s.sendto(filenamestr, ('127.0.0.1', int(TPORT))) #send packet
#end of send file name packet
#send file content packet
#reading while loop goes here
with open(localfile, 'rb', 0) as f: #open the file
while True:
fstr = f.read(1000)
if not fstr:
print "NOTHING"
break
#put together the main packet base
filecontentstr = buildPacketHeader(IP,PORT,3,bit)
print bit
filecontentbytearray = bytearray(1000) #create ytear array
filecontentbytearray = fstr #assign fstr to byte array
filecontentsend = ''.join(filecontentstr) #copy filecontentstr to new string since we will be using filecontentstr again in the future for other packets
filecontentsend += filecontentbytearray #append read data to be sent
s.sendto(filecontentsend, ('127.0.0.1', int(TPORT))) #send the file content packet
#end of send file content packet
s.close()
In this code, every time that buildPacketHeader is called, it performs flipBit as part of its operations. flipBit is supposed to flip the bit's value for ABP. I have prints set up to print out the new value of bit after all calls to buildPacketHeader, so as to track the value. Whenever I run the program, however, I always see the same value for the ABP bit.
I've tried several methods, including changing to a bool. Here are some changes to flipBit I have tried:
def flipBit(val): #flip ABP bit from 0 to 1 and vice versa
if val == 0:
val = 1
else:
val = 0
return val
and some with bools instead:
def flipBit(val):
val = not val
return val
def flipBit(val):
val = (True, False)[val]
return val
I figure that many of these methods are in fact working options due to past experience. That said, I am completely baffled as to why it is not working as expected in this program. I would assume that it is my inexperience with python that is at fault, for despite having now used it for a decent amount of time, there still are peculiarities which escape me. Any help is greatly appreciated.
I don't understand what your objection is to Python ints, but the ctypes module provides a world of low-level mutable objects; e.g.,
>>> import ctypes
>>> i = ctypes.c_ushort(12) # 2-byte unsigned integer, initial value 12
>>> i
c_ushort(12)
>>> i.value += 0xffff - 12
>>> hex(i.value)
'0xffff'
>>> i.value += 1
>>> i.value # silently overflowed to 0
0
This is my first time ever answering my own SO question, so hopefully this turns out right. If anyone has additions or further answers, then feel free to answer as well, or comment on this one.
I ended up solving the problem by adding another return value to the return statement of buildPacketHeader, so that in addition to returning a string I also return the new value of bit as well. I confirmed that it was working by setting up the following prints inside of buildPacketHeader:
#add ABP
print "before:",ABP #test line for flipBit
ABP = flipBit(ABP)
abp = str(ABP)
print "after:",ABP #test line for flipBit
The output of which is shown here (I ended it early but the proof of functionality is still visible)
% python ftpc.py 164.107.112.70 4000 8000 manygettysburgs.txt
before: 1
after: 0
Timeout, resending packet...
before: 0
after: 1
Timeout, resending packet...
before: 1
after: 0
As can be seen, the before of the second packet is the after of the first packet, and the before of the third packet is the after of the second packet. Through this, you can see that the program is now flipping bits correctly.
The change made to buildPacketHeader is shown below:
return Header
becomes
return Header, ABP
and calls to buildPacketHeader:
filesizestr = buildPacketHeader(IP,PORT,1,bit)
become
filesizestr, bit = buildPacketHeader(IP,PORT,1,bit)
Very simple for such a bother. Make sure you return values if you want to make them change.
I need a python program to use a TCP connection(to another program) to retrieve sets of 9 bytes.
The first of the nine bytes represents a char, and the rest represent a double.
How can I extract this information from a python socket? Will I have to do the maths manually to convert the stream data or is there a better way?
Take a look at python struct
http://docs.python.org/library/struct.html
So something like
from struct import unpack
unpack('cd',socket_read_buffer)
--> ('c', 3.1415)
Be careful of Endianness.
If both client and server are written in python, I'd suggest you use pickle. It lets you convert python variables to bytes and then back to python variables with their original type.
#SENDER
import pickle, struct
#convert the variable to bytes with pickle
bytes = pickle.dumps(original_variable)
#convert its size to a 4 bytes int (in bytes)
#I: 4 bytes unsigned int
#!: network (= big-endian)
length = struct.pack("!I", len(bytes))
a_socket.sendall(length)
a_socket.sendall(bytes)
#RECEIVER
import pickle, struct
#This function lets you receive a given number of bytes and regroup them
def recvall(socket, count):
buf = b""
while count > 0:
newbuf = socket.recv(count)
if not newbuf: return None
buf += newbuf
count -= len(newbuf)
return buf
length, = struct.unpack("!I", recvall(socket, 4)) #we know the first reception is 4 bytes
original_variable = pickle.loads(recval(socket, length))