I found a basic space invaders pygame on Youtube and I want to modify it in order that, as of right now, the server is doing all the processing and drawing, and the client only sends keyboard input(all run on localhost). The problem is that the game is no longer that responsive after I implemented this mechanism. It appears to be about 1 second delay after I press a key to when the ship is actually moving (when starting the game from pycharm, when it starts from cmd it's much worse).
I don't have any idea why this is happening because there isn't really anything heavy to process and I could really use your help.
I also monitored the Ethernet traffic in wireshark and there seems to be sent about 60-70 packets each second.
Here is the github link with all the necesary things: https://github.com/PaaulFarcas/C-S-Game
I would expect this code in the main loop is the issue:
recv = conn.recv(661)
keys = pickle.loads(recv)
The socket function conn.recv() will block until 661 bytes are received, or there is some socket event (like being closed). So your program is blocking every iteration of the main loop waiting for the data to arrive.
You could try using socket.setblocking( False ) as per the manual.
However I prefer to use the select module (manual link), as I like the better level of control it gives. Basically you can use it to know if any data has arrived on the socket (or if there's an error). This gives you a simple select-read-buffer type logic loop:
procedure receiveSocketData
Use select on the socket, with an immediate timeout.
Did select indicate any data arrived on my socket?
Read the data, appending it to a Rx-buffer
Does the Rx-buffer contain enough for a whole packet?
take the packet-chunk from the head of the Rx-buffer
decode & return it
Else
Keep the Rx-Buffer somewhere safe
return None
Did any errors happen on my socket
clear Rx-Buffer
close socket
return error
I guess using an unknown-sized packet, you could try to un-pickle it, and return OK when successful... this is quite inefficient though. I would use a fixed size packet and the struct module to pack and unpack it in network-byte-order.
Related
I am a little confused about the select module in python. As PyMOTW describes:
select monitors sockets, open files, and pipes (anything with a
fileno() method that returns a valid file descriptor) until they
become readable or writable, or a communication error occurs.
I am confused about what do readable and writable mean? What's the difference between them?
Besides, it describes:
select() returns three new lists, containing subsets of the contents
of the lists passed in. All of the sockets in the readable list have
incoming data buffered and available to be read. All of the sockets in
the writable list have free space in their buffer and can be written
to. The sockets returned in exceptional have had an error (the actual
definition of “exceptional condition” depends on the platform).
So in my understanding, the select module is such a tool which monitors multiple sockets when they are open and working. Select can tell a specific socket whether it should read the data, write data or there is an error. Is that right? Could someone explain to me that how does it achieve multi-connection socket communication?
select doesn't tell sockets anything. It just watches them.
Say you have a building with one entrance. You post a receptionist there. He watches the door, and when there's someone at the door, he goes and opens the door for the guest.
Now you build a back entrance, but you are too cheap to hire a second receptionist. So while the front receptionist is staring at the front entrance, the back entrance is piling up with very angry people staring at the stubbornly closed door.
If only there was a surveillance system, so that the poor receptionist could see both doors at the same time...
That's what select does.
Normally when you f.read() (on a blocking file descriptor), your program stops till some data shows up. When you f.write() but the other side has signalled their buffer is full, your program will stop till the other end clears some space in their buffer and signals it's okay to receive again. And when your program is stuck on some IO operation, it can't do anything else - while incoming data in other sockets is piling up, or maybe while a user on some other socket is impatiently waiting for their response.
With select, it waits until any file descriptor has something you can do about. It will wait until something becomes actionable; then it will tell you where f.read will be instantaneously responded to, and where write will be instantly sent, without further blocking.
Here's a basic idea of the threads that I am creating in my program:
Main thread
|
ListenerCreator(The WebSocketServer thread) ---> Several listener threads(using log())
So the main thread creates a ListenerCreator thread, which connects to a number of clients and creates a listener thread for each client. Here's briefly what a listener thread does:
EDIT1 :
I'm using WebSockets to read/write data off my client. I've made my own server for this purpose. There is a framing protocol which the standard specifies -- and I am using that. On the client side I am simply using WebSocket.send() and "unmasking" the messages according to the instructions given in the protocol(see section 5.3 in the link above).
I would be willing to provide the server code if someone requests it, however, here's a brief outline:
class WebSocketServer:
def start():
#Open server socket, bind to host:port
while True:
#Accept client socket, start a new listener thread for self.log(client)
def log(client):
#Receive data using socket.socket.recv(1024)
#Unmask data as per the protocol
#Decode using data.decode("utf-8")
#Append to data_q while holding data_q_lock
There are other methods - those to facilitate sending, closing, handshaking and so on.
Meanwhile in the main thread:
while breaking!=len(client_list):
#time.sleep(0.5)
with data_q_lock:
for i in range(len(data_q)):
mes = data_q.pop()
for m in client_list:
if "#DONE"== mes:
breaking += 1
if(mes[:len("#COUNT:")] == "#COUNT:"):
print(mes)
So basically what this loop does is: Loop thru the data_q, if the message starts with "#COUNT", print the message, and after getting a certain number of "#DONE" messages, exit the loop.
If the time.sleep is uncommented, then this code works, however without time.sleep I get an UnicodeDecodeError in the log function.
Also I only get the error sometimes , sometimes the program works perfectly.
(The client is sending the same data every time, by the way)
So, my question is, why is the time.sleep required?
I thought it was something to do with the GIL in python, as time.sleep releases the GIL. However, even after reading about it I couldn't solve the question
Currently there is no information about how the listener is reading data off the socket. It seems likely however that this is being caused by the usual misunderstanding of sockets.
Data sent down a socket is not "framed" in any way by the socket. Imagine if I sent the message "hello" three times down a socket. Then, like writing to a file without line breaks, the following would flow on the socket:
hellohellohello
Now consider the reader ... when reading the data, how does it know where one message ("hello") starts and and the next? It cannot, unless the sender and receiver agree about how that data should be "framed". This could be done by agreeing on some protocol like:
null-terminating data; or
fixed size messages; or
size prefixed messages.
It gets more complicated of course, even once you've decided how the data should be framed, you cannot guarantee that socket.recv will return a "whole" message ... it will simply return whatever data happens to be in the buffer at the time. It may be a half a message, or a message and a half. Its your job to collate the data read from the socket and divide it into messages.
Turning to your problem, where you are sending utf-8 data. How does the reader know it has read a full utf-8 data message? Most likely, what is happening here is that you have only received a partial message ... there is still more to arrive.
In particular, a valid utf-8 character may consist of more than one byte. So if your partial message ends in the middle of a multi-byte utf-8 representation of a character, then you can certainly not decode it.
I am unable to grasp this with the help of Programming concepts in general with the following scenario:
Note: All Data transmission in this scenario is done via UDP packets using socket module of Python3
I have a Server which sends some certain amount of data, assume 300 Packets over a WiFi Channel
At the other end, I have a receiver which works on a certain Decoding process to decode the data. This Decoding Process is kind of Infinite Loop which returns Boolean Value true or false at every iteration depending on certain aspects which can be neglected as of now
a Rough Code Snippet is as follows:Python3
incomingPacket = next(bringNextFromBuffer)
if decoder.consume_data(incomingPacket):
# this if condition is inside an infinite loop
# unless the if condition becomes True keep
# keep consuming data in a forever for loop
print("Data has been received")
Everything as of moment works since the Server and Client are in proximity and the data can be decoded. But in practical scenarios I want to check the loop that is mentioned above. For instance, after a certain amount of time, if the above loop is still in the Forever (Infinite) state I would like to send out something back to the server to start the data sending again.
I am not much clear with multithreading concept, but can I use a thread over here in this scenario?
For Example:
Thread a Process for a certain amount of time and keep checking the decoder.consume_data() function and if the time expires and the output is still False can I then send out a kind of Feedback to the server using struct.pack() over sockets.
Of course the networking logic, need NOT be addressed as of now. But is python capable of MONITORING THIS INFINITE LOOP VIA A PARALLEL THREAD OR OTHER CONCEPT OF PROGRAMMING?
Caveats
Unfortunately the Receiver in question is a dumb receiver i.e. No user control is specified. Only thing Receiver can do is decode the data and perhaps send a Feedback to the Server stating whether the data is received or not and that is possible only when the above mentioned LOOP is completed.
What is a possible solution here?
(Would be happy to share more information on request)
Yes you can do this. Roughly it'll look like this:
from threading import Thread
from time import sleep
state = 'running'
def monitor():
while True:
if state == 'running':
tell_client()
sleep(1) # to prevent too much happening here
Thread(target=monitor).start()
while state == 'running':
receive_data()
I was wondering if there is a way I can tell python to wait until it gets a response from a server to continue running.
I am writing a turn based game. I make the first move and it sends the move to the server and then the server to the other computer. The problem comes here. As it is no longer my turn I want my game to wait until it gets a response from the server (wait until the other player makes a move). But my line:
data=self.sock.recv(1024)
hangs because (I think) it's no getting something immediately. So I want know how can I make it wait for something to happen and then keep going.
Thanks in advance.
The socket programming howto is relevant to this question, specifically this part:
Now we come to the major stumbling block of sockets - send and recv operate on the
network buffers. They do not necessarily handle all the bytes you hand them (or expect
from them), because their major focus is handling the network buffers. In general, they
return when the associated network buffers have been filled (send) or emptied (recv).
They then tell you how many bytes they handled. It is your responsibility to call them
again until your message has been completely dealt with.
...
One complication to be aware of: if your conversational protocol allows multiple
messages to be sent back to back (without some kind of reply), and you pass recv an
arbitrary chunk size, you may end up reading the start of a following message. You’ll
need to put that aside >and hold onto it, until it’s needed.
Prefixing the message with it’s length (say, as 5 numeric characters) gets more complex,
because (believe it or not), you may not get all 5 characters in one recv. In playing
around, you’ll get away with it; but in high network loads, your code will very quickly
break unless you use two recv loops - the first to determine the length, the second to
get the data part of the message. Nasty. This is also when you’ll discover that send
does not always manage to get rid of everything in one pass. And despite having read
this, you will eventually get bit by it!
The main takeaways from this are:
you'll need to establish either a FIXED message size, OR you'll need to send the the size of the message at the beginning of the message
when calling socket.recv, pass number of bytes you actually want (and I'm guessing you don't actually want 1024 bytes). Then use LOOPs because you are not guaranteed to get all you want in a single call.
That line, sock.recv(1024), blocks until 1024 bytes have been received or the OS detects a socket error. You need some way to know the message size -- this is why HTTP messages include the Content-Length.
You can set a timeout with socket.settimeout to abort reading entirely if the expected number of bytes doesn't arrive before a timeout.
You can also explore Python's non-blocking sockets using setblocking(0).
I have a long running python process running headless on a raspberrypi (controlling a garden) like so:
from time import sleep
def run_garden():
while 1:
/* do work */
sleep(60)
if __name__ == "__main__":
run_garden()
The 60 second sleep period is plenty of time for any changes happening in my garden (humidity, air temp, turn on pump, turn off fan etc), BUT what if i want to manually override these things?
Currently, in my /* do work */ loop, i first call out to another server where I keep config variables, and I can update those config variables via a web console, but it lacks any sort of real time feel, because it relies on the 60 second loop (e.g. you might update the web console, and then wait 45 seconds for the desired effect to take effect)
The raspberryPi running run_garden() is dedicated to the garden and it is basically the only thing taking up resources. So i know i have room to do something, I just dont know what.
Once the loop picks up the fact that a config var has been updated, the loop could then do exponential backoff to keep checking for interaction, rather than wait 60 seconds, but it just doesnt feel like that is a whole lot better.
Is there a better way to basically jump into this long running process?
Listen on a socket in your main loop. Use a timeout (e.g. of 60 seconds, the time until the next garden update should be performed) on your socket read calls so you get back to your normal functionality at least every minute when there are no commands coming in.
If you need garden-tending updates to happen no faster than every minute you need to check the time since the last update, since read calls will complete significantly faster when there are commands coming in.
Python's select module sounds like it might be helpful.
If you've ever used the unix analog (for example in socket programming maybe?), then it'll be familiar.
If not, here is the select section of a C sockets reference I often recommend. And here is what looks like a nice writeup of the module.
Warning: the first reference is specifically about C, not Python, but the concept of the select system call is the same, so the discussion might be helpful.
Basically, it allows you to tell it what events you're interested in (for example, socket data arrival, keyboard event), and it'll block either forever, or until a timeout you specify elapses.
If you're using sockets, then adding the socket and stdin to the list of events you're interested in is easy. If you're just looking for a way to "conditionally sleep" for 60 seconds unless/until a keypress is detected, this would work just as well.
EDIT:
Another way to solve this would be to have your raspberry-pi "register" with the server running the web console. This could involve a little bit extra work, but it would give you the realtime effect you're looking for.
Basically, the raspberry-pi "registers" itself, by alerting the server about itself, and the server stores the address of the device. If using TCP, you could keep a connection open (which might be important if you have firewalls to deal with). If using UDP you could bind the port on the device before registering, allowing the server to respond to the source address of the "announcement".
Once announced, when config. options change on the server, one of two things usually happen:
A) You send a tiny "ping" (in the general sense, not the ICMP host detection protocol) to the device alerting it that config options have changed. At this point the host would immediately request the full config. set, acquiring the update with it.
B) You send the updated config. option (or maybe the entire config. set) back to the device. This decreases the number of messages between the device and server, but would probably take more work as it seems like more a deviation from your current setup.
Why not use an event based loop instead of sleeping for a certain amount of time.
That way your loop will only run when a change is detected, and it will always run when a change is detected (which is the point of your question?).
You can do such a thing by using:
python event objects
Just wait for one or all of your event objects to be triggered and run the loop. You can also wait for X events to be done, etc, depending if you expect one variable to be updated a lot.
Or even a system like:
broadcasting events