I am a little confused about the select module in python. As PyMOTW describes:
select monitors sockets, open files, and pipes (anything with a
fileno() method that returns a valid file descriptor) until they
become readable or writable, or a communication error occurs.
I am confused about what do readable and writable mean? What's the difference between them?
Besides, it describes:
select() returns three new lists, containing subsets of the contents
of the lists passed in. All of the sockets in the readable list have
incoming data buffered and available to be read. All of the sockets in
the writable list have free space in their buffer and can be written
to. The sockets returned in exceptional have had an error (the actual
definition of “exceptional condition” depends on the platform).
So in my understanding, the select module is such a tool which monitors multiple sockets when they are open and working. Select can tell a specific socket whether it should read the data, write data or there is an error. Is that right? Could someone explain to me that how does it achieve multi-connection socket communication?
select doesn't tell sockets anything. It just watches them.
Say you have a building with one entrance. You post a receptionist there. He watches the door, and when there's someone at the door, he goes and opens the door for the guest.
Now you build a back entrance, but you are too cheap to hire a second receptionist. So while the front receptionist is staring at the front entrance, the back entrance is piling up with very angry people staring at the stubbornly closed door.
If only there was a surveillance system, so that the poor receptionist could see both doors at the same time...
That's what select does.
Normally when you f.read() (on a blocking file descriptor), your program stops till some data shows up. When you f.write() but the other side has signalled their buffer is full, your program will stop till the other end clears some space in their buffer and signals it's okay to receive again. And when your program is stuck on some IO operation, it can't do anything else - while incoming data in other sockets is piling up, or maybe while a user on some other socket is impatiently waiting for their response.
With select, it waits until any file descriptor has something you can do about. It will wait until something becomes actionable; then it will tell you where f.read will be instantaneously responded to, and where write will be instantly sent, without further blocking.
Related
I am trying to simulate a communication protocol where I am following a pattern, so I constantly loop though looking for the same set of characters to reply information. I'm using an RS-232 adapter and the protocol I am simulating is asynchronous and half-duplex where the rx/tx lines are tied together by design and that causes a sort of echo when reading after writing.
That said, I need to be able to clear the input buffer after every write I send out in order to avoid reading what I just wrote. So whenever I use reset_input_buffer() it does not clear the last message I sent out. I have tried to fix this using a couple of methods, such as: using reset_output_buffer() together with reset_input_buffer(), using reset_input_buffer() twice, and using flush(). None of these methods make any difference, the only other method that works to clear the buffer is closing and immediately opening the port but this causes a delay that messes with the timing as it is critical at certain times.
I'm open to any suggestions, please help!
I found a basic space invaders pygame on Youtube and I want to modify it in order that, as of right now, the server is doing all the processing and drawing, and the client only sends keyboard input(all run on localhost). The problem is that the game is no longer that responsive after I implemented this mechanism. It appears to be about 1 second delay after I press a key to when the ship is actually moving (when starting the game from pycharm, when it starts from cmd it's much worse).
I don't have any idea why this is happening because there isn't really anything heavy to process and I could really use your help.
I also monitored the Ethernet traffic in wireshark and there seems to be sent about 60-70 packets each second.
Here is the github link with all the necesary things: https://github.com/PaaulFarcas/C-S-Game
I would expect this code in the main loop is the issue:
recv = conn.recv(661)
keys = pickle.loads(recv)
The socket function conn.recv() will block until 661 bytes are received, or there is some socket event (like being closed). So your program is blocking every iteration of the main loop waiting for the data to arrive.
You could try using socket.setblocking( False ) as per the manual.
However I prefer to use the select module (manual link), as I like the better level of control it gives. Basically you can use it to know if any data has arrived on the socket (or if there's an error). This gives you a simple select-read-buffer type logic loop:
procedure receiveSocketData
Use select on the socket, with an immediate timeout.
Did select indicate any data arrived on my socket?
Read the data, appending it to a Rx-buffer
Does the Rx-buffer contain enough for a whole packet?
take the packet-chunk from the head of the Rx-buffer
decode & return it
Else
Keep the Rx-Buffer somewhere safe
return None
Did any errors happen on my socket
clear Rx-Buffer
close socket
return error
I guess using an unknown-sized packet, you could try to un-pickle it, and return OK when successful... this is quite inefficient though. I would use a fixed size packet and the struct module to pack and unpack it in network-byte-order.
I'm about to write a program for a racecar, that creates a txt and continuously adds new lines to it. Unfortunately I can't close the file, because when the car shuts off the raspberry (which the program is running on) gets also shut down. So I have no chance of closing the txt.
Is this a problem?
Yes and no. Data is buffered at different places in the process of writing: the file object of python, the underlying C-functions, the operating system, the disk controller. Even closing the file, does not guarantee, that all these buffers are written physically. Only the first two levels are forced to write their buffers to the next level. The same can be done by flushing the filehandle without closing it.
As long as the power-off can occur anytime, you have to deal with the fact, that some data is lost or partially written.
Closing a file is important to give free limited resources of the operating system, but this is no concern in your setup.
I was wondering if there is a way I can tell python to wait until it gets a response from a server to continue running.
I am writing a turn based game. I make the first move and it sends the move to the server and then the server to the other computer. The problem comes here. As it is no longer my turn I want my game to wait until it gets a response from the server (wait until the other player makes a move). But my line:
data=self.sock.recv(1024)
hangs because (I think) it's no getting something immediately. So I want know how can I make it wait for something to happen and then keep going.
Thanks in advance.
The socket programming howto is relevant to this question, specifically this part:
Now we come to the major stumbling block of sockets - send and recv operate on the
network buffers. They do not necessarily handle all the bytes you hand them (or expect
from them), because their major focus is handling the network buffers. In general, they
return when the associated network buffers have been filled (send) or emptied (recv).
They then tell you how many bytes they handled. It is your responsibility to call them
again until your message has been completely dealt with.
...
One complication to be aware of: if your conversational protocol allows multiple
messages to be sent back to back (without some kind of reply), and you pass recv an
arbitrary chunk size, you may end up reading the start of a following message. You’ll
need to put that aside >and hold onto it, until it’s needed.
Prefixing the message with it’s length (say, as 5 numeric characters) gets more complex,
because (believe it or not), you may not get all 5 characters in one recv. In playing
around, you’ll get away with it; but in high network loads, your code will very quickly
break unless you use two recv loops - the first to determine the length, the second to
get the data part of the message. Nasty. This is also when you’ll discover that send
does not always manage to get rid of everything in one pass. And despite having read
this, you will eventually get bit by it!
The main takeaways from this are:
you'll need to establish either a FIXED message size, OR you'll need to send the the size of the message at the beginning of the message
when calling socket.recv, pass number of bytes you actually want (and I'm guessing you don't actually want 1024 bytes). Then use LOOPs because you are not guaranteed to get all you want in a single call.
That line, sock.recv(1024), blocks until 1024 bytes have been received or the OS detects a socket error. You need some way to know the message size -- this is why HTTP messages include the Content-Length.
You can set a timeout with socket.settimeout to abort reading entirely if the expected number of bytes doesn't arrive before a timeout.
You can also explore Python's non-blocking sockets using setblocking(0).
I'd like to write a simple command line proxy in Python to sit between a Telnet/SSH connection and a local serial interface. The application should simply bridge I/O between the two, but filter out certain unallowed strings (matched by regular expressions). (This for a router/switch lab in which the user is given remote serial access to the boxes.)
Basically, a client established a Telnet or SSH connection to the daemon. The daemon passes the client's input out (for example) /dev/ttyS0, and passes input from ttyS0 back out to the client. However, I want to be able to blacklist certain strings coming from the client. For instance, the command 'delete foo' should not be allowed.
I'm not sure how best to approach this. Communication must be asynchronous; I can't simply wait for a carriage return to allow the buffer to be fed out the serial interface. Matching regular expressions against the stream seems tricky too, as all of the following must be intercepted:
delete foo(enter)
del foo(enter)
el foo(ctrl+a)d(enter)
dl(left)e(right) foo(enter)
...and so forth. The only solid delimiter is the CR/LF.
I'm hoping someone can point me in the right direction. I've been looking through Python modules but so far haven't come up with anything.
Python is not my primary language, so I'll leave that part of the answer for others. I do alot of security work, though, and I would urge a "white list" approach, not a "black list" approach. In other words, pick a set of safe commands and forbid all others. This is much much easier than trying to think of all the malicious possibilities and guarding against all of them.
As all the examples you show finish with (enter), why is it that...:
Communication must be asynchronous; I
can't simply wait for a carriage
return to allow the buffer to be fed
out the serial interface
if you can collect incoming data until the "enter", and apply the "edit" requests (such as the ctrl-a, left, right in your examples) to the data you're collecting, then you're left with the "completed command about to be sent" in memory where it can be matched and rejected or sent on.
If you must do it character by character, .read(1) on the (unbuffered) input will allow you to, but the vetting becomes potentially more problematic; again you can keep an in-memory image of the edited command that you've sent so far (as you apply the edit requests even while sending them), but what happens when the "enter" arrives and your vetting shows you that the command thus composed must NOT be allowed -- can you e.g. send a number of "delete"s to the device to wipe away said command? Or is there a single "toss the complete line" edit request that would serve?
If you must send every character as you receive it (not allowed to accumulate them until decision point) AND there is no way to delete/erase characters already sent, then the task appears to be impossible (though I don't understand the "can't wait for the enter" condition AT ALL, so maybe there's hope).
After thinking about this for a while, it doesn't seem like there's any practical, reliable method to filter on client input. I'm going to attempt this from another angle: if I can identify persistent patterns in warning messages coming from the serial devices (e.g. confirmation prompts) I may be able to abort reliably. Thanks anyway for the input!
Fabric is doing a similar thing.
For SSH api you should check paramiko.