So here's the problem, I have a small server script in Python that is supposed to accept multiple clients and based on the message they are sending, receiving a certain command back to them. It's a simple concept and it's working like I want to, with one really big problem: I put each connection on hold and in separate thread, and I want when a certain connected users puts EXIT to close the connection...Which works, with one really big problem - the thread is kept alive and there is no way to kill it and that really bothers me.
sock = socket()
sock.bind((host,port))
sock.listen(50)
def clientthread(conn):
while True:
data = conn.recv(1024).strip()
if(data == "HELO"):
conn.send("HELO")
elif(data == "EXIT"):
conn.close()
break
return
while True:
conn,addr = sock.accept()
start_new_thread(clientthread, (conn,))
conn.close()
sock.close()
I searched of a way to terminate a thread but just couldn't find it, .join() is not working here since it detects the thread as "dummy", it does not recognize the __stop() and since a couple of searches on google for this topic I'm really out of options. Any idea? I'll be really grateful, thanks.
AFAIK, you can't kill a thread from another - you have to arrange for the thread-to-be-killed to notice some flag has changed, and terminate itself.
BTW, your socket code looks a little off - you need a loop around your send's and recv's unless you use something like twisted or bufsock. IMO, bufsock is much easier and less error prone than twisted, but I may be biased because I wrote bufsock. http://stromberg.dnsalias.org/~strombrg/bufsock.html
The problem with what I'm seeing is that TCP reserves the right to split or aggregate transmission units. Usually it won't, but under high load, or with a changing Path MTU, or even just Nagle, it probably will.
Assuming you're using Python v2.4+, you should be using the newer Threading module. Check out a tutorial on it here - It explains the use of the threading module you're using now and how and why you should use the newer Threading module.
Related
I'm building photovoltaic motorized solar trackers. They're controlled by Raspberry Pi's running python script. RPI's are connected to my public openVPN server for remote control and continuous software development. That's working fine. Recently a passionate customer asked me for some sort of telemetry data for his tracker - let's say, it's current orientation, measured wind speed etc.. By being new to python, I'm really struggling with this part.
I've decided to use socket approach from guides like this. Python script listens on a socket, and my openVPN server, which is also web server, connects to it using PHP fsockopen. Python sends telemetry data, PHP makes it user friendly and displays it on the web. Everything so far works, however I don't know how to design my python script around it.
The problem is, that my script has to run continuously, and socket.accept() halts it's execution, waiting for a connection. Didn't find any obvious solution on the web. Would multi-threading work for this? Sounds a bit like overkill.
Is there a way to run socket listening asynchronously? Like, for example, pigpio callback's which I'm using abundantly?
Or alternatively, is there a better way to accomplish my goal?
I tried with remote accessing status file that my script is maintaining, but that proved to be extremely involved with setup and prone to errors when the file was being written.
I also tried running the second script. Problem is, then I have no access to relevant data, or I need to read beforementioned status file, and that leads to the same problems as above.
Relevant bit of code is literally only this:
# Main loop
try:
while True:
# Telemetry
conn, addr = S.accept()
conn.send(data.encode())
conn.close()
Best regards.
For a simple case like this I would probably just wrap the socket code into a separate thread.
With multithreading in python, the Global Interpreter Lock (GIL) means that only one thread executes at a time, so you don't really need to add any further locks to the data if you're just reading the values, and don't care if it's also being updated at the same time.
Your code would essentially read something like:
from threading import Thread
def handle_telemetry_requests():
# Main loop
try:
while True:
# Telemetry
conn, addr = S.accept()
conn.send(data.encode())
conn.close()
except:
# Error handling here (this will cause thread to exit if any error occurs)
pass
socket_thread = Thread(target=handle_telemetry_requests)
socket_thread.daemon = True
socket_thread.start()
Setting the daemon flag means that when the main application ends, the thread will also be terminated.
Python does provide the asyncio module - which may provide the callbacks you're looking for (though I don't have any experience with this).
Other options are to run a flask server in the python apps which will handle the sockets for you and you can just code the endpoints to request the data. Or think about using an MQTT broker - the current data can be written to that - and other apps can subscribe to updates.
I found a basic space invaders pygame on Youtube and I want to modify it in order that, as of right now, the server is doing all the processing and drawing, and the client only sends keyboard input(all run on localhost). The problem is that the game is no longer that responsive after I implemented this mechanism. It appears to be about 1 second delay after I press a key to when the ship is actually moving (when starting the game from pycharm, when it starts from cmd it's much worse).
I don't have any idea why this is happening because there isn't really anything heavy to process and I could really use your help.
I also monitored the Ethernet traffic in wireshark and there seems to be sent about 60-70 packets each second.
Here is the github link with all the necesary things: https://github.com/PaaulFarcas/C-S-Game
I would expect this code in the main loop is the issue:
recv = conn.recv(661)
keys = pickle.loads(recv)
The socket function conn.recv() will block until 661 bytes are received, or there is some socket event (like being closed). So your program is blocking every iteration of the main loop waiting for the data to arrive.
You could try using socket.setblocking( False ) as per the manual.
However I prefer to use the select module (manual link), as I like the better level of control it gives. Basically you can use it to know if any data has arrived on the socket (or if there's an error). This gives you a simple select-read-buffer type logic loop:
procedure receiveSocketData
Use select on the socket, with an immediate timeout.
Did select indicate any data arrived on my socket?
Read the data, appending it to a Rx-buffer
Does the Rx-buffer contain enough for a whole packet?
take the packet-chunk from the head of the Rx-buffer
decode & return it
Else
Keep the Rx-Buffer somewhere safe
return None
Did any errors happen on my socket
clear Rx-Buffer
close socket
return error
I guess using an unknown-sized packet, you could try to un-pickle it, and return OK when successful... this is quite inefficient though. I would use a fixed size packet and the struct module to pack and unpack it in network-byte-order.
Every UDP server example I can find uses a while True loop to listen for incoming data. I'm attempting to use a single UDP socket server as part a kivy window that's also doing other things. As soon as I implement the server's while True loop everything locks up, as I guess I would expect it to do.
How do I listen on a UDP port and also have the rest of the program continue functioning?
I've tried moving the UDP server handling to another (udp_server.py) file and then importing the function, but since I'm importing the while loop nothing changes.
I've also tried assigning the received data to a variable inside udp_server.py and then just importing that variable, with udp_server.py already running separately, but even that is locking up my main program.
I'm 99.99% sure it's just some basic thing that I should already know, but I'm new to Python. Thanks in advance for any help.
Thank you Chris!!!!!!
I'm sure I'm understating the complexity of threading, but it works great now and the only thing I had to add was:
def thread_function():
from udp_server import amx_rx
# do stuff with amx_rx...
# class TouchPanel stuff...
if __name__ == '__main__':
x = threading.Thread(target=thread_function, daemon=True)
x.start()
try:
TouchPanel().run()
except KeyboardInterrupt:
raise
Now I have a running program with a UDP socket listening in the background! Thank you!!!
So I've used a bit of threading in python 2.6, but I'm confused how to do it in python 3, and it's been a while. Basically, I have a function that I pass 2 variables. I want to thread that function so that every time it's called, it creates a new thread instance, and when it finishes, the thread closes. It shouldn't be that difficult, but for some reason, after reading a ton of documentation, I can't seem to figure it out. Here's an example.
```
def accept_connection(clientsocket, addr):
print("Got a connection from %s" % str(addr)) #some serverside debugging output
msg = clientsocket.recv(1024) #recieve a message from the client
ClientAppend = str(addr) + str(msg, 'ascii')
Clients.append(ClientAppend)
print(Clients)
accept_connection(clientsocket, addr)
In case it's unclear, I want the accept_connection() function to be threaded, so that I can accept multiple connections at once and they aren't denied because one is already being processed. I'm not really sure how to do this, and I appreciate any and all help I can get.
So what you're looking for has already been nicely described in this here. You want to look into multithread tcp file transfer on localhost.
They have a nice tutorial from scratch which you can use here.
They have a five part tutorial with really simple examples which are well explained. I feel the tutorial would explain much better than I could.
I feel like this should be an easy solution but it's the end of the day and I'm brain-dead.
I am currently spawning a couple of processes, one process is receiving and storing data to a file. Another is parsing the data and third is waiting for user input to know when to stop the storing of data.
What I need to know how to do is breakout of my while loop. I'd like to not use global variables set by the parent process but if that is required I can do that.
Right now my code looks something like this:
while(packetReceived < totalToReceive):
data, addr = sock.recvfrom(packetSize)
My thoughts were something like this:
breakout = 0
while(packetReceived < totalToReceive || breakout != 0):
data, addr = sock.recvfrom(packetSize)
but then I need to set breakout somehow. Any help would be greatly appreciated.
You can't share state just by having a global variable in the parent process. This may appear to work, but it only works sometimes; it's neither reliable nor predictable. Except that on Windows, it reliably and predictably never works; each child will always have its own independent copy of the flag, and therefore you will never quit.
If you really want to do this by sharing a variable, see Sharing state between processes in the docs, but the short version is: You create a multiprocessing.Value. And then you use a multiprocessing.Condition to protect that value against races, because otherwise, there's no guarantee that the child processes will ever see a change from the parent.
Of course you can fake this by, e.g. creating an mmap of minimum size and just using m[0] as a flag and m.flush() instead of the condition, but that's not really any simpler.
The alternative way to do this is to use a multiprocessing.Pipe or similar to pass a "shut down now" message. The child processes can each spawn a thread to block on the pipe, or you can toss the pipe and your socket into a select together, or all the other usual tricks.
There may be another, simpler option in this case: don't use multiprocessing in the first place. Clearly your background task is not CPU-bound, since it's just looping around reading from a socket, so why not just threading?
Also, it strikes me that you might be able to simplify your design in other ways, which could remove this problem entirely. Do you need a file between the reading and processing jobs instead of, say, a queue, or even just a direct sequential pipeline? Can you toss the user input and the socket into the same event loop (plain old select if user input is stdin and you don't care about Windows; use a QSocket instead of a socket.socket if user input is a Qt GUI; twisted if you're willing to learn twisted; etc.). Or, is there real user input, or just "quit now" (or "shut down the socket and process the remaining messages now"), which you could handle with ^C?
Instead of using a multiprocessing variable, consider checking for the presence of "poison pill" to break out of your loop.
For example, change:
data, addr = sock.recvfrom(packetSize)
to something like:
received = sock.recvfrom(packetSize)
if received is None:
break
data, addr = received
You can signal the process to break out of its loop by sending it a None value. I'm not sure if your sock can send/receive None, but the general idea is the same.