We Keep Coding

Python: socket.rcvfrom() lock all script

I have a simple code in python 3 using schedule and socket:
import schedule
import socket
from time import sleep
def readDataFromFile():
data = []
with open("/tmp/tmp.txt", "r") as f:
for singleLine in f.readlines():
data.append(str(singleLine))
if(len(data)>0):
writeToBuffer(data)
def readDataFromUDP():
udpData = []
rcvData, addr = sock.recvfrom(256)
udpData.append(rcvData.decode('ascii'))
if(len(udpData)>0):
writeToBuffer(udpData)
.
.
.
def main():
schedule.every().second.do(readDataFromFile)
schedule.every().second.do(readDataFromUDP)
while(1):
schedule.run_pending()
sleep(1)
UDP_IP = "192.xxx.xxx.xxx"
UDP_PORT = xxxx
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((UDP_IP, UDP_PORT))
main()
The problem is, script hung up on the sock.rcvfrom() instruction, and wait until data come.
How force python to run this job independently? Better idea is to run this in threads?

You can use threads here, and it'll work fine, but it will require a few changes. First, the scheduler on your background thread is going to try to kick off a new recvfrom every second, no matter how long the last one took. Second, since both threads are apparently trying to call the same writeToBuffer function, you're probably going to need a Lock or something else to synchronize them.
Rewriting the whole program around an asynchronous event loop is almost certainly overkill here.
Just changing the socket to be nonblocking and doing a hybrid is probably the simplest change, e.g., by using settimeout:
# wherever you create your socket
sock.settimeout(0.8)
# ...
def readDataFromUDP():
udpData = []
try:
rcvData, addr = sock.recvfrom(256)
except socket.timeout:
return
udpData.append(rcvData.decode('ascii'))
if(len(udpData)>0):
writeToBuffer(udpData)
Now, every time you call recvfrom, if there's data available, you'll handle it immediately; if not, it'll wait up to 0.8 seconds, and then raise an exception, which means you have no data to process, so go back and wait for the next loop. (There's nothing magical about that 0.8; I just figured something a little less than 1 second would be a good idea, so there's time left to do all the other work before the next schedule time hits.)
Under the covers, this works by setting the OS-level socket to non-blocking mode and doing some implementation-specific thing to wait with a timeout. You could do the same yourself by using setblocking(False) and using the select or selectors module to wait up to 0.8 seconds for the socket to be ready, but it's easier to just let Python take care of that for you.

Python multiple loops at the same time

I know that it is not possible to run multiple loops at the same time in Python.
Anyhow, what I need to achieve is that I have one loop running reading loads of sensor data, every 0.25 seconds.
At the same time I have signal devices running in parallel that need to send signals every 3 seconds.
My question is what way is best practice to achieve this?
Does it make sense to write two scripts and run them in parallel?
Does it make sense to use threading?
Is there any other possibility in order to make this work?
I would be greatful for code samples.
Thank you!
Edit:
Both loops are absolutely independent.
So, let's say while script 1 is running, reading the sensor data, when one of the sensors received a value < 300, it should run script 2 which will send the signals. At the same time when the sensors data gets > 300 it should stop script 2.

"Python multiple loops at the same time. I know that it is not possible [...]" - this looks really funny.
It is possible to run two loops at the same time, exactly how you described it. And both ways make much sense, depending on what do you actually need and want. If the tasks are completely independent you should run it as two scripts. If you need those two loops to realize one task and it makes sense for them to be in one file you can use multiprocessing.
Tested for python 2.7.5+ and 3.3.2+.
Here is some minimal example:
from multiprocessing import Process
import time
def f(name):
print('hello', name)
time.sleep(10)
def d(name):
print('test2', name)
time.sleep(10)
if __name__ == '__main__':
p1 = Process(target=f, args=('bob',))
p2 = Process(target=d, args=('alice',))
p1.start()
p2.start()
p1.join()
p2.join()
Script runs for 10s and both strings are printed right away, which means everything works.
time python3 ./process.py
hello bob
test2 alice
real 0m10.073s
user 0m0.040s
sys 0m0.016s

It is also possible by running multiple scripts and some as .pyw for convenience and having them exchange information by UDP sockets. Note 127.0.0.1 is to send to yourself under ANY circumstance. Also for port, just make sure no other programs use the port you use. As in other programs, I mean ANY program that uses ports or even basic router settings.
Sample (send)
import os
from sockets import *
host = "ip"
port = "9000"
addr = (host, port)
UDPSock = socket(AF_INET, SOCK_DGRAM)
data = "Random Text"
send = data.encode("ascii")
UDPSock.sendto(send, addr)
UDPSock.close()
Sample (Receive)
import os
from socket import *
host = ""
port = 9000
addr = (host, port)
UDPSock = socket(AF_INET, SOCK_DGRAM)
UDPSock.bind(addr)
(data, addr) = UDPSock.recvfrom(1024)#1024 is MAX bytes to receive
data = data.decode('ascii')
UDPSock.close()
You can use these to run separate loops and tell what to do from two separate programs.

Timeout and high cpu load problems using multiple telnet connection threads in python

I want to connect to multiple telnet hosts using threading in python, but I stumbled about an issue I'm not able to solve.
Using the following code on MAC OS X Lion / Python 2.7
import threading,telnetlib,socket
class ReaderThread(threading.Thread):
def __init__(self, ip, port):
threading.Thread.__init__(self)
self.ip = ip
self.port = port
self.telnet_con = telnetlib.Telnet()
def run(self):
try:
print 'Start %s' % self.ip
self.telnet_con.open(self.ip,self.port,30)
print 'Done %s' % self.ip
except socket.timeout:
print 'Timeout in %s' % self.ip
def join(self):
self.telnet_con.close()
ta = []
t1 = ReaderThread('10.0.1.162',9999)
ta.append(t1)
t2 = ReaderThread('10.0.1.163',9999)
ta.append(t2)
for t in ta:
t.start()
print 'Threads started\n'
In general it works, but either one of the threads (it is not always the same one) takes a long time to connect (about 20 second and sometimes even runs into a timeout). During that awfully long connection time (in an all local network), cpu load also goes up to 100 %.
Even more strange is the fact that if I'm using only one thread in the array it always works flawlessly. So it must have something to do with the use of multiple threads.
I already added hostname entries for all IP addresses to avoid a DNS lookup issue. This didn't make a difference.
Thanks in advance for your help.
Best regards
senexi

Ok, You have overridden join(), and you are not supposed to do that. The main thread calls join() on each thread when the main thread finishes, which is right after the last line in your code. Since your join() method returns before your telnet thread actually exits, Python gets confused and tries to call join() again, and this is what causes the 100% cpu usage. Try to put a 'print' statement in your join() method.
Your implementation of join() tries to close the socket (probably while the other thread is still trying to open a connection), and this might be what causing your telnet threads to never finish.

Python: Listen on two ports

import socket
backlog = 1 #Number of queues
sk_1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sk_2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
local = {"port":1433}
internet = {"port":9999}
sk_1.bind (('', internet["port"]))
sk_1.listen(backlog)
sk_2.bind (('', local["port"]))
sk_2.listen(backlog)
Basically, I have this code. I am trying to listen on two ports: 1433 and 9999. But, this doesn't seems to work.
How can I listen on two ports, within the same python script??

The fancy-pants way to do this if you want to use Python std-lib would be to use SocketServer with the ThreadingMixin -- although the 'select' suggestion is probably the more efficient.
Even though we only define one ThreadedTCPRequestHandler you can easily repurpose it such that each listener has it's own unique handler and it should be fairly trivial to wrap the server/thread creation into a single method if thats the kind of thing you like.
#!/usr/bin/python
import threading
import time
import SocketServer
class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
print "%s wrote: " % self.client_address[0]
print self.data
self.request.send(self.data.upper())
class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
pass
if __name__ == "__main__":
HOST = ''
PORT_A = 9999
PORT_B = 9876
server_A = ThreadedTCPServer((HOST, PORT_A), ThreadedTCPRequestHandler)
server_B = ThreadedTCPServer((HOST, PORT_B), ThreadedTCPRequestHandler)
server_A_thread = threading.Thread(target=server_A.serve_forever)
server_B_thread = threading.Thread(target=server_B.serve_forever)
server_A_thread.setDaemon(True)
server_B_thread.setDaemon(True)
server_A_thread.start()
server_B_thread.start()
while 1:
time.sleep(1)

The code so far is fine, as far as it goes (except that a backlog of 1 seems unduly strict), the problem of course comes when you try to accept a connection on either listening socket, since accept is normally a blocking call (and "polling" by trying to accept with short timeouts on either socket alternately will burn machine cycles to no good purpose).
select to the rescue!-) select.select (or on the better OSs select.poll or even select.epoll or select.kqueue... but, good old select.select works everywhere!-) will let you know which socket is ready and when, so you can accept appropriately. Along these lines, asyncore and asynchat provide a bit more organization (and third-party framework twisted, of course, adds a lot of such "asynchronous" functionality).
Alternatively, you can devote separate threads to servicing the two listening sockets, but in this case, if the different sockets' functionality needs to affect the same shared data structures, coordination (locking &c) may become ticklish. I would certainly recommend trying the async approach first -- it's actually simpler, as well as offering potential for substantially better performance!-)

error: can't start new thread

I have a site that runs with follow configuration:
Django + mod-wsgi + apache
In one of user's request, I send another HTTP request to another service, and solve this by httplib library of python.
But sometimes this service don't get answer too long, and timeout for httplib doesn't work. So I creating thread, in this thread I send request to service, and join it after 20 sec (20 sec - is a timeout of request). This is how it works:
class HttpGetTimeOut(threading.Thread):
def __init__(self,**kwargs):
self.config = kwargs
self.resp_data = None
self.exception = None
super(HttpGetTimeOut,self).__init__()
def run(self):
h = httplib.HTTPSConnection(self.config['server'])
h.connect()
sended_data = self.config['sended_data']
h.putrequest("POST", self.config['path'])
h.putheader("Content-Length", str(len(sended_data)))
h.putheader("Content-Type", 'text/xml; charset="utf-8"')
if 'base_auth' in self.config:
base64string = base64.encodestring('%s:%s' % self.config['base_auth'])[:-1]
h.putheader("Authorization", "Basic %s" % base64string)
h.endheaders()
try:
h.send(sended_data)
self.resp_data = h.getresponse()
except httplib.HTTPException,e:
self.exception = e
except Exception,e:
self.exception = e
something like this...
And use it by this function:
getting = HttpGetTimeOut(**req_config)
getting.start()
getting.join(COOPERATION_TIMEOUT)
if getting.isAlive(): #maybe need some block
getting._Thread__stop()
raise ValueError('Timeout')
else:
if getting.resp_data:
r = getting.resp_data
else:
if getting.exception:
raise ValueError('REquest Exception')
else:
raise ValueError('Undefined exception')
And all works fine, but sometime I start catching this exception:
error: can't start new thread
at the line of starting new thread:
getting.start()
and the next and the final line of traceback is
File "/usr/lib/python2.5/threading.py", line 440, in start
_start_new_thread(self.__bootstrap, ())
And the answer is: What's happen?
Thank's for all, and sorry for my pure English. :)

The "can't start new thread" error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.
You should probably look at the number of threads you're creating; the maximum number you will be able to create will be determined by your environment, but it should be in the order of hundreds at least.
It would probably be a good idea to re-think your architecture here; seeing as this is running asynchronously anyhow, perhaps you could use a pool of threads to fetch resources from another site instead of always starting up a thread for every request.
Another improvement to consider is your use of Thread.join and Thread.stop; this would probably be better accomplished by providing a timeout value to the constructor of HTTPSConnection.

You are starting more threads than can be handled by your system. There is a limit to the number of threads that can be active for one process.
Your application is starting threads faster than the threads are running to completion. If you need to start many threads you need to do it in a more controlled manner I would suggest using a thread pool.

I was running on a similar situation, but my process needed a lot of threads running to take care of a lot of connections.
I counted the number of threads with the command:
ps -fLu user | wc -l
It displayed 4098.
I switched to the user and looked to system limits:
sudo -u myuser -s /bin/bash
ulimit -u
Got 4096 as response.
So, I edited /etc/security/limits.d/30-myuser.conf and added the lines:
myuser hard nproc 16384
myuser soft nproc 16384
Restarted the service and now it's running with 7017 threads.
Ps. I have a 32 cores server and I'm handling 18k simultaneous connections with this configuration.

I think the best way in your case is to set socket timeout instead of spawning thread:
h = httplib.HTTPSConnection(self.config['server'],
timeout=self.config['timeout'])
Also you can set global default timeout with socket.setdefaulttimeout() function.
Update: See answers to Is there any way to kill a Thread in Python? question (there are several quite informative) to understand why. Thread.__stop() doesn't terminate thread, but rather set internal flag so that it's considered already stopped.

I completely rewrite code from httplib to pycurl.
c = pycurl.Curl()
c.setopt(pycurl.FOLLOWLOCATION, 1)
c.setopt(pycurl.MAXREDIRS, 5)
c.setopt(pycurl.CONNECTTIMEOUT, CONNECTION_TIMEOUT)
c.setopt(pycurl.TIMEOUT, COOPERATION_TIMEOUT)
c.setopt(pycurl.NOSIGNAL, 1)
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.SSL_VERIFYHOST, 0)
c.setopt(pycurl.SSL_VERIFYPEER, 0)
c.setopt(pycurl.URL, "https://"+server+path)
c.setopt(pycurl.POSTFIELDS,sended_data)
b = StringIO.StringIO()
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.perform()
something like that.
And I testing it now. Thanks all of you for help.

If you are tying to set timeout why don't you use urllib2.

I'm running a python script on my machine only to copy and convert some files from one format to another, I want to maximize the number of running threads to finish as quickly as possible.
Note: It is not a good workaround from an architecture perspective If you aren't using it for a quick script on a specific machine.
In my case, I checked the max number of running threads that my machine can run before I got the error, It was 150
I added this code before starting a new thread. which checks if the max limit of running threads is reached then the app will wait until some of the running threads finish, then it will start new threads
while threading.active_count()>150 :
time.sleep(5)
mythread.start()

If you are using a ThreadPoolExecutor, the problem may be that your max_workers is higher than the threads allowed by your OS.
It seems that the executor keeps the information of the last executed threads in the process table, even if the threads are already done. This means that when your application has been running for a long time, eventually it will register in the process table as many threads as ThreadPoolExecutor.max_workers

As far as I can tell it's not a python problem. Your system somehow cannot create another thread (I had the same problem and couldn't start htop on another cli via ssh).
The answer of Fernando Ulisses dos Santos is really good. I just want to add, that there are other tools limiting the number of processes and memory usage "from the outside". It's pretty common for virtual servers. Starting point is the interface of your vendor or you might have luck finding some information in files like
/proc/user_beancounters