I'm implementing a socket-client which opens several sockets at the same time. Any socket may have data at a different time and I want to execute code when any socket has data and is readable.
I'm not sure how to implement this, I was looking at select.select but it seems to wait for all the sockets to be readable.
I'd like to avoid using multiprocessing to handle data on the sockets, I would like it to be serial in reading from each socket but read when there is data available.
How do I wait for any socket to be readable?
# psudo code
sockets = [sock1, sock2, sock3]
while True:
if len(sockets) == 0:
break
for sock in sockets:
if sock.has_data():
do_stuff(sock)
sockets.remove(sock)
sleep(0.1)
You can use select.select for your problem:
sockets = [sock1, sock2, sock3]
while sockets:
rlist, _, _ = select.select(sockets, [], [])
for sock in rlist:
do_stuff(sock)
sockets.remove(sock)
If you are on POSIX, take a look at select.poll:
import socket
import select
p = select.poll()
s1 = socket.socket()
s2 = socket.socket()
# call connect on sockets here...
p.register(s1, select.POLLIN)
p.register(s2, select.POLLIN)
p.poll()
If you're using Python 3.4 or newer there is the selectors module in the standard library. It will use the "best" I/O multiplexing implementation that your system offers (select, poll, kqueue...) There's a simple echo server example at the end of the documentation page https://docs.python.org/3/library/selectors.html
There's a backport of this for older Python versions available as well.
Related
I have an application which communicates on a specific port, and I would like to listen to all UDP traffic which has this specific port as a source or destination.
Naively I try to do something like:
import socket
UDP_IP = "0.0.0.0"
UDP_PORT = my_port
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_UDP)
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = s.recvfrom(4096)
print("received message:", data)
however this does not work because the application in question is already bound to the port, so I get an error if I try to bind to it in my code.
My next attempt was to use scapy with something like:
from scapy.all import *
import queue
scapy.config.conf.use_pcap = True
pending_pkts = queue.Queue()
def callback(pkt):
pending_pkts.put(pkt)
def worker():
while True:
pkt = pending_pkts.get()
# I do some stuff with the packet here
pending_pkts.task_done()
t = Thread(target=worker)
t.daemon = True
t.start()
sniff(prn=callback, filter="udp and port my_port")
The idea here was that I would stuff the packets into my queue and then do the relatively costly processing on a separate thread. While this does somewhat work, I miss something like 50% of the packets I am interested in, which is unacceptable for the project. I have seen other people running into this issue and try everything suggested (using a very specific filter, using pcap, using multiple threads to avoid costly processing holding things), but evidently this is still not fast enough since I miss so many packets.
I would greatly appreciate it if someone could point me in the right direction as to how I can get ideally 100% packet capture (doesn't need to be realtime, I can accept a couple second delay in processing if it means I get everything) on a UDP port which is already in use. Ideally I would like to stick with python, but I would be willing to also try something in C++ if someone knows of a solution.
Thanks for your time :)
I'm programming python web socket chat server. I made working server using function select() to listen to clients, but when I connect more than 512 clients on Windows or 1024 clients on Linux, my server crashes. After research I found that this is system limit and I need to use poll() or epoll() for more connections.
This is part of code using select() that I need to rewrite using epoll() or poll() function:
from select import select
rList, wList, xList = select(listeners, writers, listeners, interval)
for ready in wList:
function1()
for ready in rList:
function2()
for failed in xList:
function3()
How can I do the same thing using epoll() or poll()? It still needs to call these functions above.
I believe it should be
rList, wList, xList = select.select(listeners, writers, listeners, interval)
you can change it to
poll = select.poll()
poll.register(eachconnection)
poll.poll(5000)
Here is my code. Can somebody explain to me how to make my port scanner faster? I made the port scanner by connecting to the ip with the 'connect.ex()' and maybe should I use another command?
try:
for port in range(1,1000):
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
result = sock.connect_ex((remoteHost,port))
if result == 0:
print(colored("[+] Port {}: Open".format(port), 'green'))
sock.close()
So a quick look on Github led me to find portSpider which bills itself as
A lightning fast multithreaded network scanner framework with modules.
Reading through it, I noticed that it uses socket.connect and not connect_ex. It absolutely does multithreading though to increase performance. I would tend toward using an existing solution rather than building one from scratch, unless this is just for tinkering.
To speed up your example in particular, you could do a simple optimization using multiprocessing.Pool.
def scan_port(port):
try:
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
result = sock.connect_ex((remoteHost,port))
if result == 0:
print(colored("[+] Port {}: Open".format(port), 'green'))
sock.close()
except:
pass # you should handle this error
import multiprocessing as mp
p = mp.Pool() # will parallelize to number of CPUs you have
p.map(scan_port, range(1, 1000))
I'm designing a python program that'll talk to two other process at the same time through sockets. One of the process is a C daemon so this socket will be alive all the time - no problem there. The other process is a PHP web page. So that socket isn't established all the time. Most of the time, the socket is listen()ing on a port.
If both socket are alive all the time, a simple select() call can be used to monitor input from both. But in my situation, this is not possible. How can I achieve this easily?
Thanks,
You can use select() in this case, even in a single-threaded single-process program with only blocking sockets. Here's how you would accept incoming connections with select():
daemonSocket = socket.socket()
...
phpListenSocket = socket.socket()
phpListenSocket.bind(...)
phpListenSocket.listen(...)
phpSocket = None
while True:
rlist = ...
rready, wready, eready = select(rlist, [], [])
if phpListenSocket in rready:
phpSocket, remoteAddr = phpListenSocket.accept()
import socket
backlog = 1 #Number of queues
sk_1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sk_2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
local = {"port":1433}
internet = {"port":9999}
sk_1.bind (('', internet["port"]))
sk_1.listen(backlog)
sk_2.bind (('', local["port"]))
sk_2.listen(backlog)
Basically, I have this code. I am trying to listen on two ports: 1433 and 9999. But, this doesn't seems to work.
How can I listen on two ports, within the same python script??
The fancy-pants way to do this if you want to use Python std-lib would be to use SocketServer with the ThreadingMixin -- although the 'select' suggestion is probably the more efficient.
Even though we only define one ThreadedTCPRequestHandler you can easily repurpose it such that each listener has it's own unique handler and it should be fairly trivial to wrap the server/thread creation into a single method if thats the kind of thing you like.
#!/usr/bin/python
import threading
import time
import SocketServer
class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
print "%s wrote: " % self.client_address[0]
print self.data
self.request.send(self.data.upper())
class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
pass
if __name__ == "__main__":
HOST = ''
PORT_A = 9999
PORT_B = 9876
server_A = ThreadedTCPServer((HOST, PORT_A), ThreadedTCPRequestHandler)
server_B = ThreadedTCPServer((HOST, PORT_B), ThreadedTCPRequestHandler)
server_A_thread = threading.Thread(target=server_A.serve_forever)
server_B_thread = threading.Thread(target=server_B.serve_forever)
server_A_thread.setDaemon(True)
server_B_thread.setDaemon(True)
server_A_thread.start()
server_B_thread.start()
while 1:
time.sleep(1)
The code so far is fine, as far as it goes (except that a backlog of 1 seems unduly strict), the problem of course comes when you try to accept a connection on either listening socket, since accept is normally a blocking call (and "polling" by trying to accept with short timeouts on either socket alternately will burn machine cycles to no good purpose).
select to the rescue!-) select.select (or on the better OSs select.poll or even select.epoll or select.kqueue... but, good old select.select works everywhere!-) will let you know which socket is ready and when, so you can accept appropriately. Along these lines, asyncore and asynchat provide a bit more organization (and third-party framework twisted, of course, adds a lot of such "asynchronous" functionality).
Alternatively, you can devote separate threads to servicing the two listening sockets, but in this case, if the different sockets' functionality needs to affect the same shared data structures, coordination (locking &c) may become ticklish. I would certainly recommend trying the async approach first -- it's actually simpler, as well as offering potential for substantially better performance!-)