I have recently been introduced to the threading module in python so I decided to play around with it I opened a python socket server on port 7000:
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind(('127.0.0.1',7000))
s.listen(1)
c, a = s.accept()
and made my client server try connecting to every port from 1 to 65535 until it establishes connection on port 7000. Obviously this would take very long so I multi-threaded it:
import threading
import socket
import sys
host = None
def conn(port):
try:
s.connect((host,port))
print 'Connected'
sys.exit(1)
except:
pass
global host
host = '127.0.0.1'
for i in range(65535):
t = threading.Thread(target=conn, args=(i,))
t.start()
When the client connects its suppose to return the message 'connected' however when debugging I noticed some very strange behavior with the program. Sometimes the program would return that it connected, other times the program would fail to output that it was connected to the server instead it would just terminate without printing anything.
Its obviously a problem with the threads. As when i make the client connect to port 7000 only it works 100% of the time. However threading it through all 65535 ports causes the client to sometimes not print anything. What is the reason for this and how can I prevent or circumvent it.
Edit:
I realized making it try to connect to a smaller number of ports, ports 1-10 and port 7000, gives it a higher chance of printing out connected.
If connect() fails, consider the state of the socket as unspecified. Portable applications should close the socket and create a new one for reconnecting.
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('127.0.0.1', 6999))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 61] Connection refused
>>>
>>> s.connect(('127.0.0.1', 7000))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 22] Invalid argument
>>>
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('127.0.0.1', 7000))
# Connect success.
65535 is a huge number.
Any performance gain you might get will be dwarfed by this amount of threads. An OS should plan processor time for each of this thread and then it takes time to switch between threads. In the worst case (and 7k is pretty much bad) all OS does is thread switching with little real work in between. 2-8 threads (or just a thread per physical core) would be much more performant.
Also, make sure you wait until your threads exit, and don't silence out errors with except: pass. I bet, there are a lot of interesting things happening there. At least [selectively] log these exceptions somewhere.
Edit. Use join in order to make sure that all spawned threads exit before the main thread.
threads = [threading.Thread(target=conn, args=(i,)) for i in range(8)]
for thread in threads:
thread.start()
# do whatever
for thread in threads:
thread.join()
Related
i am trying to establish connections for multiple sockets using multi-threading
this is the code
import socket as sc
if __name__ == '__main__':
#setup()
ports = [10000, 10010, 10020, 10030]
init_sockets()
init_threads()
def init_sockets():
global host_ip
global sockets
host_ip = sc.gethostname()
sockets = []
for port in ports:
socket = sc.socket()
socket.bind((host_ip, port))
socket.listen()
sockets.append(socket)
def init_threads():
threads = [
threading.Thread(target= init_connection, args= [socket])
for socket in sockets
]
for thread in threads:
thread.start()
def init_connection(socket):
client, address = socket.accept()
while running the code this error appears
ConnectionAbortedError: [Errno 53] Software caused connection abort
the error occurs in thread.start() statement in function init_threads()
i don't know why this is happening, and would really appreciate any help. i am trying to run multiple socket connections in parallel, if it's impossible this way, i am open to recommendations
solved it!
the problem seemed to be that when the main thread (program it self) is terminated, all the objects that it created are deleted, including the socket objects.
so when the secondary threads (threads that the main program started), still running, try to reference these deleted objects the error occurs
the solution for me appeared to be adding an infinite while loop in the main method. preventing the main thread from being terminated
So before anyone says its a duplicate, I have seen multiple questions with that error, but could not notice any of that being the same as my problem.
I am trying to make a small project including a socket over SSL, and when trying to catch if a user is trying to connect with a raw socket and not ssl wrapped socket (which is raising a ConnectionResetError) I get a different error.
My code:
import socket
from classes import ClientThread
import ssl
from time import sleep
server = 'localhost'
port = 12345
threads = []
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.load_cert_chain(certfile="cert.pem", keyfile="cert.pem")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((server, port))
print(f"[*] server started, listening on port {port}")
while True:
s.listen()
with context.wrap_socket(s, server_side=True) as ssock:
try:
conn, addr = ssock.accept()
client = ClientThread(conn=conn, ip=addr[0], port=addr[1])
client.start()
threads.append(client)
print(f'Threads running: {len(threads)}')
except ConnectionResetError:
print(f'Could not establish ssl handshake with a client.')
The error i get is:
Traceback (most recent call last):
File "C:/Users/x/x/server.py", line 17, in <module>
s.listen()
OSError: [WinError 10038] An operation was attempted on something that is not a socket
I tried setting some sleep time after the exception maybe it needed to reset the socket but didnt hlep, tried to play a bit with the placement of the While True, and while resetting the entire socket help, I dont want to reset all my clients thread just because of a client who didnt try to log in with a SSL socket.
I think it has something to do with the wrap_socket because it modified the socket instance passed to it , but couldnt find a way to unwrap.
Thank you in advance!
listen enables a socket to take incoming connection requests (also called a "passive socket") and establishes a backlog of how many of those requests can be pending in the network stack at any given time. accept accepts one of those connections. You call listen once and accept many times.
Pull the listen outside of the while so that is only called once to establish this as a listening socket.
I am creating a communication platform in python (3.4.4) and using the multiprocessing.managers.BaseManager class. I have isolated the problem to the code below.
The intention is to have a ROVManager(role='server') instance running in one process on the main computer and providing read/write capabilities to the system dictionary for multiple ROVManager(role='client') instances running on the same computer and a ROV (remotely operated vehicle) connected to the same network. This way, multiple clients/processes can do different tasks like reading sensor values, moving motors, printing, logging etc, all using the same dictionary. start_reader() below is one of those clients.
Code
from multiprocessing.managers import BaseManager
import multiprocessing as mp
import sys
class ROVManager(BaseManager):
def __init__(self, role, address, port, authkey=b'abc'):
super(ROVManager, self).__init__(address=(address, port),
authkey=authkey)
if role is 'server':
self.system = {'shutdown': False}
self.register('system', callable=lambda: self.system)
server = self.get_server()
server.serve_forever()
elif role is 'client':
self.register('system')
self.connect()
def start_server(server_ip, port_var):
print('starting server')
ROVManager(role='server', address=server_ip, port=port_var)
def start_reader(server_ip, port_var):
print('starting reader')
mgr = ROVManager(role='client', address=server_ip, port=port_var)
i = 0
while not mgr.system().get('shutdown'):
sys.stdout.write('\rTotal while loops: {}'.format(i))
i += 1
if __name__ == '__main__':
server_p = mp.Process(target=start_server, args=('0.0.0.0', 5050))
reader_p = mp.Process(target=start_reader, args=('127.0.0.1', 5050))
server_p.start()
reader_p.start()
while True:
# Check system status, restart processes etc here
pass
Error
This results in the following output and error:
starting server
starting reader
Total while loops: 15151
Process Process - 2:
Traceback(most recent call last):
File "c:\python34\Lib\multiprocessing\process.py", line 254, in _bootstrap
self.run()
File "c:\python34\Lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\git\eduROV\error_test.py", line 29, in start_reader
while not mgr.system().get('shutdown'):
File "c:\python34\Lib\multiprocessing\managers.py", line 640, in temp
token, exp = self._create(typeid, *args, **kwds)
File "c:\python34\Lib\multiprocessing\managers.py", line 532, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "c:\python34\Lib\multiprocessing\connection.py", line 496, in Client
c = SocketClient(address)
File "c:\python34\Lib\multiprocessing\connection.py", line 629, in SocketClient
s.connect(address)
OSError: [WinError 10048] Only one usage of each socket address(protocol / network address / port) is normally permitted
My research
Total while loops are usually in the range 15000-16000. From my understanding it seems like a socket is created and terminated each time mgr.system().get('shutdown') is called. Windows then runs out of available sockets. I can't seem to find a way to set socket.SO_REUSEADDR.
Is there a way of solving this, or isn't Managers made for this kind of communication? Thanks :)
As error suggests Only one usage of each socket address in general , you can/should bind only a single process to a socket ( unless you design your application accordingly, by passing SO_REUSEADDR option while creating socket)
. These lines
server_p = mp.Process(target=start_server, args=('0.0.0.0', 5050))
reader_p = mp.Process(target=start_reader, args=('127.0.0.1', 5050))
creates two processes on same port 5050 & so the error.
You can refer here to learn how to use SO_REUSEADDR & its implications but i am quoting the main part which should get you going
The second socket calls setsockopt with the optname parameter set to
SO_REUSEADDR and the optval parameter set to a boolean value of TRUE
before calling bind on the same port as the original socket. Once the
second socket has successfully bound, the behavior for all sockets
bound to that port is indeterminate. For example, if all of the
sockets on the same port provide TCP service, any incoming TCP
connection requests over the port cannot be guaranteed to be handled
by the correct socket — the behavior is non-deterministic.
I would like to directly access to syslog messages from Python by reading /dev/log.
My (very limited) understanding is that the correct way is to read from there is to bind a datagram socket.
import socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind('/dev/log')
sock.listen(1)
while True:
data, addr = sock.recvfrom(1024)
print(data)
Apparently /dev/log is in use:
Traceback (most recent call last):
File "readlog.py", line 4, in <module>
sock.bind('/dev/log')
OSError: [Errno 98] Address already in use
How should I read /dev/log from Python?
EDIT: per #Barmar's comment - only one process can access /dev/log so that part is clear, the device must be clean before reading from it. sudo lsof /dev/log does not show anything.
A answer in a Java thread around this subject mentioned that syslog should be shut down before. I also tried that, lsof | grep "/dev/log" was empty but I got the error nevertheless.
Isn't it possible to have several processes reading from /dev/log?
There is a socket flag to set, in order to prevent this:
socket.SO_REUSEADDR
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
This flag tells the kernel to reuse a local socket in TIME_WAIT state, without waiting for its natural timeout to expire.
Ref: https://docs.python.org/3/library/socket.html
I write a simple program in python, with asyncore and threading. I want to implement a asynchorous client without blocking anything, like this:
How to handle asyncore within a class in python, without blocking anything?
Here is my code:
import socket, threading, time, asyncore
class Client(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect((host, port))
mysocket = Client("",8888)
onethread = threading.Thread(target=asyncore.loop)
onethread.start()
# time.sleep(5)
mysocket.send("asfas\n")
input("End")
Now a exception will be throwed in send("asfas\n"), because I didn't open any server.
I think the exception in send function will call the handle_error function and won't affect the main program, but most of the time it crashes the whole program, and sometimes it works! And if I uncomment the time.sleep(5), it will only crash the thread. Why does it behave like this? Could I write a program that won't crash the whole program and don't use time.sleep() ? Thanks!
Error message:
Traceback (most recent call last):
File "thread.py", line 13, in <module>
mysocket.send("asfas\n")
File "/usr/lib/python2.7/asyncore.py", line 374, in send
result = self.socket.send(data)
socket.error: [Errno 111] Connection refused
First of all, I would suggest not using the old asyncore module but to look into more
modern and more efficient solutions: gevent, or going along the asyncio module (Python 3.4),
which has been backported somehow to Python 2.
If you want to use asyncore, then you have to know:
be careful when using sockets created in one thread (the main thread, in your case), and dispatched by another thread (managed by "onethread", in your case), sockets cannot be shared like this between threads it is not threadsafe objects by themselves
for the same reason, you can't use the global map created by default in asyncore module, you have to create a map by thread
when connecting to a server, connection may not be immediate you have to wait for it to be connected (hence your "sleep 5"). When using asyncore, "handle_write" is called when
socket is ready to send data.
Here is a newer version of your code, hopefully it fixes those issues:
import socket, threading, time, asyncore
class Client(threading.Thread, asyncore.dispatcher):
def __init__(self, host, port):
threading.Thread.__init__(self)
self.daemon = True
self._thread_sockets = dict()
asyncore.dispatcher.__init__(self, map=self._thread_sockets)
self.host = host
self.port = port
self.output_buffer = []
self.start()
def run(self):
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect((self.host, self.port))
asyncore.loop(map=self._thread_sockets)
def send(self, data):
self.output_buffer.append(data)
def handle_write(self):
all_data = "".join(self.output_buffer)
bytes_sent = self.socket.send(all_data)
remaining_data = all_data[bytes_sent:]
self.output_buffer = [remaining_data]
mysocket = Client("",8888)
mysocket.send("asfas\n")
If you have only 1 socket by thread (i.e a dispatcher's map with size 1), there is no
point using asyncore at all. Just use a normal, blocking socket in your threads. The
benefit of async i/o comes with a lot of sockets.
EDIT: answer has been edited following comments.