Python Multiprocessing: socket error timeout while remote connection to a Manager - python

newcomer and first ever question here.
I am using the multiprocessing module of Python which is currently creating a Manager and a couple (45) processes on my localhost.
My Manager is set up as following:
manager = QueueManager(address=('', 50000), authkey='abracadabra')
manager.get_server().serve_forever()
I want also to create some other client processes remotely on another computer. So, let's say my IP is a.b.c.d, the Manager of the client in the remote computer is set up as following:
manager = QueueManager(address=('a.b.c.d', 50000), authkey='abracadabra')
manager.connect()
(yes, it's copy-pasted from the documentation).
However, I run the server and all 45 processes in localhost are fine, then I run the remote client and I get this:
Traceback (most recent call last):
File "govmap-parallel-crawler-client.py", line 144, in <module>
manager.connect()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 474, in connect
conn = Client(self._address, authkey=self._authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 134, in Client
c = SocketClient(address)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 252, in SocketClient
s.connect(address)
File "<string>", line 1, in connect
socket.error: [Errno 110] Connection timed out
Both computers can ping and ssh each other without problems.
My guess: there is one (or two!) firewall in between making the connection impossible. Is this correct?
If yes: is there a way to use a safe known port in order to avoid the firewall or maybe a more polite solution?
If no: what is happening?
Thanks!!

Use an ssh tunnel for interconnect? E.g on client:
ssh a.b.c.d -L12345:localhost:50000
If the client connects to localhost port 12345, it should be tunnelled to a.b.c.d port 50000.
EDIT: Of course, using an SSH tunnel might not be the best solution in a production environment, but at least it lets you eliminate other issues.

As defined by your snippet the server listens only on localhost (127.0.0.1) and not (a.b.c.d) thus you cannot connect from remote client.
To do so use:
manager = QueueManager(address=('a.b.c.d', 50000), authkey='abracadabra')
manager.get_server().serve_forever()

Related

Python - [Errno 111] Connection refused on client side of the connection

I'm trying to create a chat between client and server written in Python, using SSL protocols with mutual authentication (i.e: server authenticates client and client authenticates server using certificates). My host machine is being used as the server, and my laptop is the client.
When attempting to connect to my host ip, I keep getting this error on my laptop:
Traceback (most recent call last):
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 88, in <module>
main()
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 75, in main
connection(ip, port, SSLSock)
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 35, in connection
sock.connect((ip, port))
File "/usr/lib/python3.10/ssl.py", line 1375, in connect
self._real_connect(addr, False)
File "/usr/lib/python3.10/ssl.py", line 1362, in _real_connect
super().connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused
And in the server - which was supposed to print a message saying that a connection was refused - nothing happens, it keeps listening for connections as if nothing happened
Connection function on client side:
def connection(ip, port, sock):
try:
sock.connect((ip, port))
print(f"Connected with {ip}")
except Exception as e:
print("Connection failed: ", e)
sock.close()
Server side:
def acceptConnection(self):
while True:
con, senderIP = self.sock.accept()
# Attempting to wrap connection with SSL socket
try:
SSLSock = self.getSSLSocket(con)
# If exception occurs, close socket and continue listening
except Exception as e:
print("Connection refused: ", e)
con.close()
continue
print(f"{senderIP} connected to the server")
# Adding connection to clients list
self.clients.append(SSLSock)
# Initializing thread to receive and communicate messages
# to all clients
threading.Thread(target=self.clientCommunication, args=(SSLSock, ), daemon=True).start()
This is the main function on my server:
def main():
serverIP = "127.0.0.1"
port = int(input("Port to listen for connections: "))
server = Server()
server.bindSocket(serverIP, port)
server.socketListen(2)
server.acceptConnection()
Everything works fine when I connect from my localhost (e.g I open a server on my host machine on one terminal and use another one on the same machine to connect to it). Both machines have the required certificates to authenticate each other, so I don't think that's the problem. Also, without the SSL implementation, the connection between this two different computers was refused by the server
I've tried using sock.bind('', port) on server side, disabling my firewall, used telnet 127.0.0.1 54321 (on my host machine) to check if the connection was working on the specified port (and it is), and also on the client machine (which showed that the connection was refused). I also tried running both scripts with admin privileges (sudo), but it also didn't work. Any suggestions?
I found what was wrong: I was trying to connect to my public IP address (which I found by searching for "What is my ip" on Google), but instead what should be done is to connect to the private IP address (I guess that's the correct name), and you can see yours using ifconfig on Linux and Mac and ipconfig on Windows using a terminal. By doing this, I could connect two computers that are on my network to my desktop server, I still haven't tested for computers in different networks, but the problem has been solved.

Thrift TTransportException in python

I'm getting a weird error while trying to execute an RPC using thrift on python. I have found similar issues online, but none of them really apply to my situation.
Here is the error I'm getting
No handlers could be found for logger "thrift.transport.TSocket"
Traceback (most recent call last):
File "experiment.py", line 71, in <module>
transport.open()
File "/usr/local/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 152, in open
return self.__trans.open()
File "/usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.py", line 113, in open
raise TTransportException(TTransportException.NOT_OPEN, msg)
thrift.transport.TTransport.TTransportException: Could not connect to any of [('192.168.178.44', 9000)]
The following is, I believe, the code which produces it.
TECS_SERVER_IP = "192.168.178.44"
TECS_SERVER_PORT = 9000
transport = TSocket.TSocket(TECS_SERVER_IP, TECS_SERVER_PORT)
transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = TTSService.Client(protocol)
transport.open()
This happens whenever I try to communicate to another machine, so I tried with the ip "127.0.0.1" and it works. However, using the IP of the localhost "192.168.178.44" which should refer to the same computer also produces the error.
To me it seems like it cannot resolve IP addresses for some reason...
Any ideas on what's causing this and how to fix it?
I'm using Python 2.7.12, thrift 0.9.3 and Ubuntu 16.04, but I also got the error on Windows 10.
This is how my thrift service starts
handler = TTSHandler()
handler.__init__()
transport = TSocket.TServerSocket(host='localhost', port=9000)
processor = TTSService.Processor(handler)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()
server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)
server.serve()
your server should bind to that address before client could connect to:
TSocket.TServerSocket(host='192.168.178.44', port=9000)
or use host='0.0.0.0' which means bind on all IPv4 addresses on the machine.

Why does my python's socket.shutdown work on Windows but not Ubuntu?

Here is what my socket code looks like, this is for a UDP connection.
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.settimeout(8)
sock.sendto(req, (host, port))
buf = sock.recv(2048)
sock.shutdown(socket.SHUT_RDWR)
sock.close()
Here is the relevant portion of my stack trace
Exception in thread Thread-6:
Traceback (most recent call last):
File "udp_test.py", line 110, in my_method
sock.shutdown(socket.SHUT_RDWR)
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 107] Transport endpoint is not connected
I don't know what OS the host is running, I assume it is some flavor of Linux. I can wrap the socket.shutdown[docs] call in a try catch and everything seems to work fine.
Does this problem have something to do with a difference between the way Windows and Linux handle sockets? Is wrapping sock.shutdown in a try catch the solution here or will I run nasty problems down the rode?
You are calling sock.shutdown() on a UDP socket. UDP doesn't have a connection to shut down. On Windows the call doesn't do much other than prevent you from writing to and reading from the socket (packets are still received and queued), on Linux calling shutdown on a UDP connection throws an error.
In either case, you shouldn't really be using shutdown at all. Just close the socket instead, or just don't send on the socket and don't read data from it.

riak / python: unable to run a store() command

Attempting to follow the tutorial here.
After doing basic CRUD stuff, the tutorial has you edit the app.config file to use the LevelDB Backend for 2i, which in my case meant updating line 83 of /usr/local/Cellar/riak/1.4.2/etc/app.config from
{storage_backend, riak_kv_bitcask_backend},
to
{storage_backend, riak_kv_eleveldb_backend},
and then restarting riak.
I had thus far been running riak on port 8098, and their tutorial here references port 10017 instead:
# Starting Client
client = riak.RiakClient(pb_port=10017, protocol='pbc')
# Creating Buckets
customer_bucket = client.bucket('Customers')
order_bucket = client.bucket('Orders')
order_summary_bucket = client.bucket('OrderSummaries')
# Storing Data
cr = customer_bucket.new(str(customer['customer_id']),
data=customer)
cr.store()
if I try to run the code as written, I get this:
File "r.py", line 104, in <module>
cr.store()
File ".../riak/riak-python-client-master/riak/riak_object.py", line 281, in store
timeout=timeout)
File ".../riak/client/transport.py", line 127, in wrapper
return self._with_retries(pool, thunk)
File ".../riak/client/transport.py", line 82, in _with_retries
raise e.args[0]
socket.error: [Errno 61] Connection refused
suggesting that, clearly, the Riak client is not running on port 10017.
However, when I change it to port 8098
client = riak.RiakClient(pb_port=8098, protocol='pbc')
the application just freezes on the cr.store() line. Is there some thing where the eleveldb backend is expecting to run on a port other than default?
never mind, 8098 is the http port, when I changed it to port 8087, the default port, it works fine

Can't connect to local MySQL server through socket '/opt/local/var/run/mysql55/mysqld.sock' (2) [duplicate]

I'm having trouble with the MySQLdb module.
db = MySQLdb.connect(
host = 'localhost',
user = 'root',
passwd = '',
db = 'testdb',
port = 3000)
(I'm using a custom port)
the error I get is:
Error 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Which doesn't make much sense since that's the default connection set in my.conf.. it's as though it's ignoring the connection info I give..
The mysql server is definitely there:
[root#baster ~]# mysql -uroot -p -P3000
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 19
Server version: 5.0.77 Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use testdb;
Database changed
mysql>
I tried directly from the python prompt:
>>> db = MySQLdb.connect(user='root', passwd='', port=3000, host='localhost', db='pyneoform')
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib64/python2.5/site-packages/MySQLdb/__init__.py", line 74, in Connect
return Connection(*args, **kwargs)
File "/usr/lib64/python2.5/site-packages/MySQLdb/connections.py", line 169, in __init__
super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)")
>>>
I'm confused... :(
Changing localhost to 127.0.0.1 solved my problem using MySQLdb:
db = MySQLdb.connect(
host = '127.0.0.1',
user = 'root',
passwd = '',
db = 'testdb',
port = 3000)
Using 127.0.0.1 forces the client to use TCP/IP, so that the server listening to the TCP port can pickle it up. If host is specified as localhost, a Unix socket or pipe will be used.
add unix_socket='path_to_socket' where path_to_socket should be the path of the MySQL socket, e.g. /var/run/mysqld/mysqld2.sock
Make sure that the mysql server is listening for tcp connections, which you can do with netstat -nlp (in *nix). This is the type of connection you are attempting to make, and db's normally don't listen on the network by default for security reasons. Also, try specifying --host=localhost when using the mysql command, this also try to connect via unix sockets unless you specify otherwise. If mysql is not configured to listen for tcp connections, the command will also fail.
Here's a relevant section from the mysql 5.1 manual on unix sockets and troubleshooting connections. Note that the error described (2002) is the same one that you are getting.
Alternatively, check to see if the module you are using has an option to connect via unix sockets (as David Suggests).
I had this issue where the unix socket file was some place else, python was trying to connect to a non-existing socket. Once this was corrected using the unix_socket option, it worked.
Mysql uses sockets when the host is 'localhost' and tcp/ip when the host is anything else. By default Mysql will listen to both - you can disable either sockets or networking in you my.cnf file (see mysql.com for details).
In your case forget about the port=3000 the mysql client lib is not paying any attention to it since you are using localhost and specify the socket as in unix_socket='path_to_socket'.
If you decided to move this script to another machine you will need to change this connect string to use the actual host name or ip address and then you can loose the unix_socket and bring back the port. The default port for mysql is 3306 - you don't need to specify that port but you will need to specify 3000 if that is the port you are using.
As far as I can tell, the python connector can ONLY connect to mysql through a internet socket: unix sockets (the default for the command line client) is not supported.
In the CLI client, when you say "-h localhost", it actually interprets localhost as "Oh, localhost? I'll just connect to the unix socket instead", rather than the internet localhost socket.
Ie, the mysql CLI client is doing something magical, and the Python connector is doing something "consistent, but restrictive".
Choose your poison. (Pun not intended ;) )
Maybe try adding the keyword parameter unix_socket = None to connect()?

Categories