i have a rabbit mq running on machine
both client and rabbitMQ are running on the same network
rabbitMQ has many clients
i can ping client from rabbitMQ and back
longest latency measured between the machine is 12.1 ms
network details : Standard Switch network (network of virtual machines running on single physical machine - using vmware VC)
im getting random timeouts when initializing RPC connection
/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py
problem is that the timeout isn't consistent and happens from time to time.
when manually testing this issue and running the blocking_connection.py 1000 times from the same machine that it fails no timeout accrue.
this is the error i get when failing :
2013-04-23 08:24:23,396 runtest-trigger.24397 24397 DEBUG producer_rabbit initiate_rpc_connection Connecting to RabbitMQ RPC queue rpcqueue_java on host: auto-db1
2013-04-23 08:24:25,350 runtest-trigger.24397 24397 ERROR testrunner go Run 1354: cought exception: timed out
Traceback (most recent call last):
File "/testrunner.py", line 193, in go
self.set_runparams(jobid)
File "/testrunner.py", line 483, in set_runparams
self.runparams.producers_testrun = self.initialize_producers_testrun(self.runparams)
File "/basehandler.py", line 114, in initialize_producers_testrun
producer.set_testcase_checkout()
File "/baseproducer.py", line 73, in set_testcase_checkout
self.checkout_handler = pm_checkout.get_producer(self.testcasecheckout)
File "/producer_manager.py", line 101, in get_producer
producer = self.load_producer(plugin_dir, producer_name)
File "/producer_manager.py", line 20, in load_producer
producer = getattr(producer_module, 'Producer')(producer_name, self.runparams)
File "/producer_rabbit.py", line 13, in __init__
self.initiate_rpc_connection()
File "/producer_rabbit.py", line 67, in initiate_rpc_connection
self.connection = pika.BlockingConnection(pika.ConnectionParameters( host=self.conf.rpc_proxy))
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py", line 32, in __init__
BaseConnection.__init__(self, parameters, None, reconnection_strategy)
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py", line 50, in __init__
reconnection_strategy)
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/connection.py", line 170, in __init__
self._connect()
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/connection.py", line 228, in _connect
self.parameters.port or spec.PORT)
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py", line 44, in _adapter_connect
self._handle_read()
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py", line 151, in _handle_read
data = self.socket.recv(self._suggested_buffer_size)
timeout: timed out
please assist
I had a similar issue. If everything looks fine, then you most likely have some sort of miss configuration, e.g. bad binding. If miss configured, then you'll get a timeout because the script can't reach where it thinks it needs to go, so the error can be miss leading in this case.
For my problem, I specifically had issues with both my rabbitmq.config file and my bindings and had to use my python solution shown in: RabbitMQ creating queues and bindings from a command line over the command line example I showed. Once updated and configured properly, everything worked fine. Hopefully this gets you in the right direction.
Pika provides some time out issue when connecting different hosts.Solution is to pass a socket_timeout argument in connection parameter.Pika should upgrade to >=0.9.14
credentials = pika.PlainCredentials(RABBITMQ_USER, RABBITMQ_PASS)
connection = pika.BlockingConnection(pika.ConnectionParameters(
credentials=credentials,
host=RABBITMQ_HOST,
socket_timeout=300))
channel = connection.channel()
Related
My web app handles http requests using several workers. Workers use some shared state, which their actions sometimes update. So far I have used a separate script/process for initializing and managing that shared state, using multiprocessing.managers.BaseManager.
This means that state_manager script would start BaseManager server, to which workers would connect as they go up and send their requests. This used to work fine on heroku. Now I'm migrating to another hosting (virtual), which for some reasons disallows TCP sockets and state_manager cannot start its (localhost) server. For context, I'm getting:
[Errno 1] Operation not permitted
As mentioned, this server is purely for local interprocess communication.
What are my alternatives for this shared state and worker management?
P.S.
On Heroku I used gunicorn, new hosting uses Passenger, if this somehow helps.
EDIT: Error message
Traceback (most recent call last):
File "state_manager.py", line 207, in <module>
server = manager.get_server()
File "/usr/lib/python3.6/multiprocessing/managers.py", line 482, in get_server
self._authkey, self._serializer)
File "/usr/lib/python3.6/multiprocessing/managers.py", line 139, in __init__
self.listener = Listener(address=address, backlog=16)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 438, in __init__
self._listener = SocketListener(address, family, backlog)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 577, in __init__
self._socket.listen(backlog)
PermissionError: [Errno 1] Operation not permitted
I have two nearly identical devices. spur will connect via ssh with one, but not the other. How do I figure out why?
>>> shell1 = spur.SshShell('10.201.140.242', 'username', 'password', missing_host_key=spur.ssh.MissingHostKey.accept)
>>> results = shell1.run(['ls', '-a'])
>>> results.output
'.\n..\n.aptitude\n.bashrc\n.cache\n.config\n.profile\n'
>>> shell2 = spur.SshShell('10.201.129.56', 'username', 'password', missing_host_key=spur.ssh.MissingHostKey.accept)
>>> results = shell2.run(['ls', '-a'])
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "E:\development\virtenv\lib\site-packages\spur\ssh.py", line 166, in run
return self.spawn(*args, **kwargs).wait_for_result()
File "E:\development\virtenv\lib\site-packages\spur\ssh.py", line 178, in spawn
channel = self._get_ssh_transport().open_session()
File "E:\development\virtenv\lib\site-packages\spur\ssh.py", line 268, in _get_ssh_transport
raise self._connection_error(error)
ConnectionError: Error creating SSH connection
Original error: ('10.201.129.56', <paramiko.ecdsakey.ECDSAKey object at 0x11328070>, <paramiko.ecdsakey.ECDSAKey object at 0x1135F350>)
I'm confused by the error message. What does returning the ip, and two key objects supposed to mean? Is there helpful info here I'm supposed to glean from that?
Both devices will accept ssh connections from the command line, so that sets aside an obvious problem.
Both are running the same version of Ubuntu, with the same login credentials. Home directories are even the same (no .ssh dir). Even further, both of their sshd_config files are identical (so, also using the same version among other config options).
The problem doesn't seem to be in the ssh settings, but the error gives no indication of where the problem could be!
Any ideas?
Enabling logging doesn't add much.
shell1:
11:32:11|[ INFO] - paramiko.transport - _log - Connected (version 2.0, client OpenSSH_5.9p1)
11:32:11|[ INFO] - paramiko.transport - _log - Authentication (password) successful!
shell2:
11:32:25|[ INFO] - paramiko.transport - _log - Connected (version 2.0, client OpenSSH_5.9p1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "E:\development\virtenv\lib\site-packages\spur\ssh.py", line 166, in run
return self.spawn(*args, **kwargs).wait_for_result()
File "E:\development\virtenv\lib\site-packages\spur\ssh.py", line 178, in spawn
File "E:\development\virtenv\lib\site-packages\spur\ssh.py", line 268, in _get_ssh_transport
raise self._connection_error(error)
ConnectionError: Error creating SSH connection
Original error: ('10.201.129.56', <paramiko.ecdsakey.ECDSAKey object at 0x1132EF10>, <paramiko.ecdsakey.ECDSAKey object at 0x11366DF0>)
It might be telling that the SSH reports connection. The failure occurs before/during authentication. But as I said above, the passwords are the same -- both connections even use the same copy-pasted pw, which works without error on a command line connection.
i have an issue.
For my final year project of uni i have my raspberry Pi taking sensor data, i need to put that data into my rabbitMQ service on another machine.
Currently i am using pika as a library to push the data into the queue.
Right now i have an issue with connection, this is the error i am getting.
WARNING:pika.adapters.base_connection:Could not connect due to "Connection timed out," retrying in 10 sec
ERROR:pika.adapters.base_connection:Could not connect: Connection timed out
Traceback (most recent call last):
File "TryOne.py", line 8, in <module>
connection = pika.AsyncoreConnection(pika.ConnectionParameters('192.168.1.93'))
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 63, in __init__
super(BaseConnection, self).__init__(parameters, on_open_callback)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 513, in __init__
self._connect()
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 804, in _connect
self._adapter_connect()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/asyncore_connection.py", line 96, in _adapter_connect
super(AsyncoreConnection, self)._adapter_connect()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 122, in _adapter_connect
self.params.retry_delay)
pika.exceptions.AMQPConnectionError: 10.0
Any help with this problem is very much appreciated.
I'm seeing a strange but consistent behaviour from the Python Riak Client when connecting to my riak AWS cluster using protocol buffers. This short python snippet produces the error:
import time
import riak
client = riak.RiakClient(
host='address_to_my_cluster_goes_here',
http_port=8098,
pb_port=8087,
protocol='pbc'
)
result = client.ping()
# Do something else for a while, > 60 seconds
time.sleep(61)
result = client.ping()
The last ping always throws an exception, with the following traceback:
Traceback (most recent call last):
File "main_causing_exception.py", line 16, in <module>
result = client.ping()
File "C:\temp\venv\lib\site-packages\riak\client\transport.py", line 127, in wrapper
return self._with_retries(pool, thunk)
File "C:\temp\venv\lib\site-packages\riak\client\transport.py", line 69, in _with_retries
return fn(transport)
File "C:\temp\venv\lib\site-packages\riak\client\transport.py", line 125, in thunk
return fn(self, transport, *args, **kwargs)
File "C:\temp\venv\lib\site-packages\riak\client\operations.py", line 92, in ping
return transport.ping()
File "C:\temp\venv\lib\site-packages\riak\transports\pbc\transport.py", line 95, in ping
msg_code, msg = self._request(MSG_CODE_PING_REQ)
File "C:\temp\venv\lib\site-packages\riak\transports\pbc\connection.py", line 43, in _request
return self._recv_msg(expect)
File "C:\temp\venv\lib\site-packages\riak\transports\pbc\connection.py", line 50, in _recv_msg
self._recv_pkt()
File "C:\temp\venv\lib\site-packages\riak\transports\pbc\connection.py", line 71, in _recv_pkt
% len(nmsglen))
riak.RiakError: 'Socket returned short packet length 0 - expected 4'
If I do the client.ping() every 30 seconds or so, the error doesn't happen, indicating that it's some kind of socket keep-alive problem I'm seeing, but this does't seem like a solution robust enough for a production environment.
The error only occurs when using the pbc protocol setting and I've never seen it when using a http configured Riak Python Client.
I'm using Python 2.7.5 on a Win7-64 platform (though the error also occurs on our Ubuntu development server) in a virtual environment with the following packages and versions:
protobuf (2.4.1)
riak (2.0.1)
riak-pb (1.4.1.1)
Any thoughts on what's going on and how to resolve it? Am I using the Python Riak Client in the wrong way?
You should use a newer version of Riak library.
There was an error with send function in PBC transport, that was resolved here: https://github.com/basho/riak-python-client/issues/381
I am trying to run a simple Python based web server given here.
And I get the following error message:
Traceback (most recent call last):
File "webserver.py", line 63, in <module>
main()
File "webserver.py", line 55, in main
server = HTTPServer(('', 80), MyHandler)
File "/usr/lib/python2.5/SocketServer.py", line 330, in __init__
self.server_bind()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 101, in server_bind
SocketServer.TCPServer.server_bind(self)
File "/usr/lib/python2.5/SocketServer.py", line 341, in server_bind
self.socket.bind(self.server_address)
File "<string>", line 1, in bind
socket.error: (13, 'Permission denied')
As far as I understand my firewall blocks access to a socket? Am I right? If it is the case, how can I change the permissions? Is it dangerous to change these permissions?
If you want to bind to port numbers < 1024, you need to be root. It's not a firewall
issue; it's enforced by the operating system. Here's a reference from w3.org,
and a FAQ entry specific to Unix.
If you want to run on a port under 1024, you'll need to be root. You can open the socket and drop root's permission for the rest of your program by switching to another user.
Most times it's easier to run a real webserver (say nginx) on port 80 and proxy the requests through to your program which you can run on a high numbered port (8080 for example). This way you don't need to worry about screwing something up during the time your process is running as root, as it never runs as root.
If it's just for testing, run the server on port 8080 and connect at http://localhost:8080/