I'm writing a python UI in kivy to manage some remote machines with fabric. As I can't use fabric's parallel implementation on Windows 10 (see here), I was hoping to use parallel-ssh to actually perform the parallel remote operations. This issue seems to be caused by the interactions between the libraries, rather than an issue with any single one of them.
I've tried manually loading my private key as suggested here:
from fabric.api import execute
import pssh
from pssh.utils import load_private_key
hosts = ['192.168.0.2']
private_key = load_private_key('C:/Users/democracy/.ssh/id_rsa')
pssh_client = pssh.ParallelSSHClient(hosts, user='XXX', password='YYY', pkey=private_key)
output = pssh_client.run_command('whoami', sudo=True)
pssh_client.join(output)
for host in output:
for line in output[host]['stdout']:
print("Host %s - output: %s" % (host, line))
The above code results in the following backtrace:
Exception: Error reading SSH protocol banner('This operation would block forever', <Hub at 0x242566ab9c8 select pending=0 ref=0>)
Traceback (most recent call last):
File "C:\environments\democracy\lib\site-packages\paramiko\transport.py", line 1884, in _check_banner
buf = self.packetizer.readline(timeout)
File "C:\environments\democracy\lib\site-packages\paramiko\packet.py", line 331, in readline
buf += self._read_timeout(timeout)
File "C:\environments\democracy\lib\site-packages\paramiko\packet.py", line 485, in _read_timeout
x = self.__socket.recv(128)
File "C:\environments\democracy\lib\site-packages\gevent\_socket3.py", line 317, in recv
self._wait(self._read_event)
File "C:\environments\democracy\lib\site-packages\gevent\_socket3.py", line 144, in _wait
self.hub.wait(watcher)
File "C:\environments\democracy\lib\site-packages\gevent\hub.py", line 630, in wait
result = waiter.get()
File "C:\environments\democracy\lib\site-packages\gevent\hub.py", line 878, in get
return self.hub.switch()
File "C:\environments\democracy\lib\site-packages\gevent\hub.py", line 609, in switch
return greenlet.switch(self)
gevent.hub.LoopExit: ('This operation would block forever', <Hub at 0x242566ab9c8 select pending=0 ref=0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\environments\democracy\lib\site-packages\paramiko\transport.py", line 1740, in run
self._check_banner()
File "C:\environments\democracy\lib\site-packages\paramiko\transport.py", line 1888, in _check_banner
raise SSHException('Error reading SSH protocol banner' + str(e))
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner('This operation would block forever', <Hub at 0x242566ab9c8 select pending=0 ref=0>)
General SSH error - Error reading SSH protocol banner('This operation would block forever', <Hub at 0x242566ab9c8 select pending=0 ref=0>)
The above code works if I import pssh before fabric. Unfortunately, it seems if I do this, any buttons on my kivy interface (which kick off any operations in a background thread) block forever on press. If I go to the console after a button press and send a keyboard interrupt, kivy stops blocking and begins cleanup, but executes the command from the button press before exiting. The stacktrace on sending this interrupt is below:
[INFO ] [Base ] Leaving application in progress...
Traceback (most recent call last):
File "machine_control_ui.py", line 7, in <module>
DemocracyControllerApp().run()
File "C:\environments\democracy\lib\site-packages\kivy\app.py", line 828, in run
runTouchApp()
File "C:\environments\democracy\lib\site-packages\kivy\base.py", line 504, in runTouchApp
EventLoop.window.mainloop()
File "C:\environments\democracy\lib\site-packages\kivy\core\window\window_sdl2.py", line 659, in mainloop
self._mainloop()
File "C:\environments\democracy\lib\site-packages\kivy\core\window\window_sdl2.py", line 405, in _mainloop
EventLoop.idle()
File "C:\environments\democracy\lib\site-packages\kivy\base.py", line 339, in idle
Clock.tick()
File "C:\environments\democracy\lib\site-packages\kivy\clock.py", line 553, in tick
current = self.idle()
File "C:\environments\democracy\lib\site-packages\kivy\clock.py", line 533, in idle
usleep(1000000 * sleeptime)
File "C:\environments\democracy\lib\site-packages\kivy\clock.py", line 717, in usleep
_usleep(microseconds, self._sleep_obj)
File "C:\environments\democracy\lib\site-packages\kivy\clock.py", line 395, in _usleep
_kernel32.WaitForSingleObject(obj, 0xffffffff)
KeyboardInterrupt
*** BUTTON PRESS OPERATION OUTPUTS HERE ***
```
Any insight into why this might be happening and how I can avoid it would be much appreciated. I could potentially investigate other parallel ssh solutions (although I imagine anything using paramiko would have the same issue), or manually kick off a thread per host to achieve the parallel operation otherwise (which probably has its own list of headaches), but I'd prefer to just use the parallel-ssh library if there's a workable solution.
I'm using parallel-ssh 0.92.2 on Python 3 and Windows 10.
From docs -
parallel-ssh uses gevent’s monkey patching to enable asynchronous use
of the Python standard library’s network I/O.
Make sure that ParallelSSH imports come before any other imports in
your code. Otherwise, patching may not be done before the standard
library is loaded which will then cause ParallelSSH to block.
If you are seeing messages like This operation would block forever,
this is the cause.
Monkey patching is only done for the clients under pssh.pssh_client
and pssh.ssh_client for parallel and single host clients respectively.
New native library based clients under pssh.pssh2_client and
pssh.ssh2_client do not perform monkey patching and are an option if
monkey patching is not suitable. These clients will become the default
in a future major release - 2.0.0.
Since monkey patching is used for the client you are using, other uses of the threading, socket etc modules in your application will also have been patched to use gevent which means they no longer run in a native thread but in a co-routine/greenlet.
This is the reason your background thread operations block as they run in a greenlet on the same thread rather than a new thread.
As of 1.2.0, a new client based on libssh2 instead of paramiko is available which does not use monkey patching:
from pssh.pssh2_client import ParallelSSHClient
<..>
Rest of your application can then use the standard library as-is.
Related
My web app handles http requests using several workers. Workers use some shared state, which their actions sometimes update. So far I have used a separate script/process for initializing and managing that shared state, using multiprocessing.managers.BaseManager.
This means that state_manager script would start BaseManager server, to which workers would connect as they go up and send their requests. This used to work fine on heroku. Now I'm migrating to another hosting (virtual), which for some reasons disallows TCP sockets and state_manager cannot start its (localhost) server. For context, I'm getting:
[Errno 1] Operation not permitted
As mentioned, this server is purely for local interprocess communication.
What are my alternatives for this shared state and worker management?
P.S.
On Heroku I used gunicorn, new hosting uses Passenger, if this somehow helps.
EDIT: Error message
Traceback (most recent call last):
File "state_manager.py", line 207, in <module>
server = manager.get_server()
File "/usr/lib/python3.6/multiprocessing/managers.py", line 482, in get_server
self._authkey, self._serializer)
File "/usr/lib/python3.6/multiprocessing/managers.py", line 139, in __init__
self.listener = Listener(address=address, backlog=16)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 438, in __init__
self._listener = SocketListener(address, family, backlog)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 577, in __init__
self._socket.listen(backlog)
PermissionError: [Errno 1] Operation not permitted
Need some help! While running the python script using Rabbit MQ RPC. I am getting a Socket 104,Socket closed when connection was open error. Below is python traceback and some code:
Traceback (most recent call last):
File "./server.py", line 34, in <module>
channel.start_consuming()
File "/usr/lib/python2.6/site-packages/pika/adapters/blocking_connection.py", line 1681, in start_consuming
self.connection.process_data_events(time_limit=None)
File "/usr/lib/python2.6/site-packages/pika/adapters/blocking_connection.py", line 656, in process_data_events
self._dispatch_channel_events()
File "/usr/lib/python2.6/site-packages/pika/adapters/blocking_connection.py", line 469, in _dispatch_channel_events
impl_channel._get_cookie()._dispatch_events()
File "/usr/lib/python2.6/site-packages/pika/adapters/blocking_connection.py", line 1310, in _dispatch_events
evt.body)
File "./server.py", line 30, in on_request
body=json.dumps(DEVICE_INFO))
File "/usr/lib/python2.6/site-packages/pika/adapters/blocking_connection.py", line 1978, in basic_publish
mandatory, immediate)
File "/usr/lib/python2.6/site-packages/pika/adapters/blocking_connection.py", line 2065, in publish
self._flush_output()
File "/usr/lib/python2.6/site-packages/pika/adapters/blocking_connection.py", line 1174, in _flush_output
*waiters)
File "/usr/lib/python2.6/site-packages/pika/adapters/blocking_connection.py", line 395, in _flush_output
raise exceptions.ConnectionClosed()
pika.exceptions.ConnectionClosed
Apologies as i am unable to comment due to low reputation. Could you provide a little more information on how you are opening your connection. Is it really open?
It might be because of loss of connection with rabbitmq server as pika doesn't deal with disconnects and often results in similar stacktrace.
I also had similar problem, in my case it was because my pika connection was dropping after sometime and my colleague was able to deal with this by adding a wait time for mq:port_number.
We were using docker container so we added following line to our invoke.sh to wait for mq:
filename.py --wait-secs 30 --port-wait mq:5672
I hope you are able to resolve this after doing that.
Otherwise it would be better to check if the connection is being dropped by pika before your python script runs or providing more information on how you are invoking it.
I have a Django app being served with nginx+gunicorn with 3 gunicorn worker processes. Occasionally (maybe once every 100 requests or so) one of the worker processes gets into a state where it starts failing most (but not all) requests that it serves, and then it throws an exception when it tries to email me about it. The gunicorn error logs look like this:
[2015-04-29 10:41:39 +0000] [20833] [ERROR] Error handling request
Traceback (most recent call last):
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 171, in handle_request
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 206, in __call__
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 196, in get_response
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 226, in handle_uncaught_exception
File "/usr/lib/python2.7/logging/__init__.py", line 1178, in error
File "/usr/lib/python2.7/logging/__init__.py", line 1271, in _log
File "/usr/lib/python2.7/logging/__init__.py", line 1281, in handle
File "/usr/lib/python2.7/logging/__init__.py", line 1321, in callHandlers
File "/usr/lib/python2.7/logging/__init__.py", line 749, in handle
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/log.py", line 122, in emit
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/log.py", line 125, in connection
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/core/mail/__init__.py", line 29, in get_connection
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 26, in import_by_path
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 21, in import_by_path
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
ImproperlyConfigured: Error importing module django.core.mail.backends.smtp: "No module named smtp"
So some uncaught exception is happening and then Django is trying to email me about it. The fact that it can't import django.core.mail.backends.smtp doesn't make sense because django.core.mail.backends.smtp should definitely be on the worker process' Python path. I can import it just fine from a manage.py shell and I do get emails for other server errors (actual software bugs) so I know that works. It's like the the worker process' environment is corrupted somehow.
Once a worker process enters this state it has a really hard time recovering; almost every request it serves ends up failing in this same manner. If I restart gunicorn everything is good (until another worker process falls into this weird state again).
I don't notice any obvious patterns so I don't think this is being triggered by a bug in my app (the URLs error'ing out are different, etc). It seems like some sort of race condition.
Currently I'm using gunicorn's --max-requests option to mitigate this problem but I'd like to understand what's going on here. Is this a race condition? How can I debug this?
I suggest you use Sentry which gives a smart way of handling errors.
You can use it as a cloud based solution (getsentry) or you can install it on your own server (github).
Before I was using django core log mailer now I always use sentry.
I do not work at Sentry but their solution is pretty awesome !
We discovered one particular view that was pegging the CPU for a few seconds every time it was loaded that seemed to be triggering this issue. I still don't understand how slamming a gunicorn worker could result in a corrupted execution environment, but fixing the high-CPU view seems to have gotten rid of this issue.
I have a python bottle application, which uses threads. due to the fact I'm using monkey.patch, the threads were blocking app execution (a dialog box fired from a thread was blocking bottle routes from responding to the client, until dismissed.)
A little research here showed I should use monkey patch without trying to patch Thread:
# Patch python's threads with greenlets
from gevent import monkey
monkey.patch_all(thread=False)
This does not block on a minimal example I wrote.
But raises these errors on intensive use with Threads, with methods like threading.setEvent()
This is the error I get:
C:\Users\IEUser\downloadloft-localserver>python mainserver.py
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Program Files\DownloadLoft\Python27\lib\threading.py", line 551, in _
_bootstrap_inner
self.run()
File "C:\Program Files\DownloadLoft\Python27\lib\threading.py", line 753, in r
un
self.finished.wait(self.interval)
File "C:\Program Files\DownloadLoft\Python27\lib\threading.py", line 403, in w
ait
self.__cond.wait(timeout)
File "C:\Program Files\DownloadLoft\Python27\lib\threading.py", line 262, in w
ait
_sleep(delay)
File "c:\users\admini~1\appdata\local\temp\easy_install-sch3hm\gevent-0.13.8-p
y2.7-win32.egg.tmp\gevent\hub.py", line 79, in sleep
switch_result = get_hub().switch()
File "c:\users\admini~1\appdata\local\temp\easy_install-sch3hm\gevent-0.13.8-p
y2.7-win32.egg.tmp\gevent\hub.py", line 135, in get_hub
raise NotImplementedError('gevent is only usable from a single thread')
NotImplementedError: gevent is only usable from a single thread
Bottle v0.12-dev server starting up (using GeventSocketIOServer())...
Listening on http://localhost:8080/
Hit Ctrl-C to quit.
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Program Files\DownloadLoft\Python27\lib\threading.py", line 551, in _
_bootstrap_inner
self.run()
File "C:\Program Files\DownloadLoft\Python27\lib\threading.py", line 753, in r
un
self.finished.wait(self.interval)
File "C:\Program Files\DownloadLoft\Python27\lib\threading.py", line 403, in w
ait
self.__cond.wait(timeout)
File "C:\Program Files\DownloadLoft\Python27\lib\threading.py", line 262, in w
ait
_sleep(delay)
File "c:\users\admini~1\appdata\local\temp\easy_install-sch3hm\gevent-0.13.8-p
y2.7-win32.egg.tmp\gevent\hub.py", line 79, in sleep
switch_result = get_hub().switch()
File "c:\users\admini~1\appdata\local\temp\easy_install-sch3hm\gevent-0.13.8-p
y2.7-win32.egg.tmp\gevent\hub.py", line 135, in get_hub
raise NotImplementedError('gevent is only usable from a single thread')
NotImplementedError: gevent is only usable from a single thread
Is this a known issue with gevent.monkeypatch? Any ideas?
Bottle applications are threaded, so you can't use gevent in any function called in bottle routes.
To help you, I need to speculate why you use threads.
If it's to accelerate your bottle website, juste use the cherrypy server :
pip install cherrypy
(or just dump the cherrypy dir in your current dir, it's a pure Python server)
Then run your bottle app this way :
bottle.run(server='cherrypy')
If it's because you want to make non blocking calls (such as fetching URLs) without blocking the response, it's easy enought to do it manually :
Create a Queue object (it's a special queue that can be filled and popped between threads).
Create and run a thread with a infinite while loop unpoping the queue and doing actions every time.
When you need a non blocking call, push the action to the queue and carry one.
i have a rabbit mq running on machine
both client and rabbitMQ are running on the same network
rabbitMQ has many clients
i can ping client from rabbitMQ and back
longest latency measured between the machine is 12.1 ms
network details : Standard Switch network (network of virtual machines running on single physical machine - using vmware VC)
im getting random timeouts when initializing RPC connection
/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py
problem is that the timeout isn't consistent and happens from time to time.
when manually testing this issue and running the blocking_connection.py 1000 times from the same machine that it fails no timeout accrue.
this is the error i get when failing :
2013-04-23 08:24:23,396 runtest-trigger.24397 24397 DEBUG producer_rabbit initiate_rpc_connection Connecting to RabbitMQ RPC queue rpcqueue_java on host: auto-db1
2013-04-23 08:24:25,350 runtest-trigger.24397 24397 ERROR testrunner go Run 1354: cought exception: timed out
Traceback (most recent call last):
File "/testrunner.py", line 193, in go
self.set_runparams(jobid)
File "/testrunner.py", line 483, in set_runparams
self.runparams.producers_testrun = self.initialize_producers_testrun(self.runparams)
File "/basehandler.py", line 114, in initialize_producers_testrun
producer.set_testcase_checkout()
File "/baseproducer.py", line 73, in set_testcase_checkout
self.checkout_handler = pm_checkout.get_producer(self.testcasecheckout)
File "/producer_manager.py", line 101, in get_producer
producer = self.load_producer(plugin_dir, producer_name)
File "/producer_manager.py", line 20, in load_producer
producer = getattr(producer_module, 'Producer')(producer_name, self.runparams)
File "/producer_rabbit.py", line 13, in __init__
self.initiate_rpc_connection()
File "/producer_rabbit.py", line 67, in initiate_rpc_connection
self.connection = pika.BlockingConnection(pika.ConnectionParameters( host=self.conf.rpc_proxy))
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py", line 32, in __init__
BaseConnection.__init__(self, parameters, None, reconnection_strategy)
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py", line 50, in __init__
reconnection_strategy)
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/connection.py", line 170, in __init__
self._connect()
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/connection.py", line 228, in _connect
self.parameters.port or spec.PORT)
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py", line 44, in _adapter_connect
self._handle_read()
File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py", line 151, in _handle_read
data = self.socket.recv(self._suggested_buffer_size)
timeout: timed out
please assist
I had a similar issue. If everything looks fine, then you most likely have some sort of miss configuration, e.g. bad binding. If miss configured, then you'll get a timeout because the script can't reach where it thinks it needs to go, so the error can be miss leading in this case.
For my problem, I specifically had issues with both my rabbitmq.config file and my bindings and had to use my python solution shown in: RabbitMQ creating queues and bindings from a command line over the command line example I showed. Once updated and configured properly, everything worked fine. Hopefully this gets you in the right direction.
Pika provides some time out issue when connecting different hosts.Solution is to pass a socket_timeout argument in connection parameter.Pika should upgrade to >=0.9.14
credentials = pika.PlainCredentials(RABBITMQ_USER, RABBITMQ_PASS)
connection = pika.BlockingConnection(pika.ConnectionParameters(
credentials=credentials,
host=RABBITMQ_HOST,
socket_timeout=300))
channel = connection.channel()