We Keep Coding

Pycharm / Django defer Oracle Database connection

I have a Django project that connects to an Oracle database - implemented in PyCharm. I would like the app to monitor the connection and display a status as to whether the database is up or down. I stopped the listener to simulate an error. However, when I start the PyCharm server to test the functionality, the app immediately attempts a database connection and fails. Is it possible to defer the database connection until I can command one within the app?
Here is the error traceback on server startup:
C:\Users\steve\PycharmProjects\Tools\venv\Scripts\python.exe C:/Users/steve/PycharmProjects/Tools/Tools/manage.py runserver
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
Exception in thread django-main-thread:
Traceback (most recent call last):
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\base\base.py", line 219, in ensure_connection
self.connect()
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\base\base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\oracle\base.py", line 229, in get_new_connection
return Database.connect(
cx_Oracle.DatabaseError: ORA-12541: TNS:no listener
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\steve\AppData\Local\Programs\Python\Python39\lib\threading.py", line 954, in _bootstrap_inner
self.run()
File "C:\Users\steve\AppData\Local\Programs\Python\Python39\lib\threading.py", line 892, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\utils\autoreload.py", line 64, in wrapper
fn(*args, **kwargs)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\core\management\commands\runserver.py", line 121, in inner_run
self.check_migrations()
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\core\management\base.py", line 486, in check_migrations
executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\migrations\executor.py", line 18, in init
self.loader = MigrationLoader(self.connection)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\migrations\loader.py", line 53, in init
self.build_graph()
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\migrations\loader.py", line 220, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\migrations\recorder.py", line 77, in applied_migrations
if self.has_table():
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\migrations\recorder.py", line 55, in has_table
with self.connection.cursor() as cursor:
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\base\base.py", line 259, in cursor
return self._cursor()
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\base\base.py", line 235, in _cursor
self.ensure_connection()
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\base\base.py", line 219, in ensure_connection
self.connect()
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\utils.py", line 90, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\base\base.py", line 219, in ensure_connection
self.connect()
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\base\base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\Users\steve\PycharmProjects\Tools\venv\lib\site-packages\django\db\backends\oracle\base.py", line 229, in get_new_connection
return Database.connect(
django.db.utils.DatabaseError: ORA-12541: TNS:no listener

Python Redis - RuntimeError pubsub connection not set

Version: redis-py=3.1.0 and redis=3.2.10
Platform: Python 2.7.5 / CentOS Linux release 7.4.1708 (Core)
Infrastructure:
two machines (worker1, worker2 ) for running celery worker services with default concurrency (=8).
one dedicated machine (redis1) for running redis server.
Issue:
After the workers running for some time, suddenly a worker running on machine1 dies due to a RuntimeError raised losing a connection to pubsub.
machine1.worker.log
[2019-02-01 13:43:39,477: CRITICAL/MainProcess] Unrecoverable error: RuntimeError(u'pubsub connection not set: did you forget to call subscribe() or psubscribe()?',)
Traceback (most recent call last):
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/worker/worker.py", line 205, in start
self.blueprint.start(self)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 322, in start
blueprint.start(self)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 598, in start
c.loop(*c.loop_args())
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/worker/loops.py", line 91, in asynloop
next(loop)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/asynchronous/hub.py", line 354, in create_loop
cb(*cbargs)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/transport/redis.py", line 1047, in on_readable
self.cycle.on_readable(fileno)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/transport/redis.py", line 344, in on_readable
chan.handlers[type]()
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/transport/redis.py", line 674, in _receive
ret.append(self._receive_one(c))
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/transport/redis.py", line 685, in _receive_one
response = c.parse_response()
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/redis/client.py", line 3032, in parse_response
'pubsub connection not set: '
RuntimeError: pubsub connection not set: did you forget to call subscribe() or psubscribe()?
While at the same time I have spotted that worker running on machine2 suffers due to not being able to connect to redis. Eventually, it managed to recover and reconnect to redis and receiving the queued tasks.
machine2.worker.log
[2019-02-01 14:43:41,722: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 322, in start
blueprint.start(self)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 40, in start
self.sync(c)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 44, in sync
replies = self.send_hello(c)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 57, in send_hello
replies = inspect.hello(c.hostname, our_revoked._data) or {}
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/app/control.py", line 143, in hello
return self._request('hello', from_node=from_node, revoked=revoked)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/app/control.py", line 95, in _request
timeout=self.timeout, reply=True,
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/celery/app/control.py", line 454, in broadcast
limit, callback, channel=channel,
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
serializer=serializer)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/pidbox.py", line 290, in _publish
serializer=serializer,
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
message, exchange, routing_key, **kwargs
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/transport/virtual/exchange.py", line 151, in deliver
exchange, message, routing_key, **kwargs)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/kombu/transport/redis.py", line 781, in _put_fanout
dumps(message),
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/redis/client.py", line 2716, in publish
return self.execute_command('PUBLISH', channel, message)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/redis/client.py", line 775, in execute_command
return self.parse_response(connection, command_name, **options)
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/redis/client.py", line 789, in parse_response
response = connection.read_response()
File "/opt/c1/cip-middleware/webapp/virtualenv/lib/python2.7/site-packages/redis/connection.py", line 636, in read_response
raise e
ConnectionError: Error while reading from socket: (u'Connection closed by server.',)
[2019-02-01 14:44:50,237: INFO/MainProcess] Received task: c1_cip_middleware.tasks.validate_purchases.run_purchases_validation[a411dd90-ab50-4101-becb-90adda3663a2]
* Questions *
I wonder what are the circumstances/scenarios when RuntimeError is raised, thus, the worker gets into the "unrecovery" stage and must be stopped?
I am in doubt what could be a root-cause of having this issue, especially that one worker managed to recover but the other one just died?

How to use service runner in nameko?

As written in nameko's document, I run this example and it works:
from nameko.runners import ServiceRunner
from nameko.testing.utils import get_container
from nameko.rpc import rpc
class ServiceA:
name = "service_a"
class ServiceB:
name = "service_b"
# create a runner for ServiceA and ServiceB
runner = ServiceRunner(config={})
runner.add_service(ServiceA)
runner.add_service(ServiceB)
# ``get_container`` will return the container for a particular service
container_a = get_container(runner, ServiceA)
# start both services
runner.start()
print('runner start')
# stop both services
runner.stop()
It shows:
runner start
but when i add a rpc method in service_a, and add a AMQP_URI in config, it doesn't work, looks like it doesn't connected to RabbitMQ. What should i do?
class ServiceA:
name = "service_a"
#rpc
def hello_a(self):
return 'hello service_a.'
config = {
'AMQP_URI': 'amqp://guest:guest#localhost',
}
runner = ServiceRunner(config=config)
when i press Ctrl+C:
Traceback (most recent call last):
File "/Users/apple/Documents/nameko_test/proxy.py", line 34, in <module>
runner.start()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/nameko/runners.py", line 65, in start
SpawningProxy(self.containers).start()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/nameko/utils/__init__.py", line 181, in spawning_method
return list(pool.imap(call, self._items))
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/greenpool.py", line 244, in next
val = self.waiters.get().wait()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 179, in wait
return self._exit_event.wait()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/event.py", line 121, in wait
return hubs.get_hub().switch()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 295, in switch
return self.greenlet.switch()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 337, in run
self.fire_timers(self.clock())
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 458, in fire_timers
timer()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
self._resolve_links()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 236, in _resolve_links
f(self, *ca, **ckw)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/nameko/containers.py", line 458, in _handle_managed_thread_exited
self._handle_thread_exited(gt)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/nameko/containers.py", line 462, in _handle_thread_exited
gt.wait()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 179, in wait
return self._exit_event.wait()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/event.py", line 125, in wait
current.throw(*self._exc)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 218, in main
result = function(*args, **kwargs)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/kombu/mixins.py", line 177, in run
for _ in self.consume(limit=None): # pragma: no cover
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/kombu/mixins.py", line 199, in consume
conn.drain_events(timeout=safety_interval)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/kombu/connection.py", line 288, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events
return connection.drain_events(**kwargs)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/amqp/connection.py", line 303, in drain_events
chanmap, None, timeout=timeout,
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/amqp/connection.py", line 366, in _wait_multiple
channel, method_sig, args, content = read_timeout(timeout)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/amqp/connection.py", line 337, in read_timeout
return self.method_reader.read_method()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/amqp/method_framing.py", line 186, in read_method
self._next_method()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/amqp/method_framing.py", line 107, in _next_method
frame_type, channel, payload = read_frame()
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/amqp/transport.py", line 154, in read_frame
frame_header = read(7, True)
File "/Users/apple/Documents/my_projects/venv/lib/python3.6/site-packages/amqp/transport.py", line 277, in _read
s = recv(n - len(rbuf))
KeyboardInterrupt

I suspect you are missing import eventlet; eventlet.monkey_patch() at the top of your file. Eventlet is mentioned in the documentation but I think it could be clearer, and possibly explicitly added to some of the examples.
Alternatively you could use the bundled service runner (which handles this for you) by making a module with just your service definitions and using nameko run from the command line like in the hello world example

Jupyter terminats/exits with an error

Recently, whenever I terminate Jupyter by hitting CTRL-C I get the following error:
Traceback (most recent call last):
File "/Users/username/anaconda3/bin/jupyter-notebook", line 6, in <module>
sys.exit(notebook.notebookapp.main())
File "/Users/username/anaconda3/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/Users/username/anaconda3/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/Users/username/anaconda3/lib/python3.5/site-packages/notebook/notebookapp.py", line 1247, in start
self.io_loop.start()
File "/Users/username/anaconda3/lib/python3.5/site-packages/zmq/eventloop/ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "/Users/username/anaconda3/lib/python3.5/site-packages/tornado/ioloop.py", line 862, in start
^C File "/Users/username/anaconda3/lib/python3.5/site-packages/zmq/eventloop/ioloop.py", line 122, in poll
z_events = self._poller.poll(1000*timeout)
File "/Users/username/anaconda3/lib/python3.5/site-packages/zmq/sugar/poll.py", line 99, in poll
return zmq_poll(self.sockets, timeout=timeout)
File "zmq/backend/cython/_poll.pyx", line 123, in zmq.backend.cython._poll.zmq_poll (zmq/backend/cython/_poll.c:2186)
File "zmq/backend/cython/_poll.pyx", line 116, in zmq.backend.cython._poll.zmq_poll (zmq/backend/cython/_poll.c:2036)
File "zmq/backend/cython/checkrc.pxd", line 12, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/_poll.c:2418)
File "/Users/username/anaconda3/lib/python3.5/site-packages/conda/gateways/signals.py", line 15, in conda_signal_handler
raise CondaSignalInterrupt(signum)
conda.exceptions.CondaSignalInterrupt: Signal interrupt SIGINT
I tried to search a little for some hint what's the cause or reason but for vain. Therefore, I ask you. Do you know what's the reason?

Celery dies with DBPageNotFoundError

I have 3 machines with celery workers and rabbitmq as a broker, one worker is running with beat flag, all of this is managed by supervisor, and sometimes celery dies with such error.
This error appears only on beat worker, but when it appears, workers on all machines dies.
(celery==3.1.12, kombu==3.0.20)
[2014-07-05 08:37:04,297: INFO/MainProcess] Connected to amqp://user:**#192.168.15.106:5672//
[2014-07-05 08:37:04,311: ERROR/Beat] Process Beat
Traceback (most recent call last):
File "/var/projects/env/local/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap
self.run()
File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 527, in run
self.service.start(embedded_process=True)
File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 453, in start
humanize_seconds(self.scheduler.max_interval))
File "/var/projects/env/local/lib/python2.7/site-packages/kombu/utils/__init__.py", line 322, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 491, in scheduler
return self.get_scheduler()
File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 486, in get_scheduler
lazy=lazy)
File "/var/projects/env/local/lib/python2.7/site-packages/celery/utils/imports.py", line 53, in instantiate
return symbol_by_name(name)(*args, **kwargs)
File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 357, in __init__
Scheduler.__init__(self, *args, **kwargs)
File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 184, in __init__
self.setup_schedule()
File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 376, in setup_schedule
self._store['entries']
File "/usr/lib/python2.7/shelve.py", line 121, in __getitem__
f = StringIO(self.dict[key])
File "/usr/lib/python2.7/bsddb/__init__.py", line 270, in __getitem__
return _DeadlockWrap(lambda: self.db[key]) # self.db[key]
File "/usr/lib/python2.7/bsddb/dbutils.py", line 68, in DeadlockWrap
return function(*_args, **_kwargs)
File "/usr/lib/python2.7/bsddb/__init__.py", line 270, in <lambda>
return _DeadlockWrap(lambda: self.db[key]) # self.db[key]
DBPageNotFoundError: (-30985, 'DB_PAGE_NOTFOUND: Requested page not found')

I've ran into this issue and the cause was a corrupted db file (usually named "celerybeat-schedule").
Solution would be to delete the existing db file and restart the process.
Relavent:bsddb.db.DBPageNotFoundError
https://mail.python.org/pipermail/python-list/2009-October/554552.html

I had to remove some temp files in the /tmp directory. One was named celeryd-<NAME_OF_WORKER>-state and also celeryd-<NAME_OF_WORKER>-state-renamed. After removing those and I was able to restart my affected worker.