We Keep Coding

Python script making OPC DA connection not running on VM Server after RDP disconnects

I am running a Python script on a Windows 2016 Server that uses the OpenOPC module to connect to an OPC DA Server and write values to a couple of tags. I need this script to execute every 15 min so I have set up a task in task scheduler to execute the script with the highest privileges whether the user is logged in or not. If I RDP connect to the server and run the script manually it works perfectly. However, as soon as I disconnect the script no longer works.
I am running other python scripts, that connect to SQL databases and query information and then save the proccessed information to excel spreadsheets, which continue to work even when I disconnect. It seems that the OpenOPC module cannot run when the user disconnects. I can't understand why this would be an issue. Take note that the VM does not log off or restart.
I tried running the script manually in IDLE while connected and putting it on a loop so that I could start it, disconnect and reconnect to see what happened. I was presented with the following error code:
Traceback (most recent call last):
File "C:\Users\Script.py", line 92, in <module>
main(time, String, opcServer)
File "C:\Users\Script.py", line 34, in main
opc.connect(opcServer)
File "C:\Python27\lib\site-packages\Pyro\core.py", line 381, in __call__
return self.__send(self.__name, args, kwargs)
File "C:\Python27\lib\site-packages\Pyro\core.py", line 456, in _invokePYRO
return self.adapter.remoteInvocation(name, Pyro.constants.RIF_VarargsAndKeywords, vargs, kargs)
File "C:\Python27\lib\site-packages\Pyro\protocol.py", line 497, in remoteInvocation
return self._remoteInvocation(method, flags, *args)
File "C:\Python27\lib\site-packages\Pyro\protocol.py", line 572, in _remoteInvocation
answer.raiseEx()
File "C:\Python27\lib\site-packages\Pyro\errors.py", line 72, in raiseEx
raise self.excObj
OPCError: Connect: The server process could not be started because the configured identity is incorrect. Check the username and password. (The server process could not be started because the configured identity is incorrect. Check the username and password.)

Google Cloud App Engine Flexible Python 2.7 Env Errors starting new threads

I know just enough devops to be dangerous. I've successfully deployed a VERY simple python flask app to App Engine that basically publishes received post data as a message to PubSub. It is almost identical to Google's sample code to do so. Only difference is it uses a service account I push with the app repository to access PubSub to circumvent this issue.
Works very well so far, but I've started seeing a very small number of errors around starting a new thread in threading.py:
1)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "/usr/lib/python2.7/threading.py", line 736, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
2)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
3)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "/usr/lib/python2.7/threading.py", line 736, in start
File "/usr/lib/python2.7/threading.py", line 736, in start
I have 2 questions, in order of importance:
This is an app that basically needs 100% uptime in order to not lose data (not confident the clients attempt retries if there is an error on my server side). Are these errors internal to how App Engine is managing my app's resources, and not resulting in errors handling actual requests? How can I determine if I ever responded with an HTTP error/didn't successfully handle a request? I don't see any errors in my nginx logs...is that the place I need to look to see if anything failed?
Is there a way I can fix this error?

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/publisher/client.py#L143
it looks like publisher.publish(topic_path, data=data) is an async operation, returning a concurrent.futures.Future object
Have you trying calling the Future's result()? https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.result
This will block until the future object is successful, fails, or timesout.
You could then forward that result as your HTTP response.
Hopefully, the result object will give you more information about the error.

Ended up changing the methodology a bit. Instead of posting a pubsub message then having dataflow ingest through GCS to BigQuery I decided to stream directly into BQ using the BigQuery python client. Updated dependencies for the python flask app to:
Flask==1.0.2
google-cloud-pubsub==0.39.1
gunicorn==19.9.0
google-cloud-bigquery==1.11.2
and I am no longer seeing any of those exceptions. It's worth noting that I'm still using a service account .json credentials file in the same directory as the app source, and I'm creating the BigQuery client with
bq_client = bigquery.Client.from_service_account_json(BQ_SVC_ACCT_FILE).
For anyone else with similar issues I'd recommend updating your dependencies (esp any Google Cloud client libraries) and create the Client you need from a local service account credentials file. I attempted to use the inherited Compute engine environment credentials (basically the default project compute engine service account) but that was less stable than pushing up an actual credential file and using that locally. However...assess your own security needs before doing the same.

using django celery beat locally I get error 'PeriodicTask' object has no attribute '_default_manager'

using django celery beat locally I get error 'PeriodicTask' object has no attribute '_default_manager'. I am using Django 1.10. When i schedule a task it works. But then a few moments later a red error traceback like the following occurs
[2016-09-23 11:08:34,962: INFO/Beat] Writing entries...
[2016-09-23 11:08:34,965: INFO/Beat] Writing entries...
[2016-09-23 11:08:34,965: INFO/Beat] Writing entries...
[2016-09-23 11:08:34,966: ERROR/Beat] Process Beat
Traceback (most recent call last):
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/billiard/process.py", line 292, in _bootstrap
self.run()
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/celery/beat.py", line 553, in run
self.service.start(embedded_process=True)
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/celery/beat.py", line 486, in start
self.scheduler._do_sync()
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/celery/beat.py", line 276, in _do_sync
self.sync()
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/djcelery/schedulers.py", line 209, in sync
self.schedule[name].save()
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/djcelery/schedulers.py", line 98, in save
obj = self.model._default_manager.get(pk=self.model.pk)
AttributeError: 'PeriodicTask' object has no attribute '_default_manager'
after this happens the next schedule wont run unless I "control+c" out of the terminal and start it again. I saw on git hub that this may be because I am using django 1.10. I have already git pushed this to my heroku server. How can I fix this issue? The git hub post said he fixed it by doing this
Model = type(self.model)
obj = Model._default_manager.get(pk=self.model.pk)
I was willing to try this but I don't know where to put this and I don't want to cause a bigger unforeseen issue that this could cause. What are my options? am I supposed to manually go inside my remote app and reset it after every time it runs? thats unfeasible and defeats the purpose of task automation.

I figured it out. At line 98 in schedulers.py it was
obj = self.model._default_manager.get(pk=self.model.pk)
so a line above it I added
Model = type(self.model)
and changed
obj = self.model._default_manager.get(pk=self.model.pk)
to
obj = Model._default_manager.get(pk=self.model.pk)
so completed it looks like this
98 Model = type(self.model)
99 obj = Model._default_manager.get(pk=self.model.pk)

Can't bring cassandra and celery together

I'm trying the the example to use celery and cassandra together:
http://datastax.github.io/python-driver/cqlengine/third_party.html
But without luck.
I get this exception the I'm starting the worker with:
$ celery -A tasks worker -l INFO
[2016-06-12 14:11:53,609: ERROR/Worker-1] Process Worker-1
Traceback (most recent call last):
File "/Users/lutz/work/truncated/truncated-worker/venv/lib/python3.5/site-packages/billiard/process.py", line 292, in _bootstrap
self.run()
File "/Users/lutz/work/truncated/truncated-worker/venv/lib/python3.5/site-packages/billiard/pool.py", line 292, in run
self.after_fork()
File "/Users/lutz/work/truncated/truncated-worker/venv/lib/python3.5/site-packages/billiard/pool.py", line 395, in after_fork
self.initializer(*self.initargs)
File "/Users/lutz/work/truncated/truncated-worker/venv/lib/python3.5/site-packages/celery/concurrency/prefork.py", line 84, in process_initializer
signals.worker_process_init.send(sender=None)
File "/Users/lutz/work/truncated/truncated-worker/venv/lib/python3.5/site-packages/celery/utils/dispatch/signal.py", line 166, in send
response = receiver(signal=self, sender=sender, **named)
TypeError: cassandra_init() got an unexpected keyword argument 'sender'
I'm Using osx el Capitan, python 3.5.1, Celery 3.1.23 and cassandra 3.5.
So any help will be welcome.

Your cassandra_init signal handler function needs to accept arbitrary keyword arguments.
Simply change the line:
def cassandra_init():
into:
def cassandra_init(**kwargs):
For more information about Celery signals, see the user guide at:
http://docs.celeryproject.org/en/latest/userguide/signals.html#basics
Note: It would be helpful if you also submitted some kind of report to the author of that tutorial. Celery signal handlers have always required the keyword arguments, so it's sad to have non-working examples out there.

How to debug intermittent errors from Django app served with gunicorn (possible race condition)?

I have a Django app being served with nginx+gunicorn with 3 gunicorn worker processes. Occasionally (maybe once every 100 requests or so) one of the worker processes gets into a state where it starts failing most (but not all) requests that it serves, and then it throws an exception when it tries to email me about it. The gunicorn error logs look like this:
[2015-04-29 10:41:39 +0000] [20833] [ERROR] Error handling request
Traceback (most recent call last):
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 171, in handle_request
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 206, in __call__
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 196, in get_response
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 226, in handle_uncaught_exception
File "/usr/lib/python2.7/logging/__init__.py", line 1178, in error
File "/usr/lib/python2.7/logging/__init__.py", line 1271, in _log
File "/usr/lib/python2.7/logging/__init__.py", line 1281, in handle
File "/usr/lib/python2.7/logging/__init__.py", line 1321, in callHandlers
File "/usr/lib/python2.7/logging/__init__.py", line 749, in handle
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/log.py", line 122, in emit
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/log.py", line 125, in connection
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/core/mail/__init__.py", line 29, in get_connection
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 26, in import_by_path
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 21, in import_by_path
File "/home/django/virtualenvs/homestead_django/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
ImproperlyConfigured: Error importing module django.core.mail.backends.smtp: "No module named smtp"
So some uncaught exception is happening and then Django is trying to email me about it. The fact that it can't import django.core.mail.backends.smtp doesn't make sense because django.core.mail.backends.smtp should definitely be on the worker process' Python path. I can import it just fine from a manage.py shell and I do get emails for other server errors (actual software bugs) so I know that works. It's like the the worker process' environment is corrupted somehow.
Once a worker process enters this state it has a really hard time recovering; almost every request it serves ends up failing in this same manner. If I restart gunicorn everything is good (until another worker process falls into this weird state again).
I don't notice any obvious patterns so I don't think this is being triggered by a bug in my app (the URLs error'ing out are different, etc). It seems like some sort of race condition.
Currently I'm using gunicorn's --max-requests option to mitigate this problem but I'd like to understand what's going on here. Is this a race condition? How can I debug this?

I suggest you use Sentry which gives a smart way of handling errors.
You can use it as a cloud based solution (getsentry) or you can install it on your own server (github).
Before I was using django core log mailer now I always use sentry.
I do not work at Sentry but their solution is pretty awesome !

We discovered one particular view that was pegging the CPU for a few seconds every time it was loaded that seemed to be triggering this issue. I still don't understand how slamming a gunicorn worker could result in a corrupted execution environment, but fixing the high-CPU view seems to have gotten rid of this issue.