I have a Tornado app which is using python firebase_admin SDK.
When I run in single process:
console_server = tornado.httpserver.HTTPServer(ConsoleApplication())
console_server.listen(options.console_port, options.bind_addr)
tornado.ioloop.IOLoop.instance().start()
firebase_admin works fine. But when I change it to run in multiprocess:
console_server = tornado.httpserver.HTTPServer(ConsoleApplication())
console_server.bind(options.console_port, options.bind_addr)
console_server.start(4)
tornado.ioloop.IOLoop.instance().start()
The last line here is getting stuck:
if (not len(firebase_admin._apps)):
cred = ...
self.app = firebase_admin.initialize_app(cred)
self.app = firebase_admin.get_app()
self.db = firestore.client()
...
ref = self.db.document(USER_DOC.format(org, value))
user_ref = ref.get()
Seems like get() is not getting resolved since I don't get any exception.
Does anyone has an idea why it's happening or at least how can I debug it?
The multiprocess fork (i.e. the start(4) call) must come very early in the life of your application. In particular, most things that touch the network must come after the fork (bind() is one of the few exceptions, and must come before the fork in this case).
You (probably) need to reorganize things so that you're creating the firebase app after the fork. This can be annoying if you're using the HTTPServer.start method, so you may want to switch to calling tornado.process.fork_processes() directly instead (this is documented as the "advanced multi-process" pattern).
I know it's an old question, but I want to share my experience regarding this issue to help future visitors.
I recently developed a script with multiprocessing that uses Firebase Admin Python SDK, everything worked fine in my local Windows machine, but when I deployed it for production in Linux server, I noticed the script is getting stuck in get() function.
After hours of searching, I found out that the default start method of a python process is different in Windows and Unix environments: Windows uses spawn as default start method, whereas Unix uses fork. You can learn more about start methods in the documentation.
So to make it work in my Linux server, I just changed the start method to spawn:
if __name__ == '__main__':
multiprocessing.set_start_method('spawn') # <-- Set spawn as start_method
# The rest of your script here
# ...
Related
I was working on a specific type of application in Dash which required the action executed by pressing the button to be performed in a separate process. This process, in turn, was parallelizable, and in some cases spawned child processes for the efficient computation. The configuration given in this cases makes the child processes to get stuck. The code below reproduces the situation described as follows:
import multiprocessing
import time
import dash
from dash import html
from dash.dependencies import Input, Output
app = dash.Dash(__name__)
app.layout = html.Div([
html.Button(id='refresh-button', children='Button'),
html.Div(id='dynamic-container1')
])
def run_function(i):
print('hello')
time.sleep(15)
print(f'hello world {i}')
def run_process():
num = 1
print('hello world 00000')
process = multiprocessing.Process(target=run_function, args=(num,))
process.start()
process.join()
print('hello world')
#app.callback(Output('dynamic-container1', 'children'), Input('refresh-button', 'n_clicks'))
def refresh_state(click):
if click == 0 or click is None:
return None
p = multiprocessing.Process(target=run_process)
p.start()
p.join()
return None
if __name__ == '__main__':
app.run_server(debug=True)
The output of this application when pressing the button is always the following:
Connected to pydev debugger (build 172.3968.37)
Dash is running on http://127.0.0.1:8050/
* Serving Flask app 'main' (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
pydev debugger: process 6884 is connecting
hello world 00000
which means that the first process utilizing the function run_process() was launched, however, the child process run_function(i) was not even started. I was trying to find an explanation in popular books on multiprocessing in Python and any guidance to these "chaining" processes, but to no avail. From my understanding, the new child process run_function(i) should occupy a separate core (if there is any free core) and not to depend on the resources consumed by the parent process run_process(). Could you, please, explain to me the mechanics of this? I have a doubt that in this code run_function(i) might be coerced to consume the same resources as run_process() does, so the system basically is just restricting any new process from starting from the same resources, but I would like to confirm it from more expert users of Python.
I used Python 3.7 and Pycharm Community 2017.2.3 on Win7 to reproduce this example
I can not reproduce your error and i think this is because you are using windows ( i am on Linux with python 3.9), so i can not find the error for you, but maybe i can give you some hints:
First: To find the error, try to reduce the code to the core of the problem (you can remove the whole dash stuff to check if this is the error). In my tests the results were the same, with and without dash
Second: Windows and Linux handle the multiprocessing stuff a bit different:
windows spawns the process:
The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process object’s run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.
Unix Systems fork the process
The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.
With Unix System you can spawn (multiprocessing.set_start_method("spawn")), with this i can't reproduce you error, but with this fork/spawn example i wanted to make clear, that there sometimes things are a bit different between windows and linux, even if the same packages are used. I think your understanding of multiprocessing is correct. (Maybe this site helps too.)
Third: In the docs are some programming guidelines you should be aware of. Maybe they will help, too. And in general, the multiprocessing package does not work well with a lot of interactive python shells (like IDLE or pycharm), this can may be better on newer versions. Maybe you should try it from terminal to check if this changes something.
I hope this helps a litle bit.
I am really new to python's multiple processing, and I have some notions about async call, yield and etc...the most basic stuff. And I came to this snippet, in which multiprocessing.Process wraps around tornado.ioloop.IOLoop.instance
# Set up the tornado web app
app = make_app(predicted_model_queue)
app.listen(8080)
server_process = Process(target=tornado.ioloop.IOLoop.instance().start)
# Start up the server to expose the metrics.
server_process.start()
It intends to start a tornado server as a server_process, but the code does not work. I got the error,
OSError: [Errno 9] Bad file descriptor
I have no experiences with both lib, and have no idea how to fix it. Can anyone please help me?
This is an unusual pattern - if you're writing a new app, I wouldn't recommend copying it.
If you're just trying to run an app that does this (looks like it came from here), the problem is that IOLoops cannot safely cross process boundaries (on some platforms it can sometimes work, but not always). To rewrite this code to correctly create the app and IOLoop in the child process, you could do this:
def run_server():
app = make_app(predicted_model_queue)
app.listen(8080)
tornado.ioloop.IOLoop.current().start()
server_process = Process(target=run_server)
server_process.start()
This way only the predicted_model_queue is shared between the two processes.
Hi I'm just learning python, threading and flask
what is the proper way to see thread is running in flask app ? I already tried storing the thread in global variable
global thread_a
thread_a = threading.start()
and tried using flask.current_app
flask.current_app.thread_a = threading.start()
then I check if thread is running using
thread_a.is_alive()
it is works fine on my local machine. however when I deploy to remote server (this case is openshift) the value of the variable is sometimes defined but sometimes not, so I thought this method doesn't always work ?
I am working on migrating an existing python GAE (Google App Engine) standard environment app to the flexible environment. I read through the guide and decided to try out the python-compact runtime, as it's always good to re-use as much code as possible.
In the standard environment app, we use background_thread.start_new_background_thread() to spawn a bunch of infinite-loop threads to work on some background work forever. However, I couldn't get start_new_background_thread working in the flexible environment, even for some really simple app. Like this sample app:
github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/background
I keep getting the following error while running the app in the cloud (it works fine locally though).
I debugged into it by using the cloud debugger, but there was no any error message available at all while the exception was raised in the background_thread.py
Any idea how I can run a long-live background thread in the flexible environment with python-compact runtime? Thanks!
One of the differences between App Engine standard and App Engine flexible is that with Flex we're really just running a docker container. I can think of 2 approaches to try out.
1. Just use Python multiprocessing
App Engine standard enforces a sandbox that mostly means no direct use of threads or processes. With Flex, you should be able to just use the standard Python lib for starting a new sub process:
https://docs.python.org/3/library/subprocess.html
2. Use supervisord and docker
If that doesn't work, another approach you could take here is to customize the docker image you're using in Flex, and use supervisord to start multiple processes. First, generate the dockerfile by cd-ing into folder with your sources and running:
gcloud preview app gen-config --custom
This will create a Dockerfile that you can customize. Now, you are going to want to start 2 processes - the process we were starting (I think for python-compat it's gunicorn) and your background process. The easiest way to do that with docker is to use supervisord:
https://docs.docker.com/engine/admin/using_supervisord/
After modifying your Dockerfile and adding a supervisord.conf, you can just deploy your app as you normally would with gcloud preview app deploy.
Hope this helps!
I wish the documentation said that background_thread was not a supported API.
Anyway, I've found some hacks to help with some thread incompatibilities. App Engine uses os.environ to read a lot of settings. The "real" threads in your application will have a bunch of environment variables set there. The background threads you start will have none. One hack I've used is to copy some of the environment variables. For example, I needed to copy set the SERVER_SOFTWARE variable in the background threads in order to get the App Engine cloud storage library to work. We use something like:
_global_server_software = None
_SERVER_SOFTWARE = 'SERVER_SOFTWARE'
def environ_wrapper(function, args):
if _global_server_software is not None:
os.environ[_SERVER_SOFTWARE] = _global_server_software
function(*args)
def start_thread_with_app_engine_environ(function, *args):
# HACK: Required for the cloudstorage API on Flexible environment threads to work
# App Engine relies on a lot of environment variables to work correctly. New threads get none
# of those variables. loudstorage uses SERVER_SOFTWARE to determine if it is a test instance
global _global_server_software
if _global_server_software is None and os.environ.get(_SERVER_SOFTWARE) is not None:
_global_server_software = os.environ[_SERVER_SOFTWARE]
t = threading.Thread(target=environ_wrapper, args=(
function, args))
t.start()
Can you recommend on a python tool / module that allows scheduling tasks on remote machine in a network?
Note that the solution must be able to not only run certain jobs/commands on remote machines, but also verify that jobs etc are still running (for example, consider the case where a machine dies after a task has been assigned to it?)
RPyC or Remote Python Call, is a transparent and symmetrical python library for remote procedure calls, clustering and distributed-computing. Here an example from Wikipedia:
import rpyc
conn = rpyc.classic.connect("hostname") # assuming a classic server is running on 'hostname'
print conn.modules.sys.path
conn.modules.sys.path.append("lucy")
print conn.modules.sys.path[-1]
# a version of 'ls' that runs remotely
def remote_ls(path):
ros = conn.modules.os
for filename in ros.listdir(path):
stats = ros.stat(ros.path.join(path, filename))
print "%d\t%d\t%s" % (stats.st_size, stats.st_uid, filename)
remote_ls("/usr/bin")
# and exceptions...
try:
f = conn.builtin.open("/non/existent/file/name")
except IOError:
pass
To check if the remote server has died after assigning it a job, you can use the ping method of the Connection class. The complete API is described here.
Fabric (http://docs.fabfile.org/en/1.0.1/index.html) is a pretty good toolkit for various sys admin and deployment tasks. It comes with a few pre defined tasks but also gives you the flexibility to add what you need.
I highly recommend it.
Should be able to use Python WMI, for *NIX based systems it's a wrap around SSH and CRON.