I am using Flask for my Web Api service.
Finding that my services sometimes (1/100 requests) respond really slow (seconds), I started debugging, which showed me that sometimes the service hangs on reading the request field.
#app.route('/scan', methods=['POST'])
def scan():
start_time = time.time()
request_description = request.form.get('requestDescription')
end_time = time.time()
app.logger.debug('delay is ' + end_time-start_time)
Here I found that delay between start_time and end_time can be up to 2 minutes.
I've read about using Flask's Werkzeug as a production server, so I tried GUnicorn as an alternative - same thing.
I feel that my problem is somehow similar to this one, with the difference that another server didn't solve the problem.
I tried to profile the app using cProfile and SnakeViz, but with the non-prod Werkzeug server - as I don't get how to profile python apps running on GUnicorn. (maybe anyone here knows how to?)
My POST requests contain description and a file. The file can vary in size, but the logs show that the issue reproduces regardless of the file size.
People also usually say that Flask should be used in Nginx-[normal server]-flask combo, but as I use the service inside Openshift, I doubt this has any meaning. (HaProxy works as a balancer)
So my settings:
Alpine 3.8.1
GUnicorn:
workers:3
threads:1
What happens under the hood when I call this?
request.form.get('requestDescription')
How can I profile Python code under GUnicorn?
Did anyone else encounter such a problem?
Any help will be appreciated
I did face this issue as well. I was uploading a video file using request.post(). Turns out that the video uploading was not the issue.
The timing bottleneck was the request.form.get(). While I am still trying to figure out the issue, you can use Flask Monitoring Dashboard to time profile the code
Turns out that the under the hood is return self._sock.recv_into(b) if you use the profiler
Related
I have a very simple python Flask app. It is just an app.py and a templates/index.html. It is deployed in Azure. The problem is that when I make changes to the index.html and re-deploy it, the browser still loads the old one although I can see the new index.html on the server. I have tried Azure Web App Service stop/start to no success.
Any ideas how to fix this?
The file is probably still cached by your browser. Try to reload the site without cache (Different for every browser and OS - shift+f5 windows, cmd+r mac etc., ...).
You can set an HTTP-Header to control how long files are cached.
I'd recommend using the #app.after_request decorator.
#app.after_request
def add_header(response):
response.cache_control.max_age = 300 # set this to 0 to force a reload every time
return response
I hope that helps. If not, please provide a little bit more code.
In short:
I have a Django application being served up by Apache on a Google Compute Engine VM.
I want to access a secret from Google Secret Manager in my Python code (when the Django app is initialising).
When I do 'python manage.py runserver', the secret is successfully retrieved. However, when I get Apache to run my application, it hangs when it sends a request to the secret manager.
Too much detail:
I followed the answer to this question GCP VM Instance is not able to access secrets from Secret Manager despite of appropriate Roles. I have created a service account (not the default), and have given it the 'cloud-platform' scope. I also gave it the 'Secret Manager Admin' role in the web console.
After initially running into trouble, I downloaded the a json key for the service account from the web console, and set the GOOGLE_APPLICATION_CREDENTIALS env-var to point to it.
When I run the django server directly on the VM, everything works fine. When I let Apache run the application, I can see from the logs that the service account credential json is loaded successfully.
However, when I make my first API call, via google.cloud.secretmanager.SecretManagerServiceClient.list_secret_versions , the application hangs. I don't even get a 500 error in my browser, just an eternal loading icon. I traced the execution as far as:
grpc._channel._UnaryUnaryMultiCallable._blocking, line 926 : 'call = self._channel.segregated_call(...'
It never gets past that line. I couldn't figure out where that call goes so I couldnt inspect it any further than that.
Thoughts
I don't understand GCP service accounts / API access very well. I can't understand why this difference is occurring between the django dev server and apache, given that they're both using the same service account credentials from json. I'm also surprised that the application just hangs in the google library rather than throwing an exception. There's even a timeout option when sending a request, but changing this doesn't make any difference.
I wonder if it's somehow related to the fact that I'm running the django server under my own account, but apache is using whatever user account it uses?
Update
I tried changing the user/group that apache runs as to match my own. No change.
I enabled logging for gRPC itself. There is a clear difference between when I run with apache vs the django dev server.
On Django:
secure_channel_create.cc:178] grpc_secure_channel_create(creds=0x17cfda0, target=secretmanager.googleapis.com:443, args=0x7fe254620f20, reserved=(nil))
init.cc:167] grpc_init(void)
client_channel.cc:1099] chand=0x2299b88: creating client_channel for channel stack 0x2299b18
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
client_channel.cc:1879] chand=0x2299b88 calld=0x229e440: created call
...
call.cc:1980] grpc_call_start_batch(call=0x229daa0, ops=0x20cfe70, nops=6, tag=0x7fe25463c680, reserved=(nil))
call.cc:1573] ops[0]: SEND_INITIAL_METADATA...
call.cc:1573] ops[1]: SEND_MESSAGE ptr=0x21f7a20
...
So, a channel is created, then a call is created, and then we see gRPC start to execute the operations for that call (as far as I read it).
On Apache:
secure_channel_create.cc:178] grpc_secure_channel_create(creds=0x7fd5bc850f70, target=secretmanager.googleapis.com:443, args=0x7fd583065c50, reserved=(nil))
init.cc:167] grpc_init(void)
client_channel.cc:1099] chand=0x7fd5bca91bb8: creating client_channel for channel stack 0x7fd5bca91b48
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
So, we a channel is created... and then nothing. No call, no operations. So the python code is sitting there waiting for gRPC to make this call, which it never does.
The problem appears to be that the forking behaviour of Apache breaks gRPC somehow. I couldn't nail down the precise cause, but after I began to suspect that forking was the issue, I found this old gRPC issue that indicates that forking is a bit of a tricky area.
I tried to reconfigure Apache to use a different 'Multi-processing Module', but as my experience in this is limited, I couldn't get gRPC to work under any of them.
In the end, I switched to using nginx/uwsgi instead of Apache/mod_wsgi, and I did not have the same issue. If you're trying to solve a problem like this and you have to use Apache, I'd advice further investigating Apache forking, how gRPC handles forking, and the different MPMs available for Apache.
I'm facing a similar issue. When running my Flask Application with eventlet==0.33.0 and gunicorn https://github.com/benoitc/gunicorn/archive/ff58e0c6da83d5520916bc4cc109a529258d76e1.zip#egg=gunicorn==20.1.0. When calling secret_client.access_secret_version it hangs forever.
It used to work fine with an older eventlet version, but we needed to upgrade to the latest version of eventlet due to security reasons.
I experienced a similar issue and I was able to solve with the following:
import grpc.experimental.gevent as grpc_gevent
from gevent import monkey
from google.cloud import secretmanager
monkey.patch_all()
grpc_gevent.init_gevent()
client = secretmanager.SecretManagerServiceClient()
I have been working on a website for over a year now, using Django and Python3 primarily. A few of my buddies and I built a front end where a user enters some parameters and submits, this goes to the GAE to run the job and return the results.
In my local dev environment, everything works well. I have two separate dev environments. One builds the entire service up in a docker container. This produces the desired results in roughly 11 seconds. The other environment runs the source files locally on my computer and connects to the Postgres database hosted in Google Cloud. The Python application runs locally. It takes roughly 2 minutes for it to run locally, a lot of latency between the cloud and the post/gets from my local machine.
Once I perform the Gcloud app deploy and attempt to run in production, it never finishes. I have some print statements built into the code, I know it gets to the part where the submitted parameters go to the Python code. I monitor via this command on my local computer: gcloud app logs read.
I suspect that since my local computer is a beast (i7-7770 processor with 64 GB of RAM), it runs the whole thing no problem. But in the GAE, I don't think it's providing the proper machines to do the job efficiently (not enough compute, not enough RAM). That's my guess.
So, I need help in how to troubleshoot this. I tried changing my app.yaml file so that resources would scale to 16 GB of memory, but it would never deploy. I received an error 13.
One other note, after it spins around trying to run the job for 60 minutes, the website crashes and displays this message:
502 Server Error
Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.
OK, so just in case anybody in the future is having a similar problem...the constant crashing of my Google App Engine workers was because of using Pandas dataframes in the production environment. I don't know exactly what Pandas was doing, but I kept getting Memory Errors, it would crash the site...and it didn't appear to be occurring in a single line of code. That is, it randomly happened somewhere in a Pandas Dataframe operation.
I am still using a Pandas Dataframe simply to read in a csv file. I then use
data_I_care_about = dict(zip(df.col1, df.col2))
#or
other_data = df.col3.values.tolist()
and then go to town with processing. As a note, on my local machine (my development environment basically) - it took 6 seconds to run from start to finish . That's a long time for a web request but I was in a hurry, thus why I used Pandas to begin with.
After refactoring, the same job completed in roughly 200ms using python lists and dicts (again, in my dev environment). The website is up and running very smoothly now. It takes a maximum of 7 seconds after pressing "Submit" for the back-end to return the data sets and render on the web page. Thanks for the help peeps!
Really newbie questions.
I made a Python bot which receives some data and has to analyze it,then prints everything. To use it, i need it to run for the whole day, the problem is that i can't leave my computer on 24/7, so i need a server or something similar for it and i need to be able to check what it prints whenever i want.
I made some research and found Heroku, but i'm having some problems understanding it: i tried to deploy it there and it's working but it prints all the stuff on a shell in the app's page and not on the webpage that heroku assigned to my app, so my problem is partially solved, since i can run it for the whole day but checking what it prints is way harder.
I was thinking of making it as a Telegram bot in order to have everything there but since it prints a lot of stuff, Telegram would not be the best platform for this kind of thing.
Is there another resource to deploy it and have it, for example, on a webpage?
You can look into renting a cheapish cloud server (from digitalocean for example).
There are multiple ways of transferring data from your python script to your bot, either directly, through a websocket, or a webpage that displays it in a JSON format or otherwise.
Since you're already using python you could look into running a flask app on your node alongside your script or even combine them together.
If ran separately you could modify your script to output it's content into a file and then read the file with your flask application to display it on a webpage. For example:
with open('/tmp/data.txt', 'w') as f:
f.write(yourdata)
then in your flask application:
from flask import Flask
app = Flask(__name__)
#app.route('/')
def show_data():
with open('/tmp/data.txt', 'r') as f:
data = f.read()
return data
There are way more efficient ways of transferring data. Example above is a quick and dirty solution I wouldn't recommend running it due to security reasons especially if you are transmitting sensitive data.
I'm having an issue with my application on Heroku where sessions aren't persisting. Specifically, flask's SecureCookieSession object is empty, every time a request is made. Contrast this with running my application on localhost, where the contents of SecureCookieSession persist the way they should.
Also I'm using flask-login + flask-seasurf, but I'm pretty sure the issue happening somewhere between flask / gunicorn / heroku.
Here are three questions that describe a similar issue:
Flask sessions not persisting on heroku
Flask session not persisting
Flask-Login and Heroku issues
Except I'm not dealing with AJAX or multiple workers here (it's a single heroku free dyno, with a single line in the Procfile). I do get the feeling that using server side sessions with redis or switching from Heroku to something like EC2 might solve my problem though.
Also, here's my git repo if it helps https://gitlab.com/collectqt/quirell/tree/develop. And I'm testing session stuff with
def _before_request(self):
LOG.debug('SESSION REQUEST '+str(flask.session))
def _after_request(self, response):
LOG.debug('SESSION RESPONSE '+str(flask.session))
return response
Got the solved with some external help, mainly by changing the secret key to use a random string I came up with, instead of os.urandom(24)
Changing to server side redis sessions helped too, if only by making testing simpler
Just in case someone else comes across this question, check APPLICATION_ROOT configuration variable. I recently deployed a Flask application to a subdirectory under nginx with a reverse-proxy and setting the APPLICATION_ROOT variable broke Flask's session. Cookies aren't being set under the correct path because of that.