Google App Engine Produces 502 Error on Python/Flask API - python

I've been all over google, and stack and read EVERY article I could find but have not been able to solve me issue. I recently wanted to take my project that is still in development and get it up on GAE so I could test it and make sure it would work as I assumed. After some trail and error I was able to get a Postgres database setup and push my project using
gcloud app deploy app.yaml
My Flask app is using Swagger and Flask-Restplus for the API. I can access the swagger page and perform some requests, however all of those requests come back with
502 Bad Gatway
While reading the live log in the console I can see the data is coming in as I would expect. There are no exceptions, and the GAE error console does not even log the 502 error. In the console I can also see..
[CRITICAL] WORKER TIMEOUT (pid:17)
So I went into the logs and see this NGINX error..
textPayload: "[error] 32#32: *4086 upstream prematurely closed connection while reading response header from upstream, client: 130.211.2.231, server: , request: "POST /auth/login_user HTTP/1.1", upstream: "http://172.17.0.1:8080/auth/login_user", host: "blah-191811.appspot.com", referrer: "http://blah-191811.appspot.com/""
I don't think it is an error connecting to my database because I can connect to the Postgres DB hosted on GAE from my local computer. I also have another API endpoint that just uses a google API to retrieve some info and that also produces a 502 error.
I am at my wits end, and honestly about to throw in the towel and try to get it going on AWS if I can't figure this out. Any help would be much appreciated.
EDIT WITH MORE INFO:
The problem still persists. However, I have two routes on my API that require a valid JWT header. When using these routes with an invalid header token, or no token the API returns the correct response, a 401. When sending the correct token the API again returns a 502. This makes me believe that there is nothing wrong with my code, but that somewhere the response is not getting sent back.
I should add that these are not data intensive calls. The login_user is just two strings, an email and a password. The database has only one entry in it.
When I run locally but CONNECT to the remote Postgres database the API works as expected. I.e if I run a Flask server locally and do; 127.0.0.1:5000/auth/login_user and send the correct information, it is able to read back from my Postgres database on my GAE project. So I don't believe the database is the issue either.
Still looking for any wisdom because this seems to be a very common issue with little resolve.

For anyone else experiencing this issue..I found the solution after much digging. It was in the way I was connecting to the potgres SQL instance on GAE.
I was using this;
SQLALCHEMY_DATABASE_URI = 'postgresql://user:password#remote.ip.addr/database_name'
This code works fine when you are accessing the SQL instance from a whitelisted IP. However this code will not work inside an instance that also runs on GAE. You need to change it to this;
SQLALCHEMY_DATABASE_URI = 'postgresql://user:password#/database_name?
host=/cloudsql/database_instance_id'

Related

Django - Wagtail - Heroku: How to troubleshoot intermittent, seemingly random, HTTP 404s?

I'm serving Wagtail pages in a very well tested web app. No errors show up in development.
When in Production (using Heroku and a hobby-tier Postgres db), those pages occasionally return an HTTP 404. If I refresh the browser a couple times, it goes back to serving the Page perfectly again.
Some details:
This happens with multiple pages;
I get emails with the 404 error when this happens, so Django's BrokenLinkEmailsMiddleware is kicking in;
Heroku web Dyno looks healthy based on its metrics;
Heroku Postgres has no logs for the Hobby tier, so I don't know what's going on on the db side;
Server is running behind a Cloudflare proxy server, but I can see the request hitting the origin and returning a 404.
I'm obviously not expecting a solution here since the above info is insufficent for that. But I'm looking for troubleshooting pointers, for example:
If the db has become under-dimensioned (too little RAM, too many simultaneous connections, etc.) - could that result in a Wagtail db query wrongly returning PageNotFound? Or would that return a 500 Server Error?
How can I test DB error-handling locally with Django?
How can I add a full Traceback for the Django 404 error, so I'm able to know exactly what line of code raised it?
What are other potential causes of intermittent HTTP 404 errors?
Any help here is welcome!

How do I get my Python Flask app on an Ubuntu ec2 to connect with a MySQL db with RDS?

I am attempting to get my app to connect to my database on an RDS, also I am using NGINX. When connecting to the ec2 remotely using a terminal I can connect to the database there fine. It is on the public-facing side where the error exists. I followed a guide given to me by a coding school step by step. The initial login/registration page will load, but if I try to create an account or login error I get a 500 Internal service error.
I set up my security group correctly as far as inbound rules go. Are there any outbound rules I need on the ec2 for it to be able to contact the RDS from the public ip?
Thank you
Note: I am not getting graded on this at this point, I already finished the Python stack, this is just something I still want to figure out. I don't like leaving something unfinished. I've taken the time and read many articles, watched videos, gone through AWS documentation and still cannot figure the issue out.
In this particular instance, the issue turned out to be that the database was not located on RDS, it was on the ec2 along with the flask app. I just needed to go into the mysqlconnection.py file and change the host to 'localhost', username to 'root', and the password.
Thanks

How to avoid Cross-Origin Resource Sharing error when hitting a Django API from another server process?

I am building a WebApp that has two separate components:
A "frontend" node.js application server (running on localhost:3099) that web-visitors will visit
A "backend" Django server (running on localhost:3098) that manages my ORM and talks to the database. Web visitors will not interact with this server directly at all. This server simply publishes a RESTful API that will be consumed by the frontend.
I will implement security restrictions that will prevent anyone except the frontend server from accessing the backend's API.
One of the API endpoint the backend publishes looks like this: http://localhost:3098/api/myApi/.
I can successfully hit that API from curl like so: curl -X POST -H "Content-Type: application/json" -d '{"myKey1":"myVal1", "myKey2":"myVal2"}' http://localhost:3098/api/myApi/
However, when I try to hit that same API from my frontend server using Javascript, I get the following error in my browser's console window:
XMLHttpRequest cannot load http://localhost:3098/api/myApi/.
No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'http://localhost:3099' is therefore not allowed access.
To solve this, took the following steps:
I installed django-cors-headers
I added 'corsheaders' to my INSTALLED_APPS
I added 'corsheaders.middleware.CorsMiddleware' to MIDDLEWARE_CLASSES above 'django.middleware.common.CommonMiddleware'
I declared the whitelist: CORS_ORIGIN_WHITELIST = ('localhost', '127.0.0.1',)
However, implementing django-cors-headers seems to have made no difference. I'm still getting the same CORS error. How can I solve this issue?
CORS is port-sensetive. Specification says that
If there is no port component of the URI:
1. Let uri-port be the default port for the protocol given by uri-scheme.
Otherwise:
2. Let uri-port be the port component of the URI.
And
If the two origins are scheme/host/port triples, the two origins are the same if, and only if, they have identical schemes, hosts, and ports.
This means that, with your spec CORS handles your whitelist as localhost:80, 127.0.0.1:80. I believe specifying localhost:3099 should resolve this issue.

Google App Engine and Cloud SQL: Lost connection to MySQL server at 'reading initial communication packet'

I have a Django app on Google App Engine app which is connected to a Google Cloud SQL, using the App Engine authentication.
Most of the time everything works fine, but from time to time the following exception is raised:
OperationalError: (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 38")
According to the docs, this error is returned when:
If Google Cloud SQL rejects the connection, for example, because the IP address your client is connecting from is not authorized.
This doesn't make much sense in my case, because the authentication is done by the App Engine server.
What might cause these sporadic errors?
I had a similar issue and ended up contacting Google for help. They explained it happens when they need to restart or move an instance. If the client instance restarted or was moved to another host server (for various versions) the IP’s won’t match and throw that error. They mentioned that the servers may restart for patches, errors and slow downs causing a similar behavior (be it the same error or similar). The server also moves to try and be closer to the instances to increase response times. If you send a request during the move it will throw errors.
They told me I need to code in retry catches incase that happens, similar to how you handle datastore timeouts. Keeping in mind to build in back off mechanics, sending too many request too quickly after a restart could cause a crash.
How often does this happen?
In our case we had renamed the instances incorrectly inside the code. When we changed back to the correct names everything worked fine. Make sure your Cloud SQL instance is named correctly both inside the Google Cloud Console and within the code you use to access it, and make sure that your Cloud SQL instance allows your Google App Engine instance to connect to it it's Access control.
In my case the issue was caused my expired server SSL certificate on the CloudSQL instance. Strangely it was not shown in the Google Cloud Console and figured it out after downloading the certificate and decoding it with openssl (openssl x509 -in server-ca.pem -text -noout).
I was able to figure out cause of the problem after trying to connect with cloud_sql_proxy; luckily it gave more meaningful error message couldn't connect to "...": x509: certificate has expired or is not yet valid.
Connection from AppEngine Standard application started to work immediately after reseting SSL configuration from Google Cloud Console. I noticed that after reset validity date appeared on the console.
I had this problem too using Django 1.10 and GAE. The application worked fine locally (connecting the cloud sql via cloud_sql_proxy), but I'd get the 38 error when using the GAE instance of the application.
My problem turned out to be my database user. The user had a hyphen in it. Once I created a new user without a hyphen and changed my application to use the new user, the GAE instance of the application worked if

Google App Engine 413 error (Request Entity Too Large)

I've implemented an app engine server in Python for processing html documents sent to it. It's all well and good when I run it locally, but when running off the App engine, I get the following error:
"413. That’s an error. Your client issued a request that was too large. That’s all we know."
The request is only 155KB, and I thought the app engine request limit was 10MB. I've verified that I haven't exceeded any of the daily quotas, so anyone know what might be going on?
Thanks in advance!
-Saswat
Looks like it was because I was making a GET request. Changing it to POST fixed it.

Categories