Every request hits server twice in Appengine - python

I have small app in appengine. Every request hits the server twice in my local dev server.
In debug mode i can see that the get method called twice.
Its happening in both chrome and firefox browsers.
here is the log from my localserver..
INFO 2011-03-12 00:44:31,076 dev_appserver.py:3317] "GET /movie/tanu_weds_manu/rating-review-video HTTP/1.1" 200 -
INFO 2011-03-12 00:44:32,345 dev_appserver.py:3317] "GET /css/fb.css HTTP/1.1" 200 -
INFO 2011-03-12 00:44:32,588 dev_appserver.py:3317] "GET /css/wianga-movie.0.1.css HTTP/1.1" 200 -
INFO 2011-03-12 00:45:46,648 dev_appserver.py:3317] "GET /movie/tanu_weds_manu/rating-review-video HTTP/1.1" 200 -
INFO 2011-03-12 00:45:46,911 dev_appserver.py:3317] "GET /img/wianga-fb-50.gif HTTP/1.1" 200 -
INFO 2011-03-12 00:45:47,177 dev_appserver.py:3317] "GET /img/arrow_green.gif HTTP/1.1" 200 -
INFO 2011-03-12 00:45:47,470 dev_appserver.py:3317] "GET /image/movie/tanu_weds_manu HTTP/1.1" 200 -
INFO 2011-03-12 00:45:47,717 dev_appserver.py:3317] "GET /js/jquery.qtip-1.0.0-rc3.min.js HTTP/1.1" 200 -
INFO 2011-03-12 00:45:47,970 dev_appserver.py:3317] "GET /js/wianga.0.1.js HTTP/1.1" 200 -
INFO 2011-03-12 00:46:37,473 dev_appserver.py:3317] "GET /movie/tanu_weds_manu/rating-review-video HTTP/1.1" 200 -
Updated: Its getting worse,i cheked the logs in app console.There it hits more than 3 times for a request.
handlers:
- url: /favicon.ico
static_files: static/img/favicon.ico
upload: static/img/favicon.ico
- url: /robots.txt
static_files: static/robots.txt
upload: static/robots.txt
- url: /img
static_dir: static/img
- url: /images
static_dir: static/images
- url: /css
static_dir: static/css
- url: /js
static_dir: static/js
- url: /image/.*
script: /wianga/pages/common/ImageController.py
- url: /task/.*
script: /wianga/pages/task/TaskController.py
- url: /browse/.*
script: /wianga/site/Browse.py
- url: /movie/(.*)/rating-review-video/
script: /wianga/site/MoviePage.py
- url: /movie/(.*)/rating-review-video
script: /wianga/site/MoviePage.py
- url: /404
script: /wianga/404/404.py
- url: /api/.*
script: /wianga/api/ApiController.py
- url: /.*
script: /wianga/site/Home.py
Am getting same behavior in hellowrld application from appengine turorial...
App.yaml
version: 1
runtime: python
api_version: 1
handlers:
- url: /.*
script: helloworld.py
Log for hello world example for single request...
INFO 2011-03-12 06:08:06,299 dev_appserver_main.py:507] Running application wianga-movie on port 8080: http://localhost:8080
INFO 2011-03-12 06:08:12,506 dev_appserver.py:3317] "GET / HTTP/1.1" 200 -
INFO 2011-03-12 06:08:13,721 dev_appserver.py:3317] "GET / HTTP/1.1" 200 -
INFO 2011-03-12 06:08:13,838 dev_appserver.py:3317] "GET /favicon.ico HTTP/1.1" 200 -
INFO 2011-03-12 06:08:13,953 dev_appserver.py:3317] "GET / HTTP/1.1" 200 -

When I set up my app.yaml I just poured through this page: http://code.google.com/appengine/docs/python/config/appconfig.html
You have to think like a regular expression parser. Anything that fails to match the preceding expressions will fall through to your wildcard handler and cause weirdness.
One thing I notice is that you should add login: admin to your task queue handler. This will prevent external requests from hitting that URL accidentally.
Also, when you are submitting tasks to your handler, if you don't include the trailing slash (/task?blah=something instead of /task/?blah=something) the request would fall through to your wildcard handler and get sent to Home.py. That could also happen for /image/, /browse/ and /api/. You should put logging calls into those handlers to make sure they are getting called when expected.
Nothing jumps out at me as obviously wrong in your app.yaml though. But since there are GETs showing up in your log, that indicates that something is requesting that URL, and it's not just the get method getting called twice internally.
Edit:
Hold on a sec, just noticed that you have this in your app.yaml twice. You shouldn't need to do that.
- url: /movie/(.*)/rating-review-video/
script: /wianga/site/MoviePage.py
- url: /movie/(.*)/rating-review-video
script: /wianga/site/MoviePage.py`
Try replacing with:
- url: /movie/(.*)/rating-review-video.*
script: /wianga/site/MoviePage.py`
Edit2:
Try adding this to the top of your get method:
logging.info('environ: ' + str(self.request._environ))
Pop the output into a beautifier ( http://jsbeautifier.org/ ) to make it more readable, and take a look at the REQUEST_METHOD, QUERY_STRING, HTTP_USER_AGENT, HTTP_COOKIE, PATH_INFO, and HTTP_REFERER params to see where the requests are actually coming from. If they are all coming from your browser then there is something weird in your html, like hidden iframes (are you using Channel API?).
Also, try going to http://localhost:8080/_ah/admin/queues and making sure there aren't any old tasks hanging around.

Related

Flask is loading static files from a different path when a route is called [duplicate]

This question already has answers here:
Link to Flask static files with url_for
(2 answers)
Why use Flask's url_for?
(1 answer)
Closed 3 years ago.
I have 2 route functions, one to get all users and one to get a specific user. Both functions render the same template. The first function works fine, the problem is with the second one. When rendering the template it tries to load static files from another directory.
I tried using a different template for each one and the problem remained.
Get all users route:
#app.route('/users')
def list_users():
users = Users.query.all()
return render_template('users.html', users=users)
Get one user route:
#app.route('/users/<username>')
def get_user(username):
user = Users.query.filter_by(name=username).first()
if user:
return render_template('users.html', users=[user])
Template rendered by get_user route function:
127.0.0.1 - - [07/Aug/2019 13:35:56] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [07/Aug/2019 13:35:57] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/Ana HTTP/1.1" 200 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/vendor/fontawesome-free/css/all.min.css HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/css/sb-admin-2.min.css HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/img/uatronica_black_transparent.png HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/vendor/jquery/jquery.min.js HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/vendor/bootstrap/js/bootstrap.bundle.min.js HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/vendor/jquery-easing/jquery.easing.min.js HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/js/sb-admin-2.min.js HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/vendor/chart.js/Chart.min.js HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/js/demo/chart-area-demo.js HTTP/1.1" 404 -
127.0.0.1 - - [07/Aug/2019 13:36:06] "GET /users/static/js/demo/chart-pie-demo.js HTTP/1.1" 404 -
It is trying to load css files from /users/static/ instead /static/. Why is that?
You are using relative paths in your template, so the final URL is relative to whatever URL you're at. When viewing /users, if the template links to static/css/admin.css, it becomes /users/static/css/admin.css. If the path starts with a /, it is an absolute URL and won't do this.
Instead, use url_for, which generates absolute URLs no matter where you are and how the app is deployed.
<link rel="stylesheet" href="{{ url_for('static', filename='css/admin.css') }}>
This becomes /static/css/admin.css.
Please use the below code.
Get one user route:
#app.route('/users/<username>')
def get_user(username):
user = Users.query.filter_by(name=username).first()
return render_template('users.html', users=user)
Additionally please share the html page users.html for review.

websocket on port 443 with /socket.io under kubernetes not working

Trying to move my development environment to run on minikube.
The page loads but my page uses websockets on the same port/protocol that the index.html is loaded with (https in this case), and the websockets do no seem to be working correctly.
Here is an example of the correct output when run through nginx / python on my local development box.
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET / HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /static/jquery.min.js HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /static/socket.io.min.js HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /socket.io/?EIO=3&transport=polling&t=MNPIg-N HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /favicon.ico HTTP/1.0" 404 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /favicon.ico HTTP/1.0" 404 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "POST /socket.io/?EIO=3&transport=polling&t=MNPIg-o&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /socket.io/?EIO=3&transport=polling&t=MNPIg-r&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "POST /socket.io/?EIO=3&transport=polling&t=MNPIg_x&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /socket.io/?EIO=3&transport=polling&t=MNPIg_w&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:40] "GET /socket.io/?EIO=3&transport=polling&t=MNPIh0L&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:45] "GET /socket.io/?EIO=3&transport=polling&t=MNPIiE3&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:50] "GET /socket.io/?EIO=3&transport=polling&t=MNPIjSI&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:55] "GET /socket.io/?EIO=3&transport=polling&t=MNPIkgS&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
Notice how there is a GET every 5 seconds (that's a timer running on the page)
When running on Kubernetes, The page loads and the timer shows up as if the websocket has worked, however I show no logs where the websocket is having a GET or POST after the initial one.
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:03] "GET /static/jquery.min.js HTTP/1.1" 304 1210 0.008244
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:03] "GET /static/socket.io.min.js HTTP/1.1" 304 1210 0.009271
(10) accepted ('172.17.0.7', 34444)
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "GET /socket.io/?EIO=3&transport=polling&t=MNPKrsy HTTP/1.1" 200 379 0.003682
(10) accepted ('172.17.0.7', 34446)
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "GET /favicon.ico HTTP/1.1" 404 1314 0.004694
(10) accepted ('172.17.0.7', 34448)
(10) accepted ('172.17.0.7', 34450)
(10) accepted ('172.17.0.7', 34452)
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "GET /socket.io/?EIO=3&transport=polling&t=MNPKrtD&sid=77d4755c524f47c2948b9c36da007b85 HTTP/1.1" 200 210 0.000749
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "POST /socket.io/?EIO=3&transport=polling&t=MNPKrtB&sid=77d4755c524f47c2948b9c36da007b85 HTTP/1.1" 200 194 0.002632
(10) accepted ('172.17.0.7', 34454)
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "GET /favicon.ico HTTP/1.1" 404 1314 0.002388
The Ingress is setup as follows:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: websitev2-cluster-ip-service
servicePort: 8080
As mentioned before, the websocket does not reside on a different port and it is instanciated in javascript as:
namespace = '/socket';
var socket = io.connect(location.protocol + '//' + document.domain + ':' + location.port + namespace);
Are there any special requirements to get websockets to work? I do not believe i need a special route because the URI needs to be sent to the same location as everything else, and on the same port.
EDIT: MORE DETAILS
My website has a form, that when submitted, executes the following code:
$('form#job').submit(function(event) {
var nameValue = JSON.stringify($(event.target).serializeArray());
console.log(nameValue)
socket.emit('job_submit', {data: nameValue});
return false;
});
On the python side, I have my socket code, which should get hit once a user clicks the submit button.
#socketio.on('job_submit', namespace='/socket')
def job_submit(message):
print('recieved job_submit message from client')
# kick off subjob in celery task
data = unstringify(message)
print('data = {0}'.format(data))
sessiondata = dict(session)
print('sessiondata = {}'.format(sessiondata))
subjobstart.delay(sessiondata, request.sid, data)
In my logs I am not receiving any "recieved job_submit message from client" messages when the submit button is clicked, which means the javascript is trying to send a websocket emit to the python server, but the message is never getting there.
Emits from python to the client page are working as the time is getting updated on the site.
So not exactly sure what changed as it worked on test box, apparently the issue lied in the form validation I was doing. Once I put in a novalidate option to temporarily bypass the form checking, the form was then able to be submitted and I did not have any websocket issues after all.
Hopefully this post can help someone with the code posted instead.

Flask - Having more than one static folder (the static folders are actually dynamic)

Im currently working on a project that has the following structure:
ROOT
-js
-dist
-index.html
-bunch of other files that index.html uses
-src
-main.py
-doc_apps
-app1
-index.html
-other resources that index.html needs
-app2
-index.html
-other resources that index.html needs
Now my flask configuration is:
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
COMPONENT_FOLDER = os.path.abspath(os.path.join(ROOT_DIR, '../'))
DIST_FOLDER = os.path.join(COMPONENT_FOLDER, 'js\\dist')
DOCUMENTATION_FOLDER = os.path.join(COMPONENT_FOLDER, 'documentation_data')
app = Flask(__name__, static_folder=DIST_FOLDER, template_folder=DIST_FOLDER, static_url_path='')
my_loader = jinja2.ChoiceLoader([
app.jinja_loader,
jinja2.FileSystemLoader(DOCUMENTATION_FOLDER),
])
app.jinja_loader = my_loader
#app.route('/')
def index():
"""
This the main entry to the application
Returns
-------
out
renders the 'index.html' file
"""
return render_template('index.html')
#app.route('/get_index_html', methods=['GET'])
def get_index_html():
index_html_path = request.args.get('path')
full_root = os.path.join(index_html_path).replace('\\','/')
return render_template(full_root)
When I start my app and go to the main page im able to render the index.html (the one in the js/dist folder) and everything works great
the problem starts when I try to render one of the inner index.html.
Im able to render the index.html by itself but I get the following errors
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/css/theme.css HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/pygments.css HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/js/modernizr.min.js HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/jquery.js HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/underscore.js HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/doctools.js HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/js/theme.js HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/jquery.js HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:43] "GET /_static/underscore.js HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:44] "GET /_static/doctools.js HTTP/1.1" 404 -
127.0.0.1 - - [02/Aug/2018 18:08:44] "GET /_static/js/theme.js HTTP/1.1" 404 -
I can tell that the problem is that it cant find the static files that are needed in order to generate this index.html correctly.
How can I tell it that for this specific index.html use some_static_folder?
I was wondering if there is something like render(html, static_folder=...)
I would like to mention that each one of the App1/index.html
has its own links and those links rely on the inner structure of App1, so how can I configure that when I send App1/index.html it will use some static_folder for everything that will be invoked from that App1/index.html
Some more info, App1/2 can be added also after the server is already running.
thank you very much for your help

Flask will only log if debug is true

I have recently created a flask app that streams a video to my local network. This is on a raspberry pi. I have attempted to make a log file using this code::
import logging
logging.basicConfig(filename='error.log',level=logging.DEBUG)
When I run the flask app with app.run(), the app will not log anything to the error.log file. But, when I run it with debug=true , then the app logs all info. The error.log file when I run it looks like this:
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:37:40] "GET /video_feed HTTP/1.1" $
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:37:40] "GET /static/css/bootstrap.$
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:39:45] "GET /video_feed HTTP/1.1" $
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:08] "GET / HTTP/1.1" 200 -
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:08] "GET /video_feed HTTP/1.1" $
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:09] "GET /logs HTTP/1.1" 200 -
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:10] "GET /invalidpass HTTP/1.1"$
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:21] "GET /logs HTTP/1.1" 200 -
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:29] "GET / HTTP/1.1" 200 -
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:30] "GET /video_feed HTTP/1.1" $
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:31] "GET /logout HTTP/1.1" 302 -
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:31] "GET /welcome HTTP/1.1" 302$
INFO:werkzeug:10.0.0.127 - - [10/Feb/2018 16:40:31] "GET /login HTTP/1.1" 200 -
I was wondering if it is possible to not have the Debug on when I run my flask app, and still log all information to the error.log file. I have tried switching the level to logging.INFO but that has not worked.
Try setting the logging level to error.
logging.basicConfig(filename='error.log',level=logging.ERROR)
You are setting the log level for the root logger but as mentioned in the flask logging docs flask configures its own logger in the 'flask' logging namespace.
The application logger is available as a property on the application object. Do
app = Flask(__name__)
app.logger.setLevel(logging.DEBUG)
if you'd like to always show logs at DEBUG level.
I couldn't figure out which way I wanted to go with logging that way, so I decided to create my own logging instance, and then log any info I wanted to it, which seemed to work. Although this is a longer and manual process, it let me control what was logged and how it was logged. The code looked similar to this::
#Import requests from flask to log things such as form data and other requests made to the server
from flask import request
#import python logging for logging to files
import logging
from logging.handlers import RotatingFileHandler
#How I logged the information I wanted to log
logInfo = 'Info on Login: ' + 'IP: ' + request.remote_addr
app.logger.info(logInfo)
#Create the logging instance test
handler = RotatingFileHandler('error.log', maxBytes=10000, backupCount=1)
handler.setLevel(logging.INFO)
app.logger.addHandler(handler)
app.logger.info('INFO TEST')

I'm getting a 404 in google cloud endpoints on app engine

In my app, the app.yaml (the relevant part) looks like this:
handlers:
- url: /favicon\.ico
static_files: favicon.ico
upload: favicon\.ico
- url: /logs
script: logviewer.main.app
- url: /static
static_dir: static
- url: /(.*\.html)
static_files: pages/\1
upload: pages/(.*\.html)
- url: /_ah/spi/.*
script: api.application
- url: .*
script: main.app
I've included all handlers, just to make sure, but I'm quite certain that app.yml isn't the problem.
The relevant part of api.py
#endpoints.api(name='quizservice',version='v01',
description='api backand for quickbuzzer')
class QuizService(remote.Service):
#endpoints.method(messages.VoidMessage , messages.CreateQuizResponse, name="createQuiz")
def createQuiz(self, request):
. . .
application = endpoints.api_server([QuizService],
restricted=False)
Now, when I visit the explorer and try running the QuiizService.createQuiz method, I get a 404 back.
Looking at the logs, I see this:
INFO 2013-04-29 17:53:15,560 server.py:561] default: "GET /_ah/api/discovery/v1/apis/quizservice/v01/rest HTTP/1.1" 200 2738
INFO 2013-04-29 17:53:22,118 server.py:561] default: "POST /_ah/spi/BackendService.getApiConfigs HTTP/1.1" 200 1585
WARNING 2013-04-29 17:53:22,119 api_config_manager.py:201] No endpoint found for path: quizservice/v01
INFO 2013-04-29 17:53:22,119 server.py:561] default: "POST /_ah/api/quizservice/v01 HTTP/1.1" 404 9
I was able to solve the issue by supplying a path parameter to the endpoints.method decorator. What I'm wondering now, is if the endpoints api could choose a default path based on my method name.
Another case where this error can be thrown is with incorrect order of url handlers declaration.
See https://stackoverflow.com/a/15675839/362953
- url: .*
script: main.app
Should come at the end, not before
- url: /_ah/spi/.*
script: api.application
In this case of OP the order is correct.

Categories