After a few days of searching and looking around for this, and I haven't been able to find the right answer.
I'm trying to run a cron job on Google App Engine (with Python). The cron job itself isn't that important, I'm just looking to run a python script every minute. For now I'm just trying to add a line with the current time in a separate text file (test.txt).
I'm pretty sure that I don't quite understand the concept of handlers, and that's what causing me problems. But I've spent hours in the documentation, and I still can't figure it out.
I sense that I should not be using main.py as my script for the cron job, but I have a hard time understanding what the url needs to be in cron.yaml, and what the handler/script should be.
Please help!
app.yaml
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
python_version: 3
handlers:
- url: /main
script: main.py
cron.yaml
cron:
- description : most recent test
url : /main
schedule: every 1 minutes
main.py
# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# [START app]
import logging
from datetime import datetime
current_time = str(datetime.now())
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello():
"""Return a friendly HTTP greeting."""
return 'Hello World! The time is ' + current_time
message = "this worked, as of " + current_time + "\n"
with open("test.txt", "a") as myfile:
myfile.write(message)
#app.errorhandler(500)
def server_error(e):
logging.exception('An error occurred during a request.')
return """
An internal error occurred: <pre>{}</pre>
See logs for full stacktrace.
""".format(e), 500
if __name__ == '__main__':
# This is used when running locally. Gunicorn is used to run the
# application on Google App Engine. See entrypoint in app.yaml.
app.run(host='127.0.0.1', port=8080, debug=True)
# [END app]
You don't have a url handler for /main. Try this:
in app.yaml:
handlers:
- url: /.* # wildcard. every url goes there
script: main.py
in main.py:
from flask import Response
#app.route('/main')
def main():
"""Return a friendly HTTP greeting."""
Response("Hello main viewer", mimetype='text/plain')
Note: Another bug in your code is that you return the html view BEFORE the script gets a chance to write the .txt file. It will never get that far.
Your app.yaml file is mixing up the standard environment Handlers element into a flexible environment configuration, so it is probably ignored.
Your cron.yaml configuration shows a /main URL for your cron job, but your app doesn't seem to have a route for such URL path, it only seems to handle the / path. I'd expect you'd see some 404 errors in the logs for the /main cron requests.
You should add a route for the /main path in your main.py. Or replace /main with / in your cron.yaml.
Side note: you have some non-executable code (it follows the return statement) inside hello() - just in case you're looking for that test.txt file...
Related
I have a Flask API and I am attempting to implement logging. Everything is working fine in my local machine, but nothing seems to work when deployed on AWS Elastic Beanstalk.
I have a file logger.py that creates and configures the logger that looks like this:
import logging
import os
if not os.path.isdir(os.environ.get('LOG_PATH', 'log')):
os.mkdir(os.environ.get('LOG_PATH', 'log'))
# Configure logger
logger = logging.getLogger(__name__)
formatter = logging.Formatter('[%(asctime)s] %(levelname)s in %(module)s: %(message)s')
debug_file_handler = logging.FileHandler(f'{os.environ.get("LOG_PATH", "log")}/debug.log')
info_file_handler = logging.FileHandler(f'{os.environ.get("LOG_PATH", "log")}/info.log')
error_file_handler = logging.FileHandler(f'{os.environ.get("LOG_PATH", "log")}/error.log')
debug_file_handler.setLevel(logging.DEBUG)
info_file_handler.setLevel(logging.INFO)
error_file_handler.setLevel(logging.ERROR)
debug_file_handler.setFormatter(formatter)
info_file_handler.setFormatter(formatter)
error_file_handler.setFormatter(formatter)
logger.addHandler(debug_file_handler)
logger.addHandler(info_file_handler)
logger.addHandler(error_file_handler)
I import this file int my app factory and add the handlers to my app.
# Configure logger
app.logger.removeHandler(default_handler)
app.logger.addHandler(debug_file_handler)
app.logger.addHandler(info_file_handler)
app.logger.addHandler(error_file_handler)
Then in the app factory still, I have a #app.before_request so that logging should occur on every request (for testing purposes).
#app.before_request
def before_request():
elif request.method == 'GET':
app.logger.debug(f'{request.method} {request.base_url}: parameters {dict(request.args)}')
else:
app.logger.debug(f'{request.method} {request.base_url}: parameters {request.json}')
This all works fine on my local machine without a problem. However when I try to deploy to AWS EB, nothing is written to the log files. The log files appear to be created and present when I pull them down via the cli or gui, but nothing is being written to it. I've followed a few tutorials and have added to my .ebextensions file but am still having no luck. Currently my .ebextensions includes a logging.config that looks like this:
files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/applogs.conf" :
mode: "000777"
owner: root
group: root
content: |
/tmp/*.log
"/opt/elasticbeanstalk/tasks/taillogs.d/applogs.conf" :
mode: "000777"
owner: root
group: root
content: |
/tmp/*.log
The log files appear to be included when I request the logs (both tail and full), but the logs are completely empty even after making multiple request. I've also seen a few answers on here saying I need to include chown wsgi:wsgi in this file, but that has not worked either. I've also tried writing the logs to var/app/current/log and that doesn't work either. The log files are always created, but not written to.
For context, the Elastic Beanstalk platform is Python 3.8 running on Amazon Linux 2.
Any help would be greatly appreciated.
Turns out all I needed to do was set the level of the logger to logging.DEBUG so I could see debug logs.
in __init__.py:
app.logger.setLevel(logging.DEBUG)
And since my app name is src when I configure it, I can grab this logger in other modules with:
import logging
logger = logging.getLogger('src')
I have been trying to run a cron job with GAE (code developed in Python), but when I trigger the job, it fails without any error message -- I can't find anything at all in the logs.
This is happening for a service for which I'm using the flexible environment.
This is the structure of my files:
my_service.yaml looks like this:
service: my_service
runtime: custom
env: flex
env_variables:
a:
b:
the my_service.app looks like this:
from __future__ import absolute_import
from flask import Flask
from flask import request
import logging
import datetime
import os
import tweepy
from google.cloud import datastore
import time
logging.basicConfig(level=logging.INFO)
app = Flask(__name__)
#app.route('/Main')
def hello():
"""A no-op."""
return 'nothing to see.'
#app.route('/my_service')
def get_service():
is_cron = request.headers.get('X-Appengine-Cron', False)
logging.info("is_cron is %s", is_cron)
# Comment out the following test to allow non cron-initiated requests.
if not is_cron:
return 'Blocked.'
## data scraping and saving in Datastore
return 'Done.'
#app.errorhandler(500)
def server_error(e):
logging.exception('An error occurred during a request.')
return """
An internal error occurred: <pre>{}</pre>
See logs for full stacktrace.
""".format(e), 500
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080, debug=True)
Then I have a dispatch.yaml with this structure:
dispatch:
- url: "*/my_service*"
service: my_service
And a cron.yaml:
cron:
- description: run my service
url: /my_service
schedule: 1 of month 10:00
target: my_service
Not sure what I'm doing wrong here.
EDIT
A bit of context. This is something I'm editing starting from this repo.
The service called backend that is defined in there works perfectly (it has also the same schedule in the cron job as my_service but when I trigger it in a day different from the one in which it's scheduled, it works just fine). What I did was to create an additional service with its own yaml file, which looks exactly the same as the beckend.yaml, its own my_service.py and adding it to the dispacth.yamland the cron.yaml. In theory this should work, since the structure is exactly the same, but it doesn't.
This service was originally developed in the standard environment and there it was working, the problem originated when I moved it to the flex environment.
EDIT 2:
The problem was actually in the Dockerfile, that was calling a service that I was not using.
EDIT:
def get(self): may have some issues.
First, get may be reserved. Second, you aren't able to send self to that function. Change that to:
def get_service():
EDIT2:
You also need to import logging at the top of any page that uses it. And, you have not imported Flask and its components:
from flask import Flask, request, render_template # etc...
import logging
Your 1 of month 10:00 cron schedule specification can most likely be the culprit: it specifies to run the job at 10:00 only on the first day of each month! From Defining the cron job schedule:
Example: For the 1,2,3 of month 07:00 schedule, the job runs one
time at 07:00 on the first three days of each month.
So the last execution happened 3 days ago (if this cron config was deployed at the time) and no other attempt will happen until Nov 1st :)
Change the schedule to something easier to test with, like every 5 minutes or every 1 hours and revert the change once you're happy it works as expected.
I have a Python Cron job that, according to the Logs, runs to completion without errors. However, none of the "logging.error()" messages I have included in the code are included in the log and none of the required processing is done.
So that I can run this manually, I have a link in my HTML menu "Assign Rental Payments Due" that does the processing required and logs error logging messages correctly.
----
Section of app.yaml
- url: /rhrentassign.html
script: frhrentassign.app
----
Full cron.yaml
cron:
- description: Rental Payments Due
url: /rhrentassign
schedule: every day 14:00
----
Full python code (file is frhrentassign.py)
import os
import logging
import webapp2
from CronRH import *
class rhrentassignhandler(webapp2.RequestHandler):
def get(self):
swork = trhrenttopaycron()
swork.allnamespaces()
app = webapp2.WSGIApplication([('/rhrentassign.html', rhrentassignhandler)], debug=True)
----
Any thoughts on what I have done wrong would be most appreciated.
Many Thanks, David
Your handler is mapped to '/rhrentassign.html', but the cron is going to '/rhrentassign'.
Generally unless you have a very very good reason, there's no need to put 'html' in route names.
I'm trying to set up python and flask on the arduino yun. I've managed to run python files via the /etc/config/uhttpd configuration file:
...
list interpreter ".py=/usr/bin/python"
...
The default path for the website's root is: /www in which I've placed a soft link (apps) to the sd card. So now I can run python programs: http://[ip arduino]/apps/helloworld.py
And when I make my first helloflask.py program and run that via python helloflask.py I can see the result at: http://[ip arduino]:5000
But now I want to configure the uhttpd mini webserver (which is capable to exchange information via CGI) to use the flask setup. The URI: http://flask.pocoo.org/docs/deploying/cgi/#server-setup shows some instructions... but I just don't get it. I've made a directory ../apps/uno in which I've placed a __init__.py file with the following content:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "He Flask!"
In the apps dir I've put a file: cgi.py with this content:
from wsgiref.handlers import CGIHandler
from uno import app
CGIHandler().run(app)
Now I when I browse: http://[ip arduino]/cgi.py get a server error occured, contact the administrator (I think this is the CGI interface from uhttpd).
I just don't grasp the CGI configuration for Flask/uhttpd
I looked into this too and got a little further, I was able to setup a simple hello world but once I tried to do something non-trivial I ran into a big issue that uhttpd doesn't support URL rewriting/aliasing. This means your flask app can only be served at the URL of its .py file instead of at a root like http:// (arduino IP) /flaskapp/. None of the routes inside the app will be visible and makes the whole thing unusable.
However, instead of trying to force flask into uhttpd I had great success running the built in server that flask provides. Take a look at this guide I wrote up that uses flask to serve data from a Yun: https://learn.adafruit.com/smart-measuring-cup/overview
The thing to do is add a call to app.run when the script is run, for example make your flask app look like:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello Flask!"
if __name__ == '__main__':
app.run(host='0.0.0.0', debug=True, threaded=True)
Then log in to the Yun and run the script using python. Flask's built in server should start serving the app on http:// (arduino IP) :5000/. Make sure to include the host='0.0.0.0' as it's required to listen on the Yun's external network interface. You probably also want debug=True so there are better error messages (and live reloading of the server when the code changes), and I found threaded=True helps because the default server only handles one connection at a time. The Yun is a relatively slow processor so don't expect to service a lot of concurrent requests, however it's quite capable for providing a simple REST API or web application for a few users.
If you want this server to always run on bootup, edit the /etc/rc.local file to include a call to python and your script.
I have never written a python script in my life, but I have a question that can hopefully be solved pretty quickly...
I'm using Google App Engine and Dropbprox. The script uses a custom domain to point to your public DropBox folder for better DropBox URLs. I'd like to be able to redirect users to my main site (jacob.bearce.me) if they visit my dropbox url (dl.bearce.me).
The problems that I'm having:
I've never used GAE or Python before, so I have no idea where to even begin
Putting a index.html file in my GAE project didn't fix it (I was hoping it'd just default to that if there was no filename specified, like it would on a normal site, but no cigar.)
Just a simple redirect if a users visits the main URL is all I'm after, nothing fancy.
My Python file: http://dl.bearce.me/mirror.py
Here's a main.py that issues a redirect for all requests, using the Python 2.5 runtime environment:
from google.appengine.ext import webapp
from google.appengine.ext.webapp import util
class MainHandler(webapp.RequestHandler):
def get(self):
self.redirect('http://jacob.bearce.me/')
application = webapp.WSGIApplication([('/.*', MainHandler)],
debug=True)
def main():
util.run_wsgi_app(application)
if __name__ == '__main__':
main()
And here's the app.yaml file you need to route URLs to this handler:
application: myapp
version: 1
runtime: python
api_version: 1
handlers:
- url: .*
script: main.py
(Replace myapp with your actual app ID.)
For more information about creating and uploading an App Engine app in Python, see the Getting Started tutorial.