I have been trying to run a cron job with GAE (code developed in Python), but when I trigger the job, it fails without any error message -- I can't find anything at all in the logs.
This is happening for a service for which I'm using the flexible environment.
This is the structure of my files:
my_service.yaml looks like this:
service: my_service
runtime: custom
env: flex
env_variables:
a:
b:
the my_service.app looks like this:
from __future__ import absolute_import
from flask import Flask
from flask import request
import logging
import datetime
import os
import tweepy
from google.cloud import datastore
import time
logging.basicConfig(level=logging.INFO)
app = Flask(__name__)
#app.route('/Main')
def hello():
"""A no-op."""
return 'nothing to see.'
#app.route('/my_service')
def get_service():
is_cron = request.headers.get('X-Appengine-Cron', False)
logging.info("is_cron is %s", is_cron)
# Comment out the following test to allow non cron-initiated requests.
if not is_cron:
return 'Blocked.'
## data scraping and saving in Datastore
return 'Done.'
#app.errorhandler(500)
def server_error(e):
logging.exception('An error occurred during a request.')
return """
An internal error occurred: <pre>{}</pre>
See logs for full stacktrace.
""".format(e), 500
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080, debug=True)
Then I have a dispatch.yaml with this structure:
dispatch:
- url: "*/my_service*"
service: my_service
And a cron.yaml:
cron:
- description: run my service
url: /my_service
schedule: 1 of month 10:00
target: my_service
Not sure what I'm doing wrong here.
EDIT
A bit of context. This is something I'm editing starting from this repo.
The service called backend that is defined in there works perfectly (it has also the same schedule in the cron job as my_service but when I trigger it in a day different from the one in which it's scheduled, it works just fine). What I did was to create an additional service with its own yaml file, which looks exactly the same as the beckend.yaml, its own my_service.py and adding it to the dispacth.yamland the cron.yaml. In theory this should work, since the structure is exactly the same, but it doesn't.
This service was originally developed in the standard environment and there it was working, the problem originated when I moved it to the flex environment.
EDIT 2:
The problem was actually in the Dockerfile, that was calling a service that I was not using.
EDIT:
def get(self): may have some issues.
First, get may be reserved. Second, you aren't able to send self to that function. Change that to:
def get_service():
EDIT2:
You also need to import logging at the top of any page that uses it. And, you have not imported Flask and its components:
from flask import Flask, request, render_template # etc...
import logging
Your 1 of month 10:00 cron schedule specification can most likely be the culprit: it specifies to run the job at 10:00 only on the first day of each month! From Defining the cron job schedule:
Example: For the 1,2,3 of month 07:00 schedule, the job runs one
time at 07:00 on the first three days of each month.
So the last execution happened 3 days ago (if this cron config was deployed at the time) and no other attempt will happen until Nov 1st :)
Change the schedule to something easier to test with, like every 5 minutes or every 1 hours and revert the change once you're happy it works as expected.
Related
I am trying to build REST API with only one call.
Sometimes it takes up to 30 seconds for a program to return a response. But if user thinks that service is lagging - he makes a new call and my app returns response with error code 500 (Internal Server Error).
For now it is enough for me to block any new requests if last one is not ready. Is there any simple way to do it?
I know that there is a lot of queueing managers like Celery, but I prefer not to overload my app with any large dependencies/etc.
You could use Flask-Limiter to ignore new requests from that remote address.
pip install Flask-Limiter
Check this quickstart:
from flask import Flask
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address
app = Flask(__name__)
limiter = Limiter(
app,
key_func=get_remote_address,
default_limits=["200 per day", "50 per hour"]
)
#app.route("/slow")
#limiter.limit("1 per day")
def slow():
return "24"
#app.route("/fast")
def fast():
return "42"
#app.route("/ping")
#limiter.exempt
def ping():
return "PONG"
As you can see, you could ignore the remote IP address for a certain amount of time meanwhile you finish the process you´re running
DOCS
Check these two links:
Flasf-Limiter Documentation
Flasf-Limiter Quick start
After a few days of searching and looking around for this, and I haven't been able to find the right answer.
I'm trying to run a cron job on Google App Engine (with Python). The cron job itself isn't that important, I'm just looking to run a python script every minute. For now I'm just trying to add a line with the current time in a separate text file (test.txt).
I'm pretty sure that I don't quite understand the concept of handlers, and that's what causing me problems. But I've spent hours in the documentation, and I still can't figure it out.
I sense that I should not be using main.py as my script for the cron job, but I have a hard time understanding what the url needs to be in cron.yaml, and what the handler/script should be.
Please help!
app.yaml
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
python_version: 3
handlers:
- url: /main
script: main.py
cron.yaml
cron:
- description : most recent test
url : /main
schedule: every 1 minutes
main.py
# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# [START app]
import logging
from datetime import datetime
current_time = str(datetime.now())
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello():
"""Return a friendly HTTP greeting."""
return 'Hello World! The time is ' + current_time
message = "this worked, as of " + current_time + "\n"
with open("test.txt", "a") as myfile:
myfile.write(message)
#app.errorhandler(500)
def server_error(e):
logging.exception('An error occurred during a request.')
return """
An internal error occurred: <pre>{}</pre>
See logs for full stacktrace.
""".format(e), 500
if __name__ == '__main__':
# This is used when running locally. Gunicorn is used to run the
# application on Google App Engine. See entrypoint in app.yaml.
app.run(host='127.0.0.1', port=8080, debug=True)
# [END app]
You don't have a url handler for /main. Try this:
in app.yaml:
handlers:
- url: /.* # wildcard. every url goes there
script: main.py
in main.py:
from flask import Response
#app.route('/main')
def main():
"""Return a friendly HTTP greeting."""
Response("Hello main viewer", mimetype='text/plain')
Note: Another bug in your code is that you return the html view BEFORE the script gets a chance to write the .txt file. It will never get that far.
Your app.yaml file is mixing up the standard environment Handlers element into a flexible environment configuration, so it is probably ignored.
Your cron.yaml configuration shows a /main URL for your cron job, but your app doesn't seem to have a route for such URL path, it only seems to handle the / path. I'd expect you'd see some 404 errors in the logs for the /main cron requests.
You should add a route for the /main path in your main.py. Or replace /main with / in your cron.yaml.
Side note: you have some non-executable code (it follows the return statement) inside hello() - just in case you're looking for that test.txt file...
I have a cron job which runs on my db and creates tasks to process certain db records. The cron job runs as expected, but the tasks added with taskqueue.add() don't really function. They are logged in the logs of the app, but nothing happens. I also put some logging.info() code in the tasks, but it's not executing. This is really puzzling beacuse I'm not getting any errors, and if I access the task directly (through a URL in the browser) it executes fine.
Here's the log:
Here's the empty_task.py code:
import webapp2
import logging
class EmptyHandler(webapp2.RequestHandler):
def get(self):
logging.info("empty task!")
app = webapp2.WSGIApplication([
('/tasks/empty_task', EmptyHandler),
], debug=True)
EDIT:
Here's the cron.yaml code:
cron:
- description: the shots dispatcher
url: /tasks/run_schedules
schedule: every 15 minutes
And here's the run_schedules code:
import webapp2
import logging
from google.appengine.datastore.datastore_query import Cursor
from google.appengine.api import taskqueue
from models.schedule import Schedule
MAX_FETCH = 20
class RunSchedulesHandler(webapp2.RequestHandler):
def get(self):
cursor = Cursor(urlsafe=self.request.get('cursor'))
scheds, next_curs, more = Schedule.query().fetch_page(MAX_FETCH, keys_only=True, start_cursor=cursor)
for key in scheds:
logging.info("adding to taskqueue schedule: {}".format(key.id()))
taskqueue.add(url='/tasks/process_schedule', params={'sid': key.id()})
taskqueue.add(url='/tasks/empty_task', params={'sid': key.id()})
if more and next_curs:
taskqueue.add(url='/tasks/run_schedules', params={'cursor': next_curs})
app = webapp2.WSGIApplication([
('/tasks/run_schedules', RunSchedulesHandler),
], debug=True)
The default method of task execution is POST. And you have only defined the GET method in EmptyHandler. See the appengine console status. Its returning 405 which is method not allowed.
You can use 2 solution for this.
Solution 1:
Change the method name to post.
class EmptyHandler(webapp2.RequestHandler):
def post(self):
logging.info("empty task!")
Solution 2: Define method explicitly in taskqueue.add()
taskqueue.add(url='/tasks/empty_task', params={'sid': key.id()}, method='GET')
I have a Python Cron job that, according to the Logs, runs to completion without errors. However, none of the "logging.error()" messages I have included in the code are included in the log and none of the required processing is done.
So that I can run this manually, I have a link in my HTML menu "Assign Rental Payments Due" that does the processing required and logs error logging messages correctly.
----
Section of app.yaml
- url: /rhrentassign.html
script: frhrentassign.app
----
Full cron.yaml
cron:
- description: Rental Payments Due
url: /rhrentassign
schedule: every day 14:00
----
Full python code (file is frhrentassign.py)
import os
import logging
import webapp2
from CronRH import *
class rhrentassignhandler(webapp2.RequestHandler):
def get(self):
swork = trhrenttopaycron()
swork.allnamespaces()
app = webapp2.WSGIApplication([('/rhrentassign.html', rhrentassignhandler)], debug=True)
----
Any thoughts on what I have done wrong would be most appreciated.
Many Thanks, David
Your handler is mapped to '/rhrentassign.html', but the cron is going to '/rhrentassign'.
Generally unless you have a very very good reason, there's no need to put 'html' in route names.
I've been trying for a few days now to get Google App Engine to run a cron Python script which will simply execute a script hosted on a server of mine.
It doesn't need to post any data to the page, simply open a connection, wait for it to finish then email me.
The code I've previously written has logged as "successful" but I never got an email, nor did I see any of the logging.info code I added to test things.
Ideas?
The original and wrong code that I originally wrote can be found at Google AppEngine Python Cron job urllib - just so you know I have attempted this before.
Mix of weird things was happening here.
Firstly, app.yaml I had to place my /cron handler before the root was set:
handlers:
- url: /cron
script: assets/backup/main.py
- url: /
static_files: assets/index.html
upload: assets/index.html
Otherwise I'd get crazy errors about not being able to find the file. That bit actually makes sense.
The next bit was the Python code. Not sure what was going on here, but in the end I managed to get it working by doing this:
#!/usr/bin/env python
# import logging
from google.appengine.ext import webapp
from google.appengine.api import mail
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.api import urlfetch
import logging
class CronMailer(webapp.RequestHandler):
def get(self):
logging.info("Backups: Started!")
urlStr = "http://example.com/file.php"
rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, urlStr)
mail.send_mail(sender="example#example.com",
to="email#example.co.uk",
subject="Backups complete!",
body="Daily backups have been completed!")
logging.info("Backups: Finished!")
application = webapp.WSGIApplication([('/cron', CronMailer)],debug=True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()
Whatever it was causing the problems, it's now fixed.