We Keep Coding

GAE - cron job failing, with no error message in logs

I have been trying to run a cron job with GAE (code developed in Python), but when I trigger the job, it fails without any error message -- I can't find anything at all in the logs.
This is happening for a service for which I'm using the flexible environment.
This is the structure of my files:
my_service.yaml looks like this:
service: my_service
runtime: custom
env: flex
env_variables:
a:
b:
the my_service.app looks like this:
from __future__ import absolute_import
from flask import Flask
from flask import request
import logging
import datetime
import os
import tweepy
from google.cloud import datastore
import time
logging.basicConfig(level=logging.INFO)
app = Flask(__name__)
#app.route('/Main')
def hello():
"""A no-op."""
return 'nothing to see.'
#app.route('/my_service')
def get_service():
is_cron = request.headers.get('X-Appengine-Cron', False)
logging.info("is_cron is %s", is_cron)
# Comment out the following test to allow non cron-initiated requests.
if not is_cron:
return 'Blocked.'
## data scraping and saving in Datastore
return 'Done.'
#app.errorhandler(500)
def server_error(e):
logging.exception('An error occurred during a request.')
return """
An internal error occurred: <pre>{}</pre>
See logs for full stacktrace.
""".format(e), 500
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080, debug=True)
Then I have a dispatch.yaml with this structure:
dispatch:
- url: "*/my_service*"
service: my_service
And a cron.yaml:
cron:
- description: run my service
url: /my_service
schedule: 1 of month 10:00
target: my_service
Not sure what I'm doing wrong here.
EDIT
A bit of context. This is something I'm editing starting from this repo.
The service called backend that is defined in there works perfectly (it has also the same schedule in the cron job as my_service but when I trigger it in a day different from the one in which it's scheduled, it works just fine). What I did was to create an additional service with its own yaml file, which looks exactly the same as the beckend.yaml, its own my_service.py and adding it to the dispacth.yamland the cron.yaml. In theory this should work, since the structure is exactly the same, but it doesn't.
This service was originally developed in the standard environment and there it was working, the problem originated when I moved it to the flex environment.
EDIT 2:
The problem was actually in the Dockerfile, that was calling a service that I was not using.

EDIT:
def get(self): may have some issues.
First, get may be reserved. Second, you aren't able to send self to that function. Change that to:
def get_service():
EDIT2:
You also need to import logging at the top of any page that uses it. And, you have not imported Flask and its components:
from flask import Flask, request, render_template # etc...
import logging

Your 1 of month 10:00 cron schedule specification can most likely be the culprit: it specifies to run the job at 10:00 only on the first day of each month! From Defining the cron job schedule:
Example: For the 1,2,3 of month 07:00 schedule, the job runs one
time at 07:00 on the first three days of each month.
So the last execution happened 3 days ago (if this cron config was deployed at the time) and no other attempt will happen until Nov 1st :)
Change the schedule to something easier to test with, like every 5 minutes or every 1 hours and revert the change once you're happy it works as expected.

404 from cron job on google app engine django app

So, everything else works... to preface this. But, I haven't really moved outside the admin interface. I'm trying to get data from an API and insert it into the database if there's changes. I've managed to write a script that can do that (in theory... it can do it locally), but I can't get the app in the cloud to recognize its existence. I've followed Google's suggestion of adding it to the app.yaml and cron.yaml to no avail.
Do I need to add this to a urls.py? I haven't mucked with teh handlers at all thus far and I'm not sure what settings.py makes happen, what the yaml files make happen, and how much of this is pixie dust.
here are teh relevant files...
app.yaml
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT mysite.wsgi
threadsafe: yes
beta_settings:
cloud_sql_instances: [redacted]
runtime_config:
python_version: 3
health_check:
enable_health_check: False
handlers:
- url: /static
static_dir: static/
- url: /run/get_data/
script: JSONdownload.app
login: admin
- url: .*
script: mysite.wsgi.application
cron.yaml
cron:
- description: "get data"
url: /run/get_data/
schedule: every 5 minutes
JSONdownload.py
#!/usr/bin/env python
# /var/spool/cron/crontabs
import json
import urllib2
from django.http import HttpResponse
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from .models import Game
from .models import Team
class JSONdownloadHandler(webapp.RequestHandler):
def get(self):
self.response.write('cron')
class MainHandler(webapp.RequestHandler):
def get(self):
self.response.write('yay')
I'm finding great resources for a basic python app... but none for this situation really... anyone know of something better than what I'm doing, PLEASE let me know!

You're mixing up elements of the flexible environment app.yaml with those of the standard environment app.yaml
In particular the script: JSONdownload.app portion is ignored. You need to add the handler for the /run/get_data/ path inside your mysite.wsgi app, maybe from there invoking the JSONdownload.py code.
Somehow related: cron job in google app engine not working.

So... it was that I didn't route to teh location. I was actually able to just set a route (with admin against it) to the URL and then able to call it via the cron.yaml.

Cannot run Cron Job on Google App Engine (Python)

After a few days of searching and looking around for this, and I haven't been able to find the right answer.
I'm trying to run a cron job on Google App Engine (with Python). The cron job itself isn't that important, I'm just looking to run a python script every minute. For now I'm just trying to add a line with the current time in a separate text file (test.txt).
I'm pretty sure that I don't quite understand the concept of handlers, and that's what causing me problems. But I've spent hours in the documentation, and I still can't figure it out.
I sense that I should not be using main.py as my script for the cron job, but I have a hard time understanding what the url needs to be in cron.yaml, and what the handler/script should be.
Please help!
app.yaml
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
python_version: 3
handlers:
- url: /main
script: main.py
cron.yaml
cron:
- description : most recent test
url : /main
schedule: every 1 minutes
main.py
# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# [START app]
import logging
from datetime import datetime
current_time = str(datetime.now())
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello():
"""Return a friendly HTTP greeting."""
return 'Hello World! The time is ' + current_time
message = "this worked, as of " + current_time + "\n"
with open("test.txt", "a") as myfile:
myfile.write(message)
#app.errorhandler(500)
def server_error(e):
logging.exception('An error occurred during a request.')
return """
An internal error occurred: <pre>{}</pre>
See logs for full stacktrace.
""".format(e), 500
if __name__ == '__main__':
# This is used when running locally. Gunicorn is used to run the
# application on Google App Engine. See entrypoint in app.yaml.
app.run(host='127.0.0.1', port=8080, debug=True)
# [END app]

You don't have a url handler for /main. Try this:
in app.yaml:
handlers:
- url: /.* # wildcard. every url goes there
script: main.py
in main.py:
from flask import Response
#app.route('/main')
def main():
"""Return a friendly HTTP greeting."""
Response("Hello main viewer", mimetype='text/plain')
Note: Another bug in your code is that you return the html view BEFORE the script gets a chance to write the .txt file. It will never get that far.

Your app.yaml file is mixing up the standard environment Handlers element into a flexible environment configuration, so it is probably ignored.
Your cron.yaml configuration shows a /main URL for your cron job, but your app doesn't seem to have a route for such URL path, it only seems to handle the / path. I'd expect you'd see some 404 errors in the logs for the /main cron requests.
You should add a route for the /main path in your main.py. Or replace /main with / in your cron.yaml.
Side note: you have some non-executable code (it follows the return statement) inside hello() - just in case you're looking for that test.txt file...

How to start mapreduce job from cron on GAE Python

I have mapreduce job defined in mapreduce.yaml:
mapreduce:
- name: JobName
mapper:
input_reader: google.appengine.ext.mapreduce.input_readers.DatastoreInputReader
handler: handler_name
params:
- name: entity_kind
default: KindName
How to start it from cron? Is there some url that can run it?

You can start a mapreduce task from any kind of AppEngine handler using control.py
from mapreduce import control
mapreduce_id = control.start_map(
"My Mapper",
"main.my_mapper",
"mapreduce.input_readers.DatastoreInputReader",
{"entity_kind": "models.MyEntity"},
shard_count=10)

Yes, if you look at the Getting Started page, it shows that you set the URL in your app.yaml:
handlers:
- url: /mapreduce(/.*)?
script: mapreduce/main.py
login: admin
You then can just cron it in the usual App Engine fashion, which in this example would be writing a cron.yaml like this:
cron:
- description: daily summary job
url: /mapreduce
schedule: every 24 hours

Google App Engine Python Cron

I've been trying for a few days now to get Google App Engine to run a cron Python script which will simply execute a script hosted on a server of mine.
It doesn't need to post any data to the page, simply open a connection, wait for it to finish then email me.
The code I've previously written has logged as "successful" but I never got an email, nor did I see any of the logging.info code I added to test things.
Ideas?
The original and wrong code that I originally wrote can be found at Google AppEngine Python Cron job urllib - just so you know I have attempted this before.

Mix of weird things was happening here.
Firstly, app.yaml I had to place my /cron handler before the root was set:
handlers:
- url: /cron
script: assets/backup/main.py
- url: /
static_files: assets/index.html
upload: assets/index.html
Otherwise I'd get crazy errors about not being able to find the file. That bit actually makes sense.
The next bit was the Python code. Not sure what was going on here, but in the end I managed to get it working by doing this:
#!/usr/bin/env python
# import logging
from google.appengine.ext import webapp
from google.appengine.api import mail
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.api import urlfetch
import logging
class CronMailer(webapp.RequestHandler):
def get(self):
logging.info("Backups: Started!")
urlStr = "http://example.com/file.php"
rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, urlStr)
mail.send_mail(sender="example#example.com",
to="email#example.co.uk",
subject="Backups complete!",
body="Daily backups have been completed!")
logging.info("Backups: Finished!")
application = webapp.WSGIApplication([('/cron', CronMailer)],debug=True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()
Whatever it was causing the problems, it's now fixed.