I've been trying for a few days now to get Google App Engine to run a cron Python script which will simply execute a script hosted on a server of mine.
It doesn't need to post any data to the page, simply open a connection, wait for it to finish then email me.
The code I've previously written has logged as "successful" but I never got an email, nor did I see any of the logging.info code I added to test things.
Ideas?
The original and wrong code that I originally wrote can be found at Google AppEngine Python Cron job urllib - just so you know I have attempted this before.
Mix of weird things was happening here.
Firstly, app.yaml I had to place my /cron handler before the root was set:
handlers:
- url: /cron
script: assets/backup/main.py
- url: /
static_files: assets/index.html
upload: assets/index.html
Otherwise I'd get crazy errors about not being able to find the file. That bit actually makes sense.
The next bit was the Python code. Not sure what was going on here, but in the end I managed to get it working by doing this:
#!/usr/bin/env python
# import logging
from google.appengine.ext import webapp
from google.appengine.api import mail
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.api import urlfetch
import logging
class CronMailer(webapp.RequestHandler):
def get(self):
logging.info("Backups: Started!")
urlStr = "http://example.com/file.php"
rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, urlStr)
mail.send_mail(sender="example#example.com",
to="email#example.co.uk",
subject="Backups complete!",
body="Daily backups have been completed!")
logging.info("Backups: Finished!")
application = webapp.WSGIApplication([('/cron', CronMailer)],debug=True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()
Whatever it was causing the problems, it's now fixed.
Related
I have been trying to run a cron job with GAE (code developed in Python), but when I trigger the job, it fails without any error message -- I can't find anything at all in the logs.
This is happening for a service for which I'm using the flexible environment.
This is the structure of my files:
my_service.yaml looks like this:
service: my_service
runtime: custom
env: flex
env_variables:
a:
b:
the my_service.app looks like this:
from __future__ import absolute_import
from flask import Flask
from flask import request
import logging
import datetime
import os
import tweepy
from google.cloud import datastore
import time
logging.basicConfig(level=logging.INFO)
app = Flask(__name__)
#app.route('/Main')
def hello():
"""A no-op."""
return 'nothing to see.'
#app.route('/my_service')
def get_service():
is_cron = request.headers.get('X-Appengine-Cron', False)
logging.info("is_cron is %s", is_cron)
# Comment out the following test to allow non cron-initiated requests.
if not is_cron:
return 'Blocked.'
## data scraping and saving in Datastore
return 'Done.'
#app.errorhandler(500)
def server_error(e):
logging.exception('An error occurred during a request.')
return """
An internal error occurred: <pre>{}</pre>
See logs for full stacktrace.
""".format(e), 500
if __name__ == '__main__':
app.run(host='127.0.0.1', port=8080, debug=True)
Then I have a dispatch.yaml with this structure:
dispatch:
- url: "*/my_service*"
service: my_service
And a cron.yaml:
cron:
- description: run my service
url: /my_service
schedule: 1 of month 10:00
target: my_service
Not sure what I'm doing wrong here.
EDIT
A bit of context. This is something I'm editing starting from this repo.
The service called backend that is defined in there works perfectly (it has also the same schedule in the cron job as my_service but when I trigger it in a day different from the one in which it's scheduled, it works just fine). What I did was to create an additional service with its own yaml file, which looks exactly the same as the beckend.yaml, its own my_service.py and adding it to the dispacth.yamland the cron.yaml. In theory this should work, since the structure is exactly the same, but it doesn't.
This service was originally developed in the standard environment and there it was working, the problem originated when I moved it to the flex environment.
EDIT 2:
The problem was actually in the Dockerfile, that was calling a service that I was not using.
EDIT:
def get(self): may have some issues.
First, get may be reserved. Second, you aren't able to send self to that function. Change that to:
def get_service():
EDIT2:
You also need to import logging at the top of any page that uses it. And, you have not imported Flask and its components:
from flask import Flask, request, render_template # etc...
import logging
Your 1 of month 10:00 cron schedule specification can most likely be the culprit: it specifies to run the job at 10:00 only on the first day of each month! From Defining the cron job schedule:
Example: For the 1,2,3 of month 07:00 schedule, the job runs one
time at 07:00 on the first three days of each month.
So the last execution happened 3 days ago (if this cron config was deployed at the time) and no other attempt will happen until Nov 1st :)
Change the schedule to something easier to test with, like every 5 minutes or every 1 hours and revert the change once you're happy it works as expected.
So I am trying to build a restful API using flask, served up by apache on centos (httpd).
Basic API calls work just fine but I am not making much progress on the more advanced aspects because every time it fails I just get an HTTP 500 response which is completely useless for troubleshooting and I have no server-side logs to look at. I am literally trying to solve this through trial and error and it is making me bang my head against the wall.
In order to make any progress on this project I need to setup some basic error logging, but I do not understand the documentation or existing threads about this. It is completely over my head.
What I want to do is have flask write out all warnings and exceptions generated by my application to a specific file (it can be in the app directory to keep it simple).
I am looking for the simplest, easiest, least mind bendy way of doing this... suggestions?
Here is a very simplified version of my app... it shows the basic structure I am using, so please use that for reference.
from flask import Flask, jsonify, request
from flask_restful import reqparse, abort, Resource, Api
app = Flask(__name__)
api = Api(app)
class fetchTicket(Resource):
def post(self):
request_data = request.get_json(force=True)
r_ticket_id = request_data['ticket_id']
return jsonify(ticket_id=r_ticket_id)
api.add_resource(fetchTicket, '/ticket/fetch')
if __name__ == "__main__":
import logging
from logging.handlers import FileHandler
app.debug = True
file_handler = FileHandler("/var/www/project_folder/error.log")
file_handler.setLevel(logging.DEBUG)
app.logger.addHandler(file_handler)
app.run()
But when I run the above code no error.log file is created. I am not sure what I am doing wrong.
Note: I did set the folder permissions so that the apache user has access to write to the directory where the log file should go, as per http://fideloper.com/user-group-permissions-chmod-apache but it did not help so I don't think it is a permissions issue.
You’ll need to explicitly include the stack trace when logging, using the extra kwarg.
logger.exception('Probably something went wrong', extra={'stack': True})
This is very weird. The title says most of it, my code should say the rest. Here's my main.py file:
from google.appengine.api import urlfetch
import webapp2
import jinja2
import json
import os
jinja_environment = jinja2.Environment(
loader=jinja2.FileSystemLoader(os.path.dirname(__file__)))
class MainPage(webapp2.RequestHandler):
def get(self):
response = urlfetch.fetch("http://localhost:8080/api/helloworld?name=totty", method=urlfetch.GET)
if response.status_code == 200:
result = json.loads(response.content)
template_values = {'response': result['msg']}
template = jinja_environment.get_template('index.html')
self.response.out.write(template.render(template_values))
app = webapp2.WSGIApplication(
[('/', MainPage)],
debug=True)
Here's my api.py file:
import webapp2
import json
class HelloWorld(webapp2.RequestHandler):
def get(self):
name = self.request.get('name') or 'world'
msg = "Hello {}!".format(name)
payload = json.dumps({'msg': msg})
# payload = json.dumps({'dir': str(dir(self.request)), 'body': str(self.request.body), 'name': str(self.request.arguments())})
self.response.headers['Content-Type'] = 'application/json'
self.response.write(payload)
app = webapp2.WSGIApplication(
[('/api/helloworld', HelloWorld)],
debug=True)
And in case my app.yaml file would help:
application: stacksort
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /api/.*
script: api.app
- url: /.*
script: main.app
libraries:
- name: webapp2
version: latest
- name: jinja2
version: latest
Nothing changes even if I add deadline=30 to my urlfetch call. I tested the API using httpie and JQuery and it works perfectly fine and returns in under 5 seconds.
I looked at the other questions, but I'm still stumbling into the dark. Help, tips or refactoring would be appreciated.
I plan to add calls to the StackEchange Search API, so I suspect the problem might come in that time also. If there's a better way to do this, please tell. Thanks.
You are trying to fetch a URL to and from your application which is strongly discouraged on Google App Engine.
Locally you won't be able to call the development server because it serves only one request at a time. Multi-threading is not observed.
Note: the new experimental development server is now able to handle multiple requests at a time.
Multithreaded serving for better performance for complex applications and more correct semantics e.g. accessing your own application through urlfetch no longer deadlocks.
On production environment, GAE prevents the fetch service to call the same application.
To prevent an app from causing an endless recursion of requests, a request handler is not allowed to fetch its own URL. It is still possible to cause an endless recursion with other means, so exercise caution if your app can be made to fetch requests for URLs supplied by the user.
While upgrading the SDK, I noticed this addition to the DevServer page:
Note: dev_appserver.py can only serve one request at a time. If your
application makes URL fetch requests to itself while processing a
request, these requests will fail when using the development web
server. (They will not fail when running on App Engine.) To test such
requests, you can run a second instance of dev_appserver.py on a
different port, then code your application to use the other server
when making requests to itself.
So I guess that solves my problem (or at least gives a satisfactory explanation for it).
3 days ago I started using google appengine to see how it works.
I have set the basic app that prints "hello". Nothing complicated. I updated the simple app thet prints "hello" app and it worked perfectly.
Then I wanted to experiment a little more:
First I did is that I uploaded new program (i didint change app.yaml, just main.py)
Then i got a empty screen.
Then I uploaded a new version of app (changed version in app.yaml, and main.py), I have changed a versions in admin in backend. I still had a empty screen.
Thanks.
This is my code:
main.py
#!/usr/bin/env python
print 'Hello, World!'
app.yaml:
application: searchbarrel
version: 2
runtime: python
api_version: 1
handlers:
- url: /.*
script: main.py
The app works good on localhost
Your CGI script needs to print at least one blank line before outputting text, since the browser will take the first line to be an HTTP header:
#!/usr/bin/env python
print '\nHello, World!'
will work fine.
Ideally, you should print actual valid HTTP headers before your blank line.
dev_appserver acts a bit differently than the production servers, which is why you're seeing output running locally.
(It is a good idea to use a WSGI framework, however.)
this can't work. its not enough to write a file that has a print hello inside. you need to create a WSGI app and make a RequestHandler that processes your request and writes the hello world out.
main.py
import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
class MainPage(webapp.RequestHandler):
def get(self):
self.response.out.write('Hello, World')
app = webapp2.WSGIApplication([('/', MainPage)])
def main():
application = webapp.WSGIApplication([("/", MainPage)], debug=True)
run_wsgi_app(application)
if __name__ == "__main__":
main()
you should read the getting started guide first.
https://cloud.google.com/appengine/docs/standard/python/
I have never written a python script in my life, but I have a question that can hopefully be solved pretty quickly...
I'm using Google App Engine and Dropbprox. The script uses a custom domain to point to your public DropBox folder for better DropBox URLs. I'd like to be able to redirect users to my main site (jacob.bearce.me) if they visit my dropbox url (dl.bearce.me).
The problems that I'm having:
I've never used GAE or Python before, so I have no idea where to even begin
Putting a index.html file in my GAE project didn't fix it (I was hoping it'd just default to that if there was no filename specified, like it would on a normal site, but no cigar.)
Just a simple redirect if a users visits the main URL is all I'm after, nothing fancy.
My Python file: http://dl.bearce.me/mirror.py
Here's a main.py that issues a redirect for all requests, using the Python 2.5 runtime environment:
from google.appengine.ext import webapp
from google.appengine.ext.webapp import util
class MainHandler(webapp.RequestHandler):
def get(self):
self.redirect('http://jacob.bearce.me/')
application = webapp.WSGIApplication([('/.*', MainHandler)],
debug=True)
def main():
util.run_wsgi_app(application)
if __name__ == '__main__':
main()
And here's the app.yaml file you need to route URLs to this handler:
application: myapp
version: 1
runtime: python
api_version: 1
handlers:
- url: .*
script: main.py
(Replace myapp with your actual app ID.)
For more information about creating and uploading an App Engine app in Python, see the Getting Started tutorial.