Is it possible to open a file on GAE just to read its contents and get the last modified tag?
I get a IOError: [Errno 13] file not accessible:
I know that i cannot delete or update but i believe reading should be possible
Has anyone faced a similar problem?
os.stat(f,'r').st_mtim
You've probably declared the file as static in app.yaml. Static files are not available to your application; if you need to serve them both as static files and read them as application files, you'll need to include 2 copies in your project (ideally using symlinks, so you don't actually have to maintain an actual copy.)
Update Nov 2014:
As suggested in the comments, you can now do this with the application_readable flag:
application_readable
Optional. By default, files declared in static file handlers are
uploaded as static data and are only served to end users, they cannot
be read by an application. If this field is set to true, the files are
also uploaded as code data so your application can read them. Both
uploads are charged against your code and static data storage resource
quotas.
See https://cloud.google.com/appengine/docs/python/config/appconfig#Static_Directory_Handlers
You can read files, but they're on Goooogle's wacky GAE filesystem so you have to use a relative path. I just whipped up a quick app with a main.py file and test.txt in the same folder. Don't forget the 'e' on st_mtime.
import os
from google.appengine.ext import webapp
from google.appengine.ext.webapp import util
class MainHandler(webapp.RequestHandler):
def get(self):
path = os.path.join(os.path.split(__file__)[0], 'test.txt')
self.response.out.write(os.stat(path).st_mtime)
def main():
application = webapp.WSGIApplication([('/', MainHandler)],
debug=True)
util.run_wsgi_app(application)
if __name__ == '__main__':
main()
+1 for the new "application_readable: true" feature. Before using this new feature I did run into an issue with GAEs' "wacky" file system while getting the NLP Montylingua to import.
Issue: Monty uses the open(filename,'rb') and a file pointer to file_ptr.read() in bytes from the static files. My implementation worked on my local windows system but failed upon deployment!
The fix: Specify the expected bytes to read file_ptr.read(4) #4 binary bytes
Appears to be something related to the 64 bit GAE server wanting to read in more (8 by default) bytes. Anyways, took a while to find that issue. Montylingua loads now.
I came up strange but working solution :) Jinja :)
Serving static files directly sometimes become a headache with GAE. Possible trade-off from performance let you move straigh forward with Jinja
- url: /posts/(.*\.(md|mdown|markdown))
mime_type: text/plain
static_files: static/posts/\1
upload: posts/(.*\.(md|mdown|markdown))
from jinja2 import Environment
from jinja2.loaders import FileSystemLoader
posts = Environment(loader=FileSystemLoader('static/posts/')) # Note that we use static_files folder defined in app.yaml
post = posts.get_template('2013-11-13.markdown')
import markdown2 # Does not need of course
class Main(webapp2.RequestHandler):
def get ( self ):
self.response.headers[ 'Content-Type' ] = 'text/html'
self.response.write ( markdown2.markdown( post.render()) ) # Jinja + Markdown Render function
Did you get it ;) I tested and It worked.
With webapp2, supposing you have pages/index.html at the same path as main.py:
#!/usr/bin/env python
import webapp2, os
class MainHandler(webapp2.RequestHandler):
def get(self):
path = os.path.join(os.path.split(__file__)[0], 'pages/index.html')
with open(path, 'r') as f:
page_content = f.read()
self.response.write(page_content)
app = webapp2.WSGIApplication([
('/', MainHandler)
], debug=True)
I can't see an answer for when the file hasn't been marked as static, and you're trying to read it in mode 'rt'; apparently that doesn't work. You can however open files just fine in mode 'rb', or just plain 'r'. (I wasted about 10 minutes on that 't'.)
Related
Here is a simple static FastAPI app. With this setup even though the root path is expected to return a FileResponse of custom.html, the app still returns index.html. How can I get the root path work and render custom.html?
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
from fastapi.responses import FileResponse
app = FastAPI()
app.mount(
"/",
StaticFiles(directory="static", html=True),
name="static",
)
#app.get("/")
async def index() -> FileResponse:
return FileResponse("custom.html", media_type="html")
As per Starlette documentation:
StaticFiles
Signature: StaticFiles(directory=None, packages=None, check_dir=True)
html - Run in HTML mode. Automatically loads index.html for directories if such file exists.
In addtion, as shown from the code snippet you provided, you have mounted StaticFiles to the root directory (i.e., '/'), instead of, for example, /static (or some other path name), as shown below:
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
app = FastAPI()
app.mount('/static', StaticFiles(directory='static'), name='static')
As per FastAPI documentation:
"Mounting" means adding a complete "independent" application in a
specific path, that then takes care of handling all the sub-paths.
Hence, any path that starts with '/' will be handled by that StaticFiles application, and due to specifying html=True in the arguments, index.html will be automatically loaded; regardless of creating a separate endpoint pointing to the root path / and trying to return something else, as demonstrated in the example given in your question.
Important
If, for example, you moved app.mount("/",StaticFiles(... line after defining your #app.get("/") endpoint, you would see that order matters and index.html would not be automatically loaded anymore, as endpoints are evaluated in order. Note that, in your case, you might get an Internal Server Error, as your #app.get("/") endpoint would be called and attempt to find custom.html, but if this file exists under 'static' directory (as shown from your code) and not under '/', you would then get a File does not exist error, and you should instead return FileResponse('static/custom.html').
Even if you removed html=True, but keep StaticFiles mounted to the root directory (and defined before your '/' endpoint), you would get a {"detail":"Not Found"} error response when attempting to access http://localhost:8000/. This is because that request is still handled by that application (as mentioned earlier) and you should now need to specify the file that you would like to access, e.g., http://localhost:8000/index.html. Even if you defined other endpoints in your code (e.g., /register, /login, /hello)—as long as StaticFiles is mounted to the root directory (i.e., '/') and defined in your code before all other endpoints—all requests to those routes would be handled by StaticFiles application and lead to a {"detail":"Not Found"} error response.
The html=True simply provides an easy way to serve a directory of web content with just one line of code. If you only need to serve static files, such as package docs directory, then this is the way to go. If, however, you need to serve different HTML files that will get dynamically updated, as well as you wish to create additional routes/endpoints, you should have a look at Templates (not FileResponse), as well as mount your StaticFiles to a different directory (e.g., /static), rather than root directory (and without using html=True).
I'm using Flask-Uploads to upload a file and output the URL. However, for some reason the URL always points to a 404! I can see the file in the correct folder, but the URL seems to not be able to find it. Here are the configurations I'm using...
UPLOADS_DEFAULT_URL = os.environ.get("UPLOADS_URL", "http://localhost:5000/")
UPLOADS_DEFAULT_DEST = "app/uploads"
I also have an Upload set defined as:
productUploadSet = UploadSet('plist', extensions=('xlsx', 'csv', 'xls'))
The file is found in "app/uploads/plist/filename.csv" and I'll get a url returned to me like "http://localhost:5000/productlist/filename.csv" but whenever I open the URL it is always a 404. I know that the url method from Flask-Uploads doesn't actually check fi the file exists, but I can see the file is actually there. Is it looking in the wrong place somehow? Thanks for any help.
From Flask's guide to uploading files:
Now one last thing is missing: the serving of the uploaded files. As of Flask 0.5 we can use a function that does that for us:
from flask import send_from_directory
#app.route('/uploads/<filename>')
def uploaded_file(filename):
return send_from_directory(app.config['UPLOAD_FOLDER'],
filename)
Alternatively you can register uploaded_file as build_only rule and use the SharedDataMiddleware. This also works with older versions of Flask:
from werkzeug import SharedDataMiddleware
app.add_url_rule('/uploads/<filename>', 'uploaded_file',
build_only=True)
app.wsgi_app = SharedDataMiddleware(app.wsgi_app, {
'/uploads': app.config['UPLOAD_FOLDER']
})
Without implementing one of these, Flask has no idea how to serve the uploaded files.
I want to read a INI file from python, and I came up with the following script:
import webapp2
import urllib
import ConfigParser
class MainPage(webapp2.RequestHandler):
def get(self):
getstreet = self.request.get('street')
getlocation = self.request.get('location')
self.response.headers['Content-Type'] = 'text/plain'
map = ConfigParser.RawConfigParser()
map.read('Map.ini')
coords = map.get(getlocation, getstreet)
self.response.out.write(coords)
app = webapp2.WSGIApplication([('/map', MainPage)],
debug=True)
From outside I just need to read:
xxx.appspot.com/map?location=mylocation&street=mystreet
And I double checked the INI file is uploaded and available:
xxx.appspot.com/Map.ini
It works just fine locally, but I get an error when deploying.
I've seen mentions of url fetch, but all examples I could find were the other way around
Any idea?
If your file is available via the URL it's almost certainly marked as static. One of the limitations of AppEngine is that static files cannot be read from code. You should be able to fix it by removing the static declaration.
There's loads more information about this at Read a file on App Engine with Python?
So I have successfully deployed an app using webapp2/jinja2 and a Paste server, but am having trouble serving static stylesheets.
I have had luck accessing static files via this method, as well as implementing a StaticFileHandler I found with some google-fu:
import os
import mimetypes
import webapp2
import logging
class StaticFileHandler(webapp2.RequestHandler):
def get(self, path):
abs_path = os.path.abspath(os.path.join(self.app.config.get('webapp2_static.static_file_path', 'static'), path))
if os.path.isdir(abs_path) or abs_path.find(os.getcwd()) != 0:
self.response.set_status(403)
return
try:
f = open(abs_path, 'r')
self.response.headers.add_header('Content-Type', mimetypes.guess_type(abs_path)[0])
self.response.out.write(f.read())
f.close()
except:
self.response.set_status(404)
where my main app routing looks like:
app = webapp2.WSGIApplication([('/', HelloWorld),
(r'/display', DisplayHandler),
(r'/static/(.+)', StaticFileHandler)
], debug=True)
My css files are in a folder under the app root: /static/css/main.css
I can access the file via direct url, and even link it as a stylesheet, but the styles won't apply. Any ideas? Is there another way to serve stylesheets? Some way to implement an app.yaml similar to GAE?
you dont need a static file handler.
upload the app with the static file folder by adding this to your app.yaml
- url: /static/
static_dir: static
docs are here: https://developers.google.com/appengine/docs/python/config/appconfig#Static_Directory_Handlers
edit:
see answer below in comments
#Mnemon, hats off to you for solving my problem. I would upvote you but I'm not allowed to do that. You convinced me that if it's not the only webapp2 way without GAE, it's at least a way that will work.
But also I can contribute that your solution is now installable as "pip install webapp2_static", from pipi--- by an author who seems to be using his real name... you I'm sure. Other webapp2 docs that I found helpful are available here.
I'm implementing your code on a Linux desktop development server, using paste, which you also used:
def main():
from paste import httpserver
httpserver.serve(app, host='127.0.0.1', port='8080')
But with the code as you have it above (which appears to be utterly identical to that of webapp2_static.py file), I don't find that putting my css files in a folder named static in the app root works as you said.
For example, I have /home/user/proj/public_html/app/app.py, where the py file contains your code plus other "views" for my ultra-simple site. (I don't know how paste really works, so maybe for now the public_html is just in there for reference so that I don't become confused when I'm uploading stuff onto the production server.)
So if I put the css stylesheets into a folder named /static, then, if I put /static in as either a subdirectory of /app or of /public_html I find that neither location works; I must instead make it a subdirectory of /proj.
I wasn't expecting that, but the cure for me is to change the default 'static' in your app.configure.get(..., 'static') call, to 'public_html/app/static'. Then it works, with the /static folder inside /app.
Similarly using the pipi code with './app/static/ in place of the default 'static' doesn't work; I found that I need ./public_html/app/static instead (or maybe it was just /public_html/app/static or even public_html/app/static... I forgot... one of those worked).
I tested how your computation of abs_path works and have reworked it in the code below, in which I have junked your approach in favor of something more Djangoesque. To wit, in my one app py file I put at the top the following:
STATIC_DIR = os.sep + 'tostatic' + os.path.abspath(os.path.dirname(__file__)) + os.sep + 'static'
Then in the page to which I want to add css, my Home page in my case, I put a very readable:
<link href="{{STATIC_DIR}}/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css">
For the "view" that generates my Home page I have (env is a jinja2 Environment object that takes a template loader as an argument):
class Home(webapp2.RequestHandler):
def get(self):
template = env.get_template('index.html')
template_values = {'STATIC_DIR': STATIC_DIR }
self.response.write(template.render(template_values))
And finally the URL routing is as in:
app = webapp2.WSGIApplication(
[
(r'/', Home),
(r'/tostatic/(.+)', StaticView),
], debug=True)
The view for the static file serving is now:
class StaticView(webapp2.RequestHandler):
def get(self, path):
path = os.sep + path
try:
f = open(path, 'r')
self.response.headers.add_header('Content-Type', mimetypes.guess_type(path)[0])
self.response.out.write(f.read())
f.close()
except Exception, e:
print 'Problem in StaticView:', e
self.response.set_status(404)
To finally close, the problem that I had with your approach is the one that I and other near noobs have with the departure of URLs from the legacy association with the file system. In your approach "static" is both a sub-directory and a string between slashes at the front of the URL that tells the interpreter which view (which webapp2.RequestHandler subclass) to run. You take the /static from the rest of the URL and then later hard-code it back on. And when it comes time to decide what to put in for href in the tag the HTML page coder has to remember that duplicity. With the {{STATIC_DIR}} template variable approach it's clear what to do. And it's easy to redefine the location of the static files--- only the STATIC_DIR declaration has to be changed.
I found that self.response.set_status(404) shows up in Firebug, but not Firefox. Evidently with webapp2 you must provide and serve your own HTTP status code pages.
self.response.headers.add_header('Content-Type', mimetypes.guess_type(abs_path)[0])
self.response.headers['Content-Type'] = mimetypes.guess_type(abs_path)[0]
When I run this code on my computer with the help of "Google App Engine SDK", it displays (in my browser) the HTML code of the Google home page:
from google.appengine.api import urlfetch
url = "http://www.google.com/"
result = urlfetch.fetch(url)
print result.content
How can I make it display the page itself? I mean I want to see that page in my browser the way it would normally be seen by any user of the internet.
Update 1:
I see I have received a few questions that look a bit complicated to me, although I definitely remember I was able to do it, and it was very simple, except i don't remember what exactly i changed then in this code.
Perhaps, I didn't give You all enough details on how I run this code and where I found it. So, let me tell You what I did. I only installed Python 2.5 on my computer and then downloaded "Google App Engine SDK" and installed it, too. Following the instructions on "GAE" page (http://code.google.com/appengine/docs/python/gettingstarted/helloworld.html) I created a directory and named it “My_test”, then I created a “my_test.py” in it containing that small piece of the code that I mentioned in my question.
Then, continuing to follow on the said instructions, I created an “app.yaml” file in it, in which my “my_test.py” file was mentioned. After that in “Google App Engine Launcher” I found “My_test” directory and clicked on Run button, and then on Browse. Then, having visited this URL http://localhost:8080/ in my web browser, I saw the results.
I definitely remember I was able to display any page in my browser in this way, and it was very simple, except I don’t remember what exactly I changed in the code (it was a slight change). Now, all I can see is a raw HTML code of a page, but not a page itself.
Update 2:
(this update is my response to wescpy)
Hello, wescpy!!! I've tried Your updated code and something didn't work well there. Perhaps, it's because I am not using a certain framework that I am supposed to use for this code. Please, take a look at this screen shot (I guess You'll need to right-click this image to see it in better resolution):
(source: narod.ru)
Is not that easy, you have to parse content and adjust relative to absolute paths for images and javascripts.
Anyway, give it a try adding the correct Content-Type:
from google.appengine.api import urlfetch
url = "http://www.google.com/"
result = urlfetch.fetch(url)
print 'Content-Type: text/html'
print ''
print result.content
a more complete example would look something like this:
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.api import urlfetch
class MainHandler(webapp.RequestHandler):
def get(self):
url = "http://www.google.com/"
result = urlfetch.fetch(url)
self.response.out.write(result.content)
application = webapp.WSGIApplication([
('/', MainHandler),
], debug=True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()
but as others' have said, it's not that easy to do because you're not in the server's domain, meaning the pages will likely not look correct due to missing static content (JS, CSS, and/or images)... unless full pathnames are used or everything that's needed is embedded into the page itself.
UPDATE 1:
as mentioned before, you cannot just download the HTML source and expect things to render correctly because you don't necessarily have access to the static data. if you really want to render it as it was meant to be seen, you have to just redirect... here's the modified piece of code:
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.api import urlfetch
class MainHandler(webapp.RequestHandler):
def get(self):
url = "http://www.google.com/"
self.redirect(url)
application = webapp.WSGIApplication([
('/', MainHandler),
], debug=True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()
UPDATE 2:
sorry! it was a cut-n-paste error. now try it.
special characters such as <> etc are likely encoded, you'd have to decode them again for the browser to interpet it as code.