datastore use BlobKey to download - python

In the google appengine datastore, there is a BlobKey (labled as csv). The key is in the following format: encoded_gs_file:we1o5o7klkllfekomvcndhs345uh5pl31l. I would like to provide a download button to save this information. My question is, what is the endpoint that i can use to access this. More information about the BlobKey is below.
The web app is being run using dev_appserver.py and uses python 2.7 (Django) as the backend. Currently, a button exists, but when clicking on it, it returns a 404 error. The download link that the button provides is:
https://localhost:8080/data?key=encoded_gs_file:dwndjndwamwljioihkm
My question is, how can i use the blobkey to generate a url that can be downloaded; or how can i check my code base to find how the url that i can use is being generated?
class BlobstoreDataServer(blobstore_handlers.BlobstoreDownloadHandler):
def get(self):
k = str(urllib.unquote(self.request.params.get('key','')))
logging.debug(k)
blob_info = blobstore.BlobInfo.get(k)
logging.debug(blob_info)
if (not blob_info) or (not blob_info.size):
self.error(404)
return
self.response.headers['X-Blob-Size'] = str(blob_info.size)
self.response.headers['Content-Type'] = blob_info.content_type
self.response.headers['Content-Disposition'] = (u'attachment; filename=%s' % blob_info.filename).encode('utf-8')
self.send_blob(blob_info)
Edit: New Images

Do you have a Request Handler for the route /data that does something like this?
from google.appengine.ext import blobstore
class DisplayBlob(blobstore_handlers.BlobstoreDownloadHandler):
def get(self):
blob_key = self.request.GET['key']
self.send_blob(ndb.BlobKey(blob_key))
self.response.headers['Content-Type'] = 'text/plain'
EDIT:
Ok so the 404 is probably being thrown by you by this line: self.error(404) right? Add a logging.warn('BlobstoreDataServer is throwing 404') right before to make sure. Also are you seeing this line logging.debug(k) print (I want to confirm that BlobstoreDataServer is even getting hit)? You may need to do logging.getLogger().setLevel(logging.DEBUG) to see it.
So that means blobstore.BlobInfo.get(k) is returning None. Focus on making sure that is working first, you can do this in the interactive console.
Go to http://localhost:8000/blobstore
Open one of them and copy the Key (encoded_gs_file:dwndjndwamwljioih...)
Go to the Interactive console (http://localhost:8000/console) and enter this code and hit 'EXECUTE' and make sure it is able to find it:
If that step didn't work, then then something is up with your dev_appserver.py's blobstore emulator
If that works, then just manually paste that same key at the end of your download link:
https://localhost:8080/data?key=<paste_encoded_gs_file_key_here>
If this step didn't work then something is up with your download handler, maybe this line is transforming the key somehow str(urllib.unquote(self.request.params.get('key','')))
If this step worked then something is up with your code that generates this link https://localhost:8080/data?key=..., maybe you're actually writing to a different gcs_filename than what you are constructing a different BlobKey for.

Related

unable to create table on database in flask_sqlalchemy on repl.it

it seems that this problem only arises when i am using flask application factory method
here is the link to repl, and
here is the link to post i made asking fro help (Kindly read the comment section)
I have commented the app.run() part so that it won't start server
#!main.py l:3
#if __name__ == '__main__':
# app.run(host='0.0.0.0')
After running the repl once on the console side i typed
from app.main.models import *
with app.app_context():
db.create_all()
now in theory these line of code should create database with table defined init such that if i do Food.query.all it should return (in theory) an empty list => [] but it returns an error RuntimeError: No application found.
note : Kindly fork the repl you don't need an account for that just click on the fork button beside run because it might show wired when many people run it together
It seems that you've not commit the session & it's not the best practice to import function by *. You should specify specicfic module or Maybe you haven't insert any values in db so you are getting an empty list. I'm not sure about this.But your solution might be in between.

How to store an object between script runs?

I'm using a python script based on this Mint API to pull personal finance info every hour.
To connect to Mint I do mint = mintapi.Mint(email, password) which opens a Chrom instance via selenium and logs into Mint, and creates an object of <class 'mintapi.api.Mint'>
To refresh info I just need to do mint.initiate_account_refresh().
But every time I run the script, it does the whole logging in thing again.
Can I store the mint object on disk somehow so I can skip that step and just do the account refresh?
Ah the wonders of open source.
Curious, I went and looked at the mintapi you linked, to see if there was anything obvious and simple I could do to recreate the object instance without the arduous setup.
It turns out there isn't, really. :(
Here is what is called when you instantiate the Mint object:
def __init__(self, email=None, password=None):
if email and password:
self.login_and_get_token(email, password)
As you can see, if you don't give it a truthy email and password, it doesn't do anything. (As a sidenote, it should really be checking is None, but whatever).
So, we can avoid having do go through the setup process nice and easily, but now we need to find out how to fake the setup process based on previous data.
Looking at .login_and_get_token(), we see the following:
def login_and_get_token(self, email, password):
if self.token and self.driver:
return
self.driver = get_web_driver(email, password)
self.token = self.get_token()
Nice and simple, again. If it already has a token, it's done, so it goes away. If not, it sets a driver, and sets .token by calling .get_token().
This makes the whole process really easy to override. Simply instantiate a Mint object with no arguments like so:
mint = mintapi.Mint()
Then set the .token on it:
mint.token = 'something magical'
Now you have an object that is in an almost ready state. The problem is that it relies on self.driver for basically every method call, including your .initiate_account_refresh():
def initiate_account_refresh(self):
self.post(
'{}/refreshFILogins.xevent'.format(MINT_ROOT_URL),
data={'token': self.token},
headers=JSON_HEADER)
...
def post(self, url, **kwargs):
return self.driver.request('POST', url, **kwargs)
This looks like it's a simple POST that we could replace with a requests.post() call, but I suspect that seeing as it's doing it through the web browser that it's relying on some manner of cookies or session storage.
If you wanted to experiment, you could subclass Mint like so:
class HeadlessMint(Mint):
def post(self, url, **kwargs):
return requests.post(url, **kwargs)
But my guess is that there will be more issues with this that will surface over time.
The good news is that this mintapi project looks reasonably simple, and rewriting it to not rely on a web browser doesn't look like an unreasonable project for someone with a little experience, so keep that in your back pocket.
As for pickling, I don't believe that will work, for the same reason that I don't believe subclassing will - I think the existence of the browser is important. Even if you pickle your mint instance, it will be missing its browser when you try to load it.
The simplest solution might very well be to make the script a long-running one, and instead of running it every hour, you run it once, and it does what it needs to, then sleeps for an hour, before doing it again. This way, you'd log in once at the very beginning, and then it could keep that session for as long as it's running.
To store objects in Python you can use the pickle module.
Let's say you have an object mint
import pickle
mint = Something.Somefunc()
with open('data.pickle','wb') as storage:
pickle.dump(mint,storage)
The object will be saved as a sequence of binary bytes in a file named data.pickle.
To access it just use the pickle.load() function.
import pickle
with open('data.pickle','rb') as storage:
mint = pickle.load(storage)
>>>mint
>>><class 'something' object>
NOTE:
Although it doesn't matter here but the pickle module has a flaw that it can execute some function objects while loading them from a file, so don't use it when reading pickle stored object from a third party source.
Use a library pickle to save and load object
SAVE
import pickle
mint = mintapi.Mint(email, password)
with open('mint .pkl', 'wb') as output:
pickle.dump(mint , output, pickle.HIGHEST_PROTOCOL)
LOAD
import pickle
with open('mint.pkl', 'rb') as input:
mint= pickle.load(input)
mint.initiate_account_refresh()

How can I generate multiple URL paths with cherrypy?

I feel like I'm running into a brick wall, as I'm not getting anywhere with this, and I believe, simple task.
I'm trying to generate a URL by the likes of '/path/to/url', but upon gazing at multiple StackOverflow Q&A's, the official documentation for cherrypy, I still cannot seem to wrap my head around the issue.
Here's my code so far:
import details
import input_checker as input
import time
import cherrypy
class Index(object):
#cherrypy.expose
def input(self):
return input.check_input()
#cherrypy.expose
def stream(self):
while True:
return 'Hey'
#return input.check_input()
time.sleep(3)
if __name__ == '__main__':
index = Index()
cherrypy.tree.mount(index.stream(), '/input/stream', {})
cherrypy.config.update(
{'server.socket_host': '0.0.0.0'})
cherrypy.quickstart(index)
So essentially, I want to be able to visit http://127.0.0.1:8080/input/stream, and I will be returned with the given result.
After executing this code, and multiple variants of it, I'm still being returned with a 404 not found error, and I'm not sure what I need to do, in order to get it working.
Any tips and/or supporting documentation that I may have skimmed over?
Thanks guys.
So there are couple problems here, why do you use MethodDispatcher do you actually need it?
To serve you stream function on /input/stream you have to mount it as such:
cherrypy.tree.mount(index.stream(), '/input/stream', your_config)
note /input/stream instead of /stream.
But because you're using MethodDispatcher this will likely make your endpoint return 405 as GET is not allowed on this endpoint - to fix that just remove the MethodDispatcher bit.
But if you do require MethodDispatcher you will have to refactor a bit to something like that:
class Stream:
exposed = True # to let cherrypy know that we're exposing all methods in this one
def GET(self):
return something
stream = Stream()
cherrypy.tree.mount(stream , '/index/stream',
{'/':
{'request.dispatch': cherrypy.dispatch.MethodDispatcher()}
}
)
Also make sure to not actually call your methods when mounting them into cherrypy tree, just pass in the name of the function/class

Python: Mako template lookups per app

I'm using cherrypy with Mako as a template engine.
I want Mako to lookup different directories based on what app is being requested.
I.e.
I have three 'apps': Site, Admin and Install.
They all have their own template folder, structure looking something like:
/template
/template/site
/template/admin
/template/install
/template/system
/system contains some system wide templates, like 404 pages, etc.
I'm using Twiseless as a reference whilst trying to get to grips with cherrypy / mako, but I'm stuck with how to do this.
Read on for a brief overview of how I've tried to do this, but a warning: I think I'm going about this completely the wrong way! :) So, if you have any ideas/pointers, it might be a good idea to save yourself the trouble of reading any further than this.
In my main file, server.py, I do something like:
from libs.plugins.template import MakoTemplatePlugin
engine = cherrypy.engine
makoTemplate = MakoTemplatePlugin(engine, self.base_dir)
setTemplateDirs(makoTemplate, self.template_path)
MakoTemplatePlugin is a slightly modified version of the plugin by the same name found in Twiseless, linked above.
What this code does is set the TemplateLookup to use the default template directories from my global config file. i.e.
/template
/template/system
Then, each time an app is loaded, I call a function (setTemplateDirs) to update the directories where Mako searches.
I thought this would work, but it doesn't. Initially I made the error of creating a new instance of MakoTemplatePlugin for each app. This just resulted in them all being called on each page load, starting with the first one instantiated, containing just the basic, non-app specific directories.
As this was called first, it was triggering a 404 error, as it was searching in the wrong folders.
I instead made sure to pass a reference to the MakeTemplatePlugin to all of my apps. I thought if I ran setTemplateDirs each time each app is called, this would solve the problem... but it doesn't.
I don't know where to put the function so it will run every time a page is requested...
e.g.
# /apps/site/app.py
import somemodule.setTemplateDirs
class Site(object, params):
def __init__(self):
self.params = params
self.makoTemplate = params['makoTemplate']
self.base_path = params['base_path']
setTemplateDirs(self.makoTemplate, self.base_path, '', '/')
#cherrypy.expose
#cherrypy.tools.render(template='index.html')
def index(self):
pass
This obviously just works when the application is first loaded... I tried moving the update function call into a seperate method update and tried calling that for each page, e.g:
#cherrypy.exposed
#cherrypy.tools.render(template='index.html')
#update
def index(self):
pass
But this just gives me config related errors.
Rather than to continue to mess about with this, there must be an easier way.
How would you do it?
Thanks a lot,
Tom
I got this working. Thanks to stephan for providing the link to the mako tool example: http://tools.cherrypy.org/wiki/Mako.
I just modified that slightly to get it working.
If anyone's wondering, the basis of it is that you define tools.mako.directories in your global config, you can then override that in individual app config files.
e.g.
server.conf
...
tools.mako.directories: ['', 'system']
...
site.conf
...
tools.mako.directories: ['site', 'system']
...
I did some extra work to translate the relative URIs to absolute paths, but the crux of it is explained above.

How to match all .sass request to a particular controller in pylons?

I'm using pylons, and want to use clever css.
I created a controller SassController to handle .sass requests, but in the config/routing.py, I don't know how to write the mapping.
What I want is:
client request: http://localhost:5000/stylesheets/questions/index.sass
all such requests will be handled by SassController#index
I tried:
map.connect('/{path:.*}.sass', controller='sass', action='index')
But found only: http://localhost:5000/xxx.sass will be handled, but http://localhost:5000/xxx/yyy.sass won't.
What should I do now?
The routing code using regular expressions so you can make it eat everything in the url regardless of slashes.
The docs are here
It'll look something like:
map.connect(R'/{path:.*?}.sass', controller='SassController', action='index')
#in the SassController
def index(self, path):
return path
http://localhost:5000/blah/blah/something.sass will call SassController.index with path = blah/blah/something

Categories