I have a directory structure on the filesystem on my server. It can contain an arbitrary level of nested subdirectories, themselves which eventually end up containing “reports”/“queries” which are basically just xml files containing data. I'd like to display this directory structure as a nested collapsible menu list view on my site with the "reports" being the leaves in the menu list view and displaying the data from those "reports" in another area of the page once clicked on. In each of the menu headings I’d like to display the directory name (actually, I have already configured a file that maps the directory names to a more user friendly display name).
The structure of the data on disk looks like this (where the categories and subcategories are directories on the filesystem and the queries represent the files whose information I want to display):
My overall question is:
What’s the best way to read the filesystem directory structure into a python data structure so that I can eventually display it on a website using collapsible nested menus to represent the directory structure.
Since the data is being collected from the filesystem to be displayed on a page request, I’m not sure how to use django templates (if I even should) to generate these collapsible menus since the html being generated for the list structure is read on a page request and can and often does change on the filesystem. I'm not sure how to get a static Django template to work with something like that.
What is the best front-end framework out there that integrates well with Django, and preferably supports nested menus or blinds to an arbitrary depth that would allow me to display these menus and eventually the output of the reports files?
Maybe I should I use a django template to display the initial top level of the directory structure and use ajax to grab the next (and next after that) sub-level of data when a user tries to drill down into a category/directory? Ideally, I’d like to avoid the latency of making a request and waiting for a response every time a list item is clicked so I’d like to build the entire page on the server before sending it to the client. But again, I’m not sure if it’s even possible to use django templates to display arbitrary data like this.
Any suggestions and advice are much appreciated, thanks.
PS: I think I can just construct the html programmatically to display the directory structure and send that to the client but I'd like to avoid doing it that way if possible.
Just a suggestion but I would consider keeping a database table of all of the entries with their disk path, file size, file name, etc. Fill that with a scheduled job. Then search in the dbms against those paths/names.
That allows you to show a little bit more information to a user, and makes everything on the django/front end side use tooling that you'd normally expect to work with.
Take the file system out of the equation and think about what the user really wants to be able to do, odds are they don't care about how it is stored on the server.
Related
I have a server which generates some data on a daily basis. I am using D3 to visualise the data (d3.csv("path")).
The problem is I can only access the files if they are under my static_dir in the project.
However, if I put them there, they do eventually get cached and I stop seeing the updates, which is fine for css and js files but not for the underlying data.
Is there a way to put these files maybe in a different folder and prevent caching on them? Under what path will I be able to access them?
Or alternatively how would it be advisable to structure my project differently in order to maybe avoid this operation in the first place. Atm, I have a seperate process that generates the data and stores it the given folder which is independent from the server.
Many thanks,
Tony
When accessing the files you can always add ?t=RANDOM to the request in order to get a "new" data all the time.
Because the request (on the server-side) is "new" - there will be no cache, and from the client side it doesn't really matter.
To get a new random you can use Date.now():
url = "myfile.csv?t="+Date.now()
So I have a Django site that works perfectly and displays everything I want it to in the US. It automatically displays the data from the US data model.
What I want to be able to do is basically have an exact clone of my site, maybe under like mysite.com/canada for example, that displays the data from canada.
One approach was for me to just add in all the data into the database and add a field that says which country it's from, but I'd rather for each countries data to be in a completely different model.
With pure HTML/CSS this would be easy, I would just copy the entire site directory into a sub directory and that would be it for the country. Was wondering if there is something similiar I can do with Django.
Based on what you're describing you should probably be setting up parallel stacks and using either your DNS, Apache, your whatever your HTTP routing tech of choice is to do the separation.
Use a separate database, possibly even a separate server (or WSGI configuration), and keep your code clean.
Creating duplicate "models" based on the value of a field like you're describing breaks a lot of Python's DRY principles.
The question is simply as stated in the title. But the reason is probably important for a good answer.
I need to generate static html files at runtime. Generating the files is not the problem. Using jinja2 and webapp2 I can painlessly generate dynamic html files at runtime and serve them to my users. My problem is that generating these files is expensive. So I want to be able to save them as static html files; so that, when user tries to access them, I just serve the static html if it exists. If it does not exist, then I can create the file and then serve it.
Again I am already using jinja2 to create my string (i.e. file content). So my question is, how do I save the file to a path that my app.yaml file can map to?
Also if you are thinking memcache I am already using that. It is not sufficient storage. So while it’s a buffer, it’s not as good as having static html files.
Some back story:
If I can generate the files in localhost, that'd be even better. The thing is the site has a huge number of pages. But the structure is common to all the pages. So we store the relevant data in the datastore and use jinja2 to inflate the different pages. But because of heavy usage, memcache is not able to keep up. So now it's looking more cost-effective to create static html pages from the data stored in the datastore. Creating those pages by hand is obscene and error prone. So if we could generate the html pages as usual (i.e. jinja2 template and datastore) and then have an automated system for creating the html pages, it would be awesome. We could do the app.yaml part manually.
So after I do
template = jinja_environment.get_template(‘template.html')
content = template.render(template_values)
What do I do to save the file to say ./relevant/path/filename.html
IfI understand your question properly, the best way to do this would be to store your static files in Google Cloud Storage (GCS). Easy to read and write to GCS. And easy to delete the file when you no longer need it.
You can't write to files in App Engine -- see https://cloud.google.com/appengine/kb/java?csw=1#writefile for explanation.
Why don't you store the post-rendered page content back in datastore? I'm assuming the "expensive" part of the approach you're using now is the building of the templated content from the data. Once you've done that, instead of writing it off to local files, you could either put it as an additional property on the original data entity or you could create an entirely new model just for the rendered pages.
Essentially this would be a durable datastore implementation instead of memcache, if the issue is that pages are getting evicted too often from memcache.
A single fetch (directly from the key, since you can use keynames that you can create based on the page you're fetching, so you don't have to even do a query) is lightweight. Then, if you're still using memcache, then you'd check memcache first; if not there, you fetch the entity from the datastore and check to see if the rendered property contains data. If not, you use the data in the entity to render it with Jinja, store it back off in the rendered property, put the entity, store it off in memcache, and return the rendered content back to the client.
Sorry if the question seems stupid. I'm currently trying to create a website using Flask. This website will let user entry with an images. Let's say there is a price, a title, a description and an image. My problem here is with the images. I can store all the information I need in the databse but I really don't know what to do about the images. For example, when people enter the website, I'd want to display a couple of those images.
BUT, where do I store them, how do I serve them when they are called ? (For example, if I look at a user profile I'd like to see his post, images, etc...) I just don't know what to do about those images. If someone could explain like i'm an idiot or link me some information about that.
(For example, when you look at someone's profil on instagram you'll see every on his images, that's what i'd like too achieve)
Thanks !
You could store the images on the filesystem of your web server, then configure your webserver to serve them statically (if using Apache, for example, this would be done via the Alias Directive).
This would involve mapping URLs (e.g. mysite.com/img/) to a directory on the web server's filesystem (e.g. /home/images). Of course, your webserver will most likely be serving a directory by default, for example /var/www/, and you could simply store the images here.
Where you store the images (and how they are served) will depend on things specific to your application, such as security (note that if you serve a particular directory on your filesystem, any client could potentially download the contents).
EDIT
As per your comment - This would depend mostly on the size of the available storage on your web server. Also keep in mind that you should use images of a reasonable size, and a compressed format such as PNG or GIF.
I have a fairly basic GAE app that takes some input, fetches some data from a webpage, parses, then presents it to the user. Right now, the fairly spare input HTML form POSTs the arguments to the output 'file' which is wholly generated by the handler for that URL.
I'd like to do a couple things with the data (e.g. graph it at a landing page perhaps, then write it to an output file), but I don't know how I should pass the parsed data between the different handlers. I could maybe encode it then successively POST it to other handlers, but my gut says that I shouldn't need to HTTP the data back and forth within my app—it seems terribly inefficient (my gut is also hungry...).
In fairly broad swaths (or maybe a link to an example), how should my handlers handle this?
Later thoughts (edit)
My very rough idea now is to have the form submitted to a page that 1) enters the subsequent query into a database (datastore?) keyed to some hash, then uses that to 2) grab and parse all the data. The parsed data would be stored in memory (memcache?) for near-immediate use to graph it and/or process it into a variety of tabular formats for download. The script that does said parsing redirects to a unique URL based on the hash which can be referred to in order to get the data.
The thought would be that you could save the URL, then if you visit it later when the data has been lost, it can re-query the source to get it back/update it.
Reasonable? Should I be looking at other things?