Below, I print out the html for a form defined externally. Is there a difference in the way each string is retrieved and used in foo.py, other than the syntax? If so, in what circumstances would one method be preferred over the other? For example, would I be better off defining a number of html files in a module as strings and access them that way, as opposed keeping them in separate .html files and using open over and over?
mod.py
form = """\
<form type="POST" action="test.py">
Enter something:<input type="text" name="somethign">
</form>
"""
form.html
<form type="POST" action="test.py">
Enter something:<input type="text" name="something">
</form>
foo.py
import mod
print mod.form
with open('form.html', 'r') as form:
print form.read()
Having .html files is better. Sure, you will have some overhead opening a file, reading its content and then closing it, but you'll have many advantages:
.html file can be edited by any person who knows HTML syntax.
.html files can be edited without restarting your program, it is very useful for services.
You can eliminate open/read/close overhead by introducing some caching technique.
It's a lot easier for designers to edit discrete HTML files than to deal with HTML embedded in code.
Related
I am using the mako template library to generate a test.txt file, but the generated file contains additional empty lines in between each row.
I have found a similar question about this issue here and the solution proposed suggests the use of markupsafe, but I am not convinced that this is also suitable in my case as it considers passing the text to be formatted as an argument when rendering the template, which is not what I want to do (I don't want to change most of the text in the template, just a few variables I am inputting).
I think it's also worth mentioning that if I print the rendered template in Python, it prints with the correct formatting; the extra lines only appear in the file I write the template data to (test.txt) using Python's file write().
Piggy-backing off of this answer, the solution is to open the file for write-binary instead of just write. Then you need to convert your string to bytes and write that to your file. The following worked for me (tl;dr last two lines):
templates = TemplateLookup(directories=[input_dir,], module_directory=mako_module_dir)
try:
rendered_output = templates.get_template(target_template).render_unicode(**data)
except:
print(mako_exceptions.text_error_template().render())
return
with open(f'{output_dir}{os.path.sep}{os.path.basename(output_filename)}', 'wb') as outfile:
outfile.write(rendered_output.encode())
I am using the application Zim Wiki(cross-platform, FOSS), which I am using to keep a personal wiki with lots of data coming from tables, copy and pasting, my own writing, and downloading and attaching .png and .html files for viewing and offline access.
The data that is not written or pasted can be stored in tables in the form of names, url addresses, and the names and locations of images and other attachments.
To insert into zim, I can use the front end with WSIWYG, or to make the skeleton of each entry, I could modify a template text entry. If I do this, nothing matters except for the location and identity of each character in each line.
By supplying the text in this image:
DandelionDemo source text,
--I can make this entry for Dandelion:
DandelionDemo Wiki.
So, I can generate and name the Wiki entry in Zim, which creates the .txt file for me, and inserts the time stamp and title, so, the template for this type of entry without the pasted fields would be:
**Full Scientific Name: **[[|]]**[syn]**
**Common Name(s): **
===== =====
**USDA PLANTS entry for Code:** [[https://plants.usda.gov/core/profile?symbol=|]] **- CalPhotos available images for:** [[https://calphotos.berkeley.edu/cgi/img_query?query_src=photos_index&where-taxon=|]]
**---**
**From - Wikipedia **[[wp?]] **- **[[/Docs/Plants/]]
{{/Docs/Plants/?height=364}}{{/Docs/Plants/?height=364}}
**()** //,// [[|(source)]]
**()** //// [[|(source)]]
**Wikipedia Intro: **////
---
So the first line with content, after the 31st character(which is a tab), you paste "http... {etc}. Then the procedure would insert "Taraxacum officinale... {etc}" after the "|", or what was the 32nd character, and so on. This data could be from "table1" and "table2", or combining the tables to make an un-normalized "table1table2", where each row could be converted to text or a .csv or I don't know, what do you think?
Is there a way, in LibreOffice to do this? I have used LibreOffice Base to generate a "book" form that populated fields, but it was much less complex data, without wiki liking and drag-and-drop pasting of images and attachments. So maybe the answer is to go simpler? The tables are not currently part of a registered database, but I could do that, once I have decided on the method of doing this.
I am ultimately looking for a "way", hopefully an "easy" way. However, that may not be in LibreOffice. If not, I know that I could do this in Python, but I haven't learned much about Python yet. If it involves a language, that is the first and only one I don't know that I will invest in learning for this project. If you know a "way" to do this in Python, let me know, and my first project and way of framing my study process will be in learning the methods that you share.
If you know of some other Linux GUI, I am definitely interested, but only in active free and open source builds that involve minimal/no compiling. I know the basics of SQL and DBMS's. In the past, have gotten Microsoft SQL server lite to work, but not DBeaver, yet. If you know of a CLI way also let me know, but I am a self-taught outdoors-loving Linux newb and mostly know about how to tweak little settings in programs, how to use moderately easy programs like ImageMagick, and I have built a few Lamp stacks for Drupal and Wordpress (no BASH etc...).
Thank you very much!
Ok, since you want to learn some python, let me propose you a way to do it this. First you need a template engine -like jinja2 (there are many others)-, a data source in our example a .csv file, -could be other like a db- and finally some code that reads the csv line by line and mix the content with the template.
Sample CSV file:
1;sample
2;dandelion
3;just for fun
Sample template:
**Full Scientific Name: **[[|]]**[syn]**
**Common Name(s): *{{name}}*
===== =====
USDA PLANTS entry for Code: *{{symbol}}*
---
Sample code:
#!/usr/bin/env/python
#
# Using the file system load
#
# We now assume we have a file in the same dir as this one called
# test_template.ziim
#
from jinja2 import Environment, FileSystemLoader
import os
import csv
# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
def print_zim_doc():
# Create the jinja2 environment.
# Notice the use of trim_blocks, which greatly helps control whitespace.
j2_env = Environment(loader=FileSystemLoader(THIS_DIR),
trim_blocks=True)
template = j2_env.get_template('test_template.zim')
with open('example.csv') as File:
reader = csv.reader(File, delimiter=';')
for row in reader:
result = template.render(
symbol=row[0]
, name=row[1]
)
# to save the results
with open(row[0]+".txt", "wt") as fh:
fh.write(result)
fh.close()
if __name__ == '__main__':
print_zim_doc()
The code is pretty simple, reads the template located in the same folder as the python code, opens the csv file (also located in the same place), iterates over each line of the csv and renders the template using the values of the csv columns to fill the {{var_name}} in the template, finally saves the rendered result in a new file named as one of the csv column values. This sample will generate 3 files (1.txt, 2.txt, 3.txt). From here you can extend and improve the code to get your desired results.
I'm trying to write a really simply webapp with PythonAnywhere and Flask that has lets the user upload a text file, generates a csv file, then lets the user download the csv file. It doesn't have to be fancy, it only has to work. I have already written the program for generating the csv from a txt file on the drive.
Right now, my function opens the file on the drive with:
with open(INPUTFILE, "r") as fname:
and writes the csv with:
with open(OUTPUTFILE, 'w') as fname:
with INPUTFILE and OUTPUTFILE being filename strings.
Would it be better for me to handle the files as objects, returned by the flask/html somehow?
I don't know how to do this. How should I structure this program? How many HTML Templates do I need? I would prefer to work on the files wihthout saving them anywhere but if I have to save them to the PythonAnywhere directory, I could. How can I do that?
PythonAnywhere dev here. This is a good question about Flask and web development in general rather than specific to our system, so I'll try to give a generic answer without anything specific to us :-)
There are a few things that I'd need to know to give a definitive answer to your question, so I'll start by listing the assumptions I'm making -- leave me a comment if I'm wrong with any of them and I'll update the answer appropriately.
I'm assuming that the files you're uploading aren't huge and can fit into a reasonable amount of memory -- let's say, smaller than a megabyte.
I'm assuming that the program that you've already written to generate the CSV from the text file is in Python, and that it has (or, perhaps more likely, could be easily changed to have) a function that takes a string containing the contents of the text file, and returns the contents that need to be written into the CSV.
If both of those are the case, then the best way to structure your Flask app would be to handle everything inside Flask. A code sample is worth a thousand words, so here's a simple one I put together that allows the user to upload a text file, runs it through a function called transform (which is where the function from your conversion program would slot in -- mine just replaces = with , throughout the file), and sends the results back to the browser. There's a live version of this app on PythonAnywhere here.
from flask import Flask, make_response, request
app = Flask(__name__)
def transform(text_file_contents):
return text_file_contents.replace("=", ",")
#app.route('/')
def form():
return """
<html>
<body>
<h1>Transform a file demo</h1>
<form action="/transform" method="post" enctype="multipart/form-data">
<input type="file" name="data_file" />
<input type="submit" />
</form>
</body>
</html>
"""
#app.route('/transform', methods=["POST"])
def transform_view():
request_file = request.files['data_file']
if not request_file:
return "No file"
file_contents = request_file.stream.read().decode("utf-8")
result = transform(file_contents)
response = make_response(result)
response.headers["Content-Disposition"] = "attachment; filename=result.csv"
return response
Regarding your other questions:
Templates: I didn't use a template for this example, because I wanted it all to fit into a single piece of code. If I were doing it properly then I'd put the stuff that's generated by the form view into a template, but that's all.
Can you do it by writing to files -- yes you can, and the uploaded file can be saved by using the save(filename) method on the file object that I'm using the stream property of. But if your files are pretty small (as per my assumption above) then it probably makes more sense to process them in-memory like the code above does.
I hope that all helps, and if you have any questions then just leave a comment.
Better to add
response.headers["Cache-Control"] = "must-revalidate"
response.headers["Pragma"] = "must-revalidate"
response.headers["Content-type"] = "application/csv"
If you don't add the content type, FF 48.0 reported it as html and opened Save dialog once for HTML and then for CSV. If you don't add Cache-Control your result may get cached, and if you serve active content this is not what you want. If you use must-revalidate with no age, it will effectively serve as no-cache - see here and here for an explanation.
I'm trying to download an ipython notebook file as a .py file. It's working fairly well, except the .py file is interspersed with "In: []" at cell boundaries. I could live with them being there, but I'd rather they weren't.
Any simple fix?
Example output (what I see in the .py file):
# In[4]:
# Get names of all files
text_files = glob.glob('hw3data/*')
#print text_files
# In[5]:
def file_contents(file_name):
with open(file_name) as f:
return f.read()
Edit: Essentially, I'm wondering if it's possible to make notebook itself not output #In[ ]. Is there a commandline option, or a utility, or some kind of %magic?
Edit: Going by https://github.com/ipython/ipython/issues/5780, it looks like the suggested solution is just to use a custom template. From minrk:
It's just a marker that indicates where the cells were. It is by design, but it has no effect on the Python, since it is a comment. If you want to remove them, you can use a custom exporter template that doesn't add these comments.
try
def rc(in_, out_):
in_ = open(in_, 'r')
out_ = open(out_,'w')
out_.write(in_.readline())
for line in in_:
if not line.lstrip().startswith('# In['):
out_.write(line)
in_.close()
out_.close()
rc('~/in_file.py', '~/out_file.py')
As a temporary hack, I found that removing lines seven through nine of the python export template of nbconvert (ipython/IPython/nbconvert/templates/python.tpl in site-packages) stops the input prompt from being output. This still isn't ideal, as I'm modifying site packages.
Affected lines:
{% block in_prompt %}
# In[{{ cell.prompt_number if cell.prompt_number else ' ' }}]:
{% endblock in_prompt %}
I am using Bottle for uploading rather large files. The idea is that when the file is uploaded, the web app run (and forget) a system command with the uploaded file-path as an argument. Except for starting the system command with the correct file-path as an argument I do not need to save the file, but I need to be certain that the file will be available until the process completes the processing.
I use the exact code described here:
http://bottlepy.org/docs/dev/tutorial.html#post-form-data-and-file-uploads
My questions are:
Do bottle store uploaded file in memory or on a specific place on the disk (or perhaps like Flask, a bit of both)?
Will the uploaded file be directly available to other tools without .read() and then manually saving the bytes to a specified file on disk?
What would be the best way to start the system command with the file as an argument? Is it possible to just pass the path to an existing file directly?
Ok, let's break this down.
The full code is:
HTML:
<form action="/upload" method="post" enctype="multipart/form-data">
<input type="text" name="name" />
<input type="file" name="data" />
</form>
PYTHON CODE:
from bottle import route, request
#route('/upload', method='POST')
def do_upload():
name = request.forms.name
data = request.files.data
if name and data and data.file:
raw = data.file.read() # This is dangerous for big files
filename = data.filename
return "Hello %s! You uploaded %s (%d bytes)." % (name, filename, len(raw))
return "You missed a field."
(From the doc's you provided)
So, first of all, we can see that we first pull the information from the name and the data in the html form, and assign them to the variables name and data. Thats pretty straight forward. However, next we assign the variable raw to data.file.read(). This is basically taking all of the file uploaded into the variable raw. This being said, the entire file is in memory, which is why they put "This is dangerous for big files" as a comment next to the line.
This being said, if you wanted to save the file out to disk, you could do so (but be careful) using something like:
with open(filename,'w') as open_file:
open_file.write(data.file.read())
As for your other questions:
1."What would be the best way to start the system command with the file as an argument? Is it possible to just pass the path to an existing file directly?"
You should see the subprocess module, specifically Popen: http://docs.python.org/2/library/subprocess.html#popen-constructor
2."Will the uploaded file be directly available to other tools without .read() and then manually saving the bytes to a specified file on disk?"
Yes, you can pass the file data around without saving it to disk, however, be warned that memory consumption is something to watch. However, if these "tools" are not in python, you may be dealing with pipes or subprocesses to pass the data to these "tools".
with open(filename,'w') as open_file:
open_file.write(data.file.read())
dont work
you can use
data = request.files.data
data.save(Path,overwrite=True)
The file will be handled by the routine you use. That means your read handles the connection (the file should not be there, according to wsgi spec)
with open(filename, "wb") as file:
Data = data.file.read()
if type(Data) == bytes: file.write(Data)
elif type(Data) == str: file.write(Data.encode("utf-8"))
Easy :D