We Keep Coding

JSONDecodeError when convert OpenStreetMap query in jupyter notebook

I need help related to OpenStreetMap. I'm using python (jupyter notebook) to get data of hospitals in Bali area, Indonesia. Here is my code and query:
import pandas as pd
import requests
import json
overpass_api = "http://overpass-api.de/api/interpreter"
query_hospital = """
[out:json];
{{geocodeArea:'Provinsi Bali'}}->.searchArea;
node[amenity='hospital'](area.searchArea);
out;
"""
response_hospital = requests.get(overpass_api, params={'data':query_hospital})
but when I run the next code,
data_hospital = response_hospital.json()
it returns error JSONDecodeError: Expecting value: line 1 column 1 (char 0)
the query works well in Overpass Turbo but when I put in notebook, it returns error.

I've found the solution. Looks like python can't parse the double curly braces {{ }} in the openstreetmap query. So I modify the query into like this
query_hospital = """
[out:json];
area[name=Bali];
node[amenity='hospital'](area);
out;
"""
or if we use area name in local language
query_hospital = """
[out:json];
area['name:id'='Provinsi Bali'];
node[amenity='hospital'](area);
out;
"""
the query returns same result and now python can parse it.

Run Python script in Django template through a button

I have a very small project in Django where I get fx_rates through a Python script (details below). These are used to convert amounts in different currencies to GBP. I would like to know how I can just have a button on the website that allows to refresh this query instead of having to run it manually in the IDE. Can I create a view for this? How would I show this in a template? Thanks
fx_rates.py
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE','fx_rates_project.settings')
import django
django.setup()
from fx_rates_app.models import fx_table
import pandas_datareader.data as web
from datetime import datetime
os.environ["ALPHAVANTAGE_API_KEY"] = '#########'
fx_gbp_to_eur = web.DataReader("GBP/EUR","av-forex")
eur = float(fx_gbp_to_eur[4:5].values[0][0])
fx_gbp_to_aud = web.DataReader("GBP/AUD","av-forex")
aud = float(fx_gbp_to_aud[4:5].values[0][0])
fx_gbp_to_usd = web.DataReader("GBP/USD","av-forex")
usd = float(fx_gbp_to_usd[4:5].values[0][0])
from datetime import datetime
now = datetime.now()
dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
webpg1 = fx_table.objects.get_or_create(eur_to_gbp=eur,aud_to_gbp=aud,usd_to_gbp=usd,date_time=dt_string)[0]
In the template I included:
<form method="post">
{% csrf_token %}
<button type="submit" name="run_script">Refresh</button>
</form>
However, I don't think the best way to do this would be to copy the all script in views with request.POST. Is there another way to leave the script in another file and just create a view to run it (e.g. if request.method == 'POST' run fx_rates.py)?

This can be definitely done by creating a function update_fx to wrap your script and then call that function in your view.
Now for the tricky part.
Do you need the values to render them or just to update them?
Rendering will be easy, just return them from your function and you will
be able to use them in your template.
This though includes the waiting time to fetch these values. When your view will be executed you will have to wait for the function to fetch the values.
The alternative is to "trigger" a second job that updates the values from your
view. This pattern is called a task queue and there are many solutions that you can leverage already, depending on how complicated things should be (what happens if the job has an error? how many times should we retry? ...).
This also has the unfortunate effect that now you run a second python script somewhere that listens for new tasks and executes them.
As always your choice is based on tradeoffs, personally if the server has good internet connection and the request is completed fast I would not mind putting it
in my view as is.
Hope this is helpful.
Let's see an example
# update.py
from fx_rates_app.models import fx_table
import pandas_datareader.data as web
from django.conf import settings
from datetime import datetime
import os
def get_price():
os.environ["ALPHAVANTAGE_API_KEY"] = '#########'
fx_gbp_to_eur = web.DataReader("GBP/EUR","av-forex") # here we are reading stuff from the network I guess
eur = float(fx_gbp_to_eur[4:5].values[0][0])
fx_gbp_to_aud = web.DataReader("GBP/AUD","av-forex")
aud = float(fx_gbp_to_aud[4:5].values[0][0])
fx_gbp_to_usd = web.DataReader("GBP/USD","av-forex")
usd = float(fx_gbp_to_usd[4:5].values[0][0])
now = datetime.now()
dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
# here you have two choices,
# either you return the raw values
# or you put them into a model
return {
'timestamp': dt_string,
'value': 1,
'currency': 'gbp',
'eur': eur,
'aud': aud,
'us': us,
}
webpg1 = fx_table.objects.get_or_create(eur_to_gbp=eur,aud_to_gbp=aud,usd_to_gbp=usd,date_time=dt_string)[0]
return webpg1
# views.py
from django.http import HttpResponse
from update import get_price
def refresh_from_values(request):
data = get_price() # here we don't have a model, just data
webpg1 = fx_table.objects.get_or_create(
eur_to_gbp=data['eur'],
aud_to_gbp=data['aud'],
usd_to_gbp=data['usd'],
date_time=data['timestamp'],
)[0]
# here your render the model using the template
def refresh_from_model(request):
fx = get_price() # here we get back the model instead
# you can render again the model as you can refresh the page
In the future you might be interested in running this thing continuously
and then you would have to move to an event driven app which is more compilicated but can be very satisfying to get right. In django everything
would start with signals and more likely you would have to write some javascript to
update part of the page without refreshing it.

Python flask application not displaying generated html file for second time

I have a Python flask application which takes input id's and dynamically generates data into a html file. Below is my app.py file.
#app.route('/execute', methods=['GET', 'POST'])
def execute():
if request.method == 'POST':
id = request.form['item_ids']
list = [id]
script_output = subprocess.Popen(["python", "Search_Script.py"] + list)
# script_output = subprocess.call("python Search_Script.py "+id, shell=True)
# render_template('running.html')
script_output.communicate()
#driver = webdriver.Chrome()
#driver.get("home.html")
#driver.execute_script("document.getElementById('Executed').style.display = '';")
return render_template('execute.html')
#app.route('/output')
def output():
return render_template('output.html')
output.html file has below code at the bottom.
<div class="container" style="text-align: center;">
{% include 'itemSearchDetails.html' %}
</div>
itemSearchDetails.html is generated every time dynamically based on the input. I check for different inputs and it is generating perfectly. When I run it with some input(assume 2) values for the first time, it runs perfectly and shows the output correctly. But, when I run for different values(assume 4) for the next time, the file 'itemSearchDetails.html' is generated for those 4 values but the browser only shows output for the first 2 values. No matter how many times I run it, browser shows only output with the first run values.
So, every time only the first inputted values are shown no matter how many times I run. I am not sure if it is browser cache issue since I tried "disabling cache" in chrome. Still it didn't work. Please let me know if there is something I am missing.

Try solution from this answer:
Parameter TEMPLATES_AUTO_RELOAD
Whether to check for modifications of the template source and reload
it automatically. By default the value is None which means that Flask
checks original file only in debug mode.
Original documentation could be found here.

Looks like Jinja is caching the included template.
If you don't need to interpret the HTML as a Jinja template, but instead just include its contents as-is, read the file first and pass the contents into the template:
with open('itemSearchDetails.html', 'r') as infp:
data = infp.read()
return render_template('execute.html', data=data)
...
{{ data|safe }}
(If you do need to interpret the HTML page as Jinja (as include will), you can parse a Jinja Template out of data, then use the include tag with that dynamically compiled template.)

Incorrect conversion from epoch time when scraping web

import praw,time
import sys
reload(sys)
sys.setdefaultencoding("utf-8")
username=""
password=""
r = praw.Reddit(user_agent='')
r.login(username,password,disable_warning=True)
posts=r.search('china disaster', subreddit=None, sort=None, syntax=None, period=None,limit=7)
title=[];created=[]
for index,post in enumerate(posts):
date=time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime(post.created))
title.append(post.title);created.append(post.created)
print date,title[index]
break #added so it prints one post as a example
Error:
I get incorrect times.
<time title="Fri Jan 23 01:22:20 2015 UTC" datetime="2015-01-22T17:22:20-08:00" class="">5 months ago</time>
I don't understand the issue, I think I making a mistake in time-zone conversion. But reddit posts mention UTC, thus I don't get the error.

I didn't get your exact problem about how is the "Incorrect".
There are two attributes about the created time "created" and "created_utc". Maybe you want to try the second one instead.

maintaining formatting of imported text with mako and rst2pdf

I've created a template which renders pdf files from csv input. However, when the csv input fields contain user formatting, with line breaks and indentations, it messes with the rst2pdf formatting engine. Is there a way to consistently deal with user input in a way that doesn't break the document flow, but also maintains the formatting of the input text? Example script below:
from mako.template import Template
from rst2pdf.createpdf import RstToPdf
mytext = """This is the first line
Then there is a second
Then a third
This one could be indented
I'd like it to maintain the formatting."""
template = """
My PDF Document
===============
It starts with a paragraph, but after this I'd like to insert `mytext`.
It should keep the formatting intact, though I don't know what formatting to expect.
${mytext}
"""
mytemplate = Template(template)
pdf = RstToPdf()
pdf.createPdf(text=mytemplate.render(mytext=mytext),output='foo.pdf')
I have tried adding the following function in the template to insert | at the start of each line, but that doesn't seem to work either.
<%!
def wrap(text):
return text.replace("\\n", "\\n|")
%>
Then ${mytext} would become |${mytext | wrap}. This throws the error:
<string>:10: (WARNING/2) Inline substitution_reference start-string without end-string.

Actually it turns out I was on the right track, I just needed a space between the | and the text. So the following code works:
from mako.template import Template
from rst2pdf.createpdf import RstToPdf
mytext = """This is the first line
Then there is a second
Then a third
How about an indent?
I'd like it to maintain the formatting."""
template = """
<%!
def wrap(text):
return text.replace("\\n", "\\n| ")
%>
My PDF Document
===============
It starts with a paragraph, but after this I'd like to insert `mytext`.
It should keep the formatting intact.
| ${mytext | wrap}
"""
mytemplate = Template(template)
pdf = RstToPdf()
#print mytemplate.render(mytext=mytext)
pdf.createPdf(text=mytemplate.render(mytext=mytext),output='foo.pdf')