We Keep Coding

Python MySQL Query Not working (Something went wrong format requires a mapping)

I am trying to pull a query from my database and I am receiving this error when trying to run it: Something went wrong format requires a mapping.
I'm using flask in Python and pymysql.
This is my class method that is throwing the error:
#classmethod
def get_dojo(cls, data):
query = 'SELECT * FROM dojos WHERE id = %(id)s;'
result = connectToMySQL('dojos_and_ninjas').query_db(query, data)
return cls(result[0])
I thought it might be the data I am passing through but it looks good to me, and the query runs fine in workbench. I tried restarting MySQL, VS Code, and restarting the pipenv.
The data I am passing is:
#app.route('/dojo/<int:id>')
def dojo_page(id):
dojo_current = Dojo.get_dojo(id)
return render_template('dojo_page.html', dojo = dojo_current)
My page will render and I receive no error when I enter an id in manually instead of calling the data into it.

I figured it out, I needed to add a data dictionary in the route.
#app.route('/dojo/<int:id>')
def dojo_page(id):
data = {
'id': id
}
dojo_current = Dojo.get_dojo(data)
return render_template('dojo_page.html', dojo = dojo_current)

Change bigquery view definition with python

I'm trying to update a view in bigquery via python. I've been able to create the view using the following approach;
def createView(client):
viewDataset = 'dataset'
viewName = 'name'
view_ref = client.dataset(viewDataset).table(viewName)
view = bigquery.Table(view_ref)
view_sql = """
select * from '{}.{}' where thing = 2
"""".format(viewDataSet, viewName)
view.view_query = view_sql
client.create_table(view)
(Code for explanation purposes)
This worked fine and created the view. I then wanted to run a function that updates the view definition. I reused the same code and it failed with an error saying the view exists already - this makes sense. I then followed this example here;
https://cloud.google.com/bigquery/docs/managing-views
Using the code to update a views SQL query. Basically I swapped the line
client.create_table(view)
for
client.update_table(view)
I get an error saying I have not added the fields attribute... Being a view, I though I wouldn't have to do this.
Can anyone tell me the correct way to use python to update an existing bigquery view?
Cheers

Look! You are using:
"select * from '{}.{}' where thing = 2"
Notice this:
from '{}.{}'
But a table should be referenced as:
from '{}.{}.{}'
This piece of code works to me:
from google.cloud import bigquery
if __name__ == "__main__":
client = bigquery.Client()
dataset_view_id= 'dataset_name'
table_view_id = 'view_name'
view = bigquery.Table(client.dataset(dataset_view_id).table(table_view_id))
##############
###what was in that table? request table info
##############
get_view = client.get_table(view) # API Request
# Display OLD view properties
print('View at {}'.format(get_view.full_table_id))
print('View Query:\n{}'.format(get_view.view_query))
##############
#update the table:
##############
sql_template = (
'SELECT * FROM `{}.{}.{}` where disease="POLIO" ')
source_project_id = "project_from_the_query"
source_dataset_id = "dataset_from_the_query"
source_table_id = "table_from_the_query"
view.view_query = sql_template.format(source_project_id, source_dataset_id, source_table_id)
view = client.update_table(view, ['view_query']) # API request
##############
#Now print the view query to be sure it's been updated:
##############
get_view = client.get_table(view) # API Request
# Display view properties
print('\n\n NEW View at {}'.format(get_view.full_table_id))
print('View Query:\n{}'.format(get_view.view_query))
# [END bigquery_get_view]

How can I create a new view in bigquery using the python API?

I have some code that automatically generates a bunch of different SQL queries that I would like to insert into the bigquery to generate views, though one of the issues that I have is that these views need to be generated dynamically every night because of the changing nature of the data. So what I would like to be able to do is use the google bigquery api for python to be able to make a view. I understand how to do it using the 'bq' command line tool, but I'd like to be able to have this built directly into the code as opposed to using a shell to run bq. I have played with the code provided at
https://cloud.google.com/bigquery/bigquery-api-quickstart
I don't understand how to use this bit of code to create a view instead of just returning the results of a SELECT statement. I can see the documentation about doing table inserts here
https://cloud.google.com/bigquery/docs/reference/v2/tables/insert
but that refers to using the REST API to generate new tables as opposed to the example provided above.
Is it just not possible? Should I just give in and use bq?
Thanks
*** Some additional questions in response to Felipe's comments.
The table resource document indicates that there are a number of required fields, some of which make sense even if I don't fully understand what they're asking for, others do not. For example, externalDataConfiguration.schema. Does this refer to the schema for the database that I'm connecting to (I assume it does), or the schema for storing the data?
What about externalDataConfiguration.sourceFormat? Since I'm trying to make a view of a pre-existing database, I'm not sure I understand how the source format is relevant. Is it the source format of the database I'm making a view from? How would I identify that?
ANd externalDataConfiguration.sourceUris[], I'm not importing new data into the database, so I don't understand how this (or the previous element) are required.
What about schema?
tableReference.datasetId, tableReference.projectId, and tableReference.tableId are self explanatory.
Type would be view, and view.query would be the actual sql query used to make the view. So I get why those are required for making a view, but I don't understand the other parts.
Can you help me understand these details?
Thanks,
Brad

Using https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/insert
Submit something like below, assuming you add the authorization
{
"view": {
"query": "select column1, count(1) `project.dataset.someTable` group by 1",
"useLegacySql": false
},
"tableReference": {
"tableId": "viewName",
"projectId": "projectName",
"datasetId": "datasetName"
}
}
Alternatively in Python using, assuming you have a service key setup and the environmental variable GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/key. The one caveat is that as far as I can tell this can only create views using legacy sql, and as an extension can only be queried using legacy sql, though the straight API method allows legacy or standard.
from google.cloud import bigquery
def create_view(dataset_name, view_name, project, viewSQL):
bigquery_client = bigquery.Client(project=project)
dataset = bigquery_client.dataset(dataset_name)
table = dataset.table(view_name)
table.view_query = viewSQL
try:
table.create()
return True
except Exception as err:
print(err)
return False

Note: this changed a little bit with 0.28.0 of the library - see the following for further details:
Google BigQuery: creating a view via Python google-cloud-bigquery version 0.27.0 vs. 0.28.0
my example function
# create a view via python
def create_view(dataset_name, view_name, sqlQuery, project=None):
try:
bigquery_client = bigquery.Client(project=project)
dataset_ref = bigquery_client.dataset(dataset_name)
table_ref = dataset_ref.table(view_name)
table = Table(table_ref)
table.view_query = sqlQuery
table.view_use_legacy_sql = False
bigquery_client.create_table(table)
return True
except Exception as e:
errorStr = 'ERROR (create_view): ' + str(e)
print(errorStr)
raise

Everything that web UI or the bq tool does is made through the BigQuery API, so don't give up yet :).
Creating a view is akin to creating a table, just be sure to have a table resource that contains a view property when you call tables.insert().
https://cloud.google.com/bigquery/querying-data#views
https://cloud.google.com/bigquery/docs/reference/v2/tables#resource

bigquery.version -> '1.10.0'
def create_view(client, dataset_name, view_name, view_query):
try:
dataset_ref = client.dataset(dataset_name)
view = dataset_ref.table(view_name)
# view.table_type = 'VIEW'
view.view_query = view_query
view.view_query_legacy_sql = False
client.create_table(view)
pass
except Exception as e:
errorStr = 'ERROR (create_view): ' + str(e)
print(errorStr)
raise
create a table not a view !!!!
This is the right code to create a view:
def create_view(client, dataset_name, view_name, view_query):
try:
dataset_ref = client.dataset(dataset_name)
view_ref = dataset_ref.table(view_name)
table = bigquery.Table(view_ref)
table.view_query = view_query
table.view_use_legacy_sql = False
client.create_table(table)
except Exception as e:
errorStr = 'ERROR (create_view): ' + str(e)
print(errorStr)
raise
Is necessary
table = bigquery.Table(view_ref)

How do you call a python function instead of a script using a form?

I've been reading the book 'Head First Python' where the writer talks about creating dynamic webpages using a module he created called 'yate', an HTML template engine (which I renamed to site_yate in the code below). The example he works through is a hypothetical coach wanting his athletes to be able to check their times online. The design is as follows: first you enter the homepage which has a link to run a script which generates a webpage where you can select the athlete whose times you want to view. Then when you select your athlete and click submit the form calls another script called "site_generate_timing_data.py" where you can views the athlete's top times. So I decided to take it further and add functionality to add a time for the athlete, using this extra line of code in my python script.
print(site_yate.do_form("addtime.py", [athlete_id]))
The HTML this will generate will be this:
<form action="addtime.py" method="POST">
<h1>Want to add a time?</h1>
<input type="Text" name="1" size=40> //"1" is the athlete's id in this example
<input type="Submit" value="Submit">
</form>
As you can see this code calls the script 'addtime.py' which has the following code:
import cgi
import sqlite3
data = cgi.FieldStorage().value[0] #this attribute will be in the form MininFieldStorage(name, value)
id = data.name #this attribute is the input's name i.e. athlete's id
time = data.value #this attribute is the input's value i.e. the time
connection = sqlite3.connect("NUACDB.sqlite") #my DB's name
cursor = connection.cursor()
cursor.execute("""INSERT INTO timing_data (athlete_id, time)
VALUES (?, ?)""",
(id, time)) #just SQL stuff
connection.commit()
connection.close()
Which works fine, however I want to change a few thing about this, since it leaves the user on a blank page. I could generate some HTML code to provide links to the homepage etc. or even JavaScript code to redirect the user automatically, but I want to keep this script HTML-free so that I can also use it elsewhere.
What I want to do instead is make the script execute on the same page. Not only that, but I would also prefer if I could put the addtime.py code as a function in another module called 'athletemodel.py' and call it form there, i.e. athletemodel.addtime() (or I could do from athletemodel import addtime so I can call the function directly). How can I call a python function using HTML code? I'm aware of the onsubmit="" form attribute but apparently that is for JavaScript functions. Another thing I'm unsure about is whether the data submitted in the form will still be accessible through CGI FieldStorage and hence whether my addtime.py code will still work as it is.
This stuff is so confusing! All help is appreciated.

Not sure if you already had it in mind, but I would use ajax (remember to include the jQuery library). Here's a rough example to get you started if this is what you want. It'll keep them on the same page:
JavaScript file:
$('#submitButtonId').click(function (event) {
event.preventDefault();
$('#submitButtonId').hide();
$('#thinking').show(); //some div with a nice ajax loader gif...
$.ajax({
type: 'POST',
data: $('#formId').serialize(),
url: '/URL_path_to_function',
success: function (data) {
$('#loading').hide();
var response = data.result //now do stuff with your response
}
error: function(error){
console.log('Error')}
});
Python view/function:
import jsonify
if request.method == 'POST':
value = request.form['input value'] #flask...
#Do stuff
return jsonify(result='Some response')

How to display data using openlayers with OpenStreetMap in geodjango?

I've got geodjango running using openlayers and OpenStreetMaps with the admin app.
Now I want to write some views to display the data. Basically, I just want to add a list of points (seen in the admin) to the map.
Geodjango appears to use a special openlayers.js file to do it's magic in the admin. Is there a good way to interface with this?
How can I write a view/template to display the geodjango data on a open street map window, as is seen in the admin?
At the moment, I'm digging into the openlayers.js file and api looking for an 'easy' solution. (I don't have js experience so this is taking some time.)
The current way I can see to do this is add the following as a template, and use django to add the code needed to display the points. (Based on the example here)
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Draw Feature Example</title>
<script src="http://www.openlayers.org/api/OpenLayers.js"></script>
<script type="text/javascript">
var map;
function init(){
map = new OpenLayers.Map('map');
var layer = new OpenLayers.Layer.WMS( "OpenLayers WMS",
"http://labs.metacarta.com/wms/vmap0", {layers: 'basic'} );
map.addLayer(layer);
/*
* Layer style
*/
// we want opaque external graphics and non-opaque internal graphics
var layer_style = OpenLayers.Util.extend({}, OpenLayers.Feature.Vector.style['default']);
layer_style.fillOpacity = 0.2;
layer_style.graphicOpacity = 1;
/*
* Blue style
*/
var style_blue = OpenLayers.Util.extend({}, layer_style);
style_blue.strokeColor = "blue";
style_blue.fillColor = "blue";
style_blue.graphicName = "star";
style_blue.pointRadius = 10;
style_blue.strokeWidth = 3;
style_blue.rotation = 45;
style_blue.strokeLinecap = "butt";
var vectorLayer = new OpenLayers.Layer.Vector("Simple Geometry", {style: layer_style});
// create a point feature
var point = new OpenLayers.Geometry.Point(-111.04, 45.68);
var pointFeature = new OpenLayers.Feature.Vector(point,null,style_blue);
// Add additional points/features here via django
map.addLayer(vectorLayer);
map.setCenter(new OpenLayers.LonLat(point.x, point.y), 5);
vectorLayer.addFeatures([pointFeature]);
}
</script>
</head>
<body onload="init()">
<div id="map" class="smallmap"></div>
</body>
</html>
Is this how it's done, or is there a better way?

Another solution is to create a form that utilizes the GeoDjango Admin widget.
To do this, I:
Setup a GeneratePolygonAdminClass:
class GeneratePolygonAdmin(admin.GeoModelAdmin):
list_filter=('polygon',)
list_display=('object', 'polygon')
Where the form is built:
geoAdmin=GeneratePolygonAdmin(ModelWithPolygonField, admin.site)
PolygonFormField=GeneratePolygon._meta.get_field('Polygon')
PolygonWidget=geoAdmin.get_map_widget(PolygonFormField)
Dict['Polygon']=forms.CharField(widget=PolygonWidget()) #In this case, I am creating a Dict to use for a dynamic form
Populating the widget of the form:
def SetupPolygonWidget(form, LayerName, MapFileName, DefaultPolygon=''):
form.setData({'Polygon':DefaultPolygon})
form.fields['Polygon'].widget.params['wms_layer']=LayerName
form.fields['Polygon'].widget.params['wms_url']='/cgi-bin/mapserv?MAP=' + MapFileName
form.fields['Polygon'].widget.params['default_lon']=-80.9
form.fields['Polygon'].widget.params['default_lat']=33.7
form.fields['Polygon'].widget.params['default_zoom']=11
form.fields['Polygon'].widget.params['wms_name']=YOURWMSLayerName
form.fields['Polygon'].widget.params['map_width']=800
form.fields['Polygon'].widget.params['map_height']=600
form.fields['Polygon'].widget.params['map_srid']=YOUR_SRID
form.fields['Polygon'].widget.params['modifiable']=True
form.fields['Polygon'].widget.params['map_options']={}
form.fields['Polygon'].widget.params['map_options']['buffer'] = 0
return form
Based on the code at:
http://code.djangoproject.com/browser/django/branches/gis/django/contrib/gis/admin/options.py?rev=7980
It looks like you can use the extra_js option to include OpenStreetMap (I have not tested this).

This is quite old, and I wouldn't go around creating a template hack as I originally was thinking. Now I would use leaflet.js with an ajax request to a django view that returns geojson to a leaflet geojson layer.
This makes the django side super easy.
Sample Django View:
# -*- coding: utf-8 -*-
'''
'''
import json
from django.http import HttpResponse, HttpResponseBadRequest
from django.contrib.gis.geos import Polygon
from models import ResultLayer, MyModel
def get_layer_polygons(request, layer_id):
"""
Return the polygons for the given bbox (bounding box)
"""
layer = ResultLayer.objects.get(id=layer_id)
bbox_raw = request.GET.get("bbox", None)
# Make sure the incoming bounding box is correctly formed!
bbox = None
if bbox_raw and bbox_raw.count(",") == 3:
bbox = [float(v) for v in bbox_raw.split(",")]
if not bbox:
msg = "Improperly formed or not given 'bbox' querystring option, should be in the format '?bbox=minlon,minlat,maxlon,maxlat'"
return HttpResponseBadRequest(msg)
bbox_poly = Polygon.from_bbox(bbox)
bbox_poly.srid = 900913 # google
bbox_poly.transform(layer.srid) # transform to the layer's srid for querying
bin_size = int(bin_size)
# build vector polygons from bin
results = MyModel.objects.filter(layer=layer, poly__intersects=bbox_poly).transform(900913, field_name="poly")
geojson_data = []
for r in results:
# loading json in order to dump json list later
gjson = r.poly.geojson
py_gjson = json.loads(gjson)
geojson_data.append(py_gjson)
return HttpResponse(json.dumps(geojson_data), mimetype='application/json')

I think your solution is workable and probably the easiest approach. Just templatize the javascript and use Django to inject your data points as the template is rendered.
If you wanted to get fancier, you could have a Django view that served up the data points as JSON (application/json) and then use AJAX to call back and retrieve the data based on events that are happening in the browser. If you want your application to be highly interactive above and beyond what OpenLayers provides, this might be worth the added complexity, but of course it all depends on the needs of your application.

You could consider using FloppyForms. In the end, I usually end up customizing the solution to my own needs, but it's a nice way to get started.

Checkout this tutorial from the geodjango-basic-apps project:
http://code.google.com/p/geodjango-basic-apps/wiki/FOSS4GWorkshop
maybe you don't have to hack up your own javascript just yet