We Keep Coding

S3 Select Query JSON for nested value when keys are dynamic

I have a JSON object in S3 which follows this structure:
<code> : {
<client>: <value>
}
For example,
{
"code_abc": {
"client_1": 1,
"client_2": 10
},
"code_def": {
"client_2": 40,
"client_3": 50,
"client_5": 100
},
...
}
I am trying to retrieve the numerical value with an S3 Select query, where the "code" and the "client" are populated dynamically with each query.
So far I have tried:
sql_exp = f"SELECT * from s3object[*][*] s where s.{proc}.{client_name} IS NOT NULL"
sql_exp = f"SELECT * from s3object s where s.{proc}[*].{client_name}[*] IS NOT NULL"
as well as without the asterisk inside the square brackets, but nothing works, I get ClientError: An error occurred (ParseUnexpectedToken) when calling the SelectObjectContent operation: Unexpected token found LITERAL:UNKNOWN at line 1, column X (depending on the length of the query string)
Within the function defining the object, I have:
resp = s3.select_object_content(
Bucket=<bucket>,
Key=<filename>,
ExpressionType="SQL",
Expression=sql_exp,
InputSerialization={'JSON': {"Type": "Document"}},
OutputSerialization={"JSON": {}},
)
Is there something off in the way I define the object serialization? How can I fix the query so I can retrieve the desired numerical value on the fly when I provide ”code” and “client”?

I did some tinkering based on the documentation, and it works!
I need to access the single event in the EventStream (resp) as follows:
event_stream = resp['Payload']
# unpack successful query response
for event in event_stream:
if "Records" in event:
output_str = event["Records"]["Payload"].decode("utf-8") # bytes to string
output_dict = json.loads(output_str) # string to dict
Now the correct SQL expression is:
sql_exp= f"SELECT s['{code}']['{client}'] FROM S3Object s"
where I have gotten (dynamically) my values for code and client beforehand.
For example, based on the dummy JSON structure above, if code = "code_abc" and client = "client_2", I want this S3 Select query to return the value 10.
The f-string resolves to sql_exp = "SELECT s['code_abc']['client_2'] FROM S3Object s", and when we call resp, we retrieve output_dict = {'client_2': 10} (Not sure if there is a clear way to get the value by itself without the client key, this is how it looks like in the documentation as well).
So, the final step is to retrieve value = output_dict['client_2'], which in our case is equal to 10.

How to use a value in a JSON file within an if statement

I'm trying to create an if statement based on the value from an API I'm using. This API contains a status code value, "status". if this is 404 (or others) I want to return an error, else carry on.
An example of the JSON:
{
"data": {
"test_index": {
"test_a": [...], // 429 items
"test_b": [...] // 182 items
}
},
"status": 200
}
When running the code below:
import json
import urllib.request as ur
API = ur.urlopen('https://example.com')
data = json.loads(API.read())
if data['status'][0] == 404:
print("404")
else:
print("Not 404")
I get the following error:
TypeError: 'int' object is not subscriptable
Which is referring to line 7, the if statement.
How do I convert this JSON value to something I can work with?

data['status'] is an integer, and you cannot subscript an integer with an index, like you do for a list, Change your code as follows.
if data['status'] == 404:
print("404")
else:
print("Not 404")

Can we return Data in JSON format in flask using Neo4j queries?

I want to return data in JSON format from Flask method. My Code is written in Flask and Neo4j is used for storing Data.
As of now, my code is:
result = session.run("MATCH (p:Person {name:$username})-[:PURCHASED]->(:Product)<-[:PURCHASED]-(p2:Person)-[:PURCHASED]->(pd2:Product)"
"WHERE NOT (p)-[:PURCHASED]->(pd2)"
"RETURN pd2.title as product_title, pd2.description as product_details" , username=username)
for record in result:
print("%s %s" % (record["product_title"], record["product_details"]))
return 'Loop Entered'
return 'OK'
I tried using 'jsonify' as below but got Type error as : TypeError: ('product_title', 'product_details')
return jsonify([record[("product_title","product_details")] for record in result])
Please let me know what I am missing here.

Maybe you wanted to do this:
return jsonify({record["product_title"]: record["product_details"] for record in result})
or even
return jsonify([
{
"product_title": record["product_title"],
"product_details": record["product_details"],
}
for record in result
])
?
You can access only one key at a time when using dict type in Python

Return JSON object to View with Flask and Ajax

I am trying to return a python dictionary to the view with AJAX and reading from a JSON file, but so far I am only returning [object Object],[object Object]...
and if I inspect the network traffic, I can indeed see the correct data.
So here is how my code looks like. I have a class and a method which based on the selected ID (request argument method), will print specific data. Its getting the data from a python discretionary. the problem is not here, have already just tested it. But just in case I will link it.
# method to create the directionary - just in case #
def getCourselist_byClass(self, classid):
"""
Getting the courselist by the class id, joining the two tables.
Will only get data if both of them exist in their main tables.
Returning as a list.
"""
connection = db.session.connection()
querylist = []
raw_sql = text("""
SELECT
course.course_id,
course.course_name
FROM
course
WHERE
EXISTS(
SELECT 1
FROM
class_course_identifier
WHERE
course.course_id = class_course_identifier.course_id
AND EXISTS(
SELECT 1
FROM
class
WHERE
class_course_identifier.class_id = class.class_id
AND class.class_id = :classid
)
)""")
query = connection.engine.execute(raw_sql, {'classid': classid})
for column in query:
dict = {
'course_id' : column['course_id'],
'course_name' : column['course_name']
}
querylist.append(dict)
return querylist
my jsonify route method
#main.route('/task/create_test')
def get_courselist():
#objects
course = CourseClass()
class_id = request.args.get('a', type=int)
#methods
results = course.getCourselist_byClass(class_id)
return jsonify(result=results)
HTML
and here is how the input field and where it should link the data looks like.
<input type="text" size="5" name="a">
<span id="result">?</span>
<p>click me
and then I am calling it like this
<script type=text/javascript>
$(function() {
$('a#link').bind('click', function() {
$.getJSON("{{ url_for('main.get_courselist') }}", {
a: $('input[name="a"]').val()
}, function(data) {
$("#result").text(data.result);
});
return false;
});
});
</script>
but every time I enter a id number in the field, i am getting the correct data. but it is not formatted correctly. It is instead printing it like [object Object]
source, followed this guide as inspiration: flask ajax example

The data return by your server is like: {result: [{course_id: 'xxx', course_name: 'xxx'}]}, in which data.result is a JS Array.
when you set it to $("#result").text(), JS convert a array to string, so the result is [object Object].
You should iterate over the array to construct a string, then set the string in DOM, like:
courseStr = data.result.map(function(course) {return course.course_id + '-' + course.course_name; }).join(',');
$("#result").text(courseStr);

The API description for flask.json.jsonify indicates it's expecting keyword parameters. What you actually want to do seems to be serialize a list object containing dictionaries, have you tried flask.json.dumps instead? Assuming you've got the dumps symbol imported, instead of your jsonify call you can try:
return dumps(results)

while loop for deleting data in datastore

I have attempted to clean up and revise code in an answer here for my needs where I only want to delete from the Model Reservations for data records prior to the date expressed in the get as yy,mm,dd.
If I am correctly anticipating the action of cleanTable/2012/10/5 against the routing ('/cleanTable/([\d]+)/([\d]+)/([\d]+)', CleanTable) then my code would only delete at most 50 (10*nlimit) data records.
Btw, the author of the original code (who likely no longer subscribes to SO), claimed his main trick for accomplishing this code was "to include redirect in html instead of using self.redirect".
I am unfamiliar with raise Exception and the like, but my instinct would be to add a raise Exception or raise StopIteration to the for loop after it is made into a while loop. But it is not clear to me whether raising an StopIteration exception actually causes iteration to stop or if more is needed. Also, I don't know how to revise so the html ends smoothly upon early exit.
class CleanTable(BaseHandler):
def get(self, yy,mm,dd):
nlimit=5
iyy=int(yy)
imm=int(mm)
idd=int(dd)
param=date(iyy,imm,idd)
q=Reservations.all(keys_only=True)
q.filter("date < ", dt(iyy,imm,idd))
results = q.fetch(nlimit)
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write("""
<html>
<meta HTTP-EQUIV="REFRESH" content="url=http://yourapp.appspot.com/cleanTable">
<body>""")
try:
for i in range(10):
db.delete(results)
results = q.fetch(nlimit, len(results))
for r in results:
logging.info("r.name: %s" % r.name)
self.response.out.write("<p> "+str(nlimit)+" removed</p>")
self.response.out.write("""
</body>
</html>""")
except Exception, inst:
logging.info("inst: %s" % inst)
self.response.out.write(str(inst))

This is not the best approach to clean your models. A better approach would be to get all the keys of your entities and create Task Queues. Each queue will get a batch of keys for the entities that need to be modified.
Another approach would also be to create a cron job that will query for the x number of oldest modified entities, fix them and then store them back.
Finally, if your number of entities is so huge, you could also consider the use of Backends.
Hope this helps.

Here is my update routine and it has converted 500.000 entities. Be sure to run it on a backend instance (You can target a Queue to a backend instance). Notice that I am using a cursor, thats the only way you can consistently iterate through data (Never use offset!).
Queue queue = QueueFactory.getQueue("grinderQueue");
queue.add(TaskOptions.Builder.withPayload(new DeferredTask() { //lets generate
private static final long serialVersionUID = 1L;
#Override
public void run() {
String cursor = null;
boolean done = false;
Date now = new Date(1346763868L * 1000L); // 09/04/2012
while(!done) {
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
Query query = new Query("Venue");
query.setFilter(new FilterPredicate("timeOfLastUpdate", Query.FilterOperator.LESS_THAN,now));
PreparedQuery pq = datastore.prepare(query);
FetchOptions fetchOptions = FetchOptions.Builder.withLimit(1000);
if(cursor != null)
fetchOptions.startCursor(Cursor.fromWebSafeString(cursor));
QueryResultList<Entity> results = pq.asQueryResultList(fetchOptions);
List<Entity> updates = new ArrayList<Entity>();
List<Entity> oldVenueUpdates = new ArrayList<Entity>();
int tuples = 0;
for(Entity en : results) {
tuples++;
try {
if(en.getProperty(Venue.VENUE_KEY) == null)
continue;
Entity newVenue = new Entity("CPVenue",(String)en.getProperty(Venue.VENUE_KEY));
newVenue.setPropertiesFrom(en);
newVenue.removeProperty("timeOfLastVenueScoreCalculation");
newVenue.removeProperty("actionsSinceLastVenueScoreCalculation");
newVenue.removeProperty("venueImageUrl");
newVenue.removeProperty("foursquareId");
newVenue.setProperty("geoCell", GeoCellCalculator.calcCellId(Double.valueOf((String)en.getProperty("lng")), Double.valueOf((String)en.getProperty("lat")),8));
newVenue.setProperty(Venue.TIME_SINCE_LAST_UPDATE, new Date());
updates.add(newVenue);
Venue v = new Venue(newVenue);
//Set timestamp on Venue
en.setProperty("timeOfLastUpdate", now);
oldVenueUpdates.add(en);
}catch(Exception e) {
logger.log(Level.WARNING,"",e);
}
}
done = tuples == 0;
tuples = 0;
if(results.getCursor() != null)
cursor = results.getCursor().toWebSafeString();
else
done = true;
System.out.println("Venue Conversion LOOP updates.. " + updates.size() + " cursor " + cursor);
datastore.put(updates);
datastore.put(oldVenueUpdates);
}
System.out.println("Venue Conversion DONE");
}}));