We Keep Coding

flask API querying items, JSON

I am using a flask API as my rest point for my Angular application. Currently I am testing the API. I tested my /users point to make sure I got all the users.
//importing db, app, models, schema etc.
from flask import jsonify, request
#app.route('/users')
def get_users():
# fetching from database
users_objects = User.query.all()
# transforming into JSON-serializable objects
users_schema = UserSchema(many=True)
result = users_schema.dump(users_objects)
# serializing as JSON
return jsonify(result.data)
That worked. However, now that I am trying to get other data(which has more than 9000 objects.. it doesn't work(when I try querying all of them). I first just grabbed the first item
#app.route('/aggregated-measurements')
def get_aggregated_measurements():
aggregated_measurements_objects = AggregatedMeasurement.query.first()
# transforming into JSON-serializable objects
aggregated_measurement_schema = AggregatedMeasurementSchema()
result = aggregated_measurement_schema.dump(aggregated_measurements_objects)
return jsonify(result.data)
That showed me the first AggregatedMeasurement. However when I try to query all of them aggregated_measurements_objects = AggregatedMeasurement.query.all() Nothing displays. I did the same thing on my jupyter notebook and that displayed them. I then thought that maybe this was too much info, so I tried to just limit the query like this aggregated_measurements_objects = AggregatedMeasurement.query.all()[:5]. That works on the jupyter notebook, but displays nothing when I hit the route.
I don't understand why when I hit the /users point I can see all of them, but when I try to do the same for aggregated-measurements I get nothing(even when I limit the query). I am using flask_sqlalchemy with sqlite db.
**update with model and schema **
from datetime import datetime
# ... import db
import pandas as pd
from marshmallow import Schema, fields
class AggregatedMeasurement(db.Model):
id = db.Column(db.Integer, primary_key=True)
created = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
time = db.Column(db.DateTime, nullable=False)
speed = db.Column(db.Float, nullable=False)
direction = db.Column(db.Float, nullable=False)
# related fields
point_id = db.Column(db.Integer, db.ForeignKey('point.id'), nullable=False)
point = db.relationship('Point',backref=db.backref('aggregated_measurements', lazy=True))
class AggregatedMeasurementSchema(Schema):
id = fields.Int(dump_only=True)
time = fields.DateTime()
speed = fields.Number()
direction = fields.Number()
point_id = fields.Number()
SECOND UPDATE found the error.
After verifying that indeed it was hitting the db( thank you #gbozee) I noticed that on the /aggregated-measurements route when I made the schema I did it for just one object. I forgot to include the many = True like I did in the users_schema. Therefore that is why only one point appeared and when I tried more, it did not. I was using the marshmallow(an object serialization package).

Idle Timeout of Authentication Tokens

I have an app using Flask and SQLAlchemy that allows users to make calls to a RESTful API using authentication tokens which I provide to them when they log into the application. One of the requirements of the application is that if a user is idle for 15 minutes, their token should expire and they will be required to log into the application again to get a new one.
Currently I am doing this by storing a user's session in a table
class Session(Model):
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey('user.id'))
user = relationship('User')
creation_time = Column(DateTime, default=datetime.now)
last_active = Column(DateTime, default=datetime.now)
_serializer = Serializer('foobar')
def is_active(self):
return (datetime.now() - self.last_active) < timedelta(15*60)
def create_token(self, token):
return self._serializer.dumps({
'session_id': self.id,
'user_id': self.user_id,
})
#staticmethod
def from_token(self, token):
try:
data = self._serializer.loads(token)
except BadSignature:
return None
session = Session.query.get(data.get('session_id'))
if session and session.is_active() and session.user_id == data.get('user_id'):
# Refresh the session
session.last_active = datetime.now()
return session
class User(Model):
id = Column(Integer, primary_key=True)
username = Column(String)
def generate_token(self):
return Session(self.id).create_token()
def verify_auth_token(token):
session = Session.from_token(token)
if session:
return session.user
Is there a more elegant and efficient way of doing this? I have seen many examples where the token stores the creation time and the tokens are only valid for a specific amount of time (which allows for an easy test without having to know what user it is, etc.), but I haven't been able to find any information about the expiration based on a user's idle time.

Apart from the potential bug because session.last_active = … is set without a call to session.flush() (where that's the SQLAlchemy Session), this is a totally reasonable way to do it.
In general, sites use a non-relational datastore (ex, Redis) for session management because it can be more performant (RDBMSs tend to be slower for writes, and especially in this case you'll be doing a lot of writes).
The logic would still be the same, though:
def session_from_token(token):
key = "session:%s" %(token, )
session_json = redis.get(key)
if not session_json:
return None
session = json.loads(session_json)
if not session.get("is_valid"):
return None
redis.expire(key, 15 * 60)
return session

can I store a Dictionary as the property of an object?

Using Python/Flask/SQLAlchemy/Heroku.
Want to store dictionaries of objects as properties of an object:
TO CLARIFY
class SoccerPlayer(db.Model):
name = db.Column(db.String(80))
goals_scored = db.Column(db.Integer())
^How can I set name and goals scored as one dictionary?
UPDATE: The user will input the name and goals_scored if that makes any difference.
Also, I am searching online for an appropriate answer, but as a noob, I haven't been able to understand/implement the stuff I find on Google for my Flask web app.

I would second the approach provided by Sean, following it you get properly
normalized DB schema and can easier utilize RDBMS to do the hard work for you. If,
however, you insist on using dictionary-like structure inside your DB, I'd
suggest to try out hstore
data type which allows you to store key/value pairs as a single value in
Postgres. I'm not sure if hstore extension is created by default in Postgres
DBs provided by Heroku, you can check that by typing \dx command inside
psql. If there are no lines with hstore in them, you can create it by
typing CREATE EXTENSION hstore;.
Since hstore support in SQLAlchemy is available in version 0.8 which is not
released yet (but hopefully will be in coming weeks), you need to install it
from its Mercurial repository:
pip install -e hg+https://bitbucket.org/sqlalchemy/sqlalchemy#egg=SQLAlchemy
Then define your model like this:
from sqlalchemy.dialects.postgresql import HSTORE
from sqlalchemy.ext.mutable import MutableDict
class SoccerPlayer(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80), nullable=False, unique=True)
stats = db.Column(MutableDict.as_mutable(HSTORE))
# Note that hstore only allows text for both keys and values (and None for
# values only).
p1 = SoccerPlayer(name='foo', stats={'goals_scored': '42'})
db.session.add(p1)
db.session.commit()
After that you can do the usual stuff in your queries:
from sqlalchemy import func, cast
q = db.session.query(
SoccerPlayer.name,
func.max(cast(SoccerPlayer.stats['goals_scored'], db.Integer))
).group_by(SoccerPlayer.name).first()
Check out HSTORE docs
for more examples.

If you are storing such information in a database I would recommend another approach:
class SoccerPlayer(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80))
team_id = db.Column(db.Integer, db.ForeignKey('Team.id'))
stats = db.relationship("Stats", uselist=False, backref="player")
class Team(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80))
players = db.relationship("SoccerPlayer")
class Stats(db.Model):
id = db.Column(db.Integer, primary_key=True)
player_id = db.Column(db.Integer, db.ForeignKey('SoccerPlayer.id'))
goals_scored = db.Column(db.Integer)
assists = db.Column(db.Integer)
# Add more stats as you see fit
With this model setup you can do crazy things like this:
from sqlalchemy.sql import func
max_goals_by_team = db.session.query(Team.id,
func.max(Stats.goals_scored).label("goals_scored")
). \
join(SoccerPlayer, Stats). \
group_by(Team.id).subquery()
players = SoccerPlayer.query(Team.name.label("Team Name"),
SoccerPlayer.name.label("Player Name"),
max_goals_by_team.c.goals_scored). \
join(max_goals_by_team,
SoccerPlayer.team_id == max_goals_by_team.c.id,
SoccerPlayer.stats.goals_scored == max_goals_by_team.c.goals_scored).
join(Team)
thus making the database do the hard work of pulling out the players with the highest goals per team, rather than doing it all in Python.

Not even django(a bigger python web framework than flask) doesn't support this by default. But in django you can install it, it's called a jsonfield( https://github.com/bradjasper/django-jsonfield ).
What i'm trying to tell you is that not all databases know how to store binaries, but they do know how to store strings and jsonfield for django is actually a string that contains the json dump of a dictionary.
So, in short you can do in flask
import simplejson
class SoccerPlayer(db.Model):
_data = db.Column(db.String(1024))
#property
def data(self):
return simplejson.loads(self._data)
#data.setter
def data(self, value):
self._data = simplejson.dumps(value)
But beware, this way you can only assign the entire dictionary at once:
player = SoccerPlayer()
player.data = {'name': 'Popey'}
print player.data # Will work as expected
{'name': 'Popey'}
player.data['score'] = '3'
print player.data
# Will not show the score becuase the setter doesn't know how to input by key
{'name': 'Popey'}

taskqueue and non-idempotent tasks

I'm working on a voting app, where the user can upload a list of email addresses for all of the voters. After doing some error checking, I create a Voter entity for each voter. Since there can be a large number of voters, I create the Voter entities in a taskqueue to avoid the 30 second limit and the task looks like this:
put_list = []
for email, id in itertools.izip(voter_emails, uuids):
put_list.append(Voter(election = election,
email = email,
uuid = id))
election.txt_voters = ""
put_list.append(election)
db.put(put_list)
This task, however, isn't idempotent. Is there a way to make this task idempotent? Or is there a better way to do this?

use a key_name rather than a uuid property to prevent creating duplicate voter entities.

Google App Engine Python Datastore

Basically what Im trying to make is a data structure where it has the users name, id, and datejoined. Then i want a "sub-structure" where it has the users "text" and the date it was modified. and the user will have multiple instances of this text.
class User(db.Model):
ID = db.IntegerProperty()
name = db.StringProperty()
datejoined = db.DateTimeProperty(auto_now_add=True)
class Content(db.Model):
text = db.StringProperty()
datemod= db.DateTimeProperty(auto_now_add = True)
Is the code set up correctly?

One problem you will have is that making User.ID unique will be non-trivial. The problem is that two writes to the database could occur on different shards, both check at about the same time for existing entries that match the uniqueness constraint and find none, then both create identical entries (with regard to the unique property) and then you have an invalid database state. To solve this, appengine provides a means of ensuring that certain datastore entities are always placed on the same physical machine.
To do this, you make use of the entity keys to tell google how to organize the entities. Lets assume you want the username to be unique. Change User to look like this:
class User(db.Model):
datejoined = db.DateTimeProperty(auto_now_add=True)
Yes, that's really it. There's no username since that's going to be used in the key, so it doesn't need to appear separately. If you like, you can do this...
class User(db.Model):
datejoined = db.DateTimeProperty(auto_now_add=True)
#property
def name(self):
return self.key().name()
To create an instance of a User, you now need to do something a little different, you need to specify a key_name in the init method.
someuser = User(key_name='john_doe')
...
someuser.save()
Well, really you want to make sure that users don't overwrite each other, so you need to wrap the user creation in a transaction. First define a function that does the neccesary check:
def create_user(username):
checkeduser = User.get_by_key_name(username)
if checkeduser is not None:
raise db.Rollback, 'User already exists!'
newuser = User(key_name=username)
# more code
newuser.put()
Then, invoke it in this way
db.run_in_transaction(create_user, 'john_doe')
To find a user, you just do this:
someuser = User.get_by_key_name('john_doe')
Next, you need some way to associate the content to its user, and visa versa. One solution is to put the content into the same entity group as the user by declaring the user as a parent of the content. To do this, you don't need to change the content at all, but you create it a little differently (much like you did with User):
somecontent = Content(parent=User.get_by_key_name('john_doe'))
So, given a content item, you can look up the user by examining its key:
someuser = User.get(somecontent.key().parent())
Going in reverse, looking up all of the content for a particular user is only a little trickier.
allcontent = Content.gql('where ancestor is :user', user=someuser).fetch(10)

Yes, and if you need more documentation, you can check here for database types and here for more info about your model classes.

An alternative solution you may see is using referenceproperty.
class User(db.Model):
name = db.StringProperty()
datejoined = db.DateTimeProperty(auto_now_add=True)
class Content(db.Model):
user = db.ReferenceProperty(User,collection_name='matched_content')
text = db.StringProperty()
datemod= db.DateTimeProperty(auto_now_add = True)
content = db.get(content_key)
user_name = content.user.name
#looking up all of the content for a particular user
user_content = content.user.matched_content
#create new content for a user
new_content = Content(reference=content.user)