I have some entities in sqlalchemy. They are User and messages which they sent UserMessage:
class User(db.Model):
__tablename__ = 'User'
email = Column(String)
class UserMessage(db.Model):
__tablename__ = 'UserMessages'
date = Column(String)
message = Column(String)
user_id = Column(Integer, ForeignKey('User.uid'))
user = relationship('User')
Now, I want to create a flask resource which let's applications post information in a way which does not map directly to the way the models are actually defined. For example, something like this:
Json I want to parse
{
"date": '12345',
"user": {
"email": "test#email.com"
},
"message": "I'm a message"
}
All the examples I see for marshmallow involve converting objects in json which are structured in the same way as the orm object. In this example, you would expect the json to look more like this:
Json marshmallow wants
{
"user": {
"email": "test#email.com",
"messages": {
"message": "I'm a message"
},
"date": "12345"
}
}
Will marshammlow allow me to define a more arbitrary schema which will translate between this json and my internal orm objects? If so, can someone please point me at documentation or an example?
Related
I am currently exploring elasticsearch in python using the elasticsearch_dsl library. I am aware that my Elasticsearch knowledge is currently limited.
I have created a model like so:
class Post(InnerDoc):
text = Text()
id = Integer()
class User(Document):
name = Text()
posts = Object(doc_class=Posts)
signed_up_at = Date()
The data for posts is an array like this:
[
{
"text": "Test",
"id": 2
},
]
Storing my posts works. However, to me this seems wrong. I specify the "posts" attribute to be a Post - not a List of Posts.
Querying works, I can:
s = Search(using=client).query("match", posts__text="test")
and will retrieve the User that has a post containing the words as a result.
What I want is that I get the user + all Posts that qualified the user to appear in the result (meaning all posts containing the search phrase). I called that the inner hits, but I am not sure if this is correct.
Help would be highly appreciated!
I tried using "nested" instead of "match" for the query, but that does not work:
[nested] query does not support [posts]
I suspect that this has to do with the fact that my index is specified incorrectly.
I updated my model to this:
class Post(InnerDoc):
text = Text(analyzer="snowball")
id = Integer()
class User(Document):
name = Text()
posts = Nested(doc_class=Posts)
signed_up_at = Date()
This allows me to do the following query:
GET users/_search
{
"query": {
"nested": {
"path": "posts",
"query": {
"match": {
"posts.text": "idea"
}
},
"inner_hits": {}
}
}
}
This translates to the following elasticsearch-dsl query in python:
s = (
Search(using=client).query(
"nested",
path="posts",
query=Q("term", **{"post.text": "Idea"}),
inner_hits={},
)
Access inner hits like this:
Using Nested might be required, because of how elasticsearch represents objects internally (https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html). As lists of objects might be flattened, it might not allow to retrieve complete inner hits that contain the correct association of text and id for a post.
Is there a way for a FastAPI application to not display a model in its schema documentation? I have some models which are slight variations of others, and with this duplication occurring for each model, the schema documentation is quite cluttered.
from pydantic import BaseModel
class A(BaseModel):
name: str
class HasID(BaseModel):
id_: int
class AWithID(A, HasID):
pass
Is there some way not to display class AWithID in the documentation?
It's not elegant, but you can manually modify the auto-generated OpenAPI schema.
See the Extending OpenAPI section of the FastAPI docs.
A FastAPI application (instance) has an .openapi() method that is
expected to return the OpenAPI schema.
As part of the application object creation, a path operation for
/openapi.json (or for whatever you set your openapi_url) is
registered.
It just returns a JSON response with the result of the application's
.openapi() method.
By default, what the method .openapi() does is check the property
.openapi_schema to see if it has contents and return them.
If it doesn't, it generates them using the utility function at
fastapi.openapi.utils.get_openapi.
From the output of get_openapi(), all the models are defined under components > schemas, where each model's name is the dictionary key.
{
"openapi": "3.0.2",
"info": {...},
"paths": {...},
"components": {
"schemas": {
"A": {
"title": "A",
"required": [...],
"type": "object",
"properties": {...}
},
"AWithID": {
"title": "AWithID",
"required": [...],
"type": "object",
"properties": {...}
},
"HasID": {
"title": "HasID",
"required": [...],
"type": "object",
"properties": {...}
},
...
}
}
}
So, you can just pop off the models you want to hide:
from fastapi import FastAPI
from fastapi.openapi.utils import get_openapi
app = FastAPI()
# Define routes before customizing the OpenAPI schema
# ...
def custom_openapi():
if app.openapi_schema:
return app.openapi_schema
openapi_schema = get_openapi(title="My App", version="1.0.0", routes=app.routes)
openapi_schema["components"]["schemas"].pop("AWithID", None)
app.openapi_schema = openapi_schema
return app.openapi_schema
app.openapi = custom_openapi
Have to make django models and take a JSON file to feed all the data for a student and classes display webapp. JSON file is what going to drive my modeling, it looks like this (truncated to couple of data points)...
{
"students": [
{
"first": "John",
"last": "Smith",
"email": "johnsmith#mailinator.com",
"studentClasses": [
{
"id": 1,
"grade": 4
},
{
"id": 2,
"grade": 3
},
]},
{...#truncated data, this follows with more students
"classes": {
"1": "Math 101",
"2": "English 101",
"3": "Science 101",
#total 8 classes declared, truncated
}
I have my data models as.....
class Student(models.Model):
first = models.CharField(max_length=200)
last = models.CharField(max_length=200)
email = models.EmailField()
class Classes(models.Model):
student = models.ForeignKey(Student)
class_name = models.CharField(max_length=50)
Here are my questions...
(1) How can I model in a way that takes in studentClasses:[{id:1, grade:4}] type relational input from JSON file and populates my database tables ? It seems I might have to declare serializer, why and how?
(2) Getting confused by ID in classes table and not ID in students table, do I explicitly have to declare primary key in modeling with ID in classes but not students models ?
(3) It seems I can load tables with "python manage.py load data myjsonfile.json" (would that be sufficient)?
(4) Do I need a third model called 'studentClasses' that keeps track of which student has taken what class and grade for that class ?
Thanks in advance for your help.
As for me, it seems better to write short function (in separate file) that updates database:
import json
from app.models import Student, Classes
data = json.loads('fixtures.json')
for student in data['students']:
# parse student; save classes and student objects
print('done')
(2) I think you should use "many to many" relation, not "one to many"
You need to add a field student_classes to Student, you can serialize it using jsonpickle. By having that field, I don't think you would need any foreign keys...You are talking about "loading tables" - that seems is where you are also confused. The model in django consists of classes just as your modeling code shows, therefore, to work with the models (and data in them), you would be importing the classes, such as "from models import Student, Classes".
I'm trying to convert bjson structure to a schema in marshmallow library.
Below is the marshmallow schema:
class GeneSchema(Schema):
"""description of class"""
id_entrez = fields.Integer(required = True, error_messages={'required': "The 'id_entrez' field is requeired."})
symbol = fields.String()
#validates('id_entrez')
def validate_id_entrez(self, data):
if data <= 0:
raise ValidationError("The 'id_entrez' field must be greater than zero.")
Below is the bjson will be converted to schema:
[{"symbol": "VAMP4", "_id": {"$oid": "57ae3b175a945932fcbdf41d"}, "id_entrez": 8674}, {"symbol": "CCT5", "_id": {"$oid": "57ae3b175a945932fcbdf41e"}, "id_entrez": 22948}]
Note that the bjson has the "_id" as ObjectId - "$oid". This is because the result of the query using the mongodb.
Please, does anyone know why not be to convert from bjson to marshmallow schema correctly ?
Thank you all!
I don't know if this question is still valid, but I would like to show my solution, how to push an ObjectId to a Marshmallow schema:
I simply use the pre processing method pre_load to convert the ObjectId to a String:
#pre_load
def convert_objectid(self, in_data, **kwargs):
if "_id" in in_data:
in_data["_id"] = str(in_data["_id"])
return in_data
You can still use your schema to parse MongoDB output, just ignore extra "_id" field. If on the other hand you do want to parse that "_id", just add extra non-required field in your schema.
Using Django 1.7.
I have a model class Topic that I want to serialize. Topic has an attribute Creator that is a ForeignKey to a class UserProfile. The output from serialization gives me the following string:
'{"fields": {"building": 1, "title": "Foobar", "created": "2015-02-13T16:14:47Z", "creator": 1}, "model": "bb.topic", "pk": 2}'
I want key "creator" to say the username of associated with UserProfile (as opposed to right now, where it is giving me the pk value associated with UserProfile. The username is held within a OneToOneField with django.contrib.auth.models class User.
I tried to implement a UserProfileManager, but either I have done it incorrectly or the following is not an appropriate strategy:
def get_by_natural_key(self, user, picture, company):
return self.get(user_id=user.id, user_pk=user.pk, username=user.get_username, picture=picture, company=company)
Finally, I looked around SO and found pointers to this: https://code.google.com/p/wadofstuff/wiki/DjangoFullSerializers but it says it was last updated in 2011 so I am assuming there is something else out there.
Thanks. Any help is much appreciated.
It looks you haven't implemented all the code needed to serialize the UserProfile with a natural key.
Actually the get_by_natural_key method is called when deserializing an object. If you want it to be serialized with a natural key instead of pk you should provide the natural_key method to the model.
Something like:
class UserProfileManager(models.Manager):
def get_by_natural_key(self, user, company):
return self.get(user__user__username=user, company=company)
class UserProfile(models.Model):
objects = UserProfileManager()
user = models.OneToOneField(User)
company = models.CharField(max_length=255)
def natural_key(self):
return (self.user.name, self.company)
Now, if you serialize a Topic instance:
import django.core.serializers
obj = serializers.serialize('json', Topic.objects.all(), indent=2, use_natural_foreign_keys=True, use_natural_primary_keys=True)
you should get an output similar to:
{
"fields": {
"building": 1,
"title": "Foobar",
"created": "2015-02-13T16:14:47Z",
"creator": [
"dummy",
"ACME Inc."
]
},
"model": "bb.topic", "pk": 2
}
supposing a dummy user exist with a company named ACME Inc. in its user profile.
Hope this helps.