We Keep Coding

Can't get list of model fields

I need to get list of model fields like:
#instance.register
class Todo(Document):
title = fields.StringField(required=True, default='Name')
description = fields.StringField()
created_at = fields.DateTimeField()
created_by = fields.StringField()
priority = fields.IntegerField()
to
[
'title',
'description',
'created_at',
'created_by',
'priority'
]
So, I have function that returns list of fields
def get_class_properties(cls):
attributes = inspect.getmembers(cls, lambda a: not (inspect.isroutine(a)))
return [attr for attr in attributes if not (attr[0].startswith('__') and attr[0].endswith('__'))][1]
But usage gives me this error
umongo.exceptions.NoDBDefinedError: init must be called to define a db
Usage:
properties=get_class_properties(Todo)
UPD
Here is my mongo initialization code:
async def mongo_client(app):
conf = app["config"]["mongo"]
client = AsyncIOMotorClient(host=conf["host"], port=conf["port"])
db = client[conf["db"]]
instance.init(db)
await Todo.ensure_indexes()
app["db_client"]: AsyncIOMotorClient = client
app["db"] = db
yield
await app["db_client"].close()

This is a copy/paste of this answer from the author of this library:
As far as I remeber, this exception raises when you're trying to use
lazy clients without initializing them properly. Any lazy class of
uMongo expects that the used database will be specified before the
usage. Everything that you need is to specify the used database and
invoke the init method of your lazy instance, like this:
from motor.motor_asyncio import AsyncIOMotorClient
from umongo import MotorAsyncIOInstance
client = AsyncIOMotorClient("mongodb://user:password#host:port/")
client = client["test_database"]
lazy_umongo = MotorAsyncIOInstance()
lazy_umongo.init(client)
As an example you can look into Auth/Auth microservice code, where
documents defined and store in the separate files from the actual
usage. Also these files with code as examples (documents.py and
prepare_mongodb.py) will help you to find a solution.

The trick was that
properties=get_class_properties(Todo)
invokes earlier than
async def mongo_client(app):
Solution is use things in right order (see comments to code)
async def init_app(argv=None):
app = web.Application(middlewares=[deserializer_middleware], logger=logger)
app["config"] = config
conf = app["config"]["mongo"]
client = AsyncIOMotorClient(host=conf["host"], port=conf["port"])
db = client[conf["db"]]
instance.init(db)
# Remove this line:
# app.cleanup_ctx.append(mongo_client)
app.cleanup_ctx.append(api_client)
register_routes(app)
return app
def register_routes(app: web.Application):
# Use here:
todo_resource = RestResource(
entity='todo',
factory=Todo,
properties=get_class_properties(Todo)
)
todo_resource.register(app.router)

Django ORM select_related AttributeError

In my project im trying to use an external .py file for manage data using my django app style like this:
In my django project i create a model:
class temp_test_keywords(models.Model):
main_id = models.ForeignKey(temp_main)
test_id = models.ForeignKey(temp_case)
key_id = models.ForeignKey(temp_keywords)
variable_id = models.ForeignKey(temp_variables, null=True, blank=True)
def __str__(self):
return '%s -> %s' % (str(self.main_id), str(self.test_id))
Well, now in my external rst.py file i start django env like this:
import sys
import os
import django
sys.path.append('core')
os.environ['DJANGO_SETTINGS_MODULE'] = 'core.settings'
django.setup()
ok, at this point i import table and create class for do some thinks with it:
from django.db import models
from django.contrib.contenttypes.fields import
GenericForeignKey,GenericRelation
from django.contrib.contenttypes.models import ContentType
from django.db.models import Count
from frontend.models import temp_test_keywords
class PrepareRst:
def __init__(self,test_id,t_type,log=False):
self.rst = self.mainprep(test_id,t_type)
def mainprep(self,test_id,t_type):
return self.tc_prep(test_id)
#TestCase rst prep method
def tc_prep(self,test_id):
maxpar = temp_test_keywords.objects.filter(main_id = test_id).values('key_id').annotate(total=Count('variable_id')).order_by('-total').first()
totpar = maxpar['total']
#Part1 list creation
count = 0
ltouple = ()
l1 = ["Test Case"]
while (count < totpar):
l1.append("")
count += 1
ltouple += (l1,)
#Query for extract keywords, values
kv = temp_test_keywords.select_related()
but when i run an AttributeError: type object 'temp_test_keywords' has no attribute 'select_related' error raise
if i start python manage.py shell from terminal the "kv = temp_test_keywords.select_related()" command works fine, why in my .py code doesn't?
Thanks in advance

Try,
kv = temp_test_keywords.objects.all().select_related()

How to use marshmallow to serialize a custom sqlalchemy field?

I just start a simple project called flask_wiki this days and I'm using some flask extensions as the follows:
Flask-SQLAlchemy
Flask-Restful
MarshMallow
Well, I just discovered that the MarshMallow project provides a class called 'ModelSchema', which reads all fields from my SQLAlchemy Model and provide a fully automated (de)serialializer.
In my case, I created a 'GUID' Field which is RDBM agnostic and inserted it on my Sqlalchemy model as follows:
from flask.ext.sqlalchemy import SQLAlchemy
from flask_wiki.backend.custom_fields import GUIDField
class Page(db.Model):
"""
Implements the Page Model.
"""
guid = db.Column(GUIDField, primary_key=True, default=uuid.uuid4)
name = db.Column(db.String, nullable=False)
raw_content = db.Column(db.Text)
rendered_content = db.Column(db.Text)
def __repr__(self):
return self.__str__()
def __str__(self):
return self.name
The GUIDField is implemented is this way:
from sqlalchemy.types import TypeDecorator, CHAR
import uuid
class GUIDField(TypeDecorator):
# Platform independent GUID Implementation that uses little endianess.
impl = CHAR
def load_dialect_impl(self, dialect):
return dialect.type_descriptor(CHAR(32))
def process_bind_param(self, value, dialect):
if value is None:
return value
else:
if isinstance(value, uuid.UUID):
return value.bytes_le
def process_result_value(self, value, dialect):
if value is None:
return value
else:
return uuid.UUID(bytes_le=value)
The code for create test objects (through mixer) is working; all the guids are generated and verified correctly.
With this in mind, a just created the following MarshMallow Serializer Field:
from marshmallow import fields
import uuid
class GUIDSerializationField(fields.Field):
def _serialize(self, value, attr, obj):
if value is None:
return value
else:
if isinstance(value, uuid.UUID):
return str(value)
else:
return None
Finally, I created the SerializerClass:
from flask_wiki.backend.backend import marsh
from flask_wiki.backend.custom_serialization_fields import GUIDSerializationField
from flask_wiki.backend.models import Page
from marshmallow import fields
class PageSchema(marsh.ModelSchema):
class Meta:
model = Page
guid = GUIDSerializationField()
page_schema = PageSchema()
pages_schema = PageSchema(many=True)
I tried to use this last code with and without inserting the guid field, but in all cases the following error occurs:
Traceback (most recent call last):
File "/home/arthas/dev/flask-wiki/flask_wiki/wiki.py", line 3, in <module>
from flask_wiki.backend import backend
File "/home/arthas/dev/flask-wiki/flask_wiki/backend/__init__.py", line 1, in <module>
import flask_wiki.backend.routes
File "/home/arthas/dev/flask-wiki/flask_wiki/backend/routes.py", line 2, in <module>
from flask_wiki.backend.views import PageView
File "/home/arthas/dev/flask-wiki/flask_wiki/backend/views.py", line 3, in <module>
from flask_wiki.backend.serializers import pages_schema, page_schema
File "/home/arthas/dev/flask-wiki/flask_wiki/backend/serializers.py", line 7, in <module>
class PageSchema(marsh.ModelSchema):
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow/schema.py", line 116, in __new__
dict_cls=dict_cls
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/schema.py", line 53, in get_declared_fields
declared_fields = mcs.get_fields(converter, opts)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/schema.py", line 77, in get_fields
return converter.fields_for_model(opts.model, fields=opts.fields, exclude=opts.exclude)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 75, in fields_for_model
field = self.property2field(prop)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 93, in property2field
field_class = self._get_field_class_for_property(prop)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 151, in _get_field_class_for_property
field_cls = self._get_field_class_for_column(column)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 121, in _get_field_class_for_column
return self._get_field_class_for_data_type(column.type)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 143, in _get_field_class_for_data_type
'Could not find field column of type {0}.'.format(types[0]))
marshmallow_sqlalchemy.exceptions.ModelConversionError: Could not find field column of type <class 'flask_wiki.backend.custom_fields.GUIDField'>.
So, I finally ask: how to use marshmallow to serialize a custom sqlalchemy field?

You need to create your own Converter.
Try something like this:
import uuid
from flask_wiki.backend.models import Page
from flask_wiki.backend.backend import marsh
from marshmallow_sqlalchemy.convert import ModelConverter
from flask_wiki.backend.custom_fields import GUIDField
from marshmallow import fields
#Here you are overwriting the list of types of the SQLAlchemy, adding your custom type
class GUIDConverter(ModelConverter):
SQLA_TYPE_MAPPING = dict(
list(ModelConverter.SQLA_TYPE_MAPPING.items()) +
[(GUIDField, fields.Str)]
)
class GUIDSerializationField(fields.Field):
def _serialize(self, value, attr, obj):
if value is None:
return value
else:
if isinstance(value, uuid.UUID):
return str(value)
else:
return None
class PageSchema(marsh.ModelSchema):
class Meta:
model = Page
model_converter = GUIDConverter #Tell to Marshmallow to use your custom converter for this model
guid = GUIDSerializationField(attribute="guid")
You can also see this link for help.
I hope it helped and sorry for my bad english

According to what I've read in the docs, you can specify a model_converter attribute in your ModelSchema Meta class. The ModelConverter class has a SQLA_TYPE_MAPPING attribute you can override in a subclass to add your custom GUID field to the types detected by the automatic schema generator.
That said, I've never used it so I don't know if this will work or not.

peewee refers to a base class table that does not exist

The following set-up illustrates my problem:
from peewee import (Model,
SqliteDatabase)
from peewee import (CharField,
ForeignKeyField,
IntegerField)
from playhouse.shortcuts import model_to_dict
database = SqliteDatabase(':memory:')
class BaseModel(Model):
class Meta:
database = database
class Description(BaseModel):
i_am = CharField(
db_column='I_AM'
)
class Meta:
db_table = 'DESCRIPTION'
# This is not a table, just a convenient class to inherit from.
class ValueBase(BaseModel):
value = IntegerField(
db_column='VALUE'
)
description = ForeignKeyField(
db_column='I_AM_ID',
rel_model=Description,
to_field='id'
)
class SpecificValue(ValueBase):
class Meta:
db_table = 'SPECIFIC_VALUE'
if __name__ == '__main__':
models = [Description, SpecificValue]
database.create_tables(models)
description = Description(
i_am = 'prime'
)
description.save()
specific = SpecificValue(
value=7,
description=description
)
specific.save()
### This works
print model_to_dict(
specific
)
### This does NOT work
print model_to_dict(
specific,
backrefs=True
)
The problem is that when I use model_to_dict and ask for back references, then the function thinks specific is in a table valuebase and not in a table called SPECIFIC_VALUE. Does anyone know how to do this sort of inheritance and correctly set table names?
EDIT
The traceback is as follows:
Traceback (most recent call last):
File "/home/lafras/Projects/Cloud/eg.py", line 61, in <module>
backrefs=True
File "build/bdist.linux-x86_64/egg/playhouse/shortcuts.py", line 96, in model_to_dict
File "build/bdist.linux-x86_64/egg/playhouse/shortcuts.py", line 117, in model_to_dict
File "build/bdist.linux-x86_64/egg/peewee.py", line 2794, in __iter__
File "build/bdist.linux-x86_64/egg/peewee.py", line 2787, in execute
File "build/bdist.linux-x86_64/egg/peewee.py", line 2470, in _execute
File "build/bdist.linux-x86_64/egg/peewee.py", line 3199, in execute_sql
File "build/bdist.linux-x86_64/egg/peewee.py", line 3048, in __exit__
File "build/bdist.linux-x86_64/egg/peewee.py", line 3191, in execute_sql
peewee.OperationalError: no such table: valuebase
If SpecificValue inherits directly from BaseModel:
class SpecificValue(BaseModel):
value = IntegerField(
db_column='VALUE'
)
description = ForeignKeyField(
db_column='I_AM_ID',
rel_model=Description,
to_field='id'
)
class Meta:
db_table = 'SPECIFIC_VALUE'
then the output is as expected:
{'id': 1, 'value': 7, 'description': {'id': 1, 'i_am': 'prime'}}

Can you explain a bit more thoroughly what's not working correctly? I don't see any back-references that would be associated with the SpecificValue table, since there are not other models that have a foreign key to it.
EDIT:
OK, so what's happening here is that when you specify backrefs, peewee is trying to go from:
SpecificValue -> Description -> ValueBase (which has a foreign key to description).
The fix is to explicitly exclude the backref:
model_to_dict(specific, backrefs=True, exclude=[Description.valuebase_set])

mongoengine: test1 is not a valid ObjectId

I got the following error message:
$ python tmp2.py
why??
Traceback (most recent call last):
File "tmp2.py", line 15, in <module>
test._id = ObjectId(i[0])
File "/home/mictadlo/.virtualenvs/unisnp/lib/python2.7/site-packages/bson/objectid.py", line 92, in __init__
self.__validate(oid)
File "/home/mictadlo/.virtualenvs/unisnp/lib/python2.7/site-packages/bson/objectid.py", line 199, in __validate
raise InvalidId("%s is not a valid ObjectId" % oid)
bson.errors.InvalidId: test1 is not a valid ObjectId
with this code:
from bson.objectid import ObjectId
from mongoengine import *
class Test(Document):
_id = ObjectIdField(required=True)
tag = StringField(required=True)
if __name__ == "__main__":
connect('dbtest2')
print "why??"
for i in [('test1', "a"), ('test2', "b"), ('test3', "c")]:
test = Test()
test._id = ObjectId(i[0])
test.char = i[1]
test.save()
How is it possible to use its own ids which are unique too?

According to the documentation: http://docs.mongoengine.org/apireference.html#fields, ObjectIdField is 'A field wrapper around MongoDB’s ObjectIds.'. So it cannot accept a string test1 as an object id.
You may have to change the code to something like this:
for i in [(bson.objectid.ObjectId('test1'), "a"), (bson.objectid.ObjectId('test2'), "b"), (bson.objectid.ObjectId('test3'), "c")]:
for your code to work (Assuming test1 etc are valid id)

Two things:
ObjectId receives a 24 hex string, you can't initialize it with that string. For instance, instead of using 'test1' you can use a string such as '53f6b9bac96be76a920e0799' or '111111111111111111111111'. You don't even need to initialize an ObjectId, you could do something like this:
...
test._id = '53f6b9bac96be76a920e0799'
test.save()
...
I don't know what are you trying to accomplish by using _id. If you are trying to produce and id field or "primary key" for you document, it's not necessary because one is generated automatically. Your code would be:
class Test(Document):
tag = StringField(required=True)
for i in [("a"), ("b"), ("c")]:
test = Test()
test.char = i[0]
test.save()
print(test.id) # would print something similar to 53f6b9bac96be76a920e0799
If you insist in using a field named _id you must know that your id will be the same, because internally, MongoDB calls it _id. If you still want to use string1 as identifier you should do:
class Test(Document):
_id = StringField(primary_key=True)
tag = StringField(required=True)