I have a mongoengine schema like this:
class Page(Document):
title = StringField(max_length=200, required=True)
date_modified = DateTimeField(default=datetime.datetime.now)
meta = {"db_alias":"page", "collection":"page_detail"}
as you can see, my collection will be save with name "page_detail".
so my problem is this:
I have data in my database already, but some record is useless, so I need to filter them. Now, I want to filter them and save the filtered data in another collection with this schema, so I have any choice?
You can use aggregation framework with $out operator, which will store the result of your query into a new collection specified in the $out.
db.yourOldCollection.aggregate([
<your filetring pipeline, e.g. $match>,
{$out: "yourNewCollection"}
])
Related
I wondered if it is possible to query documents in MongoDB by computed properties using mongoengine in python.
Currently, my model looks like this:
class SnapshotIndicatorKeyValue(db.Document):
meta = {"collection": "snapshot_indicator_key_values"}
snapshot_id = db.ObjectIdField(nullable=False)
indicator_key_id = db.ObjectIdField(nullable=False)
value = db.FloatField(nullable=False)
created_at = db.DateTimeField()
updated_at = db.DateTimeField()
#property
def snapshot(self):
return Snapshot.objects(id=self.snapshot_id).first()
def indicator_key(self):
return IndicatorKey.objects(id=self.indicator_key_id).first()
When I do for example SnapshotIndicatorKeyValue .objects().first().snapshot, I can access the snapshotproperty.
But when I try to query it, it doesn't work. For example:
SnapshotIndicatorKeyValue.objects(snapshot__date_time__lte=current_date_time)
I get the error `mongoengine.errors.InvalidQueryError: Cannot resolve field "snapshot"``
Is there any way to get this working with queries?
I need to query SnapshotIndicatorKeyValue based on a property of snapshot.
In order to query the snapshot property directly through mongoengine, you can reference the related snapshot object rather than the snapshot_id in your SnapshotIndicatorKeyValue document definition.
An amended model using a Reference field would be like this:
from mongoengine import Document, ReferenceField
class Snapshot(Document)
property_abc = RelevantPropertyHere() # anything you need
class SnapshotIndicatorKeyValue(Document):
snapshot = ReferenceField(Snapshot)
You would sucessively save an instance of Snapshot and an instance of SnapshotIndicatorKeyValue like this:
sample_snapshot = Snapshot(property_abc=relevant_value_here) # anything you need
sample_snapshot.save()
sample_indicatorkeyvalue = SnapshotIndicatorKeyValue()
sample_indicatorkeyvalue.snapshot = sample_snapshot
sample_indicatorkeyvalue.save()
You can then refer to any of the snapshot's properties through:
SnapshotIndicatorKeyValue.objects.first().snapshot.property_abc
I start to use mongoengine in Python as a Document-Object Mappe on an already established collection. The documents are schema less. Now for development, debugging and might within the application the question what fields/keys the object User has is of interest.
Is there a different approach to achieve the same, may be without querying the document every time?
class User(DynamicDocument):
field_one = StringField()
def keys(self):
return type(self).objects.as_pymongo().filter(id=self.id).first().keys()
You can use instance._data.keys() for this, it will combine both known and dynamic fields.
from mongoengine import *
connect()
class TestDoc(DynamicDocument):
f1 = StringField()
# simulate document with dynamic fields
td1 = TestDoc(f1='garbage1')
td1.other_field = 'garbage2'
td1.other_field2 = 'garbage99'
td1.save()
# show how it looks in the database
print(TestDoc.objects.as_pymongo()) # [{'_id': ObjectId('...'), 'f1': 'garbage1', 'other_field': 'garbage2', 'other_field2': 'garbage99'}]
doc = TestDoc.objects.first()
print(doc._data.keys()) # ['f1', 'id', 'other_field', 'other_field2']
You can also look at instance._dynamic_fields.keys() which will list only the dynamic ones.
I'm writing a lot of boilerplate like code to try and persist my form data. In other languages I've not had to manually map the form attribute to the model prior to saving I'm wondering does python/flask has a similar approach? I'd ideally prefer to manually map all my form fields to simply persist a db record.
[edit]
To be clear this approach works fine for me the records are persisted properly however it seems kind of inefficient. In another piece of code I was able to use the wtf library {{ wtf.quick_form(form) }}
to create a form an html form without explicitly listing all its fields. Given I have the same names for my model attributes as I do for the form attributes I was wondering if I can do the same?
[edit]
Any feedback
Example below..
I have a largish model "product"
class Product(db.Model):
field1 = db.Column(db.String(200))
field2 = db.Column(db.String(200))
....
field30 = db.Column(db.String(200))
It's represented as a flask form e.g.
class ProductForm(FlaskForm):
field1 = StringField('Field 1', validators=[DataRequired()])
field2 = TextAreaField('Field 2 Label')
field30 = TextAreaField('Field 30 Label')
submit = SubmitField('Submit')
Here I try and map and persist the product record...
def save_product(self, product, form):
product.field1 = form.field.data
#lots more tedious form to model mapping
db.session.add(product)
# commit the data to the database
db.session.commit()
In my routes.py I have the following. I see it being called and the record does indeed persist.
def crud():
form = CrudForm()
if form.validate_on_submit():
product = Product()
product.save_product(product, form, new=True)
flash(_('Product Saved'))
return redirect(url_for('main.index'))
return render_template('crud.html', title=_('Product CRUD'),
form=form)
Given that the names of your form fields are the same as the names of your database columns it is possible to cut down on the boilerplate. A dict of all of the fields in your form is accessible as the form.data property and this can be passed directly through as keyword arguments when your create the new instance of your model rather than setting them one at a time afterwards.
product = Product(**form.data)
I have mapped fields defined in an Interface to a mysql backend database through the use of saconfig and a declarative base.* On my Plone Site instance, I have a form (z3c.form) that will use these fields and will be used to search for "Assets" from a mysql database based on criteria entered in several fields. In my case, I would like to only query on fields that have data entered in them.
This is what my interface and ORMBase class look like:
class IAsset(Interface):
"""Interface class of an asset
"""
Asset_ID = schema.Int(title=u"Asset ID",
required=False
)
GPCL_Asset_ID = schema.TextLine(title=u"GPCL Asset Tracker",
required=False
)
Asset_Type = schema.Int(title=u"Asset Type",
required=False
)
Manufacturer = schema.Int(title=u"Manufacturer",
required=True
)
Model = schema.TextLine(title=u"Serial Number",
required=False
)
Serial_Number = schema.TextLine(title=u"Serial Number",
required=False
)
class Asset(ORMBase):
"""Class for asset
"""
__tablename__ = 'Assets'
Asset_ID = sqlalchemy.Column(sqlalchemy.Integer(),
primary_key=True,
autoincrement=True
)
GPCL_Asset_ID = sqlalchemy.Column(sqlalchemy.String(255),
nullable=True
)
Asset_Type = sqlalchemy.Column(sqlalchemy.Integer(),
nullable=False
)
Manufacturer = sqlalchemy.Column(sqlalchemy.Integer(),
nullable=False
)
Model = sqlalchemy.Column(sqlalchemy.String(255),
nullable=True
)
Serial_Number = sqlalchemy.Column(sqlalchemy.String(255),
nullable=True
)
For testing purposes, I am using the view class of an Asset, and I have a function (current called through a template for testing purposes) that builds a dictionary based on fields that are filled out. I also have a utility class called AssetUtils and there is a function in the class called queryAssets that accepts a dictionary containing the criteria, and returns the search results found.
class View(grok.View):
"""View class
"""
...grok definitions and fields being set for form
def searchAssets(self):
#get data
....
criteria = {}
#build dictionary
#if field (i.e. GPCL_Asset_ID) is not empty, add to the dictionary GPCL_Asset_ID:valueEnteredByUser
assets = queryAssets(criteria)
class AssetUtils(grok.GlobalUtility):
def queryAssets(searchCriteria):
I have a session object defined at the root of the class. In queryAssets I would like to use session.query(Asset).filter() to query the Assets table. The problem I am having though is figuring out what to pass into filter(). I know how to use the query function, like session.query(Asset).filter(GPCL_Asset_ID.like('D%'),Manufacturer==15).
However, say Serial Number was present, but GPCL_Asset_ID was not, so I would like to use .filter(Serial_Number.like("EX12%")), and then another case where only Model and Manufacturer were present. .filter(Model.like("A3%"),Manufacturer==32)
So my question is, how can I make it so that .filter filters on fields that were defined/chosen in the dictionary passed into the queryAssets function I have, as opposed to fields being pre-defined in the .filter function?
*I am following along with the book "Professional Plone 4 Development" by Martin Aspeli, chapter 12 and a slideshow presentation (http://www.slideshare.net/jbellis/pycon-2010-sqlalchemy-tutorial) - Slide 27, 29. Compared to the slideshow, I am actually using saconfig as found in Martin Aspeli's book.
Looking at your class, from the searchAssets method you simply need to access the self.request.form dict-like object and read all the data submitted (by HTTP GET or POST) to that view.
You can then build your list of filters parameters like:
params = []
if 'param1' in self.request.form:
params.append(Asset_ID.like(...))
if ...
session.query(*params)
Is it possible to store a mongo_id as an ObjectId object in a MongoAlchemy field? I've been able to store an ObjectId inside of a document I defined manually, but it seems as though I'm restricted to storing the string value of the id in the context of the MongoAlchemy ORM.
Here's some of my code:
class Group(db.Document):
name = db.StringField()
trial_id = db.StringField(required=False)
participants = db.ListField(
db.DictField(db.AnythingField()), default_empty=True, required=False)
def add_participant(self, participant):
self.participants.append({
'participant_id': participant.mongo_id,
'start': datetime.utcnow(),
})
class Trial(db.Document):
name = db.StringField()
groups = db.ListField(
db.DocumentField(Group), default_empty=True, required=False)
def add_group(self, group):
group.trial_id = str(self.mongo_id)
group.save()
def get_group(self, group):
return Group.query.filter(
{'name': group, 'trial_id': str(self.mongo_id)}).first()
You'll see that I'm able to store a mongo_id as an ObjectId object in the Group method add_participant (since it's creating document manually, not through the MongoAlchemy ORM), but am forced to convert the mongo_id to a string in order to store it in a db.StringField.
I tried storing the original ObjectId in a db.AnythingField, but was then unable to filter by it.
Does anyone know if it's possible to store an ObjectId in a MongoAlchemy field and then filter by it in a database query?
Thank you!
You want an ObjectIdField: http://www.mongoalchemy.org/api/schema/fields.html#mongoalchemy.fields.ObjectIdField
This is the type of field which is used for mongo_id (although that one is special-cased)
try
id = db.ObjectIdField().gen()
This would automatically generate the object id for each instance of the mongo db object/document - as would id's in relational dbs