I am running a Flask application with SQLAlchemy (1.1.0b3) and Postgres.
With Flask I provide an API over which the client is able to GET all instances of a type database and POST them again on a clean version of the Flask application, as a way of local backup. When the client posts them again, they should again have the same ID as they had when he downloaded them.
I don't want to disable the "increment" option for primary keys for normal operation but if the client provides an ID with a POST and wishes to give a new resource said ID I would like to set it accordingly without breaking the SQLAlchemy. How can I access/reset the current maximum value of ids?
#app.route('/objects', methods = ['POST'])
def post_object():
if 'id' in request.json and MyObject.query.get(request.json['id']) is None: #1
object = MyObject()
object.id = request.json['id']
else: #2
object = MyObject()
object.fillFromJson(request.json)
db.session.add(object)
db.session.commit()
return jsonify(object.toDict()),201
When adding a bunch of object WITH an id #1 and then trying to add on WITHOUT an id or with a used id #2, I get.
duplicate key value violates unique constraint "object_pkey"
DETAIL: Key (id)=(2) already exists.
Usually, the id is generated incrementally but when that id is already used, there is no check for that. How can I get between the auto-increment and the INSERT?
After adding an object with a fixed ID, you have to make sure the normal incremental behavior doesn't cause any collisions with future insertions.
A possible solution I can think of is to set the next insertion ID to the maximum ID (+1) found in the table. You can do that with the following additions to your code:
#app.route('/objects', methods = ['POST'])
def post_object():
fixed_id = False
if 'id' in request.json and MyObject.query.get(request.json['id']) is None: #1
object = MyObject()
object.id = request.json['id']
fixed_id = True
else: #2
object = MyObject()
object.fillFromJson(request.json)
db.session.add(object)
db.session.commit()
if fixed_id:
table_name = MyObject.__table__.name
db.engine.execute("SELECT pg_catalog.setval(pg_get_serial_sequence('%s', 'id'), MAX(id)) FROM %s;" % (table_name, table_name))
return jsonify(object.toDict()),201
The next object (without a fixed id) inserted into the table will continue the id increment from the biggest id found in the table.
Related
I have a problem when I try to update hstore field. I have the following translation hybrid and database model.
translation_hybrid = TranslationHybrid(
current_locale='en',
default_locale='de'
)
class Book:
__tablename__ = "Book"
id = Column(UUID(as_uuid=True), primary_key=True)
title_translations = Column(MutableDict.as_mutable(HSTORE), nullable=False)
title = translation_hybrid(title_translations)
I want to update title with the current locale using a single orm query. When I try the following query
query(Book).filter(Book.id == id).update({"title": "new_title"})
ORM converts this to the following sql:
UPDATE "Book" SET coalesce(title_translations -> 'en', title_translations -> 'de') = "new_title" WHERE "Book".id = id
And this sql gives the syntax error. What is the best way to update it without fetching the model first and assigning the value to the field?
We got this to run eventually, documenting here for the benefit of others that might run into this issue; Note that we're using the new select methodology and async.
As you already suggested, we solved this by assigning the updated values directly to the record object. We're basically implementing this solution from the SQLAlchemy docu:
updated_record: models.Country = None # type: ignore
try:
# fetch current data from database and lock for update
record = await session.execute(
select(models.Country)
.filter_by(id=str(country_id))
.with_for_update(nowait=True)
)
updated_record = record.scalar_one()
logger.debug(
"update() - fetched current data from database",
record=record,
updated_record=vars(updated_record),
)
# merge country_dict (which holds the data to be updated) with the data in the DB
for key, value in country_dict.items():
setattr(updated_record, key, value)
logger.debug(
"update() - merged new data into record",
updated_record=vars(updated_record),
)
# flush data to database
await session.flush()
# refresh updated_record and commit
await session.refresh(updated_record)
await session.commit()
except Exception as e: # noqa: PIE786
logger.error("update() - an error occurred", error=str(e))
await session.rollback()
raise ValueError("Record can not be updated.")
return updated_record
I think I have solved a similar instance of the problem using the bulk update query variant.
In this case a PoC solution would look like this:
session.execute(update(Book), [{"id": id, "title": title}])
session.commit()
I am not sure why this does not trigger the coalesce() issue but it seems to be working. We should probably open an issue in SQLAlchemy as I don't have the time right now to debug it to its root cause.
UPDATE
I think that the original issue actually originates in sqlalchemy-util as the coalesce seems to arise from the expr_factory of the hybrid property here.
I have this firebase database structure
I want to print out the inventory list(Inventory) for each ID under Businesses.
So I tried this code
db = firebase.database()
all_users = db.child("Businesses").get()
for user in all_users.each():
userid = user.key()
inventorydb = db.child("Businesses").child(userid).child("Inventory")
print(inventorydb)
but all I got is this
<pyrebase.pyrebase.Database object at 0x1091eada0>
what am I doing wrong and how can I loop through each Business ID and print out their inventory?
First, you're printing a Database object. You need to get the data still.
You seem to already know how to get that as well as the children. Or you only copied the examples without understanding it...
Either way, you can try this
db = firebase.database()
businesses = db.child("Businesses")
for userid in businesses.shallow().get().each():
inventory = businesses.child(userid).child("Inventory").get()
print( inventory.val() )
On a side note, National_Stock_Numbers looks like it should be a value of the name, not a key for a child
I am building a web scraper and trying to assign an entity a UUID.
Since one entity may be scraped at different times, I want to store the initial UUID along with the extracted id from the webpage
// example document
{
"ent_eid_type": "ABC-123",
"ent_uid_type": "123e4567-aaa-123e456"
}
below is code that runs for every id field that is found in a scraped item
# if the current ent_eid_type is a key in mongo...
if db_coll.find({ent_eid_type: ent_eid}).count() > 0:
# return the uid value
ent_uid = db_coll.find({ent_uid_type: ent_uid })
else:
# create a fresh uid
ent_uid = uuid.uuid4()
# store it with the current entity eid as key, and uid as value
db_coll.insert({ent_eid_type: ent_eid, ent_uid_type: ent_uid})
# update the current item with the stored uid for later use
item[ent_uid_type] = ent_uid
Console is returning KeyError: <pymongo.cursor.Cursor object at 0x104d41710>. Not sure how to parse the cursor for the ent_uid
Any tips/ suggests appreciated!
Pymongo Find command returns a cursor object you need to iterate or access to get the object
Access the first result (you already checked one exists), and access the ent_uid field.
Presumably, you're going to search on EID type, with ent_eid not ent_uid. No reason to search if you already have it.
ent_uid = db_coll.find({ent_eid_type: ent_eid })[0]['ent_uid']
or don't worry about the cursor and use the find_one command instead (http://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.find_one)
ent_uid = db_coll.find_one({ent_eid_type: ent_eid })['ent_uid']
I'm really new to Python & as new to Pyramid (this is the first thing I've written in Python) and am having trouble with a database query...
I have the following models (relevant to my question anyway):
MetadataRef (contains info about a given metadata type)
Metadata (contains actual metadata) -- this is a child of MetadataRef
User (contains users) -- this is linked to metadata. MetadataRef.model = 'User' and metadata.model_id = user.id
I need access to name from MetadataRef and value from Metadata.
Here's my code:
class User(Base):
...
_meta = None
def meta(self):
if self._meta == None:
self._meta = {}
try:
for item in DBSession.query(MetadataRef.key, Metadata.value).\
outerjoin(MetadataRef.meta).\
filter(
Metadata.model_id == self.id,
MetadataRef.model == 'User'
):
self._meta[item.key] = item.value
except DBAPIError:
##TODO: actually do something with this
self._meta = {}
return self._meta
The query SQLAlchemy is generating does return what I need (close enough anyway -- it needs to query model_id as part of the ON clause rather than the WHERE, but that's minor and I'm pretty sure I can figure that out myself):
SELECT metadata_refs.`key` AS metadata_refs_key, metadata.value AS metadata_value
FROM metadata_refs LEFT OUTER JOIN metadata ON metadata_refs.id = metadata.metadata_ref_id
WHERE metadata.model_id = %s AND metadata_refs.model = %s
However, when I access the objects I get this error:
AttributeError: 'KeyedTuple' object has no attribute 'metadata_value'
This leads me to think there's some other way I need to access it, but I can't figure out how. I've tried both .value and .metadata_value. .key does work as expected.
Any ideas?
You're querying separate attributes ("ORM-enabled descriptors" in SA docs):
DBSession.query(MetadataRef.key, Metadata.value)
in this case the query returns not full ORM-mapped objects, but a KeyedTuple, which is a cross between a tuple and an object with attributes corresponding to the "labels" of the fields.
So, one way to access the data is by its index:
ref_key = item[0]
metadata_value = item[1]
Alternatively, to make SA to use a specific name for column, you may use Column.label() method:
for item in DBSession.query(MetadataRef.key.label('ref_key'), Metadata.value.label('meta_value'))...
self._meta[item.key] = item.meta_value
For debugging you can use Query.column_descriptions() method which will tell you the names of the columns returned by the query.
I want to duplicate a model instance (row) in SQLAlchemy using the orm. My first thought was to do this:
i = session.query(Model)
session.expunge(i)
old_id = i.id
i.id = None
session.add(i)
session.flush()
print i.id #New ID
However, apparently the detached object still "remembers" what id it had, even though I set the id to None while it was detached. Thus, session.flush() tries to execute an UPDATE changing the primary key to null.
Is this expected behavior? How can I remove the 'memory' of this attribute, and just treat the detached object as a new object upon re-adding it to the session? How, in general, does one clone an SQLAlchemy model instance?
this case is available using the make_transient() helper function:
inst = session.query(Model).first()
session.expunge(inst)
make_transient(inst)
inst.id = None
session.add(inst)
session.flush()
print inst.id #New ID
def duplicate(self):
arguments = dict()
for name, column in self.__mapper__.columns.items():
if not (column.primary_key or column.unique):
arguments[name] = getattr(self, name)
return self.__class__(**arguments)