We Keep Coding

Sqlalchemy query on multiple relationship between two tables

I am having trouble with the following setup of a sqlalchemy ORM connected to a postgresql db.
class Map(Base):
__tablename__ = "map"
id = Column(BigInteger, Sequence(name="myseq"), primary_key=True)
cmp_1_id = Column(BigInteger, ForeignKey("component.id"))
cmp_2_id = Column(BigInteger, ForeignKey("component.id"))
class Component(Base):
__tablename__ = "component"
id = Column(BigInteger, Sequence(name="myseq"), primary_key=True)
map_1 = relationship("Map", back_populates="cmp_1", foreign_keys=Map.cmp_1_id, uselist=False)
map_2 = relationship("Map", back_populates="cmp_2", foreign_keys=Map.cmp_2_id, uselist=False)
Map.cmp_1 = relationship(
"Component", back_populates="map_1", primaryjoin=Map.cmp_1_id == Component.id
)
Map.cmp_2 = relationship(
"Component", back_populates="map_2", primaryjoin=Map.cmp_2_id == Component.id
)
Now I want to query a specific Map object, whose cmp_1 object has a certain "known_value" of other_attribute. I tried various statements, using Query API and Select API and with a colleague finally found this solution to be working:
(session.query(Map.id)
.join(Map.cmp_1)
.where(Component.other_attribute=="known_value")
.one()[0]
)
During my research on the topic I read through some other SO articles, which raised further questions. So here they come:
My main question: why can't I do the query like this:
(session.query(Alias_Map_Expanded.id)
.where(Map.cmp_1.other_attribute=="known_value")
).one()[0]
This raises the exception AttributeError: Neither 'InstrumentedAttribute' object nor 'Comparator' object associated with Map.cmp_1 has an attribute 'other_attribute'
More generally: how would I design the model better (as in: more robust and easier to jump between relations) to possibly be able to do the above? The relationships need to be One (Component) To Many (Map), i.e a Map object points to one to two (cmp_2 is optional) components. In turn a component can be pointed to from multiple Map rows/objects.
Based on this: Should I always define a foreign key along with a relationship to not break the relationship inside the db? Update: I removed this question because I now find it rather misleading and not really worth having it answered.
Based on that: I guess I also need to use the post_update to not have a circular dependency? Or do I misinterpret the use of post_update?
Thanks in advance!

After some thorough consulting of the extensive sqlalchemy docs I found some answers:
To my first question and the related query: in my ORM classes I did not specify the loading type of the data, leaving it at the default type "lazy". Therefore the other_attribute attribute's value is not loaded with the first query but rather it would take a second call to query1_result.other_attribute upon which the related content would be queried separately. Alternatively I'd need to specify an eager loading type for the proposed query to be working.
I just figured, even if I use eager-loaded queries, I still cannot filter related objects, using class-level ORM syntax, because at that point the ORM instance has not yet mapped its relative. The filtering ("where" clause) needs to be formulated on SQL level, i.e like the first example I gave above...
There is most likely no meaningful answer to that, especially without deeper knowledge of my database structure...
Third question, based on link: I think my question is somewhat strange and maybe even misleading. I will remove it from the original post.
Last question, based on 2nd link: I haven't investigated so much more on this question, being it the least important to me, but I think I got the concept of post_update wrong and will not need it for my purpose.
I got all of it from sqlalchemy docs, so in case you hit that question and have a similar problem, work your way through the extensive documentation conglomeration. The answer is most likely there.

Is it possible to have a collection on an object that does not have a foreign key relationship to each other?

I'm using an declarative SQLAlchemy class to perform computations. Part of the computations require me to perform the computations for all configurations provided by a different table which doesn't have any foreign key relationships between the two tables.
This analogy is nothing like my real application, but hopefully will help to comprehend what I want to happen.
I have a set of cars and a list of paint colors.
The car object has a factory which provides a car in all possible colors
from sqlalchemy import *
from sqlachemy.orm import *
def PaintACar(car, color):
pass
Base = declarative_base()
class Colors(Base):
__table__ = u'colors'
id = Column('id', Integer)
color= Column('color', Unicode)
class Car(Base):
__table__ = u'car'
id = Column('id', Integer)
model = Column('model', Unicode)
# is this somehow possible?
all_color_objects = collection(...)
# I know this is possible, but would like to know if there's another way
#property
def all_colors(self):
s = Session.object_session(self)
return s.query(A).all()
def CarColorFactory(self):
for color in self.all_color_objects:
yield PaintACar(self, color)
My question: Is it possible to produce all_color_objects somehow? Without having to resort to finding the session and manually issuing a query as in the all_colors property?

It's been a while, so I'm providing the best answer I saw (as a comment by zzzeek). Basically, I was looking for one-off syntactic sugar. My original 'ugly' implementation works just fine.
what better way would there be here besides getting a Session and producing the query you
want? Are you looking for being able to add to the collection and that automatically
flushes things? (just add the objects to the Session?) Do you not like using
object_session(self) >(you can build some mixin class or something that hides that for
you?) It's not really clear >what the problem is. The objects here have no relationship to
the parent class so there's no particular intelligence SQLAlchemy would be able to add.
– zzzeek Jun 17 at 5:03

Basic logic on Python OOP

I have studied Python and understand how the OOP concept work. The doubt I have is how to implement in a application that has sql interaction..
Assume I have a employee table in SQL which contains Emp name, Emp address , Emp Salary..Now, I need to give a salary raise for all the employees for which I create a class with a method.
My Database logic
db = MySQLdb.connect("localhost","testuser","test123","TESTDB" )
cursor = db.cursor()
sql = "Select * from Emp"
cursor.execute(sql)
results = cursor.fetchall()
for row in results:
name = row[0]
address = row[1]
salary = row[2]
db.close()
Class definiton:
Class Emp():
def __init__(self,name,salary):
self.name = name
self.salary = salary
def giveraise(self):
self.salary = self.salary*1.4
How do I create the object with the details fetched from the Employee table...I know you don't need to create a class to perform this small operation..but Im thinking in the lines of practical implementation.
I need to process record by record.

It sounds like you are looking for an Object-Relational Mapper (ORM). This is a way of automatically integrating an SQL database with an object-oriented representation of the data, and sounds perfect for your use case. A list of ORMs is at http://wiki.python.org/moin/HigherLevelDatabaseProgramming. Once you have the ORM set up, you can just call the giveraise method on all the Emp objects. (Incidentally, the giveraise function you have defined is not valid Python.)

As Abe has mentioned, there are libraries that do this sort of mapping for you already (such as SQLAlchmy or Django's included ORM).
One thing to keep in mind though is that a "practical implementation" is subjective. For a small application and a developer with no experience with ORMs, coming up to speed on the general idea of an ORM can add time to the project, and subtle errors to the code. Too many times will someone be confused why something like this takes forever with an orm...
for employee in database.query(Employee).all():
employee.give_raise(0.5)
Assuming a table with 100 employees, this could make anything between a few and 200+ individual calls to the database, depending on how you've set up your ORM.
It may be completely justifiable to not use OOP, or at least not in the way you've described. This is a completely valid object:
class Workforce(object):
def __init__(self, sql_connection):
self.sql_connection = sql_connection
def give_raise(self, employee_id, raise_amount):
sql = 'UPDATE Employee SET Salary=Salary + ? WHERE ID = ?'
self.sql_connection.execute(sql, raise_amount, employee_id)
By no means am I trying to knock on ORMs or dissuade you from using them. Especially if you're just experimenting for the heck of it, learning how to use them properly can make them valuable tools. Just keep in mind that they CAN be a hassle, especially if you're never used one before, and to consider that there are simpler solutions if your application is small enough.

SQLAlchemy modelling a complex relationship using reflection

I am querying a proprietary database which is maintained by a third party. The database has many tables each with large numbers of fields.
My problem refers to three tables of interest, Tree, Site and Meter.
The tree table describes nodes in a simple tree structure. Along with other data it has a foreign key referencing its own primary key. It also has an Object_Type field and an Object_ID field. The Site and Meter tables each have many fields.
A tree node has a one-to-one relationship with either be a meter or a site. If the Object_Type field is 1 then the Object_ID field refers to the primary key in the Site table. If it is 2 then it refers to the primary key in the Meter table.
following this example https://bitbucket.org/sqlalchemy/sqlalchemy/src/408388e5faf4/examples/declarative_reflection/declarative_reflection.py
I am using reflection to load the table structures like so
Base = declarative_base(cls=DeclarativeReflectedBase)
class Meter(Base):
__tablename__ = 'Meter'
class Site(Base):
__tablename__ = 'Site'
class Tree(Base):
__tablename__ = 'Tree'
Parent_Node_ID = Column(Integer, ForeignKey('Tree.Node_ID'))
Node_ID = Column(Integer, primary_key=True)
children = relationship("Tree", backref=backref('parent', remote_side=[Node_ID]))
Base.prepare(engine)
I have included the self-referential relationship and that works perfectly. How can I add the two relationships using Object_ID as the foreign key, with the appropriate check on the Object_Type field?

First a note on reflection. I've found myself much better off not relying on reflection.
it does not require a valid database connection for you to load/work with your code
it violates the python guide that explicit is better than implicit. If you look at you code you are better off seeing the elements (columns etc) rather than having them magically created outside your field of view.
This means more code but more maintainable.
The reason I suggested that is at least in part that I cannot see schema in your posting.
If you create the tables and classes in your code rather than relying on reflection, you can then have better control over mapping.
In this case you want to use polymorphic mapping
create a TreeNode class as above.
create SiteNode and MeterNode as subclasses
Your code would then include something like:
mapper(TreeNode,tree_table,polymorphic_on=tree_table.c.object_type)
mapper(SiteNode, site_table,inherits=TreeNode,
inherit_condition=site_table.c.node_id==tree_table.c.node_id,
polymorphic_identity=1)
Hope this helps.

for tree.object_id to be a foreign key that can refer either to Site or Meter, you can either have Site and Meter descend from a common base table, that is, joined table inheritance, or be mapped to the same table, that is, single table inheritance, or as someone said have Tree be mapped to two different tables as well as a common base table. This last suggestion goes well with the idea that TreeNode already has a "type" field.
The final alternative which might be easier is to use two foreign keys on TreeNode directly - site_id and meter_id, as well as two relationships, "meter" and "site"; then use a Python #property to return one or the other:
class TreeNode(Base):
# ...
#property
def object(self):
return self.meter or self.site

Django ForeignKey which does not require referential integrity?

I'd like to set up a ForeignKey field in a django model which points to another table some of the time. But I want it to be okay to insert an id into this field which refers to an entry in the other table which might not be there. So if the row exists in the other table, I'd like to get all the benefits of the ForeignKey relationship. But if not, I'd like this treated as just a number.
Is this possible? Is this what Generic relations are for?

This question was asked a long time ago, but for newcomers there is now a built in way to handle this by setting db_constraint=False on your ForeignKey:
https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.ForeignKey.db_constraint
customer = models.ForeignKey('Customer', db_constraint=False)
or if you want to to be nullable as well as not enforcing referential integrity:
customer = models.ForeignKey('Customer', null=True, blank=True, db_constraint=False)
We use this in cases where we cannot guarantee that the relations will get created in the right order.
EDIT: update link

I'm new to Django, so I don't now if it provides what you want out-of-the-box. I thought of something like this:
from django.db import models
class YourModel(models.Model):
my_fk = models.PositiveIntegerField()
def set_fk_obj(self, obj):
my_fk = obj.id
def get_fk_obj(self):
if my_fk == None:
return None
try:
obj = YourFkModel.objects.get(pk = self.my_fk)
return obj
except YourFkModel.DoesNotExist:
return None
I don't know if you use the contrib admin app. Using PositiveIntegerField instead of ForeignKey the field would be rendered with a text field on the admin site.

This is probably as simple as declaring a ForeignKey and creating the column without actually declaring it as a FOREIGN KEY. That way, you'll get o.obj_id, o.obj will work if the object exists, and--I think--raise an exception if you try to load an object that doesn't actually exist (probably DoesNotExist).
However, I don't think there's any way to make syncdb do this for you. I found syncdb to be limiting to the point of being useless, so I bypass it entirely and create the schema with my own code. You can use syncdb to create the database, then alter the table directly, eg. ALTER TABLE tablename DROP CONSTRAINT fk_constraint_name.
You also inherently lose ON DELETE CASCADE and all referential integrity checking, of course.

To do the solution by #Glenn Maynard via South, generate an empty South migration:
python manage.py schemamigration myapp name_of_migration --empty
Edit the migration file then run it:
def forwards(self, orm):
db.delete_foreign_key('table_name', 'field_name')
def backwards(self, orm):
sql = db.foreign_key_sql('table_name', 'field_name', 'foreign_table_name', 'foreign_field_name')
db.execute(sql)
Source article

(Note: It might help if you explain why you want this. There might be a better way to approach the underlying problem.)
Is this possible?
Not with ForeignKey alone, because you're overloading the column values with two different meanings, without a reliable way of distinguishing them. (For example, what would happen if a new entry in the target table is created with a primary key matching old entries in the referencing table? What would happen to these old referencing entries when the new target entry is deleted?)
The usual ad hoc solution to this problem is to define a "type" or "tag" column alongside the foreign key, to distinguish the different meanings (but see below).
Is this what Generic relations are for?
Yes, partly.
GenericForeignKey is just a Django convenience helper for the pattern above; it pairs a foreign key with a type tag that identifies which table/model it refers to (using the model's associated ContentType; see contenttypes)
Example:
class Foo(models.Model):
other_type = models.ForeignKey('contenttypes.ContentType', null=True)
other_id = models.PositiveIntegerField()
# Optional accessor, not a stored column
other = generic.GenericForeignKey('other_type', 'other_id')
This will allow you use other like a ForeignKey, to refer to instances of your other model. (In the background, GenericForeignKey gets and sets other_type and other_id for you.)
To represent a number that isn't a reference, you would set other_type to None, and just use other_id directly. In this case, trying to access other will always return None, instead of raising DoesNotExist (or returning an unintended object, due to id collision).

tablename= columnname.ForeignKey('table', null=True, blank=True, db_constraint=False)
use this in your program