I would like to have a possibility on merging two or more tables (generated by django-tables2) which are generated from
dynamic models.
Let me first describe my problem:
I have the following fields for a dynamically generated model:
...
fields = {
'colA': models.IntegerField(),
'colB': models.IntegerField(),
'colC': models.IntegerField(),
'colD': models.IntegerField(),
}
...
The dynamic handling gives me the possibility to store my tables structered, without defining redundant model classes.
Example for models derived from stored tables in the database:
myDynamicModelA = DataModels().create_model('myDynamicModelA_tablename')
myDynamicModelB = DataModels().create_model('myDynamicModelB_tablename')
myDynamicModelC = DataModels().create_model('myDynamicModelC_tablename')
myDynamicModelD = DataModels().create_model('myDynamicModelD_tablename')
....
'_tablename' is commonly shared by all tables. What differs is the prefix 'myDynamicModel A,B,C...'
This is the model-part.
According to that let me describe the table structure by using django-tables2:
Each model/table shares some columns/fields, so that I can define a django-tables2 class like this:
class Table_A(tables.Table):
colA = tables.Column()
colB = tables.Column()
The fields which are different can be handled simply by using inheritance:
class Table_B(Table_A):
colC = tables.Column()
colD = tables.Column()
def __init__(self, *args, **kwargs):
self.colname = kwargs['colname']
kwargs.pop('colname')
super(Table_B, self).__init__(*args, **kwargs)
for col in self.base_columns:
if col not in ['colA', 'colB']:
self.base_columns[col].verbose_name = '%s_%s' % (self.colname, col)
As you can see, the constructor gives a different column-name-prefix for fields that differ across the models.
Now it is possible to generate tables from the different models, eg:
Table for 'myDynamicModelA_tablename':
columns: colA, colB, myDynamicModelA_tablename_colC, myDynamicModelA_tablename_colD
Table for 'myGenericModelB_tablename':
columns: colA, colB, myDynamicModelB_tablename_colC, myDynamicModelB_tablename_colD
...
Now my question:
is it possible to merge both tables so that I receive something like that:
colA, colB, myDynamicModelA_tablename_colC, myDynamicModelB_tablename_colC, myDynamicModelA_tablename_colD, myDynamicModelB_tablename_colD
The values which will be shown should result from an intersection between the tables (this is possible, because of the common values from colA, which could be interpreted as a primary key)
It is nessecary that the result is a django-tables2 object, because I want to provide pagination as well as sorting options.
I hope my descriptions were understandable, sorry if not.
Many thanks for your time and help.
perhaps this prev question and answer of mine will help?
I access data from combined dynamically created tables via raw sql and send this data to be rendered in a Django-Tables2
If you want to specify the order of and indeed which columns are rendered, define a Meta Class as shown below (where 'sequence' is the order of the columns; nb '...' just means 'and all other columns' - check out the documentary for Tables2 [search for 'Swapping the position of columns']):
def getTable(table_name):
cursor = runSQL(table_name,"""
SELECT * FROM subscription_exptinfo,%s
WHERE %s.id = subscription_exptinfo.id
;""" %(table_name,table_name))
exptData = dictfetchall(cursor)
class Meta:
attrs = {'class': 'paleblue'}
sequence = ('id','...')
attrs = {}
attrs['Meta'] = Meta
cols=exptData[0]
for item in cols:
if item=='timeStart':
attrs[str(item)] = tables.DateTimeColumn(format='d-m-Y g:i:s',short=False)
else:
attrs[str(item)] = tables.Column()
myTable = type('myTable', (TableReport,), attrs)
#TableOptions(table).sequence = ["Id","..."]
#print ".........................." + str(TableOptions(table).sequence)
return myTable(exptData)
Related
I'm starting working with an existing database where attribute foo of table A is related to more then one other table, B.foo and C.foo. How do I form this relationship in ponyorm?
The database is organized like below.
from pony import orm
db = orm.Database()
class POI(db.Entity):
'''Point of interest on a map'''
name = orm.PrimaryKey(str)
coordinateID = orm.Optional(('cartesian', 'polar')) # doesn't work ofc
class cartesian(db.Entity):
coordinateID = orm.Required(POI)
x = orm.Required(float)
y = orm.Required(float)
class polar(db.Entity):
coordinateID = orm.Required(POI)
r = orm.Required(float)
phi = orm.Required(float)
Of course x,y from cartesian and r,phi from polar could be moved to POI, and in the database I work with, that's the same situation. But the tables are divided up between stakeholders (cartesian and polar in this example) and I don't get to change the schema anyway. I can't split coordinateID in the schema (but it would actually be nice to have different attributes of the python class).
It is not possible to relate one attribute for several enties in PonyORM except for the case when these entities are inherited from the same base entity, then you can specify base entity as the attribute type and use any of inherited entity as a real type.
If you use existing schema that you can't change, you probably can't use inheritance and need to specify raw id attribute instead of relationship:
from pony import orm
db = orm.Database()
class POI(db.Entity):
_table_ = "table_name"
name = orm.PrimaryKey(str)
coordinate_id = orm.Optional(int, column="coordinateID")
class Cartesian(db2.Entity):
_table_ = "cartesian"
id = orm.PrimaryKey(int, column="coordinateID")
x = orm.Required(float)
y = orm.Required(float)
class Polar(db2.Entity):
_table_ = "polar"
id = orm.PrimaryKey(int, column="coordinateID")
r = orm.Required(float)
phi = orm.Required(float)
And then you can perform queries like this:
left_join(poi for poi in POI
for c in Cartesian
for p in Polar
if poi.coordinate_id == c.id
and poi.coordinate_id = p.id
and <some additional conditions>)
Note that all entities used in the same query should be from the same database. If entities belongs to two different databases, you cannot use them in the same query. And need to issue separate queries:
with db_session:
poi = POI.get(id=some_id)
coord = Cartesian.get(id=poi.coordinate_id)
if coord is None:
coord = Polar.get(id=poi.coordinate_id)
<do something with poi and coord>
But in case, for example, of SQLite you can attach one database to another to make them appear as a single database.
I've got file objects of different types, which inherit from a BaseFile, and add custom attributes, methods and maybe fields. The BaseFile also stores the File Type ID, so that the corresponding subclass model can be retrieved from any BaseFile object:
class BaseFile(models.Model):
name = models.CharField(max_length=80, db_index=True)
size= models.PositiveIntegerField()
time_created = models.DateTimeField(default=datetime.now)
file_type = models.ForeignKey(ContentType, on_delete=models.PROTECT)
class FileType1(BaseFile):
storage_path = '/path/for/filetype1/'
def custom_method(self):
<some custom behaviour>
class FileType2(BaseFile):
storage_path = '/path/for/filetype2/'
extra_field = models.CharField(max_length=12)
I also have different types of events which are associated with files:
class FileEvent(models.Model):
file = models.ForeignKey(BaseFile, on_delete=models.PROTECT)
time = models.DateTimeField(default=datetime.now)
I want to be able to efficiently get all files of a particular type which have not been involved in a particular event, such as:
unprocessed_files_type1 = FileType1.objects.filter(fileevent__isnull=True)
However, looking at the SQL executed for this query:
SELECT "app_basefile"."id", "app_basefile"."name", "app_basefile"."size", "app_basefile"."time_created", "app_basefile"."file_type_id", "app_filetype1"."basefile_ptr_id"
FROM "app_filetype1"
INNER JOIN "app_basefile"
ON("app_filetype1"."basefile_ptr_id" = "app_basefile"."id")
LEFT OUTER JOIN "app_fileevent" ON ("app_basefile"."id" = "app_fileevent"."file_id")
WHERE "app_fileevent"."id" IS NULL
It looks like this might not be very efficient because it joins on BaseFile.id instead of FileType1.basefile_ptr_id, so it will check ALL BaseFile ids to see whether they are present in FileEvent.file_id, when I only need to check the BaseFile ids corresponding to FileType1, or FileType1.basefile_ptr_ids.
This could result in a significant performance difference if there are a very large number of BaseFiles, but FileType1 is only a small subset of that, because it will be doing a large amount of unnecessary lookups.
Is there a way to force Django to join on "app_filetype1"."basefile_ptr_id" or otherwise achieve this functionality more efficiently?
Thanks for the help
UPDATE:
Using annotations and Exists subquery seems to do what I'm after, however the resulting SQL still appears strange:
unprocessed_files_type1 = FileType1.objects.annotate(file_event=Exists(FileEvent.objects.filter(file=OuterRef('pk')))).filter(file_event=False)
SELECT "app_basefile"."id", "app_basefile"."name", "app_basefile"."size", "app_basefile"."time_created", "app_basefile"."file_type_id", "app_filetype1"."basefile_ptr_id",
EXISTS(
SELECT U0."id", U0."file_id", U0."time"
FROM "app_fileevent" U0
WHERE U0."file_id" = ("app_filetype1"."basefile_ptr_id"))
AS "file_event"
FROM "app_filetype1"
INNER JOIN "app_basefile" ON ("app_filetype1"."basefile_ptr_id" = "app_basefile"."id")
WHERE EXISTS(
SELECT U0."id", U0."file_id", U0."time"
FROM "app_fileevent" U0
WHERE U0."file_id" = ("app_filetype1"."basefile_ptr_id")) = 0
It appears to be doing the WHERE EXISTS subquery twice instead of just using the annotated 'file_event' label... Maybe this is just a Django/SQLite driver bug?
My general question is: can I use the data stored in a HStoreField (Django 1.8.9) to generate columns dynamically for an existing Table class of django-tables2? As an example below, say I have a model:
from django.contrib.postgres import fields as pgfields
GameSession(models.Model):
user = models.ForeignKey('profile.GamerProfile')
game = models.ForeignKey('games.Game')
last_achievement = models.ForeignKey('games.Achievement')
extra_info = pgfields.HStoreField(null=True, blank=True)
Now, say I have a table defined as:
GameSessionTable(tables.Table):
class Meta(BaseMetaTable):
model = GameSession
fields = []
orderable=False
id = tables.LinkColumn(accessor='id', verbose_name='Id', viewname='reporting:session_stats', args=[A('id')], attrs={'a':{'target':'_blank'}})
started = DateTimeColumn(accessor='startdata.when_started', verbose_name='Started')
stopped = DateTimeColumn(accessor='stopdata.when_stopped', verbose_name='Stopped')
game_name = tables.LinkColumn(accessor='game.name', verbose_name='Game name', viewname='reporting:game_stats', args=[A('mainjob.id')], attrs={'a':{'target':'_blank'}})
I want to be able to add columns for each of the keys stored in the extra_info column for all of the GameSessions. I have tried to override the init() method of the GameSessionTable class, where I have access to the queryset, then make a set of all the keys of my GameSession objects, then add them to self, however that doesn't seem to work. Code below:
def __init__(self, data, *args, **kwargs):
super(GameSessionTable, self).__init__(data, *args, **kwargs)
if data:
extra_cols=[]
# just to be sure, check that the model has the extra_info HStore field
if data.model._meta.get_field('extra_info'):
extra_cols = list(set([item for q in data if q.extra_info for item in q.extra_info.keys()]))
for col in extra_cols:
self.columns.columns[col] = tables.Column(accessor='extra_info.%s' %col, verbose_name=col.replace("_", " ").title())
Just a mention, I have had a look at https://spapas.github.io/2015/10/05/django-dynamic-tables-similar-models/#introduction but it's not been much help because the use case there is related to the fields of a model, whereas my situation is slightly different as you can see above.
Just wanted to check, is this even possible or do I have to define an entirely different table for this data, or potentially use an entirely different library altogether like django-reports-builder?
Managed to figure this out to a certain extent. The code I was running above was slightly wrong, so I updated it to run my code before the superclass init() gets run, and changed where I was adding the columns.
As a result, my init() function now looks like this:
def __init__(self, data, *args, **kwargs):
if data:
extra_cols=[]
# just to be sure, check that the model has the extra_info HStore field
if data.model._meta.get_field('extra_info'):
extra_cols = list(set([item for q in data if q.extra_info for item in q.extra_info.keys()]))
for col in extra_cols:
self.base_columns[col] = tables.Column(accessor='extra_info.%s' %col, verbose_name=col.replace("_", " ").title())
super(GameSessionTable, self).__init__(data, *args, **kwargs)
Note that I replaced self.columns.columns (which were BoundColumn instances) with self.base_columns. This allows the superclass to then consider these as well when initializing the Table class.
Might not be the most elegant solution, but it seems to do the trick for me.
I have a many to many relationship table whith some datas in the jointing base
a basic version of my model look like:
class FooLine(models.Model):
name = models.CharField(max_length=255)
class FooCol(models.Model):
name = models.CharField(max_length=255)
class FooVal(models.Model):
value = models.CharField(max_length=255)
line = models.ForeignKey(FooLine)
col = models.ForeignKey(FooCol)
I'm trying to search every values for a certain line with a null if the value is not present (basically i'm trying to display the fooval table with null values for values that haven't been filled)
a typical sql would be
SELECT value FROM FooCol LEFT OUTER JOIN
(FooVal JOIN FooLine
ON FooVal.line_id == FooLine.id AND FooLine.name = "FIXME")
ON FooCol.id = col_id;
Is there any way to modelise above query using django model
Thanks
Outer joins can be viewed as a hack because SQL lacks "navigation".
What you have is a simple if-statement situation.
for line in someRangeOfLines:
for col in someRangeOfCols:
try:
cell= FooVal.objects().get( col = col, line = line )
except FooVal.DoesNotExist:
cell= None
That's what an outer join really is -- an attempted lookup with a NULL replacement.
The only optimization is something like the following.
matrix = {}
for f in FooVal.objects().all():
matrix[(f.line,f.col)] = f
for line in someRangeOfLines:
for col in someRangeOfCols:
cell= matrix.get((line,col),None)
class Tag(models.Model):
name = models.CharField(maxlength=100)
class Blog(models.Model):
name = models.CharField(maxlength=100)
tags = models.ManyToManyField(Tag)
Simple models just to ask my question.
I wonder how can i query blogs using tags in two different ways.
Blog entries that are tagged with "tag1" or "tag2":
Blog.objects.filter(tags_in=[1,2]).distinct()
Blog objects that are tagged with "tag1" and "tag2" : ?
Blog objects that are tagged with exactly "tag1" and "tag2" and nothing else : ??
Tag and Blog is just used for an example.
You could use Q objects for #1:
# Blogs who have either hockey or django tags.
from django.db.models import Q
Blog.objects.filter(
Q(tags__name__iexact='hockey') | Q(tags__name__iexact='django')
)
Unions and intersections, I believe, are a bit outside the scope of the Django ORM, but its possible to to these. The following examples are from a Django application called called django-tagging that provides the functionality. Line 346 of models.py:
For part two, you're looking for a union of two queries, basically
def get_union_by_model(self, queryset_or_model, tags):
"""
Create a ``QuerySet`` containing instances of the specified
model associated with *any* of the given list of tags.
"""
tags = get_tag_list(tags)
tag_count = len(tags)
queryset, model = get_queryset_and_model(queryset_or_model)
if not tag_count:
return model._default_manager.none()
model_table = qn(model._meta.db_table)
# This query selects the ids of all objects which have any of
# the given tags.
query = """
SELECT %(model_pk)s
FROM %(model)s, %(tagged_item)s
WHERE %(tagged_item)s.content_type_id = %(content_type_id)s
AND %(tagged_item)s.tag_id IN (%(tag_id_placeholders)s)
AND %(model_pk)s = %(tagged_item)s.object_id
GROUP BY %(model_pk)s""" % {
'model_pk': '%s.%s' % (model_table, qn(model._meta.pk.column)),
'model': model_table,
'tagged_item': qn(self.model._meta.db_table),
'content_type_id': ContentType.objects.get_for_model(model).pk,
'tag_id_placeholders': ','.join(['%s'] * tag_count),
}
cursor = connection.cursor()
cursor.execute(query, [tag.pk for tag in tags])
object_ids = [row[0] for row in cursor.fetchall()]
if len(object_ids) > 0:
return queryset.filter(pk__in=object_ids)
else:
return model._default_manager.none()
For part #3 I believe you're looking for an intersection. See line 307 of models.py
def get_intersection_by_model(self, queryset_or_model, tags):
"""
Create a ``QuerySet`` containing instances of the specified
model associated with *all* of the given list of tags.
"""
tags = get_tag_list(tags)
tag_count = len(tags)
queryset, model = get_queryset_and_model(queryset_or_model)
if not tag_count:
return model._default_manager.none()
model_table = qn(model._meta.db_table)
# This query selects the ids of all objects which have all the
# given tags.
query = """
SELECT %(model_pk)s
FROM %(model)s, %(tagged_item)s
WHERE %(tagged_item)s.content_type_id = %(content_type_id)s
AND %(tagged_item)s.tag_id IN (%(tag_id_placeholders)s)
AND %(model_pk)s = %(tagged_item)s.object_id
GROUP BY %(model_pk)s
HAVING COUNT(%(model_pk)s) = %(tag_count)s""" % {
'model_pk': '%s.%s' % (model_table, qn(model._meta.pk.column)),
'model': model_table,
'tagged_item': qn(self.model._meta.db_table),
'content_type_id': ContentType.objects.get_for_model(model).pk,
'tag_id_placeholders': ','.join(['%s'] * tag_count),
'tag_count': tag_count,
}
cursor = connection.cursor()
cursor.execute(query, [tag.pk for tag in tags])
object_ids = [row[0] for row in cursor.fetchall()]
if len(object_ids) > 0:
return queryset.filter(pk__in=object_ids)
else:
return model._default_manager.none()
I've tested these out with Django 1.0:
The "or" queries:
Blog.objects.filter(tags__name__in=['tag1', 'tag2']).distinct()
or you could use the Q class:
Blog.objects.filter(Q(tags__name='tag1') | Q(tags__name='tag2')).distinct()
The "and" query:
Blog.objects.filter(tags__name='tag1').filter(tags__name='tag2')
I'm not sure about the third one, you'll probably need to drop to SQL to do it.
Please don't reinvent the wheel and use django-tagging application which was made exactly for your use case. It can do all queries you describe, and much more.
If you need to add custom fields to your Tag model, you can also take a look at my branch of django-tagging.
This will do the trick for you
Blog.objects.filter(tags__name__in=['tag1', 'tag2']).annotate(tag_matches=models.Count(tags)).filter(tag_matches=2)