We Keep Coding

How to use variable column name in filter in Django ORM?

I have two tables BloodBank(id, name, phone, address) and BloodStock(id, a_pos, b_pos, a_neg, b_neg, bloodbank_id). I want to fetch all the columns from two tables where the variable column name (say bloodgroup) which have values like a_pos or a_neg... like that and their value should be greater than 0. How can I write ORM for the same?
SQL query is written like this to get the required results.
sql="select * from public.bloodbank_bloodbank as bb, public.bloodbank_bloodstock as bs where bs."+blood+">0 and bb.id=bs.bloodbank_id order by bs."+blood+" desc;"
cursor = connection.cursor()
cursor.execute(sql)
bloodbanks = cursor.fetchall()

You could be more specific in your questions, but I believe you have a variable called blood which contains the string name of the column and that the columns a_pos, b_pos, etc. are numeric.
You can use a dictionary to create keyword arguments from strings:
filter_dict = {bloodstock__blood + '__gt': 0}
bloodbanks = Bloodbank.objects.filter(**filter_dict)
This will get you Bloodbank objects that have a related bloodstock with a greater than zero value in the bloodgroup represented by the blood variable.
Note that the way I have written this, you don't get the bloodstock columns selected, and you may get duplicate bloodbanks. If you want to get eliminate duplicate bloodbanks you can add .distinct() to your query. The bloodstocks are available for each bloodbank instance using .bloodstock_set.all().
The ORM will generate SQL using a join. Alternatively, you can do an EXISTS in the where clause and no join.
from django.db.models import Exists, OuterRef
filter_dict = {blood + '__gt': 0}
exists = Exists(Bloodstock.objects.filter(
bloodbank_id=OuterRef('id'),
**filter_dict
)
bloodbanks = Bloodbank.objects.filter(exists)
There will be no need for a .distinct() in this case.

Passing a query output as input to in clause for another query in SQLAlchemy

I have 2 tables (say Student and College) and a third table (StudentCollege), which has student_id and college_id foreign keys.
I want to give output from below query:
list = (
db.session.query(StudentCollegeModel.College_id)
.filter(StudentCollegeModel.student_id== student_id)
.all()
)
to below query:
(
db.session.query(CollegeModel)
.filter(CollegeModel.College_id.in_(list))
.all()
)
But its giving programming error.

You don't need to execute the first query to use it as a subquery in the second. This saves you from having to construct an in memory list of all of the College_id's before making the in_() query and means that you only make one round trip to the database.
subquery = (
db.session.query(StudentCollegeModel.College_id)
.filter(StudentCollegeModel.student_id== student_id)
)
result = (
db.session.query(CollegeModel)
.filter(CollegeModel.College_id.in_(subquery))
.all()
)

Your first query returns a list of named tuples. You need to convert it to a list. For example:
college_tuples = db.session.query(StudentCollegeModel.College_id).filter(StudentCollegeModel.student_id == student_id).all()
college_id_list = [r.College_id for r in college_tuples]
db.session.query(CollegeModel.filter(CollegeModel.College_id.in_(college_id_list)).all()
Here we fetch your college id tuples, flatten it and used the flattened version to preform your second query.

Similar as follows:
cs = StudentCollegeModel.query.filter_by(student_id=student_id).distinct('college_id').entities_with('college_id', flat=True).all()
CollegeModel.query.filter(CollegeModel.College_id.in_(cs)).all()

Identify what values in a list doesn't exist in a Table column using SQLAlchemy

I have a list cities = ['Rome', 'Barcelona', 'Budapest', 'Ljubljana']
Then,
I have a sqlalchemy model as follows -
class Fly(Base):
__tablename__ = 'fly'
pkid = Column('pkid', INTEGER(unsigned=True), primary_key=True, nullable=False)
city = Column('city', VARCHAR(45), unique=True, nullable=False)
country = Column('country', VARCHAR(45))
flight_no = Column('Flight', VARCHAR(45))
I need to check if ALL the values in given cities list exists in fly table or not using sqlalchemy. Return true only if ALL the cities exists in table. Even if a single city doesn't exist in table, I need to return false and list of cities that doesn't exist. How to do that? Any ideas/hints/suggestions? I'm using MYSQL

One way would be to create a (temporary) relation based on the given list and take the set difference between it and the cities from the fly table. In other words create a union of the values from the list1:
from sqlalchemy import union, select, literal
cities_union = union(*[select([literal(v)]) for v in cities])
Then take the difference:
sq = cities_union.select().except_(select([Fly.city]))
and check that no rows are left after the difference:
res = session.query(~exists(sq)).scalar()
For a list of cities lacking from fly table omit the (NOT) EXISTS:
res = session.execute(sq).fetchall()
1 Other database vendors may offer alternative methods for producing relations from arrays, such as Postgresql and its unnest().

petl convert data from duplicate entries

Am trying to use petl library to build an ETL process that copied data between two tables. The table contain a unique slug field on the destination. For that, I wrote my script so It would identify duplicate slugs and convert them with by appending ID to the slug value.
table = etl.fromdb(source_con, 'SELECT * FROM user')
# get whatever remains as duplicates
duplicates = etl.duplicates(table, 'slug')
for dup in [i for i in duplicates.values('id')]:
table = etl.convert(
table,
'slug',
lambda v, row: '{}-{}'.format(slugify_unicode(v), str(row.id).encode('hex')),
where=lambda row: row.id == dup,
pass_row=True
)
The above did not work as expected, it seems like the table object remains with duplicate values after the loop.
Anyone can advise?
Thanks

Django Postgresql JsonField query related dictionary keys

A part of the model that I have, which uses Django Model field, is like the following:
class SalesModel(models.Model):
some_data = models.PositiveIntegerField(db_index=True)
some_other_data = models.CharField(max_length=50)
json_data = JSONField(blank=True, null=True)
Now following is the format of the JsonData field:
[{"id": val, "contribution": "some_val", }, {"id": some_val, "contribution": "some_other_val",}, {"id": other_val, "contribution": "some_another_val"}]
i.e., the format is:
[{'id':XX, 'contribution':XX},{'id':YY, 'contribution':YY},{'id':ZZ, 'contribution':ZZ}]
Currently I can filter the Django table with the val of ID. I would now, like to know the contribution of that particular ID.
For eg, if val = 1, I would like to filter the model SalesModel which has JsonField with id = 1, and I want to show the related contribution. So, that would mean, out of the 3 possible dictionaries (as per the field construction), I would only show one dictionary (filtered by the 'ID' key of that dictionary). That would mean, if the 2nd dictionary has a matching ID, show only the 2nd contribution, if the 1st ID is matching show only the 1st contribution, and similarly for the 3rd dictionary.
Is there a way that can be done?

You could restructure your JSONField differently, by giving it a dict where the key, value pairs are id: contribution directly. This way you could use the has_key filter and KeyTransform will work, as I'm not sure it works on an array of dicts. So assuming your json_data looks like this:
{1: 'xx', 3: 'yy', 9: 'zz'}
you could query this way, based on #vanojx1 contribution:
SalesModel.filter(json_data__has_key=id)\
.annotate(contrib=KeyTransform(id, 'json_data')\
.values('contrib')
Alternatively, using raw jsonb in postgresql:
SalesModel.filter(json_data__has_key=id)\
.extra(select={'contrib': "json_data->{0}".format(id)})\
.values('contrib')

This should work DOC
SalesModel.objects.filter(json_data__id=1).values('id', 'json_data__contribution')

Yes, I guess. If I have understood it right, you will have an id to be matched or a list of ID's. So if your ID is 2:
my_id = 2
dict1 = [{"id":1, "contribution":10},{"id":2, "contribution":20},{"id":3, "contribution":30}]
for i in dict1:
if i["id"] == my_id:
print(i["contribution"])