We Keep Coding

Python Django always passes id 1 on ForeignKey with to_field

I'm new to Django and trying to create a small application that shows scanned data from virtual machines that are inserted in a table named HostsFixDataScans.
To access the scanned data in HostsFixDataScans via the Hosts model, I defined a ForeignKey with to_field.
But unfortunately, the data returned by the linked HostsFixDataScans are wrong.
I checked the SQL statements and when requesting the HostsFixDataScans table, not the id of the Host is used, but always 1.
My domain = models.ForeignKey(Domains, on_delete=models.CASCADE) definition which does not use to_field works correctly.
I'm pretty sure, I have a misunderstanding of the definition of this relationship.
Maybe you could give me some hints how to solve the problem?
Many thanks in advance!
Here are the shortened definitions of the models:
class Os(models.Model):
operatingsystem = models.CharField(max_length=32)
lsbdistcodename = models.CharField(max_length=32)
lsbdistrelease = models.CharField(max_length=32, db_collation='ascii_bin')
lsbmajdistrelease = models.IntegerField()
remarks = models.CharField(max_length=256, blank=True, null=True)
class HostsFixDataScans(models.Model):
host_id = models.PositiveIntegerField(primary_key=True, unique=True)
scan_id = models.PositiveIntegerField()
os = models.ForeignKey(Os, on_delete=models.CASCADE)
class Hosts(models.Model):
hostname = models.CharField(max_length=256)
domain = models.ForeignKey(Domains, on_delete=models.CASCADE)
is_virtual = models.PositiveIntegerField(blank=True, null=True)
os = models.ForeignKey(HostsFixDataScans, to_field='host_id', on_delete=models.CASCADE)

Sorry folks,
I made a really stupid mistake by connecting the field os_id with the field host_id on the other table.
I now added a field last_fix_data_scan_id to the Hosts table which has to be filled with every scan of a machine now.
But doing it this way I can now request the Hosts by connecting this new field with the scan_id of HostsFixDataScans.

How to add more columns to pivot table in many to many relations in Django?

I have 2 models united by a Many to Many relationship, in this case is Policies and Coverages so far it works fine, but I want to add another column to the pivot table since it wont work on any of the modeled tables (I want to add a IntegerField with the name 'amount' so I can store how much money are we covering for that specific coverage in that specific Insurance Police)
class Policy(models.Model):
"""Insurance Policies Model"""
number = models.CharField(max_length=25, unique=True)
company = models.OneToOneField(Company, on_delete=models.CASCADE)
client = models.OneToOneField(Client, on_delete=models.CASCADE)
start_date = models.DateField()
end_date = models.DateField()
comission = models.PositiveIntegerField()
salesperson = models.ForeignKey(Salesperson, on_delete=models.CASCADE)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
rif = models.CharField(max_length=50)
phone = models.CharField(max_length=100)
class Meta:
db_table = 'policies'
class Coverage(models.Model):
"""Coverage of the different types of policies"""
name = models.CharField(max_length=55)
policies = models.ManyToManyField(Policy)
class Meta:
db_table = 'coverages'
I made the migrations and the pivot table was made without problems, but I don't know how to add another field to the pivot table

You can specify a through=... parameter [Django-doc] for that. In fact if you do not specify this yourself, Django will automatically make an "implicit model". For example:
class Policy(models.Model):
# ...
class Coverage(models.Model):
# ...
policies = models.ManyToManyField(Policy, through='CoveragePolicy')
class CoveragePolicy(models.Model):
policy = models.ForeignKey(Policy, on_delete=models.CASCADE)
coverage = models.ForeignKey(Coverage, on_delete=models.CASCADE)
amount = models.IntegerField()
So now you can create a relation object between a policy p1 and a coverage c2 with:
CoveragePolicy.objects.create(policy=p1, coverage=c2, amount=1)
You can obtain the related set of CoveragePolicy objects for a policy p1 with:
p1.coveragepolicy_set.all()
and then inspect the amount and the related coverage it has.
Django however will have troubles migrating from an existing through model to another one, so you probably will have to remove the migration file, and create a fresh one with the new model instead (and undo the changes in the database of the old one).

What is the optimal way to store 100k daily database inserts in Django?

My problem is that Django inserts entries waaaaaaay too slow ( i didnt even time but it was more than 5 mins) for 100k entries from Pandas csv file. What i am doing is parsing csv file and then save those objects to postgresql in Django. It is going to be a daily cronjob with csv files differ for most of the entries(some can be duplicates from the previous days or owner could be the same)
I haven't tried raw queries, but i dont think that would help much.
and i am really stuck at this point honestly. apart from some iteration manipulations and making a generator, rather than iterator i can not somehow improve the time of insertions.
class TrendType(models.Model):
""" Описывает тип отчета (посты, видео, субъекты)"""
TREND_TYPE = Choices('video', 'posts', 'owners') ## << mnemonic
title = models.CharField(max_length=50)
mnemonic = models.CharField(choices=TREND_TYPE, max_length=30)
class TrendSource(models.Model):
""" Источник отчета (файла) """
trend_type = models.ForeignKey(TrendType, on_delete=models.CASCADE)
load_date = models.DateTimeField()
filename = models.CharField(max_length=100)
class TrendOwner(models.Model):
""" Владелец данных (группа, юзер, и т.п.)"""
TREND_OWNERS = Choices('group', 'user', '_')
title = models.CharField(max_length=50)
mnemonic = models.CharField(choices=TREND_OWNERS, max_length=30)
class Owner(models.Model):
""" Данные о владельце """
link = models.CharField(max_length=255)
name = models.CharField(max_length=255)
trend_type = models.ForeignKey(TrendType, on_delete=models.CASCADE)
trend_owner = models.ForeignKey(TrendOwner, on_delete=models.CASCADE)
class TrendData(models.Model):
""" Модель упаковка всех данных """
owner = models.ForeignKey(Owner, on_delete=models.CASCADE)
views = models.IntegerField()
views_u = models.IntegerField()
likes = models.IntegerField()
shares = models.IntegerField()
interaction_rate = models.FloatField()
mean_age = models.IntegerField()
source = models.ForeignKey(TrendSource, on_delete=models.CASCADE)
date_trend = models.DateTimeField() # << take it as a date
Basically, i would love a good solution for a 'fast' insertion to a database and is it even possible given these models.

Maybe you don't need an ORM here? You can try to implement a simple wrapper around typical SQL requests
Use bulk read/writing, using bulk_create() in Django ORM, or in your wrapper
Check https://docs.djangoproject.com/en/2.2/topics/db/optimization/

The problem is not with django but rather with postgresql itself. My suggestion would be to change your backend. Postgresql is good for UPDATE data, but there are better DBs for INSERT data. Postgresql vs TimescaleDB However, I dont think there is django ORM for TimescaleDB.
My suggestion would be to use Redis. The primary use is cache in memory but you can make it to persist your data too. And there is also ORM for python with redis called ROM

How do you write a Django query where the condition if a COUNT of related items?

I'm using Django and Python 3.7. I have these two models, related to one another through a foreign key ...
class Website(models.Model):
objects = WebsiteManager()
path = models.CharField(max_length=100)
class Article(models.Model):
website = models.ForeignKey(Website, on_delete=models.CASCADE, related_name='articlesite')
title = models.TextField(default='', null=False)
url = models.TextField(default='', null=False)
created_on = models.DateTimeField(db_index=True, default=datetime.now)
I want to write a Django query that returns the web sites who have more than 100 articles tied to them. In PostGres, I can write this query
select w.id, count(*) FROM website w, article a where w.id = a.website_id group by w.id;
but I'm unclear how to do this with a Django query. How do I write a query where the condition if a COUNT function?
Edit:
I modified my query to add a condition ...
qset = Website.objects.annotate(articlesite_count=Count('articlesite')).filter(
articlesite__edited_date__null=True,
articlesite_count__gte=100)
but now this results in the error
Unsupported lookup 'null' for DateTimeField or join on the field not permitted.

Use annotate then filter
from django.db.models import Count
Website.objects.annotate(articlesite_count=Count('articlesite')).filter(articlesite_count__gte=100)
UPDATE-1
from django.db.models import Count
Website.objects.filter(articlesite__edited_date__isnull=True).annotate(
articlesite_count=Count('articlesite')).filter(
articlesite_count__gte=100)

How to use a queryset filter with M2M models in django admin-export?

I need to export some data in Django Admin, I'm using the django-import-export lib and the export is fine, the only problem is that it gets the ID of the attribute instead of his name(of course)
Given the models, on the Operation export, I want the book atribute to get all the book's title instead of the ID's tied with the operation
class Book(models.Model):
title = models.CharField()
value = models.FloatField()
class Operation(models.Model):
book = models.ManyToManyField(Book, through='BookOperation')
sender = models.CharField()
date = models.DateField()
class BookOperation(models.Model):
book = models.ForeignKey(Book)
operation = models.ForeignKey(Operation)
quantity = models.IntegerField()
Here is the OperationResource in admin.py
class OperationResource(resources.ModelResource):
book = fields.Field(column_name='Book(s)', widget=ManyToManyWidget(Book))
class Meta(object):
model = Operation
exclude = ('id',)
def dehydrate_book(self, operation):
return operation.book.get()
With the code like this, i'm getting
get() returned more than one Book -- it returned 2!
I tried using Operation.book.filter but doesn't see to work, i must be doing something wrong !