We are using the python software Odoo which uses an ORM for the database requests (PostgreSQL). Currently I'm trying to reduce the execution times of some SQL statements. I could increase the time by adding some indexes.
Now I have one query which takes much time and can't decrease the time of execution. Can you help me to improve the speed of this query? I can't change the query itself (maybe only some small parts) but maybe we can change something on the table structure itself?
The query:
SELECT "purchase_order".id
FROM "mail_message" AS "purchase_order__message_ids",
"mail_notification" AS "purchase_order__message_ids__notification_ids",
"purchase_order"
WHERE ("purchase_order"."id"="purchase_order__message_ids"."res_id"
AND "purchase_order__message_ids"."id"="purchase_order__message_ids__notification_ids"."message_id")
AND ((("purchase_order"."state" NOT IN ('draft',
'sent',
'bid',
'confirmed'))
OR "purchase_order"."state" IS NULL)
AND (("purchase_order__message_ids"."model" = 'purchase.order')
AND (("purchase_order__message_ids__notification_ids"."partner_id" IN (3))
AND ("purchase_order__message_ids__notification_ids"."is_read" IS NULL
OR "purchase_order__message_ids__notification_ids"."is_read" = FALSE))))
ORDER BY "purchase_order"."id" DESC LIMIT 100;
Here is the output of EXPLAIN ANALYZE:
# EXPLAIN ANALYZE SELECT "purchase_order".id FROM "mail_message" as "purchase_order__message_ids","mail_notification" as "purchase_order__message_ids__notification_ids","purchase_order" WHERE ("purchase_order"."id"="purchase_order__message_ids"."res_id" AND "purchase_order__message_ids"."id"="purchase_order__message_ids__notification_ids"."message_id") AND ((("purchase_order"."state" not in ('draft','sent','bid','confirmed')) OR "purchase_order"."state" IS NULL) AND (("purchase_order__message_ids"."model" = 'purchase.order') AND (("purchase_order__message_ids__notification_ids"."partner_id" in (3)) AND ("purchase_order__message_ids__notification_ids"."is_read" IS NULL or "purchase_order__message_ids__notification_ids"."is_read" = false )))) ORDER BY "purchase_order"."id" DESC limit 100;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=0.99..10720.11 rows=100 width=4) (actual time=956.615..958.683 rows=2 loops=1)
-> Nested Loop (cost=0.99..63779.73 rows=595 width=4) (actual time=956.609..958.669 rows=2 loops=1)
-> Nested Loop (cost=0.71..63177.50 rows=1325 width=4) (actual time=956.573..958.609 rows=2 loops=1)
-> Index Scan Backward using mail_message_res_id_index on mail_message purchase_order__message_ids (cost=0.42..48385.44 rows=40025 width=8) (actual time=110.607..518.171 rows=40149 loops=1)
Filter: ((model)::text = 'purchase.order'::text)
Rows Removed by Filter: 254269
-> Index Scan using mail_notification_message_id_index on mail_notification purchase_order__message_ids__notification_ids (cost=0.29..0.36 rows=1 width=4) (actual time=0.006..0.006 rows=0 loops=40149)
Index Cond: (message_id = purchase_order__message_ids.id)
Filter: (((is_read IS NULL) OR (NOT is_read)) AND (partner_id = 3))
Rows Removed by Filter: 0
-> Index Scan using purchase_order_pkey on purchase_order (cost=0.28..0.44 rows=1 width=4) (actual time=0.017..0.019 rows=1 loops=2)
Index Cond: (id = purchase_order__message_ids.res_id)
Filter: (((state)::text <> ALL ('{draft,sent,bid,confirmed}'::text[])) OR (state IS NULL))
Planning time: 2.468 ms
Execution time: 958.792 ms
(15 rows)
You can watch the explanation on http://explain.depesz.com/s/cbG
Just for the note - the number of rows:
table | num_rows
--------------------------+----------
public.mail_notification | 42254
public.mail_message | 294474
public.purchase_order | 6566
(3 rows)
And here is the output of \d mail_message:
# \d mail_message
Table "public.mail_message"
Column | Type | Modifiers
----------------+-----------------------------+-----------------------------------------------------------
id | integer | not null default nextval('mail_message_id_seq'::regclass)
create_uid | integer |
create_date | timestamp without time zone |
write_date | timestamp without time zone |
write_uid | integer |
body | text |
model | character varying(128) |
record_name | character varying |
date | timestamp without time zone |
subject | character varying |
message_id | character varying |
parent_id | integer |
res_id | integer |
subtype_id | integer |
author_id | integer |
type | character varying |
email_from | character varying |
mail_server_id | integer |
no_auto_thread | boolean |
reply_to | character varying |
Indexes:
"mail_message_pkey" PRIMARY KEY, btree (id)
"mail_message_author_id_index" btree (author_id)
"mail_message_message_id_index" btree (message_id)
"mail_message_model_index" btree (model)
"mail_message_model_res_id_idx" btree (model, res_id)
"mail_message_parent_id_index" btree (parent_id)
"mail_message_res_id_index" btree (res_id)
"mail_message_subtype_id_index" btree (subtype_id)
Foreign-key constraints:
"mail_message_author_id_fkey" FOREIGN KEY (author_id) REFERENCES res_partner(id) ON DELETE SET NULL
"mail_message_create_uid_fkey" FOREIGN KEY (create_uid) REFERENCES res_users(id) ON DELETE SET NULL
"mail_message_mail_server_id_fkey" FOREIGN KEY (mail_server_id) REFERENCES ir_mail_server(id) ON DELETE SET NULL
"mail_message_parent_id_fkey" FOREIGN KEY (parent_id) REFERENCES mail_message(id) ON DELETE SET NULL
"mail_message_subtype_id_fkey" FOREIGN KEY (subtype_id) REFERENCES mail_message_subtype(id) ON DELETE SET NULL
"mail_message_write_uid_fkey" FOREIGN KEY (write_uid) REFERENCES res_users(id) ON DELETE SET NULL
Referenced by:
TABLE "crm_lead_forward_to_partner" CONSTRAINT "crm_lead_forward_to_partner_parent_id_fkey" FOREIGN KEY (parent_id) REFERENCES mail_message(id) ON DELETE SET NULL
TABLE "mail_compose_message" CONSTRAINT "mail_compose_message_parent_id_fkey" FOREIGN KEY (parent_id) REFERENCES mail_message(id) ON DELETE SET NULL
TABLE "mail_mail" CONSTRAINT "mail_mail_mail_message_id_fkey" FOREIGN KEY (mail_message_id) REFERENCES mail_message(id) ON DELETE CASCADE
TABLE "mail_message" CONSTRAINT "mail_message_parent_id_fkey" FOREIGN KEY (parent_id) REFERENCES mail_message(id) ON DELETE SET NULL
TABLE "mail_message_res_partner_rel" CONSTRAINT "mail_message_res_partner_rel_mail_message_id_fkey" FOREIGN KEY (mail_message_id) REFERENCES mail_message(id) ON DELETE CASCADE
TABLE "mail_notification_bcc" CONSTRAINT "mail_notification_bcc_message_id_fkey" FOREIGN KEY (message_id) REFERENCES mail_message(id) ON DELETE CASCADE
TABLE "mail_notification_cc" CONSTRAINT "mail_notification_cc_message_id_fkey" FOREIGN KEY (message_id) REFERENCES mail_message(id) ON DELETE CASCADE
TABLE "mail_notification" CONSTRAINT "mail_notification_message_id_fkey" FOREIGN KEY (message_id) REFERENCES mail_message(id) ON DELETE CASCADE
TABLE "mail_vote" CONSTRAINT "mail_vote_message_id_fkey" FOREIGN KEY (message_id) REFERENCES mail_message(id) ON DELETE CASCADE
TABLE "message_attachment_rel" CONSTRAINT "message_attachment_rel_message_id_fkey" FOREIGN KEY (message_id) REFERENCES mail_message(id) ON DELETE CASCADE
You can create a selective index for the specific query, something like
CREATE INDEX idx_order_messages on mail_message (id) where model = 'purchase.order'
A smaller index results in fewer reads etc, so it must be faster.
Related
Our IDs look something like this "CS0000001" which stands for Customer with the ID 1. Is this possible to to with SQL and Auto Increment or do i need to to that in my GUI ?
I need the leading zeroes but with auto incrementing to prevent double usage if am constructing the ID in Python and Insert them into the DB.
Is that possible?
You have few choices:
Construct the CustomerID in your code which inserts the data into
the Customer table (=application side, requires change in your code)
Create a view on top of the Customer-table that contains the logic
and use that when you need the CustomerID (=database side, requires change in your code)
Use a procedure to do the inserts and construct the CustomerID in
the procedure (=database side, requires change in your code)
Possible realization.
Create data table
CREATE TABLE data (id CHAR(9) NOT NULL DEFAULT '',
val TEXT,
PRIMARY KEY (id));
Create service table
CREATE TABLE ids (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY);
Create trigger which generates id value
CREATE TRIGGER tr_bi_data
BEFORE INSERT
ON data
FOR EACH ROW
BEGIN
INSERT INTO ids () VALUES ();
SET NEW.id = CONCAT('CS', LPAD(LAST_INSERT_ID(), 7, '0'));
DELETE FROM ids;
END
Create trigger which prohibits id value change
CREATE TRIGGER tr_bu_data
BEFORE UPDATE
ON data
FOR EACH ROW
BEGIN
SET NEW.id = OLD.id;
END
Insert some data, check result
INSERT INTO data (val) VALUES ('data-1'), ('data-2');
SELECT * FROM data;
id | val
:-------- | :-----
CS0000001 | data-1
CS0000002 | data-2
Try to update, ensure id change prohibited
UPDATE data SET id = 'CS0000100' WHERE val = 'data-1';
SELECT * FROM data;
id | val
:-------- | :-----
CS0000001 | data-1
CS0000002 | data-2
Insert one more data, ensure enumeration continues
INSERT INTO data (val) VALUES ('data-3'), ('data-4');
SELECT * FROM data;
id | val
:-------- | :-----
CS0000001 | data-1
CS0000002 | data-2
CS0000003 | data-3
CS0000004 | data-4
Check service table is successfully cleared
SELECT COUNT(*) FROM ids;
| COUNT(*) |
| -------: |
| 0 |
db<>fiddle here
Disadvantages:
Additional table needed.
Generated id value edition is disabled (copy and delete old record must be used instead, custom value cannot be set).
So I have a table, which has the first column called id as autoincrement.
Now, Suppose I have data in the table with ids- 1,2,3
And I also have some data in the csv that starts with id 1,2,3
This is the code that I am trying to use-
cur.execute("CREATE TABLE IF NOT EXISTS sub_features (id INTEGER PRIMARY KEY AUTOINCREMENT,featureId INTEGER, name TEXT, FOREIGN KEY(featureId) REFERENCES features(id))")
df = pd.read_csv(csv_location+'/sub_features_table.csv')
df.to_sql("sub_features", con, if_exists='append', index=False)
I am getting this error-
sqlite3.IntegrityError: UNIQUE constraint failed: sub_features.id
How do I make sure that data gets appended and the id gets set as per requirement and in case the entire row is a duplicate then it gets ignored?
To explain further, Say I have a table:
id | Name
1 | Abhisek
2 | Amit
And I am trying to import this csv to the same table:
id | Name
1 | Abhisek
2 | Rahul
Then my resultant table should be:
id | Name
1 | Abhisek
2 | Amit
3 | Rahul
This question already has answers here:
MySQL distinction between e and é (e acute) - UNIQUE index
(4 answers)
Closed 5 years ago.
I have a table Labels with 2 columns:
+-------------+--------------+-------------+
| Column Name | Type | Key |
+-------------+--------------+-------------+
| id | integer | primary key |
| label | varchar(255) | unique |
+-------------+--------------+-------------+
In this table, I already have a record as the following:
id: 1, label: 'café'
And now I want to add more record as the following:
id: auto, label: 'cafe'
But when I try to insert, duplicate error appear
(1062, "Duplicate entry 'cafe' for key 'label_2'") [SQL: u'INSERT INTO vocabulary (label) VALUES (%s)'] [parameters: (u'cafe',)]
Could you guys help me in that case?
Some more information about my database: character set: utf8, collate: utf8mb4_unicode_ci
UPDATE: create table
CREATE TABLE `labels` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`label` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `label_2` (`label`),
KEY `label` (`label`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
As far as label is unique key,you are not able to insert duplicate value in that column.
As you want to distinguish between café and cafe and then you need to use utf8_bin collation .
Try below query.
ALTER TABLE labels CHARACTER SET utf8 COLLATE utf8_bin;
ALTER TABLE labels CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin;
Hope this will helps.
I have the SQLite database containing four arrays:
CREATE TABLE IF NOT EXISTS EVENTS_LIST
(ID INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
DATE TEXT,
WORKER INTEGER,
EVENT INTEGER,
REGISTRATION_METHOD INTEGER)
CREATE TABLE IF NOT EXISTS WORKERS
(ID INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
WORKER TEXT)
CREATE TABLE IF NOT EXISTS EVENTS
(ID INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
EVENT TEXT)
CREATE TABLE IF NOT EXISTS REGISTRATION
(ID INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
REGISTRATION_METHOD TEXT)
I need to get the following ID's: Worker, Event and Registration replaced with the name attached to this ID in the appropriate table.
Example:
3 Workers: 0001) John 0002) Tom 0003) Mike
3 Types of registration: 0) PIN 1) Fingerprint 2) NFC Card
3 Types of events: 1) came in 2) came out 3) came out on business
I get:
DATE | WORKER | EVENT | REGISTRATION
XXXX | 0001 | 1 | 0
XXXX | 0003 | 2 | 1
I need:
DATE | WORKER | EVENT | REGISTRATION
XXXX | John | came in | PIN
XXXX | Mike | came out | Fingerprint
I found those solutions:
How to replace fetched values with another column while querying with SQL Server 2008 R2
Multiple column SQL joins in a table
The first link is very similar but related only to one column and the second link is more complicated but has a few "LEFT OUTER JOIN" commands which seems to be the good direction.
Can anybody give me directions on how to accomplish this?
Use this code:
select DATE=el.DATE,
WORKER=w.wORKER,
EVENT=e.EVENT,
REGISTRATION=r.REGISTRATION
from
IF NOT EXISTS EVENTS_LIST as el
join
IF NOT EXISTS WORKERS as w on el.WORKER=w.ID
join
IF NOT EXISTS EVENTS as e on el.EVENT=e.ID
join
IF NOT EXISTS REGISTRATION as r on el.REGISTRATION_METHOD=r.ID
This code is working for me:
select el.DATE,
w.wORKER,
e.EVENT,
r.REGISTRATION
from
EVENTS_LIST as el
join
WORKERS as w on el.WORKER=w.ID
join
EVENTS as e on el.EVENT=e.ID
join
REGISTRATION as r on el.REGISTRATION_METHOD=r.ID
Thanks!
This question already has answers here:
Django unique together constraint failure?
(4 answers)
Closed 9 years ago.
I am using django 1.5.5 and python 2.7 and MySQL
this is my model
class Foo(models.Model):
user = models.ForeignKey(User)
date = models.DateTimeField(null=True, blank=True,editable=True)
class Meta:
unique_together = ('user', 'date')
If i use this i can add 2 records with the same user and an empty date.
I would like the uniqueness to be enforced even for empty date.
table create command
CREATE TABLE `management_foo`
( `id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`date` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `user_id` (`user_id`,`date`),
KEY `management_foo_6232c63c` (`user_id`),
CONSTRAINT `user_id_refs_id_d47e5747` FOREIGN KEY (`user_id`)
REFERENCES `auth_user` (`id`))
ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1
and table describe
+-------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| user_id | int(11) | NO | MUL | NULL | |
| date | datetime | YES | | NULL | |
+-------------+--------------+------+-----+---------+----------------+
In InnoDB each NULL treats as unique value.
example:
mysql> create table x (i int, j int, primary key (i), unique key (j));
mysql> insert into x (i,j) values (NULL,NULL);
ERROR 1048 (23000): Column 'i' cannot be null
mysql> insert into x (i,j) values (1,NULL);
mysql> insert into x (i,j) values (2,NULL);
mysql> insert into x (i,j) values (3,3);
mysql> select * from x;
+---+------+
| i | j |
+---+------+
| 1 | NULL |
| 2 | NULL |
| 3 | 3 |
+---+------+
3 rows in set (0.01 sec)
mysql> insert into x (i,j) values (4,3);
ERROR 1062 (23000): Duplicate entry '3' for key 'j'
You need to add not null constraint <=> remove null=True from field definition (replace it with default for example)
BTW, it's better to write in such style: unique_together = (("field1", "field2"),) – it will be much easier to extend unique pairs
Just for the record: the SQL standard states that NULL is not a value and so must not be taken in account for unique constraints. IOW it has nothing to do with Django and actually is the expected behavior.
For a technical solution see akaRem's answer.