I have am using SQLAlchemy and inside one transaction, I want to do the following:
Delete all records that meet a certain criteria (say this is Cars.color == red).
Now, I want to insert all cars that meet a certain criteria (say Cars.type == Honda).
Now lets say that my database is just a table with 3 columns (name, color, andtype) with name as the primary key.
If my database already has cars that are red and of type Honda with name as Bob. I can't just say
Cars.query.filter(Cars.name == red).delete()
// add all Hondas
db.session.commit()
as the // add all Hondas will fail because I could potentially be adding a car with name as Bob and color red. How can I do a deletion and have deletions follow as part of one action?
Reference: I am using MySQL.
You could try this:
db.session.query(Cars).filter(Cars.name == red).delete()
// add all Hondas
for h in Hondas:
db.session.add(h)
db.session.commit()
Caveat lector - I do not think your current code does set up the transaction properly. Apart from that I don't believe that the problem you describe exists - session.commit() flushes the changes in sequences, so no matter whether session.flush() is called, you should be able to insert the Hondas at the point you marked - the red car will be deleted before the insert hits the DB.
Related
I need to write an SQL query for MS-Access 2000 so that a row is updated if it exists, but inserted if it does not. (I believe this is called an "upsert")
i.e.
If row exists...
UPDATE Table1 SET (...) WHERE Column1='SomeValue'
If it does not exist...
INSERT INTO Table1 VALUES (...)
Can this be done in one query?
You can simulate an upsert in an Access by using an UPDATE query with a LEFT JOIN.
update b
left join a on b.id=a.id
set a.f1=b.f1
, a.f2=b.f2
, a.f3=b.f3
Assuming a unique index on Column1, you can use a DCount expression to determine whether you have zero or one row with Column1 = 'SomeValue'. Then INSERT or UPDATE based on that count.
If DCount("*", "Table1", "Column1 = 'SomeValue'") = 0 Then
Debug.Print "do INSERT"
Else
Debug.Print "do UPDATE"
End If
I prefer this approach to first attempting an INSERT, trapping the 3022 key violation error, and doing an UPDATE in response to the error. However I can't claim huge benefits from my approach. If your table includes an autonumber field, avoiding a failed INSERT would stop you from expending the next autonumber value needlessly. I can also avoid building an INSERT string when it's not needed. The Access Cookbook told me string concatenation is a moderately expensive operation in VBA, so I look for opportunities to avoid building strings unless they're actually needed. This approach will also avoid creating a lock for an unneeded INSERT.
However, none of those reasons may be very compelling for you. And in all honesty I think my preference in this case may be about what "feels right" to me. I agree with this comment by #David-W-Fenton to a previous Stack Overflow question: "It's better to write your SQL so you don't attempt to append values that already exist -- i.e., prevent the error from happening in the first place rather than depending on the database engine to save you from yourself."
An "upsert" is possible, if the tables have a unique key.
This old tip from Smart Access is one of my favourites:
Update and Append Records with One Query
By Alan Biggs
Did you know that you can use an update query in Access to both update
and add records at the same time? This is useful if you have two
versions of a table, tblOld and tblNew, and you want to integrate the
changes from tblNew into tblOld.
Follow these steps:
Create an update query and add the two tables. Join the two tables by
dragging the key field of tblNew onto the matching field of tblOld.
Double-click on the relationship and choose the join option that includes all records from tblNew and only those that match from
tblOld.
Select all the fields from tblOld and drag them onto the QBE grid.
For each field, in the Update To cell type in tblNew.FieldName, where FieldName matches the field name of tblOld.
Select Query Properties from the View menu and change Unique Records to False. (This switches off the DISTINCTROW option in the SQL
view. If you leave this on you'll get only one blank record in your
results, but you want one blank record for each new record to be added
to tblOld.)
Run the query and you'll see the changes to tblNew are now in tblOld.
This will only add records to tblOld that have been added to tblNew.
Records in tblOld that aren't present in tblNew will still remain in
tblOld.
I usually run the insert statement first and then I check to see if error 3022 occurred, which indicates the row already exists. So something like this:
On Error Resume Next
CurrentDb.Execute "INSERT INTO Table1 (Fields) VALUES (Data)", dbFailOnError
If Err.Number = 3022 Then
Err.Clear
CurrentDb.Execute "UPDATE Table1 SET (Fields = Values) WHERE Column1 = 'SomeValue'", dbFailOnError
ElseIf Err.Number <> 0 Then
'Handle the error here
Err.Clear
End If
Edit1:
I want to mention that what I've posted here is a very common solution but you should be aware that planning on errors and using them as part of the normal flow of your program is generally considered a bad idea, especially if there are other ways of achieving the same results. Thanks to RolandTumble for pointing this out.
You don't need to catch the error. Instead, just run the INSERT statement and then check
CurrentDb.RecordsAffected
It will either be 1 or 0, depending.
Note: It's not good practice to execute against CurrentDB. Better to capture the database to a local variable:
Dim db As DAO.Database
Set db = CurrentDb
db.Execute(INSERT...)
If db.RecordsAffected = 0 Then
db.Execute(UPDATE...)
End If
As others have mentioned, You can UPSERT with an UPDATE LEFT JOIN using the new table as the left hand side. This will add all missing records and update matching records, leaving deleted records intact.
If we follow the Create and run an update query Article we will end up with SQL that looks like this:
UPDATE Table1
INNER JOIN NewTable1 ON Table1.ID = NewTable1.ID
SET Table1.FirstName = [NewTable1].[FirstName]
but an inner join will only update matching records, it won't add new records. So let's change that INNER to a LEFT:
UPDATE Table1
LEFT JOIN NewTable1 ON Table1.ID = NewTable1.ID
SET Table1.FirstName = [NewTable1].[FirstName]
Now save a copy of the DB. Run a test on the copy before you run this on your primary DB.
I have a table containing jobs like this
id owner collaborator privilege
90 "919886297050" "919886212378" "read"
90 "919886297050" "919886297052" "read"
88 "919886297050" "919886212378" "read"
88 "919886297050" "919886297052" "read"
primary key is a composite of id, owner and collaborator
I want to pass in details of only the collaborators i want to retain. For example, if my collaborator = "919886212378" it means I want to delete the row for "919886297052" and keep the row for "919886212378"
Is there a way to do this in one query / execution instead of fetching the details separately and then performing the delete after filtering the missing values?
EDIT: My use case might have new collaborators added and old ones deleted. However, my input will just have a set of chosen collaborators so I will need to cross check with the old list, retain existing, add new and delete missing collaborators.
DELETE FROM table WHERE collaborator NOT IN ("919886212378", "id ..")
does the delete for the specific case you mentioned. But I don't know
how you get these id's. You give too little information regarding your exact case.
If you can get these id's by a query, you could make it a subquery like:
DELETE FROM table WHERE collaborator NOT IN (SELECT ... FROM ...)
Comparing the old and new collaborator lists in python kind of did the trick for me
original_list = ["C1","C2","C3","C4"] // result from query
updated_list= ["C1","C6","C7"] // list obtained from request
# compute the differences
to_be_added = set(updated_list).difference(set(original_list) )
to_be_deleted = set(original_list).difference(set(updated_list) )
Then I use an insert and delete statement within a transaction using the above two lists to make an update.
cursor.execute('UPDATE emp SET name = %(name)s',{"name": name} where ?)
I don't understand how to get primary key of a particular record.
I have some N number of records present in DB. I want to access those record &
manipulate.
Through SELECT query i got all records but i want to update all those records accordingly
Can someone lend a helping hand?
Thanks in Advance!
Table structure:
ID CustomerName ContactName
1 Alfreds Futterkiste
2 Ana Trujillo
Here ID is auto genearted by system in postgres.
I am accessing CustomerName of two record & updating. So here when i am updating
those record the last updated is overwrtited in first record also.
Here i want to set some condition so that When executing update query according to my record.
After Table structure:
ID CustomerName ContactName
1 xyz Futterkiste
2 xyz Trujillo
Here I want to set first record as 'abc' 2nd record as 'xyz'
Note: It ll done using PK. But i dont know how to get that PK
You mean you want to use UPDATE SQL command with WHERE statement:
cursor.execute("UPDATE emp SET CustomerName='abc' WHERE ID=1")
cursor.execute("UPDATE emp SET CustomerName='xyz' WHERE ID=2")
This way you will UPDATE rows with specific IDs.
Maybe you won't like this, but you should not use autogenerated keys in general. The only exception is when you want to insert some rows and do not do anything else with them. The proper solution is this:
Create a sequencefor your table. http://www.postgresql.org/docs/9.4/static/sql-createsequence.html
Whenever you need to insert a new row, get the next value from the generator (using select nextval('generator_name')). This way you will know the ID before you create the row.
Then insert your row by specifying the id value explicitly.
For the updates:
You can create unique constraints (or unique indexes) on sets of coulmns that are known to be unique
But you should identify the rows with the identifiers internally.
When referring records in other tables, use the identifiers, and create foreign key constraints. (Not always, but usually this is good practice.)
Now, when you need to updatea row (for example: a customer) then you should already know which customer needs to be modified. Because all records are identified by the primary key id, you should already know the id for that row. If you don't know it, but you have an unique index on a set of fields, then you can try to get the id. For example:
select id from emp where CustomerName='abc' -- but only if you have a unique constraing on CustomerName!
In general, if you want to update a single row, then you should NEVER update this way:
update emp set CustomerName='newname' where CustomerName='abc'
even if you have an unique constraint on CustomerName. The explanation is not easy, and won't fit here. But think about this: you may be sending changes in a transaction block, and there can be many opened transactions at the same time...
Of course, it is fine to update rows, if you intention is to update all rows that satisfy your condition.
I will describe my situation first in order to make the following question as clear as possible.
For simplicity, let say I have a table in MySQL database (InnoDB) with records about dogs with structure as follows:
dog_id (PK) | dog_name
And there is 10,000,000 rows in the table (each represents a unique dog) and index build on the dog_name column.
My program searches through vets records that I need to process. Each record is somehow connected with a dog and there is like 100 records for each dog. And I want to find dogs which have not been inserted to the database yet.
That means that 100 times in a row the record that is being processed can be about a dog which is already in the database and therefore the dog doesn't have to be added to the database. But sometimes it happens (as mentioned before 1:100 ratio) that I need to add a dog to the database because it is the first time the program approached a record about the dog. (I hope this example makes my situation clear)
My question is:
What is the most effective way how to verify that the dog has not beed inserted into the database yet?
Load all the dog names (suppose all the dogs in the world have unique names) to the memory of the program (a set) and check if the dog is in the set or not. When it is in the set I skip the record, when it is not I insert the dog.
Define the column as UNIQUE and try to insert all the records. When there is a database error because of the uniqueness, I just skip the dog and continue.
Query the database to find out if the dog is in the database every time I process a record and if it is in the database I skip the record and if it is not I insert the dog into the table.
To give you as much information as I can. I use Python, SqlAlchemy, MySQL, InnoDB.
You should use dog_name as the primary key, and then use
INSERT INTO dogs (dog_name) VALUES ('[NAME HERE]') ON DUPLICATE KEY UPDATE dog_name='[NAME HERE]';
This will only insert unique dog names. If you still want to use a numerical ID for each dog, you can set that column to auto increment, but the primary key should be the dog names (assuming all are unique).
SQLAlchemy does not have this functionality built in, but can make force it to make a similar query with session.merge().
Something like option 2 or option 3 will work best; they should take similar amounts of time, and which one wins will depend on exactly how MySQL/InnoDB decides that a collision has occurred. I don't actually know; it's possible that insert with a UNIQUE key triggers the same operation as a SELECT. Prototype both and profile performance.
If performance is an issue, you can always hand-code the SELECT statement since it's relatively simple. This cuts out the Python MySQL overhead to construct the SQL; that's normally not a huge issue, but SQLAlchemy can add dozens of layers of function calls that support its ability to construct arbitrary queries. You can short-circuit those calls using Python string formatting.
Assuming that 's' is your SQLAlchemy Session object:
def dog_in_db(dog_name):
q = 'SELECT COUNT (*) FROM dogs WHERE dog_name = %s;' % dog_name
res = s.execute(q)
return res.first()[0] > 0
You could also try a SELECTing and check whether any rows are returned:
q 'SELECT dog_id FROM dogs WHERE dog_name = %s;' % dog_name
res = s.execute(q)
return res.rowcount() > 0
Assuming that your option 1 means loading all of the names from the database, it will be slow. MySQL will always perform any single operation it supports faster than Python can; and what you're doing here is exactly the same single operation (finding a member in a list).
First I am not even sure whether I am asking the right question, sorry for that. SQL is new to me. I have a table I create in SQLITE like this:
CREATE TABLE ENTRIES "(ID INTEGER PRIMARY KEY AUTOINCREMENT,DATA BLOB NOT NULL)"
Which is all fine if I have only additions for entries. If I create entries, they increment. Let us say I added 7 entries. Now I delete 3 entries:
DELETE FROM NODES WHERE ID = 3
DELETE FROM NODES WHERE ID = 4
DELETE FROM NODES WHERE ID = 5
Entries I now have are:
1,2,6,7.
The next time I add an entry it will have ID=8.
So, my question is:
How do I get the next 3 entries, to get the IDs 3, 4, 5 and only the 4 entry will then get 8? I realize this is similar to SQL: find missing IDs in a table, and it is maybe also a general programming (not just SQL) problem. So, I would be happy to see some Python and SQLite solutions.
Thanks,
Oz
I don't think that's the way auto incrementing fields work. SQLite keeps a counter of the last used integer. It will never 'fill in' the deleted values if you want to get the next 3 rows after
an id you could:
SELECT * FROM NODES WHERE ID > 2 LIMIT 3;
This will give you the next three rows with an id greater than 2
Additionally you could just create a deleted flag or something? so the rows are never actually removed from your database.
You can't. SQLite will never re-use deleted IDs, for database integrity reasons. Let's assume you have a second table which has a foreign key which references the first table. If, for some reason, a corresponding row is removed without removing the rows which reference it (using the primary ID) as well, it will point to the wrong row.
Example: If you remove a person record without removing the purchases as well, the purchase records will point to the wrong person once you re-assign the old ID.
───────────────────── ────────────────────
Table 1 – customers Table 2 – purchase
───────────────────── ────────────────────
*ID <───────┐ *ID
Name │ Item
Address └─────── Customer
Phone Price
This is why pretty much any database engine out there assigns primary IDs strictly incremental. They are database internals, you usually shouldn't touch them. If you need to assign your own IDs, just add a separate column (think twice before doing so).
If you want to keep track of the number of rows, you can query it like this: SELECT Count(*) FROM table_name.