I have a python script executing SQL commands and am trying to insert all my data into a table. The problem is I have a list of entries, say:
"Bob", "bob#gmail.com"
"John", "john#gmail.com", "(123)-456-7890"
"Tom", "tom#gmail.com", "(123)-456-7890"
So the first entry will not have the last value that the other two entries have. How can I insert into my table with one command? Right now I loop through the entries and attempt to insert them but receive Column count doesn't match value count at row 1.
I have tried both INSERT INTO myTable (FirstName, email, phone) VALUES(...) as well as INSERT INTO myTable VALUES(...) omitting the table columns.
Can the Python script just insert a NULL value when the value isn't available? So:
INSERT INTO myTable (FirstName, email, phone) VALUES("Bob", "bob#gmail.com", NULL)
If you are assuming the phone will be included most of the time, you can pass 'NULL' when the phone number is not provided and use the verbose
INSERT INTO myTable (FirstName, email, phone) VALUES('John', 'js#js.com', NULL)
I would suggest writing a rudimentary ORM that will take whatever you give it and adjust the INSERT statement as needed. i.e. If your object doesn't have the phone number property populated, it can generate the insert statement as such:
INSERT INTO myTable (FirstName, email) VALUES('John', 'js#js.com')
Then if the phone number is populated, it will build the insert statement with the phone number column included.
Related
I have an SQLite database. I am trying to insert values (users_id, lessoninfo_id) in table bookmarks, only if both do not exist before in a row.
INSERT INTO bookmarks(users_id,lessoninfo_id)
VALUES(
(SELECT _id FROM Users WHERE User='"+$('#user_lesson').html()+"'),
(SELECT _id FROM lessoninfo
WHERE Lesson="+lesson_no+" AND cast(starttime AS int)="+Math.floor(result_set.rows.item(markerCount-1).starttime)+")
WHERE NOT EXISTS (
SELECT users_id,lessoninfo_id from bookmarks
WHERE users_id=(SELECT _id FROM Users
WHERE User='"+$('#user_lesson').html()+"') AND lessoninfo_id=(
SELECT _id FROM lessoninfo
WHERE Lesson="+lesson_no+")))
This gives an error saying:
db error near where syntax.
If you never want to have duplicates, you should declare this as a table constraint:
CREATE TABLE bookmarks(
users_id INTEGER,
lessoninfo_id INTEGER,
UNIQUE(users_id, lessoninfo_id)
);
(A primary key over both columns would have the same effect.)
It is then possible to tell the database that you want to silently ignore records that would violate such a constraint:
INSERT OR IGNORE INTO bookmarks(users_id, lessoninfo_id) VALUES(123, 456)
If you have a table called memos that has two columns id and text you should be able to do like this:
INSERT INTO memos(id,text)
SELECT 5, 'text to insert'
WHERE NOT EXISTS(SELECT 1 FROM memos WHERE id = 5 AND text = 'text to insert');
If a record already contains a row where text is equal to 'text to insert' and id is equal to 5, then the insert operation will be ignored.
I don't know if this will work for your particular query, but perhaps it give you a hint on how to proceed.
I would advice that you instead design your table so that no duplicates are allowed as explained in #CLs answer below.
For a unique column, use this:
INSERT OR REPLACE INTO tableName (...) values(...);
For more information, see: sqlite.org/lang_insert
insert into bookmarks (users_id, lessoninfo_id)
select 1, 167
EXCEPT
select user_id, lessoninfo_id
from bookmarks
where user_id=1
and lessoninfo_id=167;
This is the fastest way.
For some other SQL engines, you can use a Dummy table containing 1 record.
e.g:
select 1, 167 from ONE_RECORD_DUMMY_TABLE
I have this department table which has an array int column.
create table department(id int,name text, emp_id int[])
I'm getting this error for the below statement while inserting data via Python.
my_cursor.execute(
"""
insert into department
select 10, 'QC', array[34,62,Null]
"""
)
The error is
psycopg2.errors.NullValueNotAllowed: array must not contain nulls
But I can insert a Null value in array and can run this SQL insert statement in SQL Client or PG Admin
insert into department
select 10, 'QC', array[34,62,Null]
I think this error is coming from the python module, as the DB allows inserting Null values in SQL statement.
How can we insert Null values in emp_id column through Python?
I create a table with primary key and autoincrement.
with open('RAND.xml', "rb") as f, sqlite3.connect("race.db") as connection:
c = connection.cursor()
c.execute(
"""CREATE TABLE IF NOT EXISTS race(RaceID INTEGER PRIMARY KEY AUTOINCREMENT,R_Number INT, R_KEY INT,\
R_NAME TEXT, R_AGE INT, R_DIST TEXT, R_CLASS, M_ID INT)""")
I want to then insert a tuple which of course has 1 less number than the total columns because the first is autoincrement.
sql_data = tuple(b)
c.executemany('insert into race values(?,?,?,?,?,?,?)', b)
How do I stop this error.
sqlite3.OperationalError: table race has 8 columns but 7 values were supplied
It's extremely bad practice to assume a specific ordering on the columns. Some DBA might come along and modify the table, breaking your SQL statements. Secondly, an autoincrement value will only be used if you don't specify a value for the field in your INSERT statement - if you give a value, that value will be stored in the new row.
If you amend the code to read
c.executemany('''insert into
race(R_number, R_KEY, R_NAME, R_AGE, R_DIST, R_CLASS, M_ID)
values(?,?,?,?,?,?,?)''',
sql_data)
you should find that everything works as expected.
From the SQLite documentation:
If the column-name list after table-name is omitted then the number of values inserted into each row must be the same as the number of columns in the table.
RaceID is a column in the table, so it is expected to be present when you're doing an INSERT without explicitly naming the columns. You can get the desired behavior (assign RaceID the next autoincrement value) by passing an SQLite NULL value in that column, which in Python is None:
sql_data = tuple((None,) + a for a in b)
c.executemany('insert into race values(?,?,?,?,?,?,?,?)', sql_data)
The above assumes b is a sequence of sequences of parameters for your executemany statement and attempts to prepend None to each sub-sequence. Modify as necessary for your code.
I have 2 tables. First table contains some products and the second table is used for temporary data storage. Both tables have the same column names.
Table `products`: contains this columns
- id (unique, autoincrement)
- name
- quantity
- price
- group
Table `temp_stor`: contains this columns
- id (unique, autoincrement)
- name
- quantity
- price
- group
I want to get from the first table one row (name,quantity,price,group) and insert it into the second table if the data does not exist. If the same data exists in temp_stor I want to update only one column (quantity).
For example:
I take from products the following line ('cola','1','2.5','soda'), I want to check the temp_stor to see if the line exist. temp_store table looks like this:
('milk 1L','1','1.5','milks')
('cola','1','2.5','soda')
('bread','1','0.9','pastry')
('7up','1','2.8','soda')
We see the second line exists, and I want to update it's quantity. The table will look like this:
('milk 1L','1','1.5','milks')
('cola','2','2.5','soda')
('bread','1','0.9','pastry')
('7up','1','2.8','soda')
If the table looks like this:
('milk 1L','1','1.5','milks')
('bread','1','0.9','pastry')
('7up','1','2.8','soda')
I want to insert the line into the table. So it would look like this:
('milk 1L','1','1.5','milks')
('bread','1','0.9','pastry')
('7up','1','2.8','soda')
('cola','1','2.5','soda')
Is this posible to do it through a sql query? I need to implement this into a python code. The python part I can handle , but I'm not that good to sql.
Thank you
UPDATE 1:
i forgot to specify maybe the most important thing and this is my fault. I want to check the existence of a product name inside the temp_stor table. Only the name should be unique. If the product exists i wan't to update it's quantity value only, if the product doesn't exist i want to insert it into the temp_stor table.
Assuming that "if the data does not exist" means "if the combo of name+quantity+price+group" is is not already there?
Add
UNIQUE(name, quantity, price, group)
Then
INSERT INTO products
(name, quantity, price, group)
SELECT name, quantity, price, group FROM temp_stor;
Minor(?) drawback: This will (I think) 'burn' ids. In your example, it will allocate 4 new values of id, but use only one of them.
after update to Question
Do not have the index above, instead, have this. (I assume product is spelled "name"??)
UNIQUE(name)
Then...
INSERT INTO products
(name, quantity, price, group)
SELECT name, quantity, price, group FROM temp_stor
ON DUPLICATE KEY -- this will notice UNIQUE(name)
UPDATE
quantity = VALUES(quantity);
It is unusual to use this construct without UPDATEing all the extra columns (quantity, price, group). What if temp_stor has a different value for price or group?
Take a look at How to connect Python programs to MariaDB to see how to connect to your DB.
After that you can select from temp_stor the row with the same id as the row you have obtained from products. Let row be the tuple of values you obtained.
cursor = mariadb_connection.cursor()
cursor.execute("SELECT * FROM temp_stor WHERE id=%s", (some_id,))
If the result of this query contains nothing, indicating that there is no such row, you can proceed to insert it, otherwise, update the row.
if len(cursor) == 0:
try:
cursor.execute("""INSERT INTO temp_stor VALUES (%s,%s,%s,%s,%s)""", row)
except mariadb.Error as error:
print("Error: {}".format(error))
else:
cursor.execute ("""
UPDATE temp_stor
SET id=%s, name=%s, quantity=%s, price=%s, group=%s
WHERE id=%s
""", (row[0], row[1], row[2], row[3], row[4], row[0]))
Update:
To perform something similar with just one query:
INSERT INTO temp_stor (id, name, quantity, price, group) VALUES(1, "cola", '2.5', 'soda') ON DUPLICATE KEY UPDATE quantity='2.5'
Here I am assuming that 1 is the id and "cola" is the name. If you want name to be unique, you should make that the key, because this query currently only compares keys, which in this case seems to be id.
I have such an INSERT statement:
mtemp = "station, calendar, type, name, date, time"
query = "INSERT INTO table (%s) VALUES ( '%s', '%s', '%s', %s, '%s', '%s' );"
query = query % (mtemp, mstation, mcalendar, mtype, mname, mdate, mtime)
curs.execute(query, )
conn.commit()
The problem is that I can not get the variables: mcalendar, mdate, mtime in this statement. They are not constant values. I would have to access each of them within a forloop. However, the values of mstation, mtype and mname are fixed. I tried to split the INSERT statement into several ones: one for each of the three variables in a forloop, and one for the three fixed values in a single forloop. The forloop is basically to define when to insert rows. I have a list of rows1 and a list of rows2, rows1 is a full list of records while rows2 lack some of them. I’m checking if the rows2 record exist in rows1. If it does, then execute the INSERT statement, if not, do nothing.
I ran the codes and found two problems:
It’s inserting way more rows than it is supposed to. It’s supposed to insert no more than 240 rows for there are only 240 time occurrences in each day for each sensor. (I wonder if it is because I wrote too many forloop so that it keeps inserting rows). Now it’s getting more than 400 new rows.
In these new rows being inserted to the table, they only have values in the columns of fixed value. For the three ones that I use the single forloop to insert data, they don’t have value at all.
Hope someone give me some tip here. Thanks in advance! I can put more codes here if needed. I’m not even sure if I’m in the right track.
I'm not sure I understand exactly your scenario, but is this the sort of thing you need?
Pseudo code
mstation = "foo"
mtype = "bar"
mname = "baz"
mtemp = "station, calendar, type, name, date, time"
queryTemplate = "INSERT INTO table (%s) VALUES ( '%s', '%s', '%s', %s, '%s', '%s' );"
foreach (mcalendar in calendars)
foreach (mdate in dates)
foreach (mtime in times)
query = queryTemplate % (mtemp, mstation, mcalendar, mtype, mname, mdate, mtime)
curs.execute(query, )
One INSERT statement always corresponds to one new row in a table. (Unless of course there is an error during the insert.) You can INSERT a row, and then UPDATE it later to add/change information but there is no such thing as splitting up an INSERT.
If you have a query which needs to be executed multiple times with changing data, the best option is a prepared statement. A prepared statement "compiles" an SQL query but leaves placeholders that can set each time it is executed. This improves performance because the statement doesn't need to be parsed each time. You didn't specify what library you're using to connect to postgres so I don't know what the syntax would be, but it's something to look in to.
If you can't/don't want to use prepared statements, you'll have to just create the query string once for each insert. Don't substitute the values in before the loop, wait until you know them all before creating the query.
Following syntax works in SQL Server 2008 but not in SQL Server 2005.
CREATE TABLE Temp (id int, name varchar(10));
INSERT INTO Temp (id, name) VALUES (1, 'Anil'), (2, 'Ankur'), (3, 'Arjun');
SELECT * FROM Temp;
id | name
------------
1 | Anil
2 | Ankur
3 | Arjun