I am using python3, postgress 10 and Psycopg2 to query multiple records like so
import psycopg2
conn = psycopg2.connect(<my connection string>)
with conn:
with conn.cursor() as cur:
cur.execute('select id,field1 from table1')
for id, field1 from cur.fetchall():
print(id,field1)
#todo: how up update field1 to be f(field1) where f is an arbitrary python function
My question is: how do i update the value of the rows that I am reading and set the value of field1 to some arbitrary python-based calculation
edit: the purpose is to update the rows in the table
You need another cursor, e.g.:
with conn:
with conn.cursor() as cur:
cur.execute('select id,field1 from table1')
for id, field1 in cur.fetchall():
print(id,field1)
with conn.cursor() as cur_update:
cur_update.execute('update table1 set field1 = %s where id = %s', (f(field1), id))
Note however that this involves as many updates as selected rows, which is obviously not efficient. The update can be done in a single query using psycopg2.extras.execute_values():
from psycopg2.extras import execute_values
with conn:
with conn.cursor() as cur:
cur.execute('select id,field1 from table1')
rows = cur.fetchall()
for id, field1 in rows:
print(id,field1)
# convert rows to new values of field1
values = [(id, f(field1)) for id, field1 in rows]
sql = '''
with upd (id, field1) as (values %s)
update table1 t
set field1 = upd.field1
from upd
where upd.id = t.id
'''
with conn.cursor() as cur:
execute_values(cur, sql, values)
Related
I have a table Employee in SQL Server as follows:
ID (AUTO, PK),
firstname (varchar),
lastname (varchar)
I want to insert data like ('John', 'Myers') into the table.
I used the following code in Python using pyodbc:
connection = pyodbc.connect(...)
cursor = connection.cursor()
cursor.execute("insert into employee(firstname, lastname) values(?, ?)", ['John','Myers'])
Is it possible to get the ID value of this newly inserted row without having to write a select query?
You can use the OUTPUT clause
cursor.execute("insert into employee(firstname, lastname) output inserted.ID values(?, ?);", ['John','Myers'])
id = cursor.fetchone()
Alternatively, use SCOPE_IDENTITY()
cursor.execute("insert into employee(firstname, lastname) values(?, ?); select SCOPE_IDENTITY();", ['John','Myers'])
id = cursor.fetchone()
I tried to update multiple rows (approx. 350000) with a single query by implementing the following function:
def update_items(rows_to_update):
sql_query = """UPDATE contact as t SET
name = e.name
FROM (VALUES %s) AS e(id, name)
WHERE e.id = t.id;"""
conn = get_db_connection()
cur = conn.cursor()
psycopg2.extras.execute_values (
cur, sql_query, rows_to_update, template=None, page_size=100
)
While trying to run the function above, only 31 records were updated. Then, I tried to update row by row with the following function:
def update_items_row_by_row(rows_to_update):
sql_query = """UPDATE contact SET name = %s WHERE id = %s"""
conn = get_db_connection()
with tqdm(total=len(rows_to_update)) as pbar:
for id, name in rows_to_update:
cur = conn.cursor()
# execute the UPDATE statement
cur.execute(sql_query, (name, id))
# get the number of updated rows
# Commit the changes to the database
conn.commit()
cur.close()
pbar.update(1)
The latter has updated all the records so far but is very slow (estimated to end in 9 hours).
Does anyone know what is the efficient way to update multiple records?
By splitting the list into chunks of size equal to page_size, it worked well:
def update_items(rows_to_update):
sql_query = """UPDATE contact as t SET
name = data.name
FROM (VALUES %s) AS data (id, name)
WHERE t.id = data.id"""
conn = get_db_connection()
cur = conn.cursor()
n = 100
with tqdm(total=len(rows_to_update)) as pbar:
for i in range(0, len(rows_to_update), n):
psycopg2.extras.execute_values (
cur, sql_query, rows_to_update[i:i + n], template=None, page_size=n
)
conn.commit()
pbar.update(cur.rowcount)
cur.close()
conn.close()
The problem with your original function appears to be that you forgot to apply commit. When you execute an insert/update query with psycopg2 a transaction is opened but not finalized until commit is called. See my edits in your function (towards the bottom).
def update_items(rows_to_update):
sql_query = """UPDATE contact as t SET
name = e.name
FROM (VALUES %s) AS e(id, name)
WHERE e.id = t.id;"""
conn = get_db_connection()
cur = conn.cursor()
psycopg2.extras.execute_values(cur, sql_query, rows_to_update)
## solution below ##
conn.commit() # <- We MUST commit to reflect the inserted data
cur.close()
conn.close()
return "success :)"
If you don't want to call conn.commit() each time you create a new cursor, you can use autocommit such as
conn = get_db_connection()
conn.set_session(autocommit=True)
Beginners question here. I wish to populate a table with many rows of data straight from a query I'm running in the same session. I wish to do it using with excutemany(). currently, I insert each row as a tuple, as shown in the script below.
Select Query to get the needed data:
This query returns data with 4 columns Parking_ID, Snapshot_Date, Snapshot_Time, Parking_Stat
park_set_stat_query = "SET #row_number = 0;"
park_set_stat_query2 = "SET #row_number2 = 0;"
# one time load to catch only the changes done in the input table
park_change_stat_query = """select in1.Parking_ID,
in1.Snapshot_Date as Snapshot_Date,
in1.Snapshot_Time as Snapshot_Time,
in1.Parking_Stat
from (SELECT
Parking_ID,
Snapshot_Date,
Snapshot_Time,
Parking_Stat,
(#row_number:=#row_number + 1) AS num1
from Fact_Parking_Stat_Input
WHERE Parking_Stat<>0) as in1
left join (SELECT
Parking_ID,
Snapshot_Date,
Snapshot_Time,
Parking_Stat,
(#row_number2:=#row_number2 + 1)+1 AS num2
from Fact_Parking_Stat_Input
WHERE Parking_Stat<>0) as in2
on in1.Parking_ID=in2.Parking_ID and in1.num1=in2.num2
WHERE (CASE WHEN in1.Parking_Stat<>in2.Parking_Stat THEN 1 ELSE 0 END=1) OR num1=1"""
Here is the insert part of the script:
as you can see below I insert each row to the destination table Fact_Parking_Stat_Input_Alter
mycursor = connection.cursor()
mycursor2 = connection.cursor()
mycursor.execute(park_set_stat_query)
mycursor.execute(park_set_stat_query2)
mycursor.execute(park_change_stat_query)
# # keep only changes in a staging table named Fact_Parking_Stat_Input_Alter
qSQLresults = mycursor.fetchall()
for row in qSQLresults:
Parking_ID = row[0]
Snapshot_Date = row[1]
Snapshot_Time = row[2]
Parking_Stat = row[3]
#SQL query to INSERT a record into the table Fact_Parking_Stat_Input_Alter.
mycursor2.execute('''INSERT into Fact_Parking_Stat_Input_Alter (Parking_ID, Snapshot_Date, Snapshot_Time, Parking_Stat)
values (%s, %s, %s, %s)''',
(Parking_ID, Snapshot_Date, Snapshot_Time, Parking_Stat))
# Commit your changes in the database
connection.commit()
mycursor.close()
mycursor2.close()
connection.close()
How can I improve the code so it will insert the data in on insert command?
Thanks
Amir
MYSQL has an INSERT INTO command that is probably far more efficient than query it in python, pulling it and re-iserting
https://www.mysqltutorial.org/mysql-insert-into-select/
I am trying to query MS SQL Server for a table.column, then insert this output into a sqlite table.
This example has one numeric column in the SQL Server source table.
I think I've almost got it by scouring the other answers.
Please let me know what I am missing.
import sqlite3
import pyodbc
#def connect_msss():
ODBC_Prod = ODBC_Prod
SQLSN = SQLSN
SQLpass = SQLpass
conn_str = ('DSN='+ODBC_Prod+';UID='+SQLSN+';PWD='+SQLpass)
conn = pyodbc.connect(conn_str)
#def connect_sqlite():
sl3Conn = sqlite3.connect('server_test.db')
c = sl3Conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS mrn_test (PTMRN NUMERIC)')
#def query_msss():
cur = conn.cursor()
cur.execute("SELECT TOP 50 PTMRN FROM dbo.atl_1234_mrntest")
rows = cur.fetchall()
for row in rows:
c.execute("INSERT INTO mrn_test VALUES (?)", row)
conn.commit()
#connect_msss()
#connect_sqlite()
#query_msss()
Error 1:
c.execute('CREATE TABLE IF NOT EXISTS mrn_test
(PTMRN NUMERIC)')
Out[117]: <sqlite3.Cursor at 0x2d1a742fc70>
Error 2:
cur = conn.cursor() cur.execute("SELECT TOP 50 PTMRN FROM
dbo.atl_1234_mrntest")
Out[118]: <pyodbc.Cursor at 0x2d1a731b990>
You're not committing the executed changes on the sqlite connection, after the c.execute step you're committing the MySQL DB connection. I think you need to replace conn.commit() at the end with sl3Conn.commit().
My code so far
conn = sqlite3.connect('databaserm/database')
curs = conn.cursor()
curs.execute('SELECT * FROM saves')
lvl = curs.fetchone()
conn.close()
ok maybes its the code i used to add the data to the db
i've tried this
cn = sqlite3.connect(/databaserm/database")
curs = cn.cursor()
curs.execute('DROP TABLE saves')
curs.execute('CREATE TABLE saves (lvl)')
#curs.execute('INSERT INTO saves VALUES (null, ?)', lvl)
query = """INSERT INTO saves (lvl)
VALUES (?)"""
data = [lvl]
curs.execute(query, data)
cn.commit
cn.close()
and this
conn = sqlite3.connect('/databaserm/database')
curs = conn.cursor()
curs.execute('INSERT INTO saves VALUES(null, ?,)', (lvl,))
conn.commit
Executing select and then fetchone will return a tuple. What else do you need?
Here's an example:
import sqlite3 as sqlite
con = sqlite.connect(':memory:')
cursor = con.cursor()
cursor.execute('''
create table names (id integer primary key,
name varchar(50), email varchar(50))''')
cursor.execute('''
insert into names values (null, "John Doe", "jdoe#jdoe.zz")''')
cursor.execute('''
insert into names values (null, "Mary Sue", "msue#msue.yy")''')
name = """Lu'k'e d"fdf" Sma"""
email = "use#force.net"
cursor.execute(
'''insert into names values (null, ?, ?)''',
(name, email))
cursor.execute('select * from names')
for c in cursor:
print c
Uses iteration over cursor (an alternative way to get result). Prints:
(1, u'John Doe', u'jdoe#jdoe.zz')
(2, u'Mary Sue', u'msue#msue.yy')
(3, u'Lu\'k\'e d"fdf" Sma', u'use#force.net')
Using fetchone instead of iteration:
print cursor.fetchone()
cursor.close()
Prints:
(1, u'John Doe', u'jdoe#jdoe.zz')