With Python's sqlite3 library, can I have a variable number of place holders in the SQL statement:
INSERT INTO table VALUES (?,?)`
where ? are the place holders, which is safe from an SQL injection attack?
I want to be able to have a general function (below) that checks number of columns and writes data into a row but it could work for any table with any number of columns.
I looked at:
Python Sqlite3: INSERT INTO table VALUE(dictionary goes here) and
PHP MYSQL 'INSERT INTO $table VALUES ......' variable number of fields
but I'm still not sure.
def rowin(self, TableName, ColumnData=[]):
# First check number columns in the table TableName to confirm ColumnData=[] fits
check = "PRAGMA table_info(%s)"%TableName
conn = sqlite3.connect(self.database_file)
c = conn.cursor()
c.execute(check)
ColCount = len(c.fetchall())
# Compare TableName Column count to len(ColumnData)
if ColCount == len(ColumnData):
# I want to be have the number of ? = ColCount
c.executemany('''INSERT INTO {tn} VALUES (?,?)'''.format(tn=TableName), ColumnData)
conn.commit()
else:
print("Input doesn't match number of columns")
def rowin(self,TableName,ColumnData=[]):
#first count number columns in the table TableName
check = "PRAGMA table_info(%s)"%TableName
conn = sqlite3.connect(self.database_file)
c = conn.cursor()
c.execute(check)
#assing number of columns to ColCount
ColCount = len(c.fetchall())
#compare TableName Column count to len(ColumnData)
qmark = "?"
#first create a place holder for each value going to each column
for cols in range(1,len(ColumnData)):
qmark += ",?"
#then check that the columns in the table match the incomming number of data
if ColCount == len(ColumnData):
#now the qmark should have an equl number of "?" to match each item in the ColumnData list input
c.execute('''INSERT INTO {tn} VALUES ({q})'''.format(tn=TableName, q=qmark),ColumnData)
conn.commit()
print "Database updated"
else:
print "input doesnt match number of columns"
Related
I have a database with around 10 columns. Sometimes I need to insert a row which has only 3 of the required columns, the rest are not in the dic.
The data to be inserted is a dictionary named row :
(this insert is to avoid duplicates)
row = {'keyword':'abc','name':'bds'.....}
df = pd.DataFrame([row]) # df looks good, I see columns and 1 row.
engine = getEngine()
connection = engine.connect()
df.to_sql('temp_insert_data_index', connection, if_exists ='replace',index=False)
result = connection.execute(('''
INSERT INTO {t} SELECT * FROM temp_insert_data_index
ON CONFLICT DO NOTHING''').format(t=table_name))
connection.close()
Problem : when I don't have all columns in the row(dic), it will insert dic fields by order (a 3 keys dic will be inserted to the first 3 columns) and not to the right columns. ( I expect the keys in dic to fit the db columns)
Why ?
Consider explicitly naming the columns to be inserted in INSERT INTO and SELECT clauses which is best practice for SQL append queries. Doing so, the dynamic query should work for all or subset of columns. Below uses F-string (available Python 3.6+) for all interpolation to larger SQL query:
# APPEND TO STAGING TEMP TABLE
df.to_sql('temp_insert_data_index', connection, if_exists='replace', index=False)
# STRING OF COMMA SEPARATED COLUMNS
cols = ", ".join(df.columns)
sql = (
f"INSERT INTO {table_name} ({cols}) "
f"SELECT {cols} FROM temp_insert_data_index "
"ON CONFLICT DO NOTHING"
)
result = connection.execute(sql)
connection.close()
Iterating through columns of a table in a database and printing number of records in it
already connected to the database, need to iterate through the columns of a table and prints count of records in each archive as name is printed below
import mysql.connector
from mysql.connector import Error
try:
mySQLconnection = mysql.connector.connect(host='localhost',
database='nmoorg',
user='root',
password='ChvU5M12')
sql_select_Query = "select * from archive"
cursor = mySQLconnection .cursor()
cursor.execute(sql_select_Query)
records = cursor.fetchall()
print("Total number of rows in python_developers is - ", cursor.rowcount)
print ("Printing each row's column values i.e. developer record")
for row in records:
print("archive name = ", row[1], )
cursor.close()
except Error as e :
print ("Error while connecting to MySQL", e )
finally:
#closing database connection.
if(mySQLconnection .is_connected()):
connection.close()
print("MySQL connection is closed")
I need to count the number of records in a column instead of table
SELECT COUNT(*) num FROM archive
is the way to count rows in a table.
SELECT COUNT(*) num, COUNT(column) num_column FROM archive
is the way to count both all the rows and the number of rows with a non-null value in archive.column.
I guess that's what you mean by count the number of records in a column instead of table.
Counting by doing SELECT * and counting the rows in the result set is shockingly inefficient. Plus, .fetchall() will blow out your RAM if the table has any interesting size. The whole point of SQL is to allow you to handle data sets orders of magnitude larger than your RAM.
I'm trying to create an SQL queries for a large list of records (>42 million) to insert into a remote database. Right now I'm building queries in the format INSERT INTO tablename (columnnames) VALUES (values)
tablename, columnnames, and values are all of varying length so I'm generating a number of placeholders equal to the number of values required.
The result is I have a string called sqcommand that looks like INSERT INTO ColName (?,?,?) VALUES (?,?,?); and a list of parameters that looks like ([Name1, Name2, Name3, Val1, Val2, Val3]).
When try to execute the query as db.execute(sqlcommand, params) I get errors indicating I'm trying to insert into columns "#P1", "#P2", "#P3" et cetera. Why aren't the values from my list properly translating? Where is it getting "#P1" from? I know I don't have a column of that name and as far as I can tell I'm not referencing a column of that name yet the execute method is still trying to use it.
UPDATE: As per request, the full code is below, modified to avoid anything that might be private. The end result of this is to move data, row by row, from an sqlite3 db file to an AWS SQL server.
newDB = pyodbc.connect(newDataBase)
oldDB = sqlite3.connect(oldDatabase)
tables = oldDB.execute("SELECT * FROM sqlite_master WHERE type='table';").fetchall()
t0 = datetime.now()
for table in tables:
print('Parsing:', str(table[1]))
t1 = datetime.now()
colInfo = oldDB.execute('PRAGMA table_info('+table[1]+');').fetchall()
cols = list()
cph = ""
i = 0
for col in colInfo:
cph += "?,"
cols.append(str(col[1]))
rowCount = oldDB.execute("SELECT COUNT(*) FROM "+table[1]+" ;").fetchall()
count = 0
while count <= int(rowCount[0][0]):
params = list()
params.append(cols)
count += 1
row = oldDB.execute("SELECT * FROM "+table[1]+" LIMIT 1;").fetchone()
ph = ""
for val in row:
ph += "?,"
params = params.append(str(val))
ph = ph[:-1]
cph = cph[:-1]
print(str(table[1]))
sqlcommand = "INSERT INTO "+str(table[1])+" ("+cph+") VALUES ("+ph+");"
print(sqlcommand)
print(params)
newDB.execute(sqlcommand, params)
sqlcommand = "DELETE FROM ? WHERE ? = ?;"
oldDB.execute(sqlcommand, (str(table[1]), cols[0], vals[0],))
newDB.commit()
Unbeknownst to me, column names can't be passed as parameters. Panagiotis Kanavos answered this in a comment. I guess I'll have to figure out a different way to generate the queries. Thank you all very much, I appreciate it.
Is it possible for me to take data stored in a sqlite3 table and use it as a Python variable? I'm looking for something that might be similar to this pseudo-code:
import sqlite3
conn = sqlite3.connect(DATABASE)
cursor = conn.cursor()
variable = cursor.execute("fetch data from table")
To read a single value from a table, use a SELECT query that returns a result with a single row and a single column:
for row in cursor.execute("SELECT MyColumn FROM MyTable WHERE ID = ?", [123]):
variable = row[0]
break
else:
variable = 0 # not found
I have a list of tuples of which i'm inserting into a Table.
Each tuple has 50 values. How do i insert without having to specify the column names and how many ? there is?
col1 is an auto increment column so my insert stmt starts in col2 and ends in col51.
current code:
l = [(1,2,3,.....),(2,4,6,.....),(4,6,7,.....)...]
for tup in l:
cur.execute(
"""insert into TABLENAME(col2,col3,col4.........col50,col51)) VALUES(?,?,?,.............)
""")
want:
insert into TABLENAME(col*) VALUES(*)
MySQL's syntax for INSERT is documented here: http://dev.mysql.com/doc/refman/5.7/en/insert.html
There is no wildcard syntax like you show. The closest thing is to omit the column names:
INSERT INTO MyTable VALUES (...);
But I don't recommend doing that. It works only if you are certain you're going to specify a value for every column in the table (even the auto-increment column), and your values are guaranteed to be in the same order as the columns of the table.
You should learn to use code to build the SQL query based on arrays of values in your application. Here's a Python example the way I do it. Suppose you have a dict of column: value pairs called data_values.
placeholders = ['%s'] * len(data_values)
sql_template = """
INSERT INTO MyTable ({columns}) VALUES ({placeholders})
"""
sql = sql_template.format(
columns=','.join(keys(data_values)),
placeholders=','.join(placeholders)
)
cur = db.cursor()
cur.execute(sql, data_values)
example code to put before your code:
cols = "("
for x in xrange(2, 52):
cols = cols + "col" + str(x) + ","
test = test[:-1]+")"
Inside your loop
for tup in l:
cur.execute(
"""insert into TABLENAME " + cols " VALUES {0}".format(tup)
""")
This is off the top of my head with no error checking