Python psycopg2 multiple columns in INSERT query - python

I want to execute an INSERT query via psycopg2, for this question let's simplify it to just:
query = """ INSERT INTO %s("%s") VALUES(%s); """
This works just fine when I do:
params = [AsIs(table_name), AsIs(column_name), value]
cursor.execute(query, params)
Now, my Pandas dataframe has about 90+ columns, I want to know what the best way to extend the query above to be able to execute it for multiple columns.
I have tried joining every column and value together as a single string and passing that in. I have also tried creating a string with 90+ "\"%s\"" and I have also tried creating a format string ie. """INSERT INTO {0} ({1}...{n}) VALUES ({n+1...n+n})""".format(...). There are unrelated issues that prevent these from working, but is there an easier way to handle this multiple column case?

I'm not familiar with pandas but you probably want something like this:
columns = ', '.join(column_names) # Where column names is a tuple or list of all the column headings you want in your query.
query = """ INSERT INTO %s("%s") VALUES(%%s); """ % (table_name, columns)
params = [value]
cursor.execute(query, params)
The point is that you need to insert the column headings and the values separately. See this post for a much better explanation than what I can provide:
Psycopg2 Insert Into Table with Placeholders

Related

What SQL statement do I need to insert data into multiple rows that are currently 'null'?

I have data in this format, that I want to enter into a column in my mysql database. The column i created is empty with null values.
[(1958000,),
(261250,),
(205480,),
(140580,),
(804320,),
(652653,),
(484990,),]
I am using python to insert data using the following code:
sql = "INSERT INTO city(Population) VALUES(%s)"
new_column_data = The data I listed above.
mycursor.executemany(sql, new_column_data)
When I run this, It inserted data into rows starting below all the 'null' values in Population.
Would something like this work?
sql = "INSERT INTO city(Population) VALUES(%s) WHERE Population = 'null'"
Any help would be greatly appreciated!
Quickest solution for you is to remove the NULL columns before inserting the new data.
You would have to do this by running something like:
DELETE c FROM city AS c WHERE c.Population IS NULL
Then run your INSERT.
However, if you are trying to UPDATE existing records in city whose Population field is NULL, then you need to write a completely different statement.
Your data set will need to include a key, kind of like updating a Python dictionary.
dataset =
[(1958000,<key_value>),
(261250,<key_value>),
(205480,<key_value>),
(140580,<key_value>),
(804320,<key_value>),
(652653,<key_value>),
(484990,<key_value>),]
Using these tuples, you will need to create an update statement. How this works depends on the Python library you are using. But something along the lines of this:
WITH ds = dataset:
statement = UPDATE c SET c.Population = ds.<population_value> WHERE c.<key_column> = ds.<key_value>
mysql.update(statment,ds)

Insert list of values into postgres database [duplicate]

This question already has answers here:
imploding a list for use in a python MySQLDB IN clause
(8 answers)
Closed 1 year ago.
I want to insert a list in my database but I can't.
Here is an example of what I need:
variable_1 = "HELLO"
variable_2 = "ADIOS"
list = [variable_1,variable_2]
INSERT INTO table VALUES ('%s') % list
Can something like this be done? Can I insert a list as a value?
When I try it, an error says that is because of an error in MySQL syntax
The answer to your original question is: No, you can't insert a list like that.
However, with some tweaking, you could make that code work by using %r and passing in a tuple:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1, variable_2]
print "INSERT INTO table VALUES %r;" % (tuple(varlist),)
Unfortunately, that style of variable insertion leaves your code vulnerable to SQL injection attacks.
Instead, we recommend using Python's DB API and building a customized query string with multiple question marks for the data to be inserted:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1,variable_2]
var_string = ', '.join('?' * len(varlist))
query_string = 'INSERT INTO table VALUES (%s);' % var_string
cursor.execute(query_string, varlist)
The example at the beginning of the SQLite3 docs shows how to pass arguments using the question marks and it explains why they are necessary (essentially, it assures correct quoting of your variables).
Your question is not clear.
Do you want to insert the list as a comma-delimited text string into a single column in the database? Or do you want to insert each element into a separate column? Either is possible, but the technique is different.
Insert comma-delimited list into one column:
conn.execute('INSERT INTO table (ColName) VALUES (?);', [','.join(list)])
Insert into separate columns:
params = ['?' for item in list]
sql = 'INSERT INTO table (Col1, Col2. . .) VALUES (%s);' % ','.join(params)
conn.execute(sql, list)
both assuming you have established a connection name conn.
A few other suggestions:
Try to avoid INSERT statements that do not list the names and order of the columns you're inserting into. That kind of statement leads to very fragile code; it breaks if you add, delete, or move columns around in your table.
If you're inserting a comma-separted list into a single-field, that generally violates principals of database design and you should use a separate table with one value per record.
If you're inserting into separate fields and they have names like Word1 and Word2, that is likewise an indication that you should be using a separate table instead.
Never use direct string substitution to create SQL statements. It will break if one of the values is, for example o'clock. It also opens you to attacks by people using SQL injection techniques.
You can use json.dumps to convert a list to json and write the json to db.
For example:
insert table example_table(column_name) values(json.dumps(your_list))

Copy row from Cassandra database and then insert it using Python

I'm using plugin DataStax Python Driver for Apache Cassandra.
I want to read 100 rows from database and then insert them again into database after changing one value. I do not want to miss previous records.
I know how to get my rows:
rows = session.execute('SELECT * FROM columnfamily LIMIT 100;')
for myrecord in rows:
print(myrecord.timestamp)
I know how to insert new rows into database:
stmt = session.prepare('''
INSERT INTO columnfamily (rowkey, qualifier, info, act_date, log_time)
VALUES (, ?, ?, ?, ?)
IF NOT EXISTS
''')
results = session.execute(stmt, [arg1, arg2, ...])
My problems are that:
I do not know how to change only one value in a row.
I don't know how to insert rows into database without using CQL. My columnfamily has more than 150 columns and writing all their names in query does not seem as a best idea.
To conclude:
Is there a way to get rows, modify one value from every one of them and then insert this rows into database without using only CQL?
First, you need to select only needed columns from Cassandra - it will be faster to transfer the data. You need to include all columns of primary key + column that you want to change.
After you get the data, you can use UPDATE command to update only necessary column (example from documentation):
UPDATE cycling.cyclist_name
SET comments ='='Rides hard, gets along with others, a real winner'
WHERE id = fb372533-eb95-4bb4-8685-6ef61e994caa
You can also use prepared statement to make it more performant...
But be careful - the UPDATE & INSERT in CQL are really UPSERTs, so if you change columns that are part of primary key, then it will create new entry...

How to use a parameter to select one or more columns in a SELECT statement

I'm trying to create a function that is able to select particular columns from a SQLite 3 table. The idea is to do something like this:
con = sqlite3.connect("my_db.db")
cursor = con.cursor()
def my_func(parameter_list):
con.execute("SELECT parameter_list FROM a_table")
return cursor.fetchall()
where the parameter_list contains the names of the columns the user wants selected.
I've tried using ? placeholders, but:
I still need to use a fixed amount in the SELECT statement itself;
for some reason the output is the names of the columns, not the contents. What I want to do is let the user determine the number of columns and which columns exactly he'd like to fetch.
you need to get a comma-separated string for the columns, right? can be done like his:
"SELECT {} FROM a_table".format(','.join(parameter_list))

Insert entry into mysqldb with entry values stored as dictionary python

I have stored the entry to be inserted into db as dictionary with dictionary keys same as field names. is there any simple command to directly do this?
currently this is the command I use
cursor.execute('INSERT INTO jobs (title, description, country, state, city)
VALUES (%(title)s, %(description)s, %(country)s, %(state)s, %(city)s)', (job_data.get_parsed_dictionary()))
Many times python has so many elegant library methods for all sorts of things. I am hoping there one such command which make it much simpler.
You could generate the column names:
data = job_data.get_parsed_dictionary()
columns = ', '.join(data.keys())
parameters = ', '.join(['%({0})s'.format(k) for k in data.keys()])
query = 'INSERT INTO jobs ({columns}) VALUES ({parameters})'.format(columns=columns, parameters=parameters)
cursor.execute(query, data)
You do need to make sure that the keys in the .get_parsed_dictionary() result are safe to interpolate into a SQL query (remember your SQL injection attack vectors). If you have a list of possible column names handy, I'd certainly check against that to filter out any stray 'extra' keys you may find in that dict.

Categories