We Keep Coding

insert dictionaries as rows in sqlite table

I have dictionaries like this:
{'id': 8, 'name': 'xyzzy', 'done': False}
the table is already created with the correct column names (keys of the dictionary). How can I insert the values in the respective columns? I want to create a new row for each dictionary.
Note that for 'done' the type defined is originally Integer since sqlite does not offer bool type.
cur = connection().cursor()
query = "insert .... tablename"

In Python, database cursors accept two parameters:
an SQL statement as a string: the statement may contain placeholders instead of some values to handle cases where the values are not known until runtime.
a collection of values to be inserted into the SQL statement. These values replace the placeholders in the SQL statement when it is executed.
Placeholders may be positional or named:
# Positional placeholders: the order of values should match the order of
# placeholders in the statement. Values should be contained with
# a tuple or list, even if there is only one.
cur.execute("""SELECT * FROM tbl WHERE name = ? AND age = ?""", ('Alice', 42))
# Named placeholders: values and placeholders are matched by name, order
# is irrelevant. Values must be contained within a mapping (dict) of
# placeholders to values.
cur.execute(
"""SELECT * FROM tbl WHERE name = :name AND age = :age""",
{'age': 42, 'name': 'Alice'}
)
You can dictionary to cursor execute and it will do the right thing as long as the values placeholders in the SQL statement used the :named format (that is, the dict key prefixed by a colon ":").
conn = sqlite3.connect()
cur = conn.cursor()
stmt = """INSERT INTO mytable (id, name, done) VALUES (:id, :name, :done)"""
cur.execute(stmt, {'id': 8, 'name': 'xyzzy', 'done': False})
# Call commit() on the connection to "save" the data.
conn.commit()
This method ensures that values are correctly quoted before being inserted into the database and protects against SQL injection attacks.
See also the docs

You could use .format() method to insert into a query string however this is much more straightforward.
dic = {'id': 8, 'name': 'xyzzy', 'done': False}
cur.execute("INSERT INTO tablename VALUES (:id,:name,:done)",{"id" : dic["id"],"name" : dic["name"],"done" : dic["done"]})

PyMySQL Query returns bizarre dictionary

Problem
I'm using PyMYSQL to query a database using the following SQL translater function.
def retrieve_column(lotkey, column="active",print = False):
result = None
try:
connection = sql_connect()
with connection.cursor() as cursor:
# Create a new record
sql = "SELECT %s FROM table_n"
val = (column)
os_print(sql + '\r\n...', end='', style='dim', flush=True)
cursor.execute(sql, val)
result = cursor.fetchall()
connection.close()
except Exception as e:
os_print(e, style='error')
os_print("ERROR: Can't connect to database!", style='error', tts='error')
return result
Which I call and print using the following lines. Note: The 'active' column is boolean.
active_col = retrieve_column(key)
print(active_col)
Which prints the following bizarre result. It seems to be a dictionary with no values present therein.
...[{'active': 'active'}, {'active': 'active'}, {'active': 'active'}, {'active': 'active'}, {'active': 'active'}, {'active': 'active'}, {'active': 'active'}]
Attempted Solutions
My first step was to run the same query in MySQL workbench which produced the following result.
Workbench Query
Which is roughly what I am trying to replicate in Python (Getting a dictionary with each row's boolean value).
Next, I used the python debugger and found that indeed the returned values from cursor.fetchall() are empty dictionaries with nothing but a single key and no values.
Has anyone encountered something similar before?

Actually using these three instructions:
sql = "SELECT %s FROM table_n"
val = (column)
cursor.execute(sql, val)
You will get the following query executed:
SELECT 'column' FROM table_n
The result is a list of 'column' values (name of the column is also 'column'). Because the parameters of the cursor.execute() method are not literals, but parameter values (in this case, a string value 'column')
If you are trying to select the column value, you need to format the SQL query content, not the paramaters:
sql = "SELECT {colname} FROM table_n"
sql.format( colname = column )
cursor.execute(sql)

Column names cannot be passed to cursor the same way argument values can be passed. For that you do actually need to format the query string.
sql = "SELECT {} FROM table_n".format(column)
cursor.execute(sql)

How to save dictionaries of different lengths to the same table in database?

What would be the most elegant way to save multiple dictionaries - most of them following the same structure, but some having more/less keys - to the same SQL database table?
The steps I can think of are the following:
Determine which dictionary has the most keys and then create a table which follows the dictionary's keys order.
Sort every dictionary to match this column order.
Insert each dictionary's values into the table. Do not insert anything (possible?) if for a particular table column no key exists in the dictionary.
Some draft code I have:
man1dict = {
'name':'bartek',
'surname': 'wroblewski',
'age':32,
}
man2dict = {
'name':'bartek',
'surname': 'wroblewski',
'city':'wroclaw',
'age':32,
}
with sqlite3.connect('man.db') as conn:
cursor = conn.cursor()
#create table - how do I create it automatically from man2dict (the longer one) dicionary, also assigning the data type?
cursor.execute('CREATE TABLE IF NOT EXISTS People(name TEXT, surname TEXT, city TEXT, age INT)')
#show table
cursor.execute('SELECT * FROM People')
print(cursor.fetchall())
#insert into table - this will give 'no such table' error if dict does not follow table column order
cursor.execute('INSERT INTO People VALUES('+str(man1dict.values())+')', conn)

Use NoSQL databases such as MongoDB for this purpose. They will handle these themselves. Using relational data for something that is not relational, this is an anti-pattern. This will break your code, degrade your application's scalability and when you want to change the table structure, it will more cumbersome to do so.

It might be easiest to save the dict as pickle and then unpickle it later. ie
import pickle, sqlite3
# SAVING
my_pickle = pickle.dumps({"name": "Bob", "age": 24})
conn = sqlite3.connect("test.db")
c = conn.cursor()
c.execute("CREATE TABLE test (dict BLOB)")
conn.commit()
c.execute("insert into test values (?)", (my_pickle,))
conn.commit()
# RETRIEVING
b = [n[0] for n in c.execute("select dict from test")]
dicts = []
for d in b:
dicts.append(pickle.loads(d))
print(dicts)
This outputs
[{"name": "Bob", "age": 24}]

Set Sqlite query results as variables [duplicate]

This question already has answers here:
How can I get dict from sqlite query?
(16 answers)
Closed 4 years ago.
Issue:
Hi, right now I am making queries to sqlite and assigning the result to variables like this:
Table structure: rowid, name, something
cursor.execute("SELECT * FROM my_table WHERE my_condition = 'ExampleForSO'")
found_record = cursor.fetchone()
record_id = found_record[0]
record_name = found_record[1]
record_something = found_record[2]
print(record_name)
However, it's very possible that someday I have to add a new column to the table. Let's put the example of adding that column:
Table structure: rowid, age, name, something
In that scenario, if we run the same code, name and something will be assigned wrongly and the print will not get me the name but the age, so I have to edit the code manually to fit the current index. However, I am working now with tables of more than 100 fields for a complex UI and doing this is tiresome.
Desired output:
I am wondering if there is a better way to catch results by using dicts or something like this:
Note for lurkers: The next snipped is made up code that does not works, do not use it.
cursor.execute_to(my_dict,
'''SELECT rowid as my_dict["id"],
name as my_dict["name"],
something as my_dict["something"]
FROM my_table WHERE my_condition = "ExampleForSO"''')
print(my_dict['name'])
I am probably wrong with this approach, but that's close to what I want. That way if I don't access the results as an index, and if add a new column, no matter where it's, the output would be the same.
What is the correct way to achieve it? Is there any other alternatives?

You can use namedtuple and then specify connection.row_factory in sqlite. Example:
import sqlite3
from collections import namedtuple
# specify my row structure using namedtuple
MyRecord = namedtuple('MyRecord', 'record_id record_name record_something')
con = sqlite3.connect(":memory:")
con.isolation_level = None
con.row_factory = lambda cursor, row: MyRecord(*row)
cur = con.cursor()
cur.execute("CREATE TABLE my_table (record_id integer PRIMARY KEY, record_name text NOT NULL, record_something text NOT NULL)")
cur.execute("INSERT INTO my_table (record_name, record_something) VALUES (?, ?)", ('Andrej', 'This is something'))
cur.execute("INSERT INTO my_table (record_name, record_something) VALUES (?, ?)", ('Andrej', 'This is something too'))
cur.execute("INSERT INTO my_table (record_name, record_something) VALUES (?, ?)", ('Adrika', 'This is new!'))
for row in cur.execute("SELECT * FROM my_table WHERE record_name LIKE 'A%'"):
print(f'ID={row.record_id} NAME={row.record_name} SOMETHING={row.record_something}')
con.close()
Prints:
ID=1 NAME=Andrej SOMETHING=This is something
ID=2 NAME=Andrej SOMETHING=This is something too
ID=3 NAME=Adrika SOMETHING=This is new!

Using instance variables in SQLite3 update?

Ok so basically I'm trying to update an existing SQLite3 Database with instance variables (typ and lvl)
#Set variables
typ = 'Test'
lvl = 6
#Print Databse
print("\nHere's a listing of all the records in the table:\n")
for row in cursor.execute("SELECT rowid, * FROM fieldmap ORDER BY rowid"):
print(row)
#Update Info
sql = """
UPDATE fieldmap
SET buildtype = typ, buildlevel = lvl
WHERE rowid = 11
"""
cursor.execute(sql)
#Print Databse
print("\nHere's a listing of all the records in the table:\n")
for row in cursor.execute("SELECT rowid, * FROM fieldmap ORDER BY rowid"):
print(row)
As an Error I'm getting
sqlite3.OperationalError: no such column: typ
Now I basically know the problem is that my variable is inserted with the wrong syntax but I can not for the life of me find the correct one. It works with strings and ints just fine like this:
sql = """
UPDATE fieldmap
SET buildtype = 'house', buildlevel = 3
WHERE rowid = 11
"""
But as soon as I switch to the variables it throws the error.

Your query is not actually inserting the values of the variables typ and lvl into the query string. As written the query is trying to reference columns named typ and lvl, but these don't exist in the table.
Try writing is as a parameterised query:
sql = """
UPDATE fieldmap
SET buildtype = ?, buildlevel = ?
WHERE rowid = 11
"""
cursor.execute(sql, (typ, lvl))
The ? acts as a placeholder in the query string which is replaced by the values in the tuple passed to execute(). This is a secure way to construct the query and avoids SQL injection vulnerabilities.

Hey I think you should use ORM to manipulate with SQL database.
SQLAlchemy is your friend. I use that with SQLite, MySQL, PostgreSQL. It is fantastic.
That can make you get away from this syntax error since SQL does take commas and quotation marks as importance.
For hard coding, you may try this:
sql = """
UPDATE fieldmap
SET buildtype = '%s', buildlevel = 3
WHERE rowid = 11
""" % (house)
This can solve your problem temporarily but not for the long run. ORM is your friend.
Hope this could be helpful!