There are 6 columns and for some reason when my program gets to this bit of code during install, it simply creates a blank file with no table.
Through trial and error, I found the only thing that did not create a blank file was removing the limit row.
I have other code that runs and looks the same just for different databases and it works fine.
try:
# Connect to Database
conn = sqlite3.connect('databases/Categories.db')
cur = conn.cursor()
# Create Table
cur.execute("""CREATE TABLE categories (
priority text,
name text,
type text,
increment text,
total real,
limit real)""")
# Commit and Close
conn.commit()
conn.close()
except sqlite3.OperationalError:
pass
"limit" is an SQL keyword, for example, as in
SELECT foo
FROM bar
LIMIT 10;
If you want to use "limit" as a column name in sqlite, it needs to be quoted, in one of these ways:
'limit'
"limit"
[limit]
`limit`
So for example, your statement could be
cur.execute("""CREATE TABLE categories (
priority text,
name text,
type text,
increment text,
total real,
"limit" real)""")
Note that it must be quoted in other statements too, for example
"""INSERT INTO categories ("limit") VALUES (?);"""
I did some more testing and I fixed it by renaming the limit row to something else. Turns out, sqlite3 doesn't like rows named limit.
Related
I am making a script, that should create a schema for each customer. I’m fetching all metadata from a database that defines how each customer’s schema should look like, and then create it. Everything is well defined, the types, names of tables, etc. A customer has many tables (fx, address, customers, contact, item, etc), and each table has the same metadata.
My procedure now:
get everything I need from the metadataDatabase.
In a for loop, create a table, and then Alter Table and add each metadata (This is done for each table).
Right now my script runs in about a minute for each customer, which I think is too slow. It has something to do with me having a loop, and in that loop, I’m altering each table.
I think that instead of me altering (which might be not so clever approach), I should do something like the following:
Note that this is just a stupid but valid example:
for table in tables:
con.execute("CREATE TABLE IF NOT EXISTS tester.%s (%s, %s);", (table, "last_seen date", "valid_from timestamp"))
But it gives me this error (it seems like it reads the table name as a string in a string..):
psycopg2.errors.SyntaxError: syntax error at or near "'billing'"
LINE 1: CREATE TABLE IF NOT EXISTS tester.'billing' ('last_seen da...
Consider creating tables with a serial type (i.e., autonumber) ID field and then use alter table for all other fields by using a combination of sql.Identifier for identifiers (schema names, table names, column names, function names, etc.) and regular format for data types which are not literals in SQL statement.
from psycopg2 import sql
# CREATE TABLE
query = """CREATE TABLE IF NOT EXISTS {shm}.{tbl} (ID serial)"""
cur.execute(sql.SQL(query).format(shm = sql.Identifier("tester"),
tbl = sql.Identifier("table")))
# ALTER TABLE
items = [("last_seen", "date"), ("valid_from", "timestamp")]
query = """ALTER TABLE {shm}.{tbl} ADD COLUMN {col} {typ}"""
for item in items:
# KEEP IDENTIFIER PLACEHOLDERS
final_query = query.format(shm="{shm}", tbl="{tbl}", col="{col}", typ=i[1])
cur.execute(sql.SQL(final_query).format(shm = sql.Identifier("tester"),
tbl = sql.Identifier("table"),
col = sql.Identifier(item[0]))
Alternatively, use str.join with list comprehension for one CREATE TABLE:
query = """CREATE TABLE IF NOT EXISTS {shm}.{tbl} (
"id" serial,
{vals}
)"""
items = [("last_seen", "date"), ("valid_from", "timestamp")]
val = ",\n ".join(["{{}} {typ}".format(typ=i[1]) for i in items])
# KEEP IDENTIFIER PLACEHOLDERS
pre_query = query.format(shm="{shm}", tbl="{tbl}", vals=val)
final_query = sql.SQL(pre_query).format(*[sql.Identifier(i[0]) for i in items],
shm = sql.Identifier("tester"),
tbl = sql.Identifier("table"))
cur.execute(final_query)
SQL (sent to database)
CREATE TABLE IF NOT EXISTS "tester"."table" (
"id" serial,
"last_seen" date,
"valid_from" timestamp
)
However, this becomes heavy as there are too many server roundtrips.
How many tables with how many columns are you creating that this is slow? Could you ssh to a machine closer to your server and run the python there?
I don't get that error. Rather, I get an SQL syntax error. A values list is for conveying data. But ALTER TABLE is not about data, it is about metadata. You can't use a values list there. You need the names of the columns and types in double quotes (or no quotes) rather than single quotes. And you can't have a comma between name and type. And you can't have parentheses around each pair. And each pair needs to be introduced with "ADD", you can't have it just once. You are using the wrong tool for the job. execute_batch is almost the right tool, except it will use single quotes rather than double quotes around the identifiers. Perhaps you could add a flag to it tell it to use quote_ident.
Not only is execute_values the wrong tool for the job, but I think python in general might be as well. Why not just load from a .sql file?
I'm trying to run SQL query with LIKE operator through Python to find any values that have "test" in any position. The problem seems to be with the formatting of what comes after the LIKE operator. There's no error messages, queries are just empty.
The SQL query that I'm trying to mimic is as follows, and works on when executed on Access.
SELECT Areas.ID, Areas.Name
FROM Areas
WHERE Name LIKE '*test*'
Here's how the connection and test data is made. No issue in there.
import pyodbc
# Connect to database
conn_str = (
r'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};'
r'DBQ=C:\Temp\TestDB.accdb;'
r'Uid=;'
r'Pwd=;'
)
# Make cursor
connection = pyodbc.connect(conn_str)
connection.setencoding('utf-8')
cursor = connection.cursor()
# Create test table
cursor.execute("CREATE TABLE Areas (ID integer, Name varchar(255))")
connection.commit()
# Create test data
cursor.execute("INSERT INTO Areas (ID, Name) VALUES (1,'Example_1');")
cursor.execute("INSERT INTO Areas (ID, Name) VALUES (2,'Example_test_2');")
cursor.execute("INSERT INTO Areas (ID, Name) VALUES (3,'Example_3');")
connection.commit()
# Query filter
Filter = "'*test*'"
Attempt 01
query_01 = cursor.execute(r"""
SELECT Areas.ID, Areas.Name
FROM Areas
WHERE Name LIKE {Filter}
""".format(Filter=Filter)).fetchall()
for row in query_01:
print(row)
Attempt 02
query_02 = cursor.execute(r"""
SELECT Areas.ID, Areas.Name
FROM Areas
WHERE Name LIKE ?
""",("%{}%".format(filter),)).fetchall()
for row in query_02:
print(row)
Attempt 03, I would like the filter to be variable but even "hard coded" does not work.
query_03 = cursor.execute(r"""
SELECT Areas.ID, Areas.Name
FROM Areas
WHERE Name LIKE '*test*'
""").fetchall()
for row in query_03:
print(row)
To be sure something is working, I ran this and it prints the row.
query_04 = cursor.execute(r"""
SELECT Areas.ID, Areas.Name
FROM Areas
WHERE Name = 'Example_test_2'
""").fetchall()
for row in query_04:
print(row)
The ideal solution would be that the filter variable could be just a string, without the wildcards. How should I format the filter variable and the query?
For historical reasons, LIKE queries run from within the Access UI default to using * and ? as the wildcard characters. However, external applications using ODBC to query an Access database must use the more common % and _ wildcard characters.
Also, the parameter value must contain the wildcard character(s). (A LIKE condition without wildcard characters is just the same as an = condition.) The parameter placeholder in the SQL command text must be a bare question mark ?.
Finally, do not use connection.setencoding('utf-8'). Access stores text values as Unicode, but it does not use UTF-8 encoding. The default pyodbc encoding (UTF-16) works just fine.
So what you're looking for is
filter = 'test'
sql = "SELECT Areas.ID, Areas.Name FROM Areas WHERE Areas.Name LIKE ?"
param = f'%{filter}%'
rows = cursor.execute(sql, param).fetchall()
I have a table using SQL Lite with Python. The size of the table always has 3 columns and could have many rows. Each of the cells are strings. Here is example table:
serial_num date_measured status
1234A 1-1-2015 passed
4321B 6-21-2015 failed
1423C 12-25-2015 passed
......
My program prompts me for a serial number. This is saved as a variable called serialNum. How can I delete (or overwrite) an entire row if serialNum equals any of the strings in the serial_num column in my table?
I've seen many examples on how to delete (or overwrite) a row in a table if I know all the values in each cell of that row, but my trouble is that the only cell that could ever be the same in each row would be the serial number. I need to so a search through the serial_number column and if any string in that column equals the current value of my serialNum variable, I need to delete (or overwrite) that row.
import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()
c.execute('''CREATE TABLE test (serial_num text, date_measured text, status text)''')
c.execute("INSERT INTO test VALUES ('1234A', '1-1-2015', 'passed')")
c.execute("INSERT INTO test VALUES ('4321B', '6-21-2015', 'failed')")
c.execute("INSERT INTO test VALUES ('1423C', '12-25-2015', 'passed')")
conn.commit()
Does anyone know a simple way to do this? I've seen others say that an ID must be used or a temporary table, but I would hope there might be an easier way to accomplish my task. Any advice would be great.
SQL suports this: simply use delete
"delete from test where serial_num=<some input>;"
or in this case
c.execute("delete from test where serial_num=%s;", serialNum);
There's no need to search through the list when using SQL. SQL is declarative: you tell it what to do using your query, not how to do it. Don't loop though all your rows to check which to delete: tell it what to delete and the database engine will find the best/fastest way to satisfy that goal.
Hope I well interpreted your question
for row in c.execute('SELECT * FROM test WHERE serial_num = ?', serialNum'):
# do whatever you want on row
print row
I was able to figure out a working solution:
sql = "DELETE FROM test WHERE serial_num = ?"
c.execute(sql, (serialNum,))
The comma after serialNum for some reason has to be there. Thank you #Michiel Arienfor the head start
i just started out with programmming and wrote a few lines of code in pyscripter using sqlite3.
The table "gather" is created beforehand. i then select certain rows from "gather" to put them into another table. i try to sort this table by a specific column 'date'. But it doesn't seem to work. it doesn't give me an error message or something like that. It's just not sorted. If i try the same command (SELECT * FROM matches ORDER BY date) in sqlitemanager, it works fine on the exact same table! what is the problem here? i googled quite some time, but i don't find a solution. it's proobably something stupid i'm missing..
as i said i'm a total newbie. i guess you all break out in tears looking at the code. so if you have any tips how i can shorten the code or make it faster or whatever, you're very welcome :) (but everything works fine except the above mentioned part.)
import sqlite3
connection = sqlite3.connect("gather.sqlite")
cursor1 = connection.cursor()
cursor1.execute('Drop table IF EXISTS matches')
cursor1.execute('CREATE TABLE matches(date TEXT, team1 TEXT, team2 TEXT)')
cursor1.execute('INSERT INTO matches (date, team1, team2) SELECT * FROM gather WHERE team1=? or team2=?, (a,a,))
cursor1.execute("SELECT * FROM matches ORDER BY date")
connection.commit()
OK, I think I understand your problem. First of all: I'm not sure if that commit call is necessary at all. However, if it is, you'll definitely want that to be before your select statement. 'connection.commit()' is essentially saying, commit the changes I just made to the database.
Your second issue is that you are executing the select query but never actually doing anything with the results of the query.
try this:
import sqlite3
connection = sqlite3.connect("gather.sqlite")
cursor1 = connection.cursor()
cursor1.execute('Drop table IF EXISTS matches')
cursor1.execute('CREATE TABLE matches(date TEXT, team1 TEXT, team2 TEXT)')
cursor1.execute('INSERT INTO matches (date, team1, team2) SELECT * FROM gather WHERE team1=? or team2=?, (a,a,))
connection.commit()
# directly iterate over the results of the query:
for row in cursor1.execute("SELECT * FROM matches ORDER BY date"):
print row
you are executing the query, but never actually retrieving the results. There are two ways to do this with sqlite3: One way is the way I showed you above, where you can just use the execute statement directly as an iteratable object.
The other way is as follows:
import sqlite3
connection = sqlite3.connect("gather.sqlite")
cursor1 = connection.cursor()
cursor1.execute('Drop table IF EXISTS matches')
cursor1.execute('CREATE TABLE matches(date TEXT, team1 TEXT, team2 TEXT)')
cursor1.execute('INSERT INTO matches (date, team1, team2) SELECT * FROM gather WHERE team1=? or team2=?, (a,a,))
connection.commit()
cursor1.execute("SELECT * FROM matches ORDER BY date")
# fetch all means fetch all rows from the last query. here you put the rows
# into their own result object instead of directly iterating over them.
db_result = cursor1.fetchall()
for row in db_result:
print row
Try moving the commit before the SELECT * (I'm not sure 100% that this is an issue) You then just need to fetch the results of the query :-) Add a line like res = cursor1.fetchall() after you've executed the SELECT. If you want to display them like in sqlitemanager, add
for hit in res:
print '|'.join(hit)
at the bottom.
Edit: To address your issue of storing the sort order to the table:
I think what you're looking for is something like a clustered index. (Which doesn't actually sort the values in th table, but comes close; see here).
SQLIte doesn't have such such indexes, but you can simulate them by actually ordering the table. You can only do this once, as you're inserting the data. You would need an SQL command like the following:
INSERT INTO matches (date, team1, team2)
SELECT * FROM gather
WHERE team1=? or team2=?
ORDER BY date;
instead of the one you currently use.
See point 4 here, which is where I got the idea.
I created a table by following query:
with sqlite3.connect('example.db', detect_types=sqlite3.PARSE_DECLTYPES) as conn:
cur = conn.cursor()
cur.execute("CREATE TABLE Surveys(Id INTEGER PRIMARY KEY, Name TEXT, Desc TEXT, DictObject BLOB, Hash TEXT)")
conn.commit()
Now I have to add some Survey data to Surveys table for every request. Surveys table has Id as primary integer value. This has to be increased upon every insertion - And What is the proper way to do it? Do I have to fetch every row and check what the lastIdis upon every request?
sqlite will automatically provide an id for a INTEGER PRIMARY KEY column on INSERT on a table if you do not provide a value yourself. Just insert data for every column except for the id column.
with sqlite3.connect('example.db', detect_types=sqlite3.PARSE_DECLTYPES) as conn:
cur = conn.cursor()
cur.execute("INSERT INTO Surveys(Name, Desc, DictObject, Hash) VALUES (?, ?, ?, ?",
('somename', 'some description\nof sorts\n',
"{'dump': 'of a dictionary'}", '0xhash'))
You may want to add the keyword AUTOINCREMENT to your id column though; the default is to pick the highest existing row id plus 1, and if you delete data from the table on a regular basis that can lead to reuse of ids (delete the current highest id and it'll be re-used next time round). AUTOINCREMENT guarantees that each generated number will only be used once for a table, independent of deletes.
Note that when you use a sqlite3 connection as a context manager (using the with statement), the commit() is automatically applied for you. Quoting the documentation:
Connection objects can be used as context managers that automatically commit or rollback transactions. In the event of an exception, the transaction is rolled back; otherwise, the transaction is committed.
In other words, you can safely remove the conn.commit() line in your code, it is entirely redundant.