I've been following Python documentation on the SQLite tutorial and I managed to create an Employee table and write to it.
import sqlite3
conn = sqlite3.connect('employee.db')
c = conn.cursor()
firstname = "Ann Marie"
lastname = "Smith"
email = "ams#cia.com"
employee = (email, firstname, lastname)
c.execute('INSERT INTO Employee Values (?,?,?)', employee)
conn.commit()
# Print the table contents
for row in c.execute("select * from Employee"):
print(row)
conn.close()
I've been reading about the Write-Ahead Logging, but I can't find a tutorial that explains how to implement it. Can someone provide an example?
I notice Firefox, which uses SQLite, locks the file in such a way that if you attempt to delete the sqlite file while using Firefox, it will fail saying "file is open or being used"(or something similar), how do I achieve this? I'm running Python under Windows 10.
conn = sqlite3.connect('app.db', isolation_level=None)
Set journal mode to WAL:
conn.execute('pragma journal_mode=wal')
Or another way (just show how to off wal mode)
cur = conn.cursor()
cur.execute('pragma journal_mode=DELETE')
The PRAGMA journal_mode documentation says:
If the journal mode could not be changed, the original journal mode is returned. […]
Note also that the journal_mode cannot be changed while a transaction is active.
So you have to ensure that the database library does not try to be clever and automatically starts a transaction.
Related
I am using sqlite3 with python, and after connecting to the database and creating a table, sqlite3 shows an error when I try to execute a SELECT statment on the table with the name of the databse in it :
con = sqlite3.connect("my_databse")
cur = con.cursor()
cur.execute('''CREATE TABLE my_table ... ''')
cur.execute("SELECT * FROM my_database.my_table") # this works fine without the name of the database before the table name
but I get this error from sqlite3 :
no such table : my_database.my_table
Is there a way to do a SELECT statment with the name of the database in it ?
The short answer is no you can't do this with SQLite. This is because you already specify the database name with sqlite3.connect() and SQLite3 doesn't allow multiple databases in the same file.
Make sure of the database is in the same directory with the python script. In order to verify this you can use os library and os.listdir() method. After connecting the database and creating the cursor, you can query with the table name.
cur.execute('SELECT * FROM my_table')
I am trying to setup a python script to get some data and store it into a SQLite database. However when I am running the script a .fuse_hidden file is created.
On windows no .fuse_hidden file is observed but on ubuntu it generates at each call. The .fuse_hidden file seems to contain some form of sql query with input and tables.
I can delete the files without error during runtime but they are not deleted automatically. I make sure to end my connection to the db when I am finished with the query.
lsof give no information.
I am out of ideas on what to try next to get the files removed automatically. Any suggestions?
Testing
In order to confirm that it is nothing wrong with the code I made a simple script
(Assume there is an empty error.db)
import sqlite3
conn = sqlite3.connect("error.db")
cur = conn.cursor()
create_query = """
CREATE TABLE Errors (
name TEXT
);"""
try:
cur.execute(create_query)
except:
pass
cur.execute("INSERT INTO Errors (name) VALUES(?)", ["Test2"])
conn.commit()
cur.close()
conn.close()
I am getting:
sqlite3.OperationalError: unknown database schema
conn = sqlite3.connect('tick.db', detect_types=sqlite3.PARSE_DECLTYPES, timeout=20,isolation_level=None)
# conn1 = sqlite3.connect('nifty_tick.db', detect_types=sqlite3.PARSE_DECLTYPES, timeout=20,isolation_level=None)
c = conn.cursor()
# c1 = conn1.cursor()
c.execute('PRAGMA journal_mode=wal')
def tick_entry1(inst,timestamp,ltp, bid, ask):
if inst == 12335874:
c.execute('INSERT INTO niftyfut (timestamp, close, bid, ask) VALUES (?,?,?,?)',
(timestamp, ltp, bid, ask))
def on_ticks(ws, ticks):
global c, conn
for t in ticks:
if t['instrument_token'] == 12335874:
timestamp = t['timestamp']
ltp = t['last_price']
inst = t['instrument_token']
try:
tick_entry1(inst,timestamp,ltp)
except:
# print('problem with db')
pass
c.execute('PRAGMA schema.wal_checkpoint(FULL);')
I have tried:
c.execute('PRAGMA schema.wal_checkpoint(FULL);')
and
c.execute('PRAGMA schema.wal_checkpoint(FULL)')
Edit
I just tried:
c.execute('PRAGMA wal_checkpoint(FULL)')
It seems to work. Now wondering if the following should be executed at the start.:
c.execute("PRAGMA wal_autocheckpoint = 0")
The message is saying that no database has a schema named schema.
In the documentation the schema is italic, which indicates that it is symbolic rather than an actual value and would be replaced with an appropriate value, the value is used to distinguish between attached databases.
e.g. if you used
ATTACH DATABASE the_database_path AS database2 /*<<<<<<<<<< THE SCHEMA is database2 */;
then you could use
PRAGMA database2.wal_checkpoint(FULL);
to checkpoint the attached database.
The initial database's schema is main but isn't required as it's the default if no schema is supplied.
Hence why c.execute('PRAGMA wal_checkpoint(FULL)') worked (as would c.execute('PRAGMA main.wal_checkpoint(FULL)')). i.e. the are effectively the same.
If you use c.execute("PRAGMA wal_autocheckpoint = 0") then you will have to manage all the checkpointing as auto-checkpointing will be turned off (note that closing all the database connections checkpoints).
You may wish to consider :-
Disabling the automatic checkpoint mechanism. In its default
configuration, SQLite will checkpoint the WAL file at the conclusion
of any transaction when the WAL file is more than 1000 pages long.
However, compile-time and run-time options exist that can disable or
defer this automatic checkpoint. If an application disables the
automatic checkpoint, then there is nothing to prevent the WAL file
from growing excessively. Write-Ahead Logging - 6. Avoiding Excessively Large WAL Files
I'd suggest not using PRAGMA wal_autocheckpoint = 0 autocheckpointing does not hinder forced checkpointing, other than if a forced checkpoint happens after an auto checkpoint (and all pages are written) and nothing has been updated then it will do nothing (gracefully), otherwise more pages would be written to the database file.
I understand how to open a database using a python script:
import sqlite3 as lite
.
.
con = lite.connect(db_name)
However what I have is a lot of database files made with tsk_loaddb, when I am in sqlite3 I can use the following
SQLite version 3.8.7.2 2014-11-18 20:57:56
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open /work/jmjohnso1/db_project/db_IN60-1020
sqlite> .tables
blackboard_artifact_types tsk_files_derived_method
blackboard_artifacts tsk_files_path
blackboard_attribute_types tsk_fs_info
blackboard_attributes tsk_image_info
tsk_db_info tsk_image_names
tsk_file_layout tsk_objects
tsk_files tsk_vs_info
tsk_files_derived tsk_vs_parts
However, I am not understanding how to do it in python. My question is how do I translate the open command with the sqlite3 python library.
Turns out I had a newline in the name because I was reading from a file. So the following works.
def open_sql(db_folder, db_name, table):
# databases are located at /work/jmjohnso1/db_project
path_name = os.path.join(db_folder,db_name).strip()
con = lite.connect(path_name)
cur = con.cursor()
cur.execute('SELECT * FROM ' + table)
col_names = [cn[0] for cn in cur.description]
rows = cur.fetchall()
print_header(col_names)
print_output(rows)
con.close()
My simple test code is listed below. I created the table already and can query it using the SQLite Manager add-in on Firefox so I know the table and data exist. When I run the query in python (and using the python shell) I get the no such table error
def TroyTest(self, acctno):
conn = sqlite3.connect('TroyData.db')
curs = conn.cursor()
v1 = curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
print v1
conn.close()
When you pass SQLite a non-existing path, it'll happily open a new database for you, instead of telling you that the file did not exist before. When you do that, it'll be empty and you'll instead get a "No such table" error.
You are using a relative path to the database, meaning it'll try to open the database in the current directory, and that is probably not where you think it is..
The remedy is to use an absolute path instead:
conn = sqlite3.connect('/full/path/to/TroyData.db')
You need to loop over the cursor to see results:
curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
for row in curs:
print row[0]
or call fetchone():
print curs.fetchone() # prints whole row tuple
The problem is the SQL statment. you must specify the db name and after the table name...
'''SELECT * FROM db_name.table_name WHERE acctno = ? '''