Question: When does the actual writing to the sqlite3 db file take place and what happens if it is interrupted?
Info:
I have this program in python that I've been working on for a few weeks that uses sqlite3 to store large amounts of data from the simulation it is running. But there are two situations I'm worried about.
We've been having a lot of thunderstorms recently and this has knocked out power a few times, also I'm updating the file that writes to the db with some frequency, and to do so I have to kill the current running thread of the simulation. in both of these cases, especially the former, I worry about what happens when this thread gets interrupted. what if I happen to interrupt it or lose power while it is writing from the log file into the db? will the information just not get there? will it get there but corrupt? will it corrupt the whole sqlite3 db file?
Basically I want to know when does the data actually get writen to the file and not just the log file. and if this writing process does not finish for any reason, what happens to the log and the db file?
~n
SQLite uses atomic commits, so either everything or nothing is committed.
If you're concerned about the database being left in an invalid state, you need to make sure you wrap the entire "transitional" state in a BEGIN TRANSACTION ... COMMIT block.
The fine details of writing to the journal files, etc. (including through failures) are in the document "File Locking and Concurrency in SQLite Version 3".
Related
I am trying to read an sqlite3 db into memory for further processing using python like so:
con = sqlite3.connect(':memory:')
print('Reading sqlite file into memory... ', end='', flush=True)
src_con.backup(con) # <---- This line seems to hang, SOMETIMES
print('Completed')
src_con.close()
cur = con.cursor()
While the code works fine most of the times, occasionally I observe a hang in the line src_con.backup(con).
This db is dumped by a process that I dont own on a shared network disk.
Here are my observations based on the advice I have found elsewhere on the internet:
fuser filename.db does not show any processes from my user account
sqlite3 filename.db "pragma integrity_check;" returns Error: database is locked
the md5sum of the filename.db (the db that hangs) and its copy filename2.db (doesn't hang) are identical. So is the OS locking the db - because the lock info is not in the DB file itself?
This locked DB appears to occur when the process creating it did not exit cleanly.
Copying the db out (into say, filename2.db) and reading it is a workaround - but I want to understand whats going on and why - and if there is a way to read the locked DB in read-only mode.
The stance of the SQLite developers about using a database stored in a networked file system is, essentially, "Don't. Use a client-server database instead."
This simple, "remote database" approach is usually not the best way to use a single SQLite database from multiple systems, (even if it appears to "work"), as it often leads to various kinds of trouble and grief. Because these problems are inevitable with some usages, but not frequent or repeatable, it behooves application developers to not rely on early testing success to decide that their remote database use will work as desired.
Sounds like you're running into one of those problems. What I suspect is happening, because of
This locked DB appears to occur when the process creating it did not exit cleanly.
is that the networked file system you're using isn't detecting that the creating process's lock on the database is gone due to the process no longer existing, so the backup is just waiting for the lock to be released so that it can acquire it.
Even if you figure out how to broadcast that the lock is available from the creating process's computer to others with that file mounted, there's bound to be other subtle and not-so-subtle problems that pop up now and then. So your best bet is to follow the official advice:
If your data lives on a different machine from your application, then you should consider a client/server database.
You've to close the database before you backup or restore. Btw, do you have right permissions to the source (src)? This related link might help:
How to back up a SQLite db?
I'm using the python SQlite driver and trying to stream read from Table1 putting each row through a python function which computes something and creates another object that I am writing to another table, Table2.
But I'm running locking issues. Shouldn't SQLite be able to so this easily? Is there some special mode I need to turn on?
I can get this to work if I read the whole stream into memory first and then write the other table by looping over the list but that isn't streaming and has issues if Table1 can't fit into memory. Isn't there a way to permit this basic kind of streaming operation?
Update
I tried the following and perhaps it's the answer
db = sqlite3.connect(file, timeout=30.0, check_same_thread=False,
isolation_level=None)
db.execute('pragma journal_mode=wal;')
That is I added isolation_level=None and the pragma command. This puts it into WAL (write ahead logger) mode. It seems to avoid the locking issue for my use case anyway.
Sqlite does not lock tables but the whole database. So if I understand you correctly it will not be possible with sqlite.
For reference take a look at sqlite locks.
It mentiones:
An EXCLUSIVE lock is needed in order to write to the database file. Only one EXCLUSIVE lock is allowed on the file and no other locks of any kind are allowed to coexist with an EXCLUSIVE lock. In order to maximize concurrency, SQLite works to minimize the amount of time that EXCLUSIVE locks are held.
A possible workaround might be to create one transaction for all your reading and writing operations. Sqlite will only lock the database when the transaction is actually commited. See here for transactions.
I am using python and sqlalchemy to manage a sqlite database (in the future I plan to replace sqlite with postgres).
The operations I do are INSERT, SELECT and DELETE and all these operations are part of a python script that runs every hour.
Each one of these operation can take a considerable amount of time due to the large amount of data.
Now in certain circumstances the python script may be killed by an external process. How can I make sure that my database is not corrupted if the script is killed while reading / writing from the DB?
Well, you use a database.
Databases implement ACID properties (see here). To the extent possible, these guarantee the integrity of the data, even when transactions are not complete.
The issue that you are focusing on is dropped connections. I think dropped connections usually result in a transaction being rolled back (I'm not sure if there are exceptions). That is, the database ignores everything since the last commit.
So, the database protects you against internal corruption. Your data model might become invalid, if the sequence of operations is stopped at an arbitrary place. The solution to this is to wrap such operations into a transaction, so the transaction is rolled back.
There is a (small) danger of databases getting corrupted when the hardware or software they are running on suddenly "disappears". This is rare and there are safeguards. And, this is not the problem that you are concerned with (unless your SQLite instance is part of your python process).
I am trying to implement a "record manager" class in python 3x and linux/macOS. The class is relatively easy and straightforward, the only "hard" thing I want is to be able to access the same file (where results are saved) on multiple processes.
This seemed pretty easy, conceptually: when saving, acquire an exclusive lock on the file. Update your information, save the new information, release exclusive lock on the file. Easy enough.
I am using fcntl.lockf(file, fcntl.LOCK_EX) to acquire the exclusive lock. The problem is that, looking on the internet, I am finding a lot of different websites saying how this is not reliable, that it won't work on windows, that the support on NFS is shaky, and that things could change between macOS and linux.
I have accepted that the code won't work on windows, but I was hoping to be able to make it work on macOS (single machine) and on linux (on multiple servers with NFS).
The problem is that I can't seem to make this work; and after a while of debugging and after the tests passed on macOS, they failed once I tried them on the NFS with linux (ubuntu 16.04). The issue is an inconsistency between the informations saved by multiple processes - some processes have their modifications missing, which means something went wrong in the locking and saving procedure.
I am sure there is something I am doing wrong, and I suspect this may be related to the issues that I read about online. So, what is the proper way to deal multiple access to the same file that works on macOS and linux over NFS?
Edit
This is what the typical method that writes new informations to disk looks like:
sf = open(self._save_file_path, 'rb+')
try:
fcntl.lockf(sf, fcntl.LOCK_EX) # acquire an exclusive lock - only one writer
self._raw_update(sf) #updates the records from file (other processes may have modified it)
self._saved_records[name] = new_info
self._raw_save() #does not check for locks (but does *not* release the lock on self._save_file_path)
finally:
sf.flush()
os.fsync(sf.fileno()) #forcing the OS to write to disk
sf.close() #release the lock and close
While this is how a typical method that only read info from disk looks like:
sf = open(self._save_file_path, 'rb')
try:
fcntl.lockf(sf, fcntl.LOCK_SH) # acquire shared lock - multiple writers
self._raw_update(sf) #updates the records from file (other processes may have modified it)
return self._saved_records
finally:
sf.close() #release the lock and close
Also, this is how _raw_save looks like:
def _raw_save(self):
#write to temp file first to avoid accidental corruption of information.
#os.replace is guaranteed to be an atomic operation in POSIX
with open('temp_file', 'wb') as p:
p.write(self._saved_records)
os.replace('temp_file', self._save_file_path) #pretty sure this does not release the lock
Error message
I have written a unit test where I create 100 different processes, 50 that read and 50 that write to the same file. Each process does some random waiting to avoid accessing the files sequentially.
The problem is that some of the records are not kept; at the end there are some 3-4 random records missing, so I only end up with 46-47 records rather than 50.
Edit 2
I have modified the code above and I acquire the lock not on the file itself, but on a separate lock file. This prevents the issue that closing the file would release the lock (as suggested by #janneb), and makes the code work correctly on mac. The same code fails on linux with NFS though.
I don't see how the combination of file locks and os.replace() can make sense. When the file is replaced (that is, the directory entry is replaced), all the existing file locks (probably including file locks waiting for the locking to succeed, I'm not sure of the semantics here) and file descriptors will be against the old file, not the new one. I suspect this is the reason behind the race conditions causing you to lose some of the records in your tests.
os.replace() is a good technique to ensure that a reader doesn't read a partial update. But it doesn't work robustly in the face of multiple updaters (unless losing some of the updates is ok).
Another issues is that fcntl is a really really stupid API. In particular, the locks are bound to the process, not the file descriptor. Which means that e.g. a close() on ANY file descriptor pointing to the file will release the lock.
One way would be to use a "lock file", e.g. taking advantage of the atomicity of link(). From http://man7.org/linux/man-pages/man2/open.2.html:
Portable
programs that want to perform atomic file locking using a
lockfile, and need to avoid reliance on NFS support for
O_EXCL, can create a unique file on the same filesystem (e.g.,
incorporating hostname and PID), and use link(2) to make a
link to the lockfile. If link(2) returns 0, the lock is
successful. Otherwise, use stat(2) on the unique file to
check if its link count has increased to 2, in which case the
lock is also successful.
If it's Ok to read slightly stale data then you can use this link() dance only for a temp file that you use when updating the file and then os.replace() the "main" file you use for reading (reading can then be lockless). If not, then you need to do the link() trick for the "main" file and forget about shared/exclusive locking, all locks are then exclusive.
Addendum: One tricky thing to deal with when using lock files is what to do when a process dies for whatever reason, and leaves the lock file around. If this is to run unattended, you might want to incorporate some kind of timeout and removal of lock files (e.g. check the stat() timestamps).
Using randomly named hard links and the link counts on those files as lock files is a common strategy (E.g. this), and arguable better than using lockd but for far more information about the limits of all sorts of locks over NFS read this: http://0pointer.de/blog/projects/locking.html
You'll also find that this is a long standing standard problem for MTA software using Mbox files over NFS. Probably the best answer there was to use Maildir instead of Mbox, but if you look for examples in the source code of something like postfix, it'll be close to best practice. And if they simply don't solve that problem, that might also be your answer.
NFS is great for file sharing. It sucks as a "transmission" medium.
I've been down the NFS-for-data-transmission road multiple times. In every instance, the solution involved moving away from NFS.
Getting reliable locking is one part of the problem. The other part is the update of the file on the server and expecting the clients to receive that data at some specific point-in-time (such as before they can grab the lock).
NFS isn't designed to be a data transmission solution. There are caches and timing involved. Not to mention paging of the file content, and file metadata (e.g. the atime attribute). And client O/S'es keeping track of state locally (such as "where" to append the client's data when writing to the end of the file).
For a distributed, synchronized store, I recommend looking at a tool that does just that. Such as Cassandra, or even a general-purpose database.
If I'm reading the use-case correctly, you could also go with a simple server-based solution. Have a server listen for TCP connections, read messages from the connections, and then write each to file, serializing the writes within the server itself. There's some added complexity in having your own protocol (to know where a message starts and stops), but otherwise, it's fairly straight-forward.
I have two programs: the first only write to sqlite db, and the second only read. May I be sure that there are never be some errors? Or how to avoid from it (in python)?
Yes, it is generally safe.
According to the SQLite FAQ:
Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.
Since only one of your processes is writing, this is fine.
Exception: This may break if your .sqlite file is stored on a network drive (Windows or NFS).
generally, it is safe if there is only one program writing the sqlite db at one time.
(If not, it will raise exception like "database is locked." while two write operations want to write at the same time.)
By the way, it is no way to guarantee the program will never have errors. using Try ... catch to handle exception will make the program much safer.