I am using sqlite combined with tkinter to write and delete records within my Python program. The deletion works perfectly fine in my program and also when I restart the program, the record does not exist anymore.
However, I always cross check using the Linux standard software DB Browser for SQLite and look at my SQL Table. Strangely, all records still exist in the DB Browser. Now I am wondering, why's that? Why is it gone within my Python sqlite queries but not in the DB Browser? Somehow the records are still there. How can I completely destroy my records?
For deletion I use:
(The user can chose a specific entry using a listbox. Eventually, I "translate" the selected item into its specific ID and trigger the deletion.)
self.c.execute("DELETE FROM financial_table WHERE ID=?",(entry,))
self.conn.commit()
For my query I use:
(I query the data for a specific year and month.)
self.c.execute("SELECT ID, Date, Item, Price FROM financial_table WHERE strftime('%Y-%m', Date) = '{}' ORDER BY Date ".format(date))
single_dates = self.c.fetchall()
Thank you very much for your help.
The solution to my question is: I am stupid!
I was tired yesterday evening and looked at the wrong sql file in a subfolder which had the same name than the one from my python program. So it is actually working. Please excuse my stupidity.
#Bruceskyaus
Despite my stupidity I learned from your answer, especially the try ... except block. I am going to implement it. Thanks.
You may have an problem with controlling transactions on your database, but it could also be the connection itself. Make sure you don't have any uncommitted DML statements on a different connection (i.e. an INSERT, UPDATE or DELETE in your DB Browser that wasn't committed), this could cause the conn.commit() to fail. With SQLite, an uncommitted transaction could lock the entire database - for a brief period of time.
Try ensuring that there is a new cursor for the delete statement and call conn.close() after the conn.commit(). Before you execute the code, make sure that no other connections are accessing the database - including the DB Browser. Only check in the DB Browser when you have shut down the application (for this test). This eliminates multithreading or locking as a possible cause. See also SQLite - Data Persistence and SQLite - Controlling Transactions
It is also helpful to trap all errors for DML statements using a try...except block. Something like this:
import sqlite3
try:
self.conn = sqlite3.connect('mydb.db')
self.c = conn.cursor()
self.c.execute("DELETE FROM financial_table WHERE ID=?",(entry,))
self.conn.commit()
except sqlite3.Error as e:
print("An error occurred:", e.args[0])
finally:
self.conn.close()
Related
Situation
I have a plotly-dash application running in a docker container (based on python3.7-slim).
The app is accessing a postgres database and visualizes the queried data.
However, if the app has not been used for some time (I would estimate around 24-48 hours. We first noticed this issue on mondays after nobody used the app during the weekend) i.e. if no data has been queried from the database, the app freezes and the logs show some errors related to the database.
I cannot fully access the logs, but they contain this error:
AttributeError: 'Connection' object has no attribute '_Connection_connection'
and in the following, all the pieces of code which tried to query data from the database are stated (but not what exactly went wrong).
The problem was always solved with a restart of the app (and thus a new connection to the database)
Assumption
As stated above, this always occured after a period of inactivity. So my assumption is, that the engine disconnects after some idle time
Code Sample
For accessing the database, I have a DatabaseConnection class. The relevant part of the code contais something like this:
from sqlalchemy import create_engine
...
engine = create_engine(f"postgresql+psycopg2://{user}:{passw}#{url}:{port}/{db_name}")
self.engine = engine.connect()
...
Question
What is the best solution for overcoming the issue of the disconnect after some inactivity?
How could I possibly check whether the database connection is still active and if not, reconnect it somehow?
-Is there a better way the access the database than through an engine-object?
Is there something wrong with my approach in general?
Please let me know if you require further information. Thanks in Advance.
There is an error in my code. It should be
self.engine = create_engine(f"postgresql+psycopg2://{user}:{passw}#{url}:{port}/{db_name}")
and the second line should be omitted. I misunderstood what engine.connect() is doing: It returns a Connection object (not an engine, as the attribute name suggests).
Then, for each query I execute, I use the context manager like this:
with self.engine.connect() as conn
table1 = pd.read_sql_table("table1", con=conn)
That way, the Connection object is closed after it has been used. But the engine object may open new connections whenever necessary.
In my previous solution, die Connection was killed after some Idle time.
(Based on this GitHub Discussion)
Ours is flask app, sql alchemy, sql server based stack. We are seeing a weird issue issue in our production.
While doing an insert operation, we are getting
sqlalchemy.orm.exc:ObjectDeletedError
In database we see that the entity did not got created. We have a trigger function on our insert which has lot of PRINT statements in it.
Sequence of events -
obj = Object()
session.add(obj)
session.commit()
obj.id # sqlalchemy.orm.exc:ObjectDeletedError
Issue is not seen if we remove the PRINT statements.
Whats the co relation of print statements in triggers and sql alchemy?
Also the issue is not seen in our test envs but only on production.
Trying to simply update existing row in database (running on web2py),
but always getting database locked error..
Error message:
<class 'sqlite3.OperationalError'> database is locked
My setup
in models/db.py I create database and it works when using database administration (can insert, update using the web interface)
db.define_table('mytest', Field('name', 'string'))
I have added 1 row to mytest, using the web interface (so its not empty)
in controllers/test.py i have simple code to get first item and try to update the value, there it fails (I open the page is browser and it gives the internal error, with link to error log)
def index():
# connect
db = DAL('sqlite://storage.sqlite',pool_size=10,auto_import=True)
# get first record
record = db(db.mytest).select().first()
# try to update it.. database locked error here
record.update_record(name="asdfg")
# just in case needed?
db.commit()
db.close()
return "test"
Software
WinPython2.7
Running win2py.py (2.14.6) manually using Spyder ide
windows8
What i've tried so far
Different DAL settings, poolsize, without autoimport..
Close all web2py admin tools/tabs
Create new database
Restart web2py
Restart pc
Error log: http://pastebin.com/2WMWypt6
Current workaround:
- Create New Application, exact same code seems to work there
Solution was: by #GauravVichare
- Remove this line from controller (its already defined in db.py)
db = DAL('sqlite://storage.sqlite',pool_size=10,auto_import=True)
Check Whether there is no other connection (to sqlite db) open on your machine, if web2py shell is open, close it.
Check DAL is defined only once or not. Define DAL only in models/db.py, no need to define it again in controller.
Every variable defined in models is visible in controllers.
You must have defined DAL in models/db.py and you are defining once again in controller, so you have two connection open for SQLite db. Thats why you are getting error 'database is locked'.
My Suggestion is
1.First of all save the code when u make some changes.
2.Aftr saving u r new code try to reload web2py.exe
then run web2py so that u wont get Databaselocked error.
3.Dont ever create tables in Sqlite database before.
4.once u start running web2py and starts server and when ever u enter data into forms it automatically creates the tablesin sqlite database.
Try using myRecord instead of Record, since it may be a reserved word.
I know User has given people issues in web2py. I would just tend to stay away from very generic aliases.
Otherwise, is there anything currently in the db? If it is empty you would receive and error.
It might be better to :
myRecord = record = db(db.mytest).select().first()
if myRecord:
myRecord.update_record(name="asdfg")
else:
[insert statement here]
I've been trying using PyMysql and so far everything i did worked (Select/insert) but when i try to update it just doesn't work, no errors no nothing, just doesn't do anything.
import pymysql
connection = pymysql.connect(...)
cursor = connection.cursor()
cursor.execute("UPDATE Users SET IsConnected='1' WHERE Username='test'")
cursor.close()
connection.close()
and yes I've double checked that Users, IsConnected and Username are all correct and test does exist (SELECT works on it)
what's my problem here?
When you execute your update, MySQL is implicitly starting a transaction. You need to commit this transaction by calling connection.commit() after you execute your update to keep the transaction from automatically rolling back when you disconnect.
MySQL (at least when using the InnoDB engine for tables) supports transactions, which allow you to run a series of update/insert statements then have them either all commit at once effectively as a single operation, or rollback so that none are applied. If you do not explicitly commit a transaction, it will rollback automatically when you close your connection to the database.
In fact, what #JoeDay has described above has little to do with default MySQL transaction's behaviour. MySQL by default operates in auto-commit mode and normally you don't need any additional twist to persist your changes:
By default, MySQL runs with autocommit mode enabled. This means that as soon as you execute a statement that updates (modifies) a table, MySQL stores the update on disk to make it permanent. The change cannot be rolled back.
PEP-249's (DB API) authors decided to complicate things and break Zen of Python by making a transaction's start implicit, by proposing auto-commit to be disabled by default.
What I suggest to do, is to restore MySQL's default behaviour. And use transactions explicitly, only when you need them.
import pymysql
connection = pymysql.connect(autocommit=True)
I've also written about it here with a few references.
I'm trying to insert all values of a list to my sqlite3 database. When I simulate this query by using the python interactive interpreter, I am able to insert the single value to DB properly. But my code fails while using an iteration:
...
connection=lite.connect(db_name)
cursor=connection.cursor()
for name in match:
cursor.execute("""INSERT INTO video_dizi(name) VALUES (?)""",(name,))
connection.commit()
...
error:cursor.execute("""INSERT INTO video_dizi(name) VALUES (?)""",(name,))
sqlite3.OperationalError: database is locked
Any way to overcome this problem?
Do you have another connection elsewhere in your code that you use to begin a transaction that is still active (not committed) when you try to commit the operation that fails?
As this error can happen because you have opened your site.db or database file in DBbrowser type application to view in interactive database interface. Just close that it will work fine.
Because your database is use by another process or connection. If you need real concurrency, use a real RDBMS.