I'm using Django Python framework, and MySQL DBMS.
In the screenshot below, I'm creating the new_survey_draft object using the SurveyDraft.objects.create() as shown, assuming that it should create a new row in the surveydraft DB table, but as also shown in the screenshot, and after debugging my code, the new_survey_draft object was created with id=pk=270 , while the DB table shown in the other window to the right doesn't have the new row with the id=270.
Even when setting a break point in the publish_survey_draft() called after the object instantiation, I called the SurveyDraft.objects.get(pk=270) which returned the object, but still there is not id=270 in the DB table.
And finally, after resuming the code and returning from all definitions, the row was successfully added to the DB table with the id=270.
I'm wondering what's happening behind the seen, and is it possible that Django stores data in objects without persisting to DB on real-time, and only persists the data all together on some later execution point?
I've been stuck in this for hours and couldn't find anything helpful online, so I really appreciate any advice regarding the issue.
After digging deep into this issue, I just found that there is a concept called Atomic Requests that's enabled in my Django project by setting the ATOMIC_REQUESTS to True in the settings.py under the DATABASES dictionary as explained here
It works like this. Before calling a view function, Django starts a
transaction. If the response is produced without problems, Django
commits the transaction. If the view produces an exception, Django
rolls back the transaction.
That's why the changes were not persisting in the database while debugging my code using break points, since the changes will only be committed to the DB once the successful response is returned.
Related
I have many models with relational links to each other which I have to use. My code is very complicated so I cannot keep session alive after a query. Instead, I try to preload all the objects:
def db_get_structure():
with Session(my_engine) as session:
deps = {x.id: x for x in session.query(Department).all()}
...
return (deps, ...)
def some_logic(id):
struct = db_get_structure()
return some_other_logic(struct.deps[id].owner)
However, I get the following error anyway regardless of the fact that all the objects are already loaded:
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Department at 0x10476e780> is not bound to a Session; lazy load operation of attribute 'owner' cannot proceed
Is it possible to link preloaded objects with each other so that the relations will work after session get closed?
I know about joined queries (.options(joinedload(), but this approach leads to more code lines and bigger DB request, and I think this should be solved simpler, because all the objects are already loaded into Python objects.
It's even possible now to request the related objects like struct.deps[struct.deps[id].owner_id], but I think the ORM should do this and provide shorter notation struct.deps[id].owner using some "cached load".
Whenever you access an attribute on a DB entity that has not yet been loaded from the DB, SQLAlchemy will issue an implicit SQL statement to the DB to fetch that data. My guess is that this is what happens when you issue struct.deps[struct.deps[id].owner_id].
If the object in question has been removed from the session it is in a "detached" state and SQLAlchemy protects you from accidentally running into inconsistent data. In order to work with that object again it needs to be "re-attached".
I've done this already fairly often with session.merge:
attached_object = new_session.merge(detached_object)
But this will reconile the object instance with the DB and potentially issue updates to the DB if necessary. The detached_object is taken as "truth".
I believe you can do the reverse (attaching it by reading from the DB instead of writing to it) by using session.refresh(detached_object), but I need to verify this. I'll update the post if I found something.
Both ways have to talk to the DB with at least a select to ensure the data is consistent.
In order to avoid loading, issue session.merge(..., load=False). But this has some very important cavetas. Have a look at the docs of session.merge() for details.
I will need to read up on your link you added concerning your "complicated code". I would like to understand why you need to throw away your session the way you do it. Maybe there is an easier way?
So I am right now developing a backend API using GraphQL and either Python with SQLAlchemy or Node.js with Sequelize to write all the data into a SQLite database.
I am still not sure what backend I will end up with. Maybe this question and the answer to my question will lead me to using one or the other.
So my app will have a function where you add dart throws (like double 20, triple 19, ...) into a table with throws. Depending on the game chosen and the conditions chosen this throw will either substract from a total count or maybe be added into a counter table or something like that.
Everytime a throw gets added there will be a check if the game is won and some other checks as well. Those might also write (commit) data to the database.
Now, if the user types in the wrong number / or the automatic darts machine will discover the wrong throw number (malfunction) there is a throw and several changed data in the database which I will have to rollback.
In darts scorer apps you will find in the app store either ios or android you always will have a undo button which will erase the throw from the database and all the "effects" caused by entering the wrong throw.
As a former mssql administrator my first idea was transactional logs and rollback function. But after searching the net for a while I understand both SQLAlchemy and Sequelize will rollback transactions only on exception.
Can I purposely rollback the last "complete" transaction (meaning all changes which for example a function will make by the end of the function) by hitting a button / calling a rest api path like '/undo' and if so are there any code snippets for either SQLAlchemy or Sequelize I am missing?
Best regards,
Patrick
edit:
collection of provided answers:
command pattern (python)
According to Can I rollback a transaction I've already committed? (data loss) and other sources, no, you cannot rollback a committed transaction. Also this is DB dependent, so doesn't have to do much with ORM or language.
I would recommend to implement the Command pattern, which is great for Undo. So basically do the revert from code, not from the database.
I'm facing a problem which I have no more ideas how to resolve.
I have need to test data which is returned by direct query from database.
During execution of TestCase django.db.connection.cursor() is returning data from main database, not from test one, which contain fixtures prepared for this test purposes.
I've tried to use both TestCase and TransactionalTestCase.
I've tried debugging, checking variables values and found out that connection is pointed onto test database for sure.
Do you know why it's returning data from main database? Is there any case when Django is copying data from main database to this created for tests purposes?
I'm using: Python 3.6.5, Django 2.1, pytest 4.6.3, pytest-django 3.5
Thanks in advance for any support.
Follow-up
Dears,
That problem occurs only while trying to perform custom raw SQL Query directly inside Test Case. Retrieving objects by standard Django QuerySets works fine.
Do you have any idea why this specific way to retrieve data from db not working during test execution?
I found an answer in Django documentation:
If your code attempts to access the database when its modules are compiled, this will occur before the test database is set up, with potentially unexpected results.
Still - if you know any way to avoid affecting tests by production data while raw sql queries are performed I would love to know how to do it.
I'm new to Flask and web development in general. I have a Flask web-application that is using SQLAlchemy, is it ok to put session.rollback at the beginning of the app in order to keep it running even after a transaction fails?
I had a problem with my website when it stopped working after I was attempting to delete records of one table. The error log showed that the deletion failed due to entries in another table still referencing these records as their foreign key. The error log suggested using session.rollback to rollback this change, so I put it at the beginning of my app just after binding my database and creating the session and my website worked. This gave me the hint to leave that line there. Is my move right, safe and ok? Can anyone tell me what is the correct thing to do if this is somewhat endangering the functionality or logic of my website by any chance?
I'd say by that you are by definition cargo cult coding and should try to determine why you're finding these errors in the first place instead of just including a bit of code for a reason you don't understand.
The problem you're describing is the result of using foreign keys to ensure data integrity in your database. Typically SQLAlchemy will nullify all of the depending foreign keys, but since I don't know anything about your set up I can't explain why it wasn't. It is perhaps a difference between databases.
One massive problem with putting the rollback at the beginning of a route (or the entire global app) is that you might rollback data which you didn't want to. You haven't provided an MVCE so no one can really help you debug your problem.
Cargo cult coding in circumstances like this is understandable, but it's never a good practice. To solve this problem, investigate the cascades in SQLAlchemy. Also, fire up your actual SQL db interface and look at the data's structure, and set SQLALCHEMY_ECHO = 1 in your config file to see what's actually getting emitted.
Good luck!
You should not use the rollback at the beginning but when a database operation fails.
The error is due to an integrity condition in your database. Some rows in your table are being referenced by another table. So, you have to remove referencing rows first.
I have a django model, TestModel, over an SQL database.
Whenever I do
TestModel.objects.all()
I seem to be getting the same results if I run it multiple times from the same process. I tested that by manually deleting (without using ANY of the django primitives) a line from the table the model is constructed on, the query still returns the same results, even though obviously there should be less objects after the delete.
Is there a caching mechanism of some sort and django is not going to the database every time I want to retrieve the objects?
If there is, is there a way I could still force django to go to the database on each query, preferably without writing raw SQL queries?
I should also specify that by restarting the process the model once again returns the correct objects, I don't see the deleted ones anymore, but if I delete some more the issue occurs again.
This is because your database isolation level is repeatable read. In a django shell all requests are enclosed in a single transaction.
Edited
You can try in your shell:
from django.db import transaction
with transaction.autocommit():
t = TestModel.objects.all()
...
Sounds like a db transaction issue. If you're keeping a shell session open while you separately go into the database itself and modify data, the transaction that's open in the shell won't see the changes because of isolation. You'll need to exit and reload the shell to get a new transaction before you can see them.
Note that in production, transactions are tied to the request/response cycle so this won't be a significant issue.