I'm currently working on a django application. I can't add an element to my database on the admin view. I fill all the information but when I click on save button but the operation doesn't finish and I get a timeout. I use sqlite3 as database.
My question is there any one that know the origin of this problem. If not how could I investigate the problem. When I worked with other language (Java, C ...etc) when I have a problem I can use a debugger. What are the options I have?
This Problem can occur because of following reasons:
(Less Probable) You computation code is too Slow: Which is a rarity because the Timeout is set to about 1 minute or so, and code doesn't take that time to Execute
Your app is waiting on some external resource but it is not Responding. For this you will have to check for the Django logs and check if some external resource error is there
(Most Probable) Database taking too much time: This can occur either because:
App can't connect to Database: For this you have to check database logs OR try and connect manually with database through python manage.py dbshell
DB Query Taking so much time to execute: You can test this by checking database logs for how much time a query is taking OR you can connect manually via dbshell and make the same query there
Your can also use tools Like django-profiler , Django debug toolbar etc for debugging purposes. and for native python code python debugger
Related
I have a processing engine built in python and a driver program with several components that uses this engine to process some files.
See the pictorial representation here.
The Engine is used for math calculations.
The Driver program has several components.
Scanner keeps scanning a folder to check for new files, if found makes entry into DB by calling a API.
Scheduler picks new entries made by scanner and schedules them for processing (makes entry into 'jobs' table in DB)
Executer picks entries from job table and executes them using the engine and outputs new files.
All the components run as separate python process continuously. This is very in efficient, how can I improve this? The use of Django is to provide a DB (so the multiple processes could communicate) and keep a record of how many files are processed.
Next came a new requirement to manually check the processed files for errors so a UI was developed for this. Also the assess to the engine was to be made API based. See the new block diagram here
Now the entire thing is a huge mess in my opinion. For start, the Django now has to serve 2 different sets of API - one for the UI and other for the driver program. If the server stops, the UI stops working and also the Driver program stops working.
Since the engine is API based there is a huge amount of data passed to it in the request. The Engine takes several minutes (3 to 4) to process the files and most of the time the request to engine get timeout. The Driver program is started separately from terminal and it fails if Django server is not running as the DB APIs are required to schedule jobs and execute the jobs.
I want to ask what is the beast way to structure such projects.
should I keep the Engine and driver program logic inside Django? In this case how do I start the driver program?
Should I keep both of them outside Django, in which case how do I communicate with Django such that even if the Django server is down I can still keep processing the files.
I would really appreciate any sort of improvement ideas in any of the areas.
i'ev developed a chatbot web application using Django and python3 on web faction server. Basically the chatbot interacts with a store's users as a customer server.It uses the REST API to POST the user input and GET to display the chatbot output and a python file to process the input and find the output.
How the chatting works:
1. chatboy.js: it first POST the user input in the API then run the python file chabot.py.
2. chabot.py: a python file that connects to the Django backend db.sqlite3 and the conversation.sqlite3. So it select the user input from db.sqlite3 then select the matching output in the conversation.sqlite3. Finally the file will update the chatbot output in the db.sqlite3.
3. chatboy.js: will GET the last chatbot output and display it.
In development stage when i was testing the application in my local server everything works fine but problems came we i deployed the Django project in the internet. Mostly and the main ERORR that stops the application form working is :
The database is locked
I did many research and found out that:
sqlite3 is not for production and only for small or stand alone application.
sqlite3 have a multiple threads problem.
not good for realtime chatting application.
Some suggestion said either
switch all the sqlite3 to Mysql ( but how? and how i can change the sql queries in the python file to fit Mysql)
use a fast key value store like Reddis ( does it mean to make it the backend db form my project or juts the conversation.sqlite3?)
please help me because it is very frustrating and i had this error for a long time and i couldn't found and solution for the problem.
thank you,
PS: i'm using Django 10 , python3 , sqlite3, web faction server
You only need to update your settings file to change the DATABASES setting from sqlite3 to mysql or postgresql. But before you do that you will need to create a database in your Webfaction Control Panel (more information here).
I'm looking for a way to initialize a database with a set of preliminary data. I've been looking around, but haven't seem to find anything that exactly matches what I'd like to do. Any suggestions?
The Datastore Administration allow to copy data to another app. But if you need this tool for testing your app in local, as I suppose, you could write a script.
You can also do this programatically. Write a function that initializes your database and call it from inside a dedicated URL handler, e.g.
http://myapp.com/prepareDb
See here how to configure this URL so that only admins of the app have permission to access it.
An HTTP request coming from a user has 60 seconds to complete before it is aborted by app engine. If, for some reason, you need more than 60 seconds to prepare your database, do it from a cron job (which has 10 minutes to finish).
I'm using SQLAlchemy with Flask as shown here: http://flask.pocoo.org/docs/patterns/sqlalchemy/
I have a Selenium test suite that first runs with Firefox and then with Chrome.
Before the start of tests on each browsers, tables in the test database (PostgreSQL) are dropped and created.
It runs perfectly for the first browsers, but for the second browser the SQL create / drop attempt just freezes and no errors are shown.
I believe this is because of open SQLAlchemy sessions, is that correct?
I believe this is because of open SQLAlchemy sessions, is that correct?
That's most likely the case. To confirm it, connect to the postgres database and run SELECT * FROM pg_stat_activity;
I'm not sure how you handle the DB creation/dropping but you may want to call dispose() and possibly recreate() on the SQLAlchemy connection pool, after making sure that any checked out connection has been returned (for example, with session.close()).
This is something that happens to me too on running Flask unittest with SQLAlchemy and Postgres. Many a times, the culprit is an exception, which did not propagate upwards and got stuck. This exception also stops the test from cleaning up properly and hence the freeze.
If you are creating a test suite then call debug method on the suit and it will show the exception. Linked the docs of this method here.
Your observation of an open Sqlalchemy session can also be a reason. I will test a theory of mine based on this observation tomorrow. If it clears some doubts then I will post here.
Look at this answer that shows how you can trigger debugger on exception. Maybe it can help pinpoint problem.
After fighting with different things here and there, I was finally able to get BottlePY running on Apache and run a MongoDB powered site. I am used to running Django apps, so I will be relating to that a bit in my question.
The Problem
Every time a page is loaded via BottlePY, the connection to the MongoDB database located on MongoHQ.com needs to be re-authenticated (meaning it probably had to reconnect).
What I Found
I attached a db.keep_alive() function to the top of each model function, so that before any mongodb query is run, it trys to run a simple query. If it fails, it catches the OperationFailure or AutoReconnect errors and then calls the db.authenticate() function. After it reauthenticates, I have it add a log to a logs db to monitor how often it needs to reauthenticate. Currently, it needs to reauthenticate on every page load (that requires running a query). This isn't right.
Difference from Django
I use this same concept in django, and have found that the db connection only needs to be authenticated after 10-15 minutes of no queries being run.
I don't understand why creating a pymongo connection in django would be different from creating one in bottle, since I am using the same driver, functions and methods. I am not using any ORMS or anything like that either.
Versions
Bottle: 0.9.dev
Django: 1.2.1 final
PyMongo: 1.8
I appreciate the help!
Update: A friend was able to take a quick look and noticed the following that may help with answering my question.
It appears that each request is
launching a new Python process, as
opposed to Django, in which a single
process remains running for a long
period of time.
This just ended up to be a weird thing between Bottle and MongoHQ. No real solution was found, but I couldn't recreate it with other frameworks. Any other ideas are appreciated.
does your apache xxx.conf contain something like:
WSGIDaemonProcess project user=mysite group=www-data processes=5 threads=1
WSGIProcessGroup project
I think most important should be threads=1