I am working on building a gui/dashboard type of thing thru which I take input from user.. and when user press submits.. I gather response and fire a job in back end.
What I am hoping to achive is..
When the user press submits:
and that long job is being processed, I show something like: "Your job has been submitted succesfully) submitted
and when it is finished.. I refresh the page to take user to that page.
Here is how my route.py snippet looks like
#app.route('/',methods=['POST'])
def get_data():
data = request.form
for data_tuple in data:
requests_to_gb[data_tuple] = data[data_tuple]
flag = execute_request(requests_to_gb) <---Fires a job
if flag:
flash("Your request has been submitted")
else:
flash("request could not be completed!")
return render_template('request_submitted.html')
But the issue is.. that line where i execute_request() takes a long time to process.. and everything is halted until that is finished?
How do I handle this?
And how do i automatically refresh to new page as well?
Use celery which is a distributed task queue. The quickstart guide should get you going.
In a nutshell, it allows you to offload tasks to workers that run in the background so that you don't block your main user interface (which is what you are trying to prevent).
The good news is that it is very easy to integrate with flask (or django, or anything else really) since its written in Python.
Related
i have some project working with Flask and Jinja2, and the web ui just table like this image
that all Routes is triggering program backend for scraping some website, this web is working fine. but i wonder how make it more flexible.
i just thought of making auto-run that all routes with one button, and have their own status.
that column will have Status and their status like (running, done, stop, not running etc.) but i cannot imagine that logic.
i already create for auto-run and work fine, and my question just how to know their status is running, done, stop or not running in the background.
any idea really appreciate. this is my own project so i'm so excite to make this work
The simplest way of doing this is by running a log of each stage of the scraping process. So for example:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://www.google.com/')
print("Loaded Google.com")
some_task = driver.find_element(By.XPATH, '//button[text()="Some text"]')
print("Got some task")
(Locating elements as per: https://selenium-python.readthedocs.io/locating-elements.html)
However, for real-time processing of task status and for more efficiency, you can use Celery.
Celery works well for web scraping tasks as it actually allows you to asynchronously offload work from your Python app to workers and task queues.
You can then retrieve proper status reports from each worker. See: https://docs.celeryq.dev/en/stable/reference/celery.states.html
An easy and efficient approach will be to use AJAX to concurrently check a log file for the status of process and update the DOM element...
I would suggest you to have a seperate log file where the backend flask processes update the current status they are working on. something like initially everything is on status "SLEEP", and once the process is triggered it changes its corresponding log status to "RUNNING"... Once the process ends, it changes the log status to "DONE".
Use AJAX from the front end to parse the Log File every N seconds and update the DOM status element based on the status parsed from the log file by AJAX
PS. You can also add animation effects like a spinner on the DOM element on running process through AJAX
I have a view, and when user click on a button it gets called. But if a user clicks the button twice, it gets called for another even if the first is still executing or running.
This produces a problem (if I am correct), and that is: it stops execution for the first one and starts executing for the other. How can I stop this calling of views twice?
When a request has been sent to the server, Django will pick it up and send a response when it's finished. if there is no one there to receive the response then nothing happens but the processes have been done already. You should ask for verification if your process is important to be received.
When the user sends the second request, both requests will be processed and will return a response. If you're working with an API and your frontend isn't rendered in the backend server then the user will probably receive both responses and depending on the code on the front side, many things can happen. they might see both responses or just one. or the first one might get updated instantly when the second response comes in.
This might lead to problems depending on the processes you are running. for example, if you are updating a record in a database and the update can only happen once, then probably you'll get an error in the second try. it is really really rare and mostly impossible that the update part for both requests happens in the exact same time depending on your database, your code, and many other things.
There are many ways to handle such a problem which most of them depends on many things but I'm gonna list you few of these options:
1 - Change the display for the button to none so the user won't be able to click on it again. This is a very simple solution and works most of the times unless you have users trying to hurt your system intentionally.
2 - Redirect the user to another page and then wait for a response there.
3 - If your view is doing some heavy processes which are expensive to run, then just create a queue system with some limitations for each user. This is usually done in scaled projects with a lot of users.
4 - Use a rate limiting system to deny too many requests at once or block any none normal traffic.
my advice, use jquery
$("btnSubmit").click(function(event){
event.preventDefault();
//disable the submit button
$("#btnSubmit").attr("disabled", true);
// call ur view using ajax here
});
and then after ajax ended activate the button
$("#btnSubmit").attr("disabled", false);
I currently create website with Flask. After user sign up, i want to send mail & execute insert into sql, but it run slow, so i want to execute it on background and still continue the main flow and return the html without stuck on loading page.
def send_email_and_insert():
# some smtplib code here
# and some insert into sql too
# this function make the web stuck loading 5-10 second
#app.route('/sign_up/')
def sign_up():
# some code here
send_email_and_insert()
return render_template('thanks_for_sign_up.html')
I think it can be done with multiprocessing or threading, but i cant get it right. or maybe there is another solution?
I have an anchor tag that hits a route which generates a report in a new tab. I am lazyloading the report specs because I don't want to have copies of my data in the original place and on the report object. But collecting that data takes 10-20 seconds.
from flask import render_template
#app.route('/report/')
#app.route('/report/<id>')
def report(id=None):
report_specs = function_that_takes_20_seconds(id)
return render_template('report.html', report_specs=report_specs)
I'm wondering what I can do so that the server responds immediately with a spinner and then when function_that_takes_20_seconds is done, load the report.
You are right: a HTTP view is not a place for a long running tasks.
You need think your system architecture: what you can prepare outside the view and what computations must happen real time in a view.
The usual solutions include adding asynchronous properties and processing your data in a separate process. Often people use schedulers like Celery for this.
Prepare data in a scheduled process which runs for e.g. every 10 minutes
Cache results by storing them in a database
HTTP view always returns the last cached version
This, or then make a your view to do an AJAX call via JavaScript ("the spinner approach"). This requires obtaining some basic JavaScript skills. However this doesn't make the results appear to an end user any faster - it's just user experience smoke and mirrors.
I have list of players like :
player_list = Participant.objects.all()
participant_count = player_list.count()
I want to randomly select winner from this like:
winner_index = random.randint(0, participant_count-1)
winner = player_list[winner_index]
Lets say I have one million participant then I guess It will take long time to randomly generate winner. Till then my site will be hang I guess.
For this purpose should I use celery or its fine? What if my site go hang for few minutes and only display winner. Any suggestion ?
With proper indexing your database should be able to handle this without needing any special workarounds. If you make it asynchronous with celery, then you won't be able to include that data in your standard request/response cycle.
If you're worried about page speed for the user, you could load a page without the winner, then do an ajax call using javascript to get the winner and update the page, allowing you to display a loading message to the user while they wait.