I am fairly new to flask and therefore I have a question regarding the logic behind my code and if my thought process makes sense.
I have an old python script, that takes a lot of data, processes it and then produces a plot with matplotlib. This works fine.
Now I want to build a web application, where a user selects specific input parameters, clicks the submit button, my server checks (with the help of sqlite) if there already exists a plot/the data to make the plot with those parameters, if yes then make it downloadable by the user.
If this plot/the data does not exist, my flask application calls my python script and the new plot/data for it will get created, uploaded to sqlite and then the user can download it from the web application.
As you can see in my code so far the python script is an external one and my plan is to not include it in my views from my flask app, does this make sense? Should I call an external script here or just copy the code directly to my views? (I wanted to avoid it so far since it's a pretty big script)
My logic so far looks like this:
# this part works well so far, I get the user input here and redirect it to the specific page
#app.route('/plot', methods = ['GET','POST'])
def get_plot():
if request.method == 'POST':
input1 = request.form['input1']
input2 = request.form['input2']
input3 = request.form['input3']
return redirect('plot/{}/{}/{}/'.format(input1, input2, input3))
else:
return render_template('plot.html')
# here it get's a bit tricky for me
#app.route('/plot/<input1>/<input2>/<input3>/')
def create_plot(input1='1', input2='2', input3='3'):
try:
db = get_db(DB_PLOT)
cur = db.execute('SELECT * FROM {} WHERE param1 = {} AND param2={}'.format(input1, input2, input3)) # get all the data from the table
except:
return "Plot not found !"
# CALL THE EXTERNAL PYTHON SCRIPT HERE?
cur = db.execute('SELECT * FROM {} WHERE param1 = {} AND param2={}'.format(input1, input2, input3)) # python script should have updated the database, so i can call the data here
data = cur.fetchall()
return render_template('show_plot.html', data=data)
Furthermore, I have another question:
As said my python script, which I've only ever used so far on its own, takes raw data, manipulates it and then creates a plot with matplotlib.
When I want to implement it in my web application, should I still create the plot with the python script, upload the plot to sqlite and then get the image from sqlite with the web application OR should I just upload the manipulated data, then download this data from sqlite and create the plot with flask?
In the end, I want to make it possible for the user to download the plot as .jpg and .pdf file.
Thank you so much!
You can use the subprocess module to achieve this. Depending on how long your script takes to generate the plot you should consider returning a page to the user asking him to refresh the page after some time until the plot is available (you could also auto refresh using javascript frequently). A view function which takes too long can be a problem because the amount of threads Flask uses to handle requests is limited, this could make your application unavailable if too many users are generating plots at the same time.
Returning before calling your script won't work obviously.
return "Plot not found !"
# CALL THE EXTERNAL PYTHON SCRIPT HERE?
You also need to insert a entry in the database before starting to work on the plot generation so no other user can run your script again with the same parameters.
Your except case (when the plot is not is the database already) would look something like this (probably vulerable to SQL injection):
# write entry to the database with parameters but without plot
db.execute('INSERT INTO {} (param1, param2, param3) VALUES ({}, {}, {})'.format(table_name, input1, input2, input3))
# start process which does it's calculations in another process and
# updates the table we just inserted with the plot when finished
p = Popen(['/path/to/script.py', input1, input2, input3], stdin=None, stdout=None, stderr=None, close_fds=True)
return "Plot is beeing generated ..., please refresh page until the plot is available"
The script generating the plot would then update the entry as follows (probably vulerable to SQL injection):
db.execute('UPDATE {} SET plot_blob = {} WHERE param1 = {} AND param2 = {} AND param3 = {}'.format(table_name, binary_image_data, input1, input2, input3))
NOTES:
You can build the URL with url_for which might be neater:
return redirect(url_for('create_plot', input1=input1, input2=input2, input3=input3))
Your SQL query is probably vulnerable to SQL injection, although the library your using might handle this. I don't know which it is.
Related
In PyQt5, I want to read my serial port after writing (requesting a value) to it. I've got it working using readyRead.connect(self.readingReady), but then I'm limited to outputting to only one text field.
The code for requesting parameters sends a string to the serial port. After that, I'm reading the serial port using the readingReady function and printing the result to a plainTextEdit form.
def read_configuration(self):
if self.serial.isOpen():
self.serial.write(f"?request1\n".encode())
self.label_massGainOutput.setText(f"{self.serial.readAll().data().decode()}"[:-2])
self.serial.write(f"?request2\n".encode())
self.serial.readyRead.connect(self.readingReady)
self.serial.write(f"?request3\n".encode())
self.serial.readyRead.connect(self.readingReady)
def readingReady(self):
data = self.serial.readAll()
if len(data) > 0:
self.plainTextEdit_commandOutput.appendPlainText(f"{data.data().decode()}"[:-2])
else: self.serial.flush()
The problem I have, is that I want every answer from the serial port to go to a different plainTextEdit form. The only solution I see now is to write a separate readingReady function for every request (and I have a lot! Only three are shown now). This must be possible in a better way. Maybe using arguments in the readingReady function? Or returning a value from the function that I can redirect to the correct form?
Without using the readyRead signal, all my values are one behind. So the first request prints nothing, the second prints the first etc. and the last is not printed out.
Does someone have a better way to implement this functionality?
QSerialPort has asyncronous (readyRead) and syncronous API (waitForReadyRead), if you only read configuration once on start and ui freezing during this process is not critical to you, you can use syncronous API.
serial.write(f"?request1\n".encode())
serial.waitForReadyRead()
res = serial.read(10)
serial.write(f"?request2\n".encode())
serial.waitForReadyRead()
res = serial.read(10)
This simplification assumes that responces comes in one chunk and message size is below or equal 10 bytes which is not guaranteed. Actual code should be something like this:
def isCompleteMessage(res):
# your code here
serial.write(f"?request2\n".encode())
res = b''
while not isCompleteMessage(res):
serial.waitForReadyRead()
res += serial.read(10)
Alternatively you can create worker or thread, open port and query requests in it syncronously and deliver responces to application using signals - no freezes, clear code, slightly more complicated system.
What I have : Basically I have a dashboard in Python's Dash which has a button called GetData. The callback handler for this button gets data from 3 sources, draws 3 charts and sends back an output Div wrapping all of these charts nicely. I also have a text area on the client's screen which updates after all work is done on the server, serving as a console.
What I want:
A dedicated area on the client's screen to keep a running log of actions initiated on the server by the client. This would mean that,
a) I would like to update the text in the console by appending status of intermediate steps on the server. Way I see it, I can only update the value in the text area ONCE, right at the end of the button callback handler.
b) I would also like to bind this console as output to multiple button callbacks so that I can send a log of what's happening on the server with the client's request back to the client.
Thoughts?
I was able to achieve this by,
a) Writing to a logfile
b) Using a dcc.Interval component with output bound to a text area that polls the logfile and shows the last 20 rows from it
So far there is only one user/client but once there are more will need to think about filtering and the like.
For this kind of use case, i typically use variations of following approach
Generate a unique key for each log instance, e.g. if you want all logs for the current user, you could use the session id or a GUID generated on page load
During callback executing on server, logging output is written to a server side resource (a redis cache, a file etc.) identified by the key
Using a callback triggered by Interval component and with the key as State (so that the correct log can be identified), the log is read from the server
I'm not sure if this will do the job, but you could construct a logging decorator right before the callback as such:
#callback(...)
#myLogger
def function(a, b, c):
pass
def myLogger(fn):
#wraps
def wrapper(*args, **kwargs)
code_info = fn.__code__
code_vars = fn.__code__.co_names ## There are many options here
## check fn.__code__.__dict__
## in repl
retval = fn(args, kwargs)
print(f"func: {fn.__name__}, args: {args}, kwargs: {kwargs}, retval: {retval}")
return retval
return wrapper
The above is more or less psudo-code for a function decorator that can extract information from function calls decorated with it. It queries the code object from the function, evaluates the function, prints a collection of information to the terminal and returns the output just as it would normally. Notice that myLogger comes after the callback, as we expect to compose the functions as follows
callback(myLogger(function(...)))
I'm really new to programming in general and very inexperienced, and I'm learning python as I think it's more simple than other languages. Anyway, I'm trying to use Flask-Ask with ngrok to program an Alexa skill to check data online (which changes a couple of times per hour). The script takes four different numbers (from a different URL) and organizes it into a dictionary, and uses Selenium and phantomjs to access the data.
Obviously, this exceeds the 8-10 second maximum runtime for an intent before Alexa decides that it's taken too long and returns an error message (I know its timing out as ngrok and the python log would show if an actual error occurred, and it invariably occurs after 8-10 seconds even though after 8-10 seconds it should be in the middle of the script). I've read that I could just reprompt it, but I don't know how and that would only give me 8-10 more seconds, and the script usually takes about 25 seconds just to get the data from the internet (and then maybe a second to turn it into a dictionary).
I tried putting the getData function right after the intent that runs when the Alexa skill is first invoked, but it only runs when I initialize my local server and just holds the data for every new Alexa session. Because the data changes frequently, I want it to perform the function every time I start a new session for the skill with Alexa.
So, I decided just to outsource the function that actually gets the data to another script, and make that other script run constantly in a loop. Here's the code I used.
import time
def getData():
username = '' #username hidden for anonymity
password = '' #password hidden for anonymity
browser = webdriver.PhantomJS(executable_path='/usr/local/bin/phantomjs')
browser.get("https://gradebook.com") #actual website name changed
browser.find_element_by_name("username").clear()
browser.find_element_by_name("username").send_keys(username)
browser.find_element_by_name("password").clear()
browser.find_element_by_name("password").send_keys(password)
browser.find_element_by_name("password").send_keys(Keys.RETURN)
global currentgrades
currentgrades = []
gradeids = ['2018202', '2018185', '2018223', '2018626', '2018473', '2018871', '2018886']
for x in range(0, len(gradeids)):
try:
gradeurl = "https://www.gradebook.com/grades/"
browser.get(gradeurl)
grade = browser.find_element_by_id("currentStudentGrade[]").get_attribute('innerHTML').encode('utf8')[0:3]
if grade[2] != "%":
grade = browser.find_element_by_id("currentStudentGrade[]").get_attribute('innerHTML').encode('utf8')[0:4]
if grade[1] == "%":
grade = browser.find_element_by_id("currentStudentGrade[]").get_attribute('innerHTML').encode('utf8')[0:1]
currentgrades.append(grade)
except Exception:
currentgrades.append('No assignments found')
continue
dictionary = {"class1": currentgrades[0], "class2": currentgrades[1], "class3": currentgrades[2], "class4": currentgrades[3], "class5": currentgrades[4], "class6": currentgrades[5], "class7": currentgrades[6]}
return dictionary
def run():
dictionary = getData()
time.sleep(60)
That script runs constantly and does what I want, but then in my other script, I don't know how to just call the dictionary variable. When I use
from getdata.py import dictionary
in the Flask-ask script it just runs the loop and constantly gets the data. I just want the Flask-ask script to take the variable defined in the "run" function and then use it without running any of the actual scripts defined in the getdata script, which have already run and gotten the correct data. If it matters, both scripts are running in Terminal on a MacBook.
Is there any way to do what I'm asking about, or are there any easier workarounds? Any and all help is appreciated!
It sounds like you want to import the function, so you can run it; rather than importing the dictionary.
try deleting the run function and then in your other script
from getdata import getData
Then each time you write getData() it will run your code and get a new up-to-date dictionary.
Is this what you were asking about?
This issue has been resolved.
As for the original question, I didn't figure out how to make it just import the dictionary instead of first running the function to generate the dictionary. Furthermore, I realized there had to be a more practical solution than constantly running a script like that, and even then not getting brand new data.
My solution was to make the script that gets the data start running at the same time as the launch function. Here was the final script for the first intent (the rest of it remained the same):
#ask.intent("start_skill")
def start_skill():
welcome_message = 'What is the password?'
thread = threading.Thread(target=getData, args=())
thread.daemon = True
thread.start()
return question(welcome_message)
def getData():
#script to get data here
#other intents and rest of script here
By design, the skill requested a numeric passcode to make sure I was the one using it before it was willing to read the data (which was probably pointless, but this skill is at least as much for my own educational reasons as for practical reasons, so, for the extra practice, I wanted this to have as many features as I could possibly justify). So, by the time you would actually be able to ask for the data, the script to get the data will have finished running (I have tested this and it seems to work without fail).
I have created my micro web framework with flask which uses fabric to call the shell scripts which are in remote servers.
The shell script might take a longer time to get completed. I send the POST request from my browser and awaits for the results.
The fabric displays the real time contents on the flask run screen but flask returns the values to the browser after the completion of that remote script.
How can i make my flask to print that real time values on my browser screen ?
My flask piece:
#app.route("/abc/execute", methods=['POST'])
def execute_me():
value = request.json['value']
result = fabric_call(value)
result = formations(result)
return json.dumps(result)
My fabric piece:
def fabric_call(value):
with settings(host_string='my server', user='user',password='passwd',warn_only=True):
proc = run(my shell script)
return json.dumps(proc)
Update
I tried streamin` as well. But it didn't work. The output is displayed to my curl POST after script's complete execution. What am I missing ?
#app.route("/abc/execute", methods=['POST'])
def execute_me():
value = request.json['value']
def generate():
for row in formations(fabric_call(value)):
yield row + '\n'
return Response(generate(), mimetype="text/event-stream")
First of all, you need to make sure your data source (formations()) is actually a generator that yields data when available. Right now it very much looks like it runs the command and only returns a value once it has completely finished.
Also, in case you are using AJAX to call your endpoint remember that you cannot use e.g. jQuery's $.ajax(); you need to use XHR directly and poll for new data instead of using the onreadystatechange event since you want data once it's available and not only when the request finished.
I have a function like this in Django:
def uploaded_files(request):
global source
global password
global destination
username = request.user.username
log_id = request.user.id
b = File.objects.filter(users_id=log_id, flag='F') # Get the user id from session .delete() to use delete
source = 'sachet.adhikari#69.43.202.97:/home/sachet/my_files'
password = 'password'
destination = '/home/zurelsoft/my_files/'
a = Host.objects.all() #Lists hosts
command = subprocess.Popen(['sshpass', '-p', password, 'rsync', '--recursive', source],
stdout=subprocess.PIPE)
command = command.communicate()[0]
lines = (x.strip() for x in command.split('\n'))
remote = [x.split(None, 4)[-1] for x in lines if x]
base_name = [os.path.basename(ok) for ok in remote]
files_in_server = base_name[1:]
total_files = len(files_in_server)
info = subprocess.Popen(['sshpass', '-p', password, 'rsync', source, '--dry-run'],
stdout=subprocess.PIPE)
information = info.communicate()[0]
command = information.split()
filesize = command[1]
#st = int(os.path.getsize(filesize))
#filesize = size(filesize, system=alternative)
date = command[2]
users_b = User.objects.all()
return render_to_response('uploaded_files.html', {'files': b, 'username':username, 'host':a, 'files_server':files_in_server, 'file_size':filesize, 'date':date, 'total_files':total_files, 'list_users':users_b}, context_instance=RequestContext(request))
The main usage of the function is to transfer the file from the server to local machine and writes the data into the database. What I want it: There are single file which is of 10GB which will take a long time to copy. Since the copying happens using rsync in command line, I want to let user play with other menus while the file is being transferred. How can I achieve that? For example if the user presses OK, the file will be transferring in command line, so I want to show user "The file is being transferred" message and stop rolling the cursor or something like that? Is multiprocessing or threading appropriate in this case? Thanks
Assuming that function works inside of a view, your browser will timeout before the 10GB file has finished transferring over. Maybe you should re-think your architecture for this?
There are probably several ways to do this, but here are some that come to my mind right now:
One solution is to have an intermediary storing the status of the file transfer. Before you begin the process that transfers the file, set a flag somewhere like a database saying the process has begun. Then if you make your subprocess call blocking, wait for it to complete, check the output of the command if possible and update the flag you set earlier.
Then have whatever front end you have poll the status of the file transfer.
Another solution, if you make the subprocess call non-blocking as in your example, in that case you should use a thread which sits there reading the stdout and updating an intermediary store which your front end can query to get a more 'real time' update of the transfer process.
What you need is Celery.
It let's you spawn job as a parallel task and return http response.
RaviU solutions would certainly work.
Another option is to call a blocking subprocess in its own Thread. This thread could be responsible for setting a flag or information (in memcache, db, or just a file on the harddrive) as well as clearing it when it's complete. Personally, there is no love lost between reading rsyncs stdout and I so I usually just ask the OS for the filesize.
Also, if you don't need the file absolutely ASAP, adding "-c" to do a checksum can be good for those giant files. source: personal experience trying to transfer giant video files over spotty campus network.
I will say the one problem with all of the solutions so far is that it doesn't work for "N" files. Eventually, even if you make sure each file only can be transfered once at a time, if you have a lot of different files then eventually it'll bog down the system. You might be better off just using some sort of task queue unless you know it will only ever be the one file at a time. I haven't used one recently, but a quick google search yielded Celery which doesn't look to bad.
Every web server has a facility of uploading files. And what it does for large files is that it divides the file in chunks and does a merge after every chunk is received. What you can do here is that you can have a hidden tag in your html page which has a value attribute and whenever your upload webservice returns you an ok message at that point of time you can change the hidden html value to something relevant and also write a function that keeps on reading the value of that hidden html element and check whether your file uploading has been finished or not.