Function in django views run 2 times without reason - python

I have problem because I can not find the reason why my function in Django views.py sometimes runs two times. When I go to url, which call function create_db in Django view, function read json files from directory, parse it and write the data in the database. Most of the time it works perfectly, but sometimes for no reason it runs two times and write the same data in the data base. Does anyone know what can be the reason why code is sometimes done twice and how can I solve the problem?
Here is my create_db function:
def create_db(request):
response_data = {}
try:
start = time.time()
files = os.listdir()
print(files)
for filename in files:
if filename.endswith('.json'):
print(filename)
with open(f'{filename.strip()}', encoding='utf-8') as f:
data = json.load(f)
for item in data["CVE_Items"]:
import_item(item)
response_data['result'] = 'Success'
response_data['message'] = 'Baza podatkov je ustvarjena.'
except KeyError:
response_data['result'] = 'Error'
response_data['message'] = 'Prislo je do napake! Podatki niso bili uvozeni!'
return HttpResponse(json.dumps(response_data), content_type='application/json')
The console output that I expect:
['nvdcve-1.0-2002.json', 'nvdcve-1.0-2003.json', 'nvdcve-1.0-2004.json', 'nvdcve-1.0-2005.json', 'nvdcve-1.0-2006.json', 'nvdcve-1.0-2007.json', 'nvdcve-1.0-2008.json', 'nvdcve-1.0-2009.json', 'nvdcve-1.0-2010.json', 'nvdcve-1.0-2011.json', 'nvdcve-1.0-2012.json', 'nvdcve-1.0-2013.json', 'nvdcve-1.0-2014.json', 'nvdcve-1.0-2015.json', 'nvdcve-1.0-2016.json', 'nvdcve-1.0-2017.json']
nvdcve-1.0-2002.json
nvdcve-1.0-2003.json
nvdcve-1.0-2004.json
nvdcve-1.0-2005.json
nvdcve-1.0-2006.json
nvdcve-1.0-2007.json
nvdcve-1.0-2008.json
nvdcve-1.0-2009.json
nvdcve-1.0-2010.json
nvdcve-1.0-2011.json
nvdcve-1.0-2012.json
nvdcve-1.0-2013.json
nvdcve-1.0-2014.json
nvdcve-1.0-2015.json
nvdcve-1.0-2016.json
nvdcve-1.0-2017.json
Console output when error happened:
['nvdcve-1.0-2002.json', 'nvdcve-1.0-2003.json', 'nvdcve-1.0-2004.json', 'nvdcve-1.0-2005.json', 'nvdcve-1.0-2006.json', 'nvdcve-1.0-2007.json', 'nvdcve-1.0-2008.json', 'nvdcve-1.0-2009.json', 'nvdcve-1.0-2010.json', 'nvdcve-1.0-2011.json', 'nvdcve-1.0-2012.json', 'nvdcve-1.0-2013.json', 'nvdcve-1.0-2014.json', 'nvdcve-1.0-2015.json', 'nvdcve-1.0-2016.json', 'nvdcve-1.0-2017.json']
nvdcve-1.0-2002.json
['nvdcve-1.0-2002.json', 'nvdcve-1.0-2003.json', 'nvdcve-1.0-2004.json', 'nvdcve-1.0-2005.json', 'nvdcve-1.0-2006.json', 'nvdcve-1.0-2007.json', 'nvdcve-1.0-2008.json', 'nvdcve-1.0-2009.json', 'nvdcve-1.0-2010.json', 'nvdcve-1.0-2011.json', 'nvdcve-1.0-2012.json', 'nvdcve-1.0-2013.json', 'nvdcve-1.0-2014.json', 'nvdcve-1.0-2015.json', 'nvdcve-1.0-2016.json', 'nvdcve-1.0-2017.json']
nvdcve-1.0-2002.json
nvdcve-1.0-2003.json
nvdcve-1.0-2003.json
nvdcve-1.0-2004.json
nvdcve-1.0-2004.json
nvdcve-1.0-2005.json
nvdcve-1.0-2005.json
nvdcve-1.0-2006.json
nvdcve-1.0-2006.json
nvdcve-1.0-2007.json
nvdcve-1.0-2007.json
nvdcve-1.0-2008.json
nvdcve-1.0-2008.json
nvdcve-1.0-2009.json
nvdcve-1.0-2009.json
nvdcve-1.0-2010.json
nvdcve-1.0-2010.json
nvdcve-1.0-2011.json
nvdcve-1.0-2011.json
nvdcve-1.0-2012.json
nvdcve-1.0-2012.json
nvdcve-1.0-2013.json
nvdcve-1.0-2013.json
nvdcve-1.0-2014.json
nvdcve-1.0-2014.json
nvdcve-1.0-2015.json
nvdcve-1.0-2015.json
nvdcve-1.0-2016.json
nvdcve-1.0-2016.json
nvdcve-1.0-2017.json
nvdcve-1.0-2017.json

The problem is not in the code which you show us. Enable logging for the HTTP requests which your application receives to make sure the browser sends you just a single request. If you see two requests, make sure they use the same session (maybe another user is clicking at the same time).
If it's from the same user, maybe you're clicking the button twice. Could be a hardware problem with the mouse. To prevent this, use JavaScript to disable the button after the first click.

Related

Why is the code returning a TypeError - Python

#app.route("/admin/3")
def admin3_p():
return render_template("input_test.html")
#app.route("/admin/3", methods=['POST'])
def student_name():
with app.test_request_context('/admin/3', data='student'):
variable = request.form.get('student', list(''))
return variable
# Connect to CSV
def csv_func():
variable = student_name()
csv_f = "names.csv"
titles = ["Event", "Student", "Grade"]
students = [["Ev", "St", "Gr"], [variable]]
with open(csv_f, 'w') as csvfile:
csvwriter = csv.writer(csvfile)
csvwriter.writerow(titles)
csvwriter.writerows(students)
with open(csv_f, 'r') as csvfile:
csvreader = csv.reader(csvfile)
titles = next(csvreader)
for student in csvreader:
students.append(students)
print('Fields: ' + ', '.join(title for title in titles))
print(students)
csv_func()
I am trying to make a website with Flask. Th csv_func method is supposed to take the input from the html and print it to a csv file.
It returns "TypeError: The view function did not return a valid response. The return type must be a string, dict, tuple, Response instance, or WSGI callable, but it was a list" When it runs
Technically the error is because function with a route decorator is considered 'a view' and is supposed to return a page, yet yours student_name returns a tuple (of student names)
Yet I have to tell you that you got it wrong idea of web app syntax and structure. Your flow of control is opposite from what is should be. You should initiate model and csv changes from controller (student_name function), and you are doing it vise versa, by calling student_name from . The main code usually just start web app with something like
app.run(host='0.0.0.0', port=81)
So you should restructure you code in a so student_name function invokes csv changing function.
I guess you think that web app form is akin to input command in python, yet a web app is very different from python console input. The main difference is that website normally offer several different pages, and user is free to land on any page he likes. So normal webserver just wait for user landing to one or another page or sending one or another form. Thus the structure of web app is a set of pages, routes and controllers for that pages, and main code just starts the flask server. Go throw some introductory flask tutorial if it is still
unclear. E.g. https://flask.palletsprojects.com/en/1.1.x/quickstart/
Most web apps follow UI design pattern called Model-View-Controller, where user actions, such as opening a webpage on a specific web address or filling a form first hit some controlling code, which the initiates some changes in the model (data).
Get rid of the app.route(...) decorator above def student_name():.

Transform a multiple line url request into a function in Python

I try to download a serie of text files from different websites. I am using urllib.request with Python. I want to expend the list of URL without making the code long.
The working sequence is
import urllib.request
url01 = 'https://web.site.com/this.txt'
url02 = 'https://web.site.com/kind.txt'
url03 = 'https://web.site.com/of.txt'
url04 = 'https://web.site.com/link.txt'
[...]
urllib.request.urlretrieve(url01, "Liste n°01.txt")
urllib.request.urlretrieve(url02, "Liste n°02.txt")
urllib.request.urlretrieve(url03, "Liste n°03.txt")
[...]
The number of file to download is increasing and I want to keep the second part of the code short.
I tried
i = 0
while i<51
i = i +1
urllib.request.urlretrieve( i , "Liste n°0+"i"+.txt")
It doesn't work and I am thinking that a while loop can be use for string but not for request.
So I was thinking of making it a function.
def newfunction(i)
return urllib.request.urlretrieve(url"i", "Liste n°0"+1+".txt")
But it seem that I am missing a big chunk of it.
This request is working but it seem I cannot transform it for long list or URL.
As a general suggestion, I'd recommend the requests module for Python, rather than urllib.
Based on that, some naive code for a possible function:
import requests
def get_file(site, filename):
target = site + "/" + filename
try:
r = requests.get(target, allow_redirects=True)
open(filename, 'wb').write(r.content)
return r.status_code
except requests.exceptions.RequestException as e:
print("File not downloaded, error: {}".format(e))
You can then call the function, passing in parameters of site and file name:
get_file('https://web.site.com', 'this.txt')
The function will raise an exception, but not stop execution, if it cannot download a file. You could expand exception handling to deal with files not being writable, but this should be a start.
It seems as if you're not casting the variable i to an integer before your concatenating it to the url string. That may be the reason why you're code isn't working. The while-loop/for-loop approach shouldn't effect whether or not the requests get sent out. I recommend using the requests module for making requests as well. Mike's post covers what the function should kind of look like. I also recommend creating a sessions object if you're going to be making a whole lot of requests in a piece of code. The sessions object will keep the underlying TCP connection open while you make your requests, which should reduce latency, CPU usage, and network congestion (https://en.wikipedia.org/wiki/HTTP_persistent_connection#Advantages). The code would look something like this:
import requests
with requests.Session() as s:
for i in range(10):
s.get(str(i)+'.com') # make request
# write to file here
To cast to a string you would want something like this:
i = 0
while i<51
i = i +1
urllib.request.urlretrieve( i , "Liste n°0" + str(i) + ".txt")

Flask - render page without return

I'm working on an application, where user fills in form.
One of the fields asks for postcode. When request is sent, a separate thread runs python code using PYQT5, rendering page and scraping web, (don't want to pay for API from google) providing mileage between given postcode and set postcode. Results of this thread are saved in a file.
What i would like to do is to open a new webpage with information that the data is being checked. This page would run python code, which checks for results in a file (that takes up to few seconds) using while loop. if the result is in the file, the page redirect to another page.
Is there a way to render a page (or redirect to another page) without using RETURN? i understand that when I use return render_templates (page), rest of the code is ignored.
#app.route('/add_user', methods=['GET', 'POST'])
def add_user():
file = open('app/results_from_g.txt','w')
file.write('')
file.close()
form = AddForm()
if form.validate_on_submit():
url = 'https://google.co.uk/maps/dir/...'
def sendtogoogle(url):
os.system('python googlemaps_mileage.py ' +url)
thread1=threading.Thread(target=sendtogoogle, args=(url,))
thread1.start()
the next line of the code should be redirect to another page, and when results are in the file, either back here, or different page:
while result_from_file==0:
file = open('results_from_g.txt','r')
result_from_file = file.read()
if result_from_file =='': #means no data saved in the file yet
time.sleep(1)
elif wynik =='0': #means wrong postcode
//render page 'wrong postcode'
else:
//render page 'correct postcode

How do I use python requests to download a processed files?

I'm using Django 1.8.1 with Python 3.4 and i'm trying to use requests to download a processed file. The following code works perfect for a normal request.get command to download the exact file at the server location, or unprocessed file.
The file needs to get processed based on the passed data (shown below as "data"). This data will need to get passed into the Django backend, and based off the text pass variables to run an internal program from the server and output .gcode instead .stl filetype.
python file.
import requests, os, json
SERVER='http://localhost:8000'
authuser = 'admin#google.com'
authpass = 'passwords'
#data not implimented
##############################################
data = {FirstName:Steve,Lastname:Escovar}
############################################
category = requests.get(SERVER + '/media/uploads/9128342/141303729.stl', auth=(authuser, authpass))
#download to path file
path = "/home/bradman/Downloads/requestdata/newfile.stl"
if category.status_code == 200:
with open(path, 'wb') as f:
for chunk in category:
f.write(chunk)
I'm very confused about this, but I think the best course of action is to pass the data along with request.get, and somehow make some function to grab them inside my views.py for Django. Anyone have any ideas?
To use data in request you can do
get( ... , params=data)
(and you get data as parameters in url)
or
post( ... , data=data).
(and you send data in body - like HTML form)
BTW. some APIs need params= and data= in one request of GET or POST to send all needed information.
Read requests documentation

open('file_path').read() - save contents back to file

I need to upload a file to the server using urllib2. Since I cannot use any external libraries (like requests and others) because I am using OpenOffice python, I needed a simple way to post file data.
So I came with:
post_url = "http://localhost:8000/admin/oo_file_uploader?user_id=%s&file_id=%s" % (user_id, file_id)
file_path = doc.Location.replace('file://', '')
data = urllib.urlencode({"file": open(file_path).read()})
urllib2.urlopen(post_url, data)
which posts something to the server.
I wonder if it is possible to save posted contents back to the file using python/django?
This expands somewhat on #zero323's answer. You will want to make sure that you implement some sort of security to prevent random files being uploaded by unauthorized users, which is what the #file_upload_security decorator is implied to handle.
#file_upload_security
def oo_file_uploader(user_id=None, file_id=None):
if request.method == 'POST':
# Exception handling skipped if get() fails.
user = User.objects.get(id=user_id)
save_to_file = MyFiles.objects.get(id=file_id)
# You will probably want to ensure 'file' is in post data.
file_contents = save_to_file.parse_post_to_content(request.POST['file'])
with open(save_to_file.path_to_file, 'w') as fw:
fw.write(file_contents)

Categories