We Keep Coding

Function in django views run 2 times without reason

I have problem because I can not find the reason why my function in Django views.py sometimes runs two times. When I go to url, which call function create_db in Django view, function read json files from directory, parse it and write the data in the database. Most of the time it works perfectly, but sometimes for no reason it runs two times and write the same data in the data base. Does anyone know what can be the reason why code is sometimes done twice and how can I solve the problem?
Here is my create_db function:
def create_db(request):
response_data = {}
try:
start = time.time()
files = os.listdir()
print(files)
for filename in files:
if filename.endswith('.json'):
print(filename)
with open(f'{filename.strip()}', encoding='utf-8') as f:
data = json.load(f)
for item in data["CVE_Items"]:
import_item(item)
response_data['result'] = 'Success'
response_data['message'] = 'Baza podatkov je ustvarjena.'
except KeyError:
response_data['result'] = 'Error'
response_data['message'] = 'Prislo je do napake! Podatki niso bili uvozeni!'
return HttpResponse(json.dumps(response_data), content_type='application/json')
The console output that I expect:
['nvdcve-1.0-2002.json', 'nvdcve-1.0-2003.json', 'nvdcve-1.0-2004.json', 'nvdcve-1.0-2005.json', 'nvdcve-1.0-2006.json', 'nvdcve-1.0-2007.json', 'nvdcve-1.0-2008.json', 'nvdcve-1.0-2009.json', 'nvdcve-1.0-2010.json', 'nvdcve-1.0-2011.json', 'nvdcve-1.0-2012.json', 'nvdcve-1.0-2013.json', 'nvdcve-1.0-2014.json', 'nvdcve-1.0-2015.json', 'nvdcve-1.0-2016.json', 'nvdcve-1.0-2017.json']
nvdcve-1.0-2002.json
nvdcve-1.0-2003.json
nvdcve-1.0-2004.json
nvdcve-1.0-2005.json
nvdcve-1.0-2006.json
nvdcve-1.0-2007.json
nvdcve-1.0-2008.json
nvdcve-1.0-2009.json
nvdcve-1.0-2010.json
nvdcve-1.0-2011.json
nvdcve-1.0-2012.json
nvdcve-1.0-2013.json
nvdcve-1.0-2014.json
nvdcve-1.0-2015.json
nvdcve-1.0-2016.json
nvdcve-1.0-2017.json
Console output when error happened:
['nvdcve-1.0-2002.json', 'nvdcve-1.0-2003.json', 'nvdcve-1.0-2004.json', 'nvdcve-1.0-2005.json', 'nvdcve-1.0-2006.json', 'nvdcve-1.0-2007.json', 'nvdcve-1.0-2008.json', 'nvdcve-1.0-2009.json', 'nvdcve-1.0-2010.json', 'nvdcve-1.0-2011.json', 'nvdcve-1.0-2012.json', 'nvdcve-1.0-2013.json', 'nvdcve-1.0-2014.json', 'nvdcve-1.0-2015.json', 'nvdcve-1.0-2016.json', 'nvdcve-1.0-2017.json']
nvdcve-1.0-2002.json
['nvdcve-1.0-2002.json', 'nvdcve-1.0-2003.json', 'nvdcve-1.0-2004.json', 'nvdcve-1.0-2005.json', 'nvdcve-1.0-2006.json', 'nvdcve-1.0-2007.json', 'nvdcve-1.0-2008.json', 'nvdcve-1.0-2009.json', 'nvdcve-1.0-2010.json', 'nvdcve-1.0-2011.json', 'nvdcve-1.0-2012.json', 'nvdcve-1.0-2013.json', 'nvdcve-1.0-2014.json', 'nvdcve-1.0-2015.json', 'nvdcve-1.0-2016.json', 'nvdcve-1.0-2017.json']
nvdcve-1.0-2002.json
nvdcve-1.0-2003.json
nvdcve-1.0-2003.json
nvdcve-1.0-2004.json
nvdcve-1.0-2004.json
nvdcve-1.0-2005.json
nvdcve-1.0-2005.json
nvdcve-1.0-2006.json
nvdcve-1.0-2006.json
nvdcve-1.0-2007.json
nvdcve-1.0-2007.json
nvdcve-1.0-2008.json
nvdcve-1.0-2008.json
nvdcve-1.0-2009.json
nvdcve-1.0-2009.json
nvdcve-1.0-2010.json
nvdcve-1.0-2010.json
nvdcve-1.0-2011.json
nvdcve-1.0-2011.json
nvdcve-1.0-2012.json
nvdcve-1.0-2012.json
nvdcve-1.0-2013.json
nvdcve-1.0-2013.json
nvdcve-1.0-2014.json
nvdcve-1.0-2014.json
nvdcve-1.0-2015.json
nvdcve-1.0-2015.json
nvdcve-1.0-2016.json
nvdcve-1.0-2016.json
nvdcve-1.0-2017.json
nvdcve-1.0-2017.json

The problem is not in the code which you show us. Enable logging for the HTTP requests which your application receives to make sure the browser sends you just a single request. If you see two requests, make sure they use the same session (maybe another user is clicking at the same time).
If it's from the same user, maybe you're clicking the button twice. Could be a hardware problem with the mouse. To prevent this, use JavaScript to disable the button after the first click.

How do I use python requests to download a processed files?

I'm using Django 1.8.1 with Python 3.4 and i'm trying to use requests to download a processed file. The following code works perfect for a normal request.get command to download the exact file at the server location, or unprocessed file.
The file needs to get processed based on the passed data (shown below as "data"). This data will need to get passed into the Django backend, and based off the text pass variables to run an internal program from the server and output .gcode instead .stl filetype.
python file.
import requests, os, json
SERVER='http://localhost:8000'
authuser = 'admin#google.com'
authpass = 'passwords'
#data not implimented
##############################################
data = {FirstName:Steve,Lastname:Escovar}
############################################
category = requests.get(SERVER + '/media/uploads/9128342/141303729.stl', auth=(authuser, authpass))
#download to path file
path = "/home/bradman/Downloads/requestdata/newfile.stl"
if category.status_code == 200:
with open(path, 'wb') as f:
for chunk in category:
f.write(chunk)
I'm very confused about this, but I think the best course of action is to pass the data along with request.get, and somehow make some function to grab them inside my views.py for Django. Anyone have any ideas?

To use data in request you can do
get( ... , params=data)
(and you get data as parameters in url)
or
post( ... , data=data).
(and you send data in body - like HTML form)
BTW. some APIs need params= and data= in one request of GET or POST to send all needed information.
Read requests documentation

How to convert suds object to xml string

This is a duplicate to this question:
How to convert suds object to xml
But the question has not been answered: "totxt" is not an attribute on the Client class.
Unfortunately I lack of reputation to add comments. So I ask again:
Is there a way to convert a suds object to its xml?
I ask this because I already have a system that consumes wsdl files and sends data to a webservice. But now the customers want to alternatively store the XML as files (to import them later manually). So all I need are 2 methods for writing data: One writes to a webservice (implemented and tested), the other (not implemented yet) writes to files.
If only I could make something like this:
xml_as_string = My_suds_object.to_xml()
The following code is just an example and does not run. And it's not elegant. Doesn't matter. I hope you get the idea what I want to achieve:
I have the function "write_customer_obj_webservice" that works. Now I want to write the function "write_customer_obj_xml_file".
import suds
def get_customer_obj():
wsdl_url = r'file:C:/somepathhere/Customer.wsdl'
service_url = r'http://someiphere/Customer'
c = suds.client.Client(wsdl_url, location=service_url)
customer = c.factory.create("ns0:CustomerType")
return customer
def write_customer_obj_webservice(customer):
wsdl_url = r'file:C:/somepathhere/Customer.wsdl'
service_url = r'http://someiphere/Customer'
c = suds.client.Client(wsdl_url, location=service_url)
response = c.service.save(someparameters, None, None, customer)
return response
def write_customer_obj_xml_file(customer):
output_filename = r'C\temp\testxml'
# The following line is the problem. "to_xml" does not exist and I can't find a way to do it.
xml = customer.to_xml()
fo = open(output_filename, 'a')
try:
fo.write(xml)
except:
raise
else:
response = 'All ok'
finally:
fo.close()
return response
# Get the customer object always from the wsdl.
customer = get_customer_obj()
# Since customer is an object, setting it's attributes is very easy. There are very complex objects in this system.
customer.name = "Doe J."
customer.age = 42
# Write the new customer to a webservice or store it in a file for later proccessing
if later_processing:
response = write_customer_obj_xml_file(customer)
else:
response = write_customer_obj_webservice(customer)

I found a way that works for me. The trick is to create the Client with the option "nosend=True".
In the documentation it says:
nosend - Create the soap envelope but don't send. When specified, method invocation returns a RequestContext instead of sending it.
The RequestContext object has the attribute envelope. This is the XML as string.
Some pseudo code to illustrate:
c = suds.client.Client(url, nosend=True)
customer = c.factory.create("ns0:CustomerType")
customer.name = "Doe J."
customer.age = 42
response = c.service.save(someparameters, None, None, customer)
print response.envelope # This prints the XML string that would have been sent.

You have some issues in write_customer_obj_xml_file function:
Fix bad path:
output_filename = r'C:\temp\test.xml'
The following line is the problem. "to_xml" does not exist and I can't find a way to do it.
What's the type of customer? type(customer)?
xml = customer.to_xml() # to be continued...
Why mode='a'? ('a' => append, 'w' => create + write)
Use a with statement (file context manager).
with open(output_filename, 'w') as fo:
fo.write(xml)
Don't need to return a response string: use an exception manager. The exception to catch can be EnvironmentError.
Analyse
The following call:
customer = c.factory.create("ns0:CustomerType")
Construct a CustomerType on the fly, and return a CustomerType instance customer.
I think you can introspect your customer object, try the following:
vars(customer) # display the object attributes
help(customer) # display an extensive help about your instance
Another way is to try the WSDL URLs by hands, and see the XML results.
You may obtain the full description of your CustomerType object.
And then?
Then, with the attributes list, you can create your own XML. Use an XML template and fill it with the object attributes.
You may also found the magic function (to_xml) which do the job for you. But, not sure the XML format matches your need.

client = Client(url)
client.factory.create('somename')
# The last XML request by client
client.last_sent()
# The last XML response from Web Service
client.last_received()

File upload with Django via PUT

I am trying to implement a function in Django to upload an image from a client (an iPhone app) to an Amazon S3 server. The iPhone app sends a HttpRequest (method PUT) with the content of the image in the HTTPBody. For instance, the client PUTs the image to the following URL: http://127.0.0.1:8000/uploadimage/sampleImage.png/
My function in Django looks like this to handle such a PUT request and save the file to S3:
def store_in_s3(filename, content):
conn = S3Connection(settings.ACCESS_KEY, settings.PASS_KEY) # gets access key and pass key from settings.py
bucket = conn.create_bucket("somepicturebucket")
k = Key(bucket)
k.key = filename
mime = mimetypes.guess_type(filename)[0]
k.set_metadata("Content-Type", mime)
k.set_contents_from_string(content)
k.set_acl("public-read")
def upload_raw_data(request, name):
if request.method == 'PUT':
store_in_s3(name,request.raw_post_data)
return HttpResponse('Upload of raw data to S3 successful')
else:
return HttpResponse('Upload not successful')
My problem is how to tell my function the name of the image. In my urls.py I have the following but it won't work:
url(r'^uploadrawdata/(\d+)/', upload_raw_data ),
Now as far as I'm aware, d+ stands for digits, so it's obviously of no use here when I pass the name of a file. However, I was wondering if this is the correct way in the first place. I read this post here and it suggests the following line of code which I don't understand at all:
file_name = path.split("/")[-1:][0]
Also, I have no clue what the rest of the code is all about. I'm a bit new to all of this, so any suggestions of how to simply upload an image would be very welcome. Thanks!

This question is not really about uploading, and the linked answer is irrelevant. If you want to accept a string rather than digits in the URL, in order to pass a filename, you can just use w instead of d in the regex.
Edit to clarify Sorry, didn't realise you were trying to pass a whole file+extension. You probably want this:
r'^uploadrawdata/(.+)/$'
so that it matches any character. You should probably read an introduction to regular expressions, though.

How do I handle file upload via PUT request in Django?

I'm implementing a REST-style interface and would like to be able to create (via upload) files via a HTTP PUT request. I would like to create either a TemporaryUploadedFile or a InMemoryUploadedFile which I can then pass to my existing FileField and .save() on the object that is part of the model, thereby storing the file.
I'm not quite sure about how to handle the file upload part. Specifically, this being a put request, I do not have access to request.FILES since it does not exist in a PUT request.
So, some questions:
Can I leverage existing functionality in the HttpRequest class, specifically the part that handles file uploads? I know a direct PUT is not a multipart MIME request, so I don't think so, but it is worth asking.
How can I deduce the mime type of what is being sent? If I've got it right, a PUT body is simply the file without prelude. Do I therefore require that the user specify the mime type in their headers?
How do I extend this to large amounts of data? I don't want to read it all into memory since that is highly inefficient. Ideally I'd do what TemporaryUploadFile and related code does - write it part at a time?
I've taken a look at this code sample which tricks Django into handling PUT as a POST request. If I've got it right though, it'll only handle form encoded data. This is REST, so the best solution would be to not assume form encoded data will exist. However, I'm happy to hear appropriate advice on using mime (not multipart) somehow (but the upload should only contain a single file).
Django 1.3 is acceptable. So I can either do something with request.raw_post_data or request.read() (or alternatively some other better method of access). Any ideas?

Django 1.3 is acceptable. So I can
either do something with
request.raw_post_data or
request.read() (or alternatively some
other better method of access). Any
ideas?
You don't want to be touching request.raw_post_data - that implies reading the entire request body into memory, which if you're talking about file uploads might be a very large amount, so request.read() is the way to go. You can do this with Django <= 1.2 as well, but it means digging around in HttpRequest to figure out the the right way to use the private interfaces, and it's a real drag to then ensure your code will also be compatible with Django >= 1.3.
I'd suggest that what you want to do is to replicate the existing file upload behaviour parts of the MultiPartParser class:
Retrieve the upload handers from request.upload_handlers (Which by default will be MemoryFileUploadHandler & TemporaryFileUploadHandler)
Determine the request's content length (Search of Content-Length in HttpRequest or MultiPartParser to see the right way to do this.)
Determine the uploaded file's filename, either by letting the client specify this using the last path part of the url, or by letting the client specify it in the "filename=" part of the Content-Disposition header.
For each handler, call handler.new_file with the relevant args (mocking up a field name)
Read the request body in chunks using request.read() and calling handler.receive_data_chunk() for each chunk.
For each handler call handler.file_complete(), and if it returns a value, that's the uploaded file.
How can I deduce the mime type of what
is being sent? If I've got it right, a
PUT body is simply the file without
prelude. Do I therefore require that
the user specify the mime type in
their headers?
Either let the client specify it in the Content-Type header, or use python's mimetype module to guess the media type.
I'd be interested to find out how you get on with this - it's something I've been meaning to look into myself, be great if you could comment to let me know how it goes!
Edit by Ninefingers as requested, this is what I did and is based entirely on the above and the django source.
upload_handlers = request.upload_handlers
content_type = str(request.META.get('CONTENT_TYPE', ""))
content_length = int(request.META.get('CONTENT_LENGTH', 0))
if content_type == "":
return HttpResponse(status=400)
if content_length == 0:
# both returned 0
return HttpResponse(status=400)
content_type = content_type.split(";")[0].strip()
try:
charset = content_type.split(";")[1].strip()
except IndexError:
charset = ""
# we can get the file name via the path, we don't actually
file_name = path.split("/")[-1:][0]
field_name = file_name
Since I'm defining the API here, cross browser support isn't a concern. As far as my protocol is concerned, not supplying the correct information is a broken request. I'm in two minds as to whether I want say image/jpeg; charset=binary or if I'm going to allow non-existent charsets. In any case, I'm putting setting Content-Type validly as a client-side responsibility.
Similarly, for my protocol, the file name is passed in. I'm not sure what the field_name parameter is for and the source didn't give many clues.
What happens below is actually much simpler than it looks. You ask each handler if it will handle the raw input. As the author of the above states, you've got MemoryFileUploadHandler & TemporaryFileUploadHandler by default. Well, it turns out MemoryFileUploadHandler will when asked to create a new_file decide whether it will or not handle the file (based on various settings). If it decides it's going to, it throws an exception, otherwise it won't create the file and lets another handler take over.
I'm not sure what the purpose of counters was, but I've kept it from the source. The rest should be straightforward.
counters = [0]*len(upload_handlers)
for handler in upload_handlers:
result = handler.handle_raw_input("",request.META,content_length,"","")
for handler in upload_handlers:
try:
handler.new_file(field_name, file_name,
content_type, content_length, charset)
except StopFutureHandlers:
break
for i, handler in enumerate(upload_handlers):
while True:
chunk = request.read(handler.chunk_size)
if chunk:
handler.receive_data_chunk(chunk, counters[i])
counters[i] += len(chunk)
else:
# no chunk
break
for i, handler in enumerate(upload_handlers):
file_obj = handler.file_complete(counters[i])
if not file_obj:
# some indication this didn't work?
return HttpResponse(status=500)
else:
# handle file obj!

Newer Django versions allow for handling this a lot easier thanks to https://gist.github.com/g00fy-/1161423
I modified the given solution like this:
if request.content_type.startswith('multipart'):
put, files = request.parse_file_upload(request.META, request)
request.FILES.update(files)
request.PUT = put.dict()
else:
request.PUT = QueryDict(request.body).dict()
to be able to access files and other data like in POST. You can remove the calls to .dict() if you want your data to be read-only.

I hit this problem while working with Django 2.2, and was looking for something that just worked for uploading a file via PUT request.
from django.http import QueryDict
from django.http.multipartparser import MultiValueDict
from django.core.files.uploadhandler import (
SkipFile,
StopFutureHandlers,
StopUpload,
)
class PutUploadMiddleware(object):
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
method = request.META.get("REQUEST_METHOD", "").upper()
if method == "PUT":
self.handle_PUT(request)
return self.get_response(request)
def handle_PUT(self, request):
content_type = str(request.META.get("CONTENT_TYPE", ""))
content_length = int(request.META.get("CONTENT_LENGTH", 0))
file_name = request.path.split("/")[-1:][0]
field_name = file_name
content_type_extra = None
if content_type == "":
return HttpResponse(status=400)
if content_length == 0:
# both returned 0
return HttpResponse(status=400)
content_type = content_type.split(";")[0].strip()
try:
charset = content_type.split(";")[1].strip()
except IndexError:
charset = ""
upload_handlers = request.upload_handlers
for handler in upload_handlers:
result = handler.handle_raw_input(
request.body,
request.META,
content_length,
boundary=None,
encoding=None,
)
counters = [0] * len(upload_handlers)
for handler in upload_handlers:
try:
handler.new_file(
field_name,
file_name,
content_type,
content_length,
charset,
content_type_extra,
)
except StopFutureHandlers:
break
for chunk in request:
for i, handler in enumerate(upload_handlers):
chunk_length = len(chunk)
chunk = handler.receive_data_chunk(chunk, counters[i])
counters[i] += chunk_length
if chunk is None:
# Don't continue if the chunk received by
# the handler is None.
break
for i, handler in enumerate(upload_handlers):
file_obj = handler.file_complete(counters[i])
if file_obj:
# If it returns a file object, then set the files dict.
request.FILES.appendlist(file_name, file_obj)
break
any(handler.upload_complete() for handler in upload_handlers)