We Keep Coding

Programmatically attaching file to desire2learn Dropbox feedback - python

Here's my understanding of the desired sequence of operations when uploading a file to an entity's dropbox submission:
Create feedback for a user (in my case, I am populating a rubric assessment automatically)
Upload a file, receive unique file key
Attach that file to the user's feedback, using the key received in (2)
I can upload a file for the user using the web interface to d2L, but obviously would like to streamline this if I am preparing the feedback through automation anyway. My current status is:
Create feedback for a user (in my case, I am populating a rubric assessment automatically) (200 - Success)
Upload a file, receive unique file key (200 - Success)
Attach that file to the user's feedback, using the key received in (2) (404 - Not Found)
I've been struggling with how to debug uploading and attaching a file to a dropbox feedback. I followed the resumable upload instructions at http://docs.valence.desire2learn.com/basic/fileupload.html#resumable-uploads, and (generally) that went OK, but I'm getting a final 404 error when attaching that uploaded file to the dropbox folder for this entity.
Below is my debug output that demonstrates what I understand as the state of the system for me.
Note: I replaced COURSEID, DROPBOXID, ENTITYID, and a few Keys using search and replace in an external editor, to preserve the illusion of security.
POST /d2l/api/le/1.4/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID
Posts the feedback to the dropbox for this user
{}
Confirms that the post was successful
filepath=092215-0952/Testing Download Sep 21, 2015 1114 AM_upload/ENTITYID-DROPBOXID - Student Name-Due 18, YYYY TIME AM - netid_assignmentname.zip_upload.zip
length=76630
https://d2l.domain.tld/d2l/api/le/1.4/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID/upload?x_b=_________USERID___________&x_c=_________KEY_I_DONT_UNDERSTAND___________&x_a=_________APP_ID___________&x_d=_________ANOTHER_KEY_I_DONT_UNDERSTAND___________&x_t=1443053113
{'Content-Length': '76630', 'User-Agent': 'python-requests/2.7.0 CPython/3.3.6 Darwin/14.5.0', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'Accept': '*/*', 'X-Upload-Content-Type': 'application/zip', 'X-Upload-File-Name': 'temp.zip', 'X-Upload-Content-Length': '76630'}
Received new URL to which to upload this file: revising post for this fileKey
result=qafSoMUYAm
https://d2l.domain.tld/d2l/upload/qafSoMUYAm
result of revised post is 200
With this, it seems that I initiated, and completed, the resumable upload. I will say, this part was made easier by the instructions from valence, but there was a lot of muttering on my part trying to debug it until I realized that some of the built-in post commands in d2lservice would always fail since I had to change the URL after the first post. :) By the way, I can make this return a 416 if I intentionally mess up the byte count for the file length. I did this to make sure that I wasn't getting 200 for the wrong reason.
fileKey=qafSoMUYAm, attaching to the dropbox now
GET /d2l/api/le/1.1/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID
Feedback exists for org_unit_id=COURSEID,folder_id=DROPBOXID,entity_id=ENTITYID
Here, I am making sure that the feedback truly exists for this user, so I can compare the URL to the one that fails below. NOTE: I am using version 1.4 for almost everything, but in the example below I was hardcoding the version to be 1.1 in case there was a version difference in the APIs; I tried 1.1, 1.4, but nothing else.
I am also force-marking the feedback to be feedback['IsGraded']=False, in case for some reason it's not possible to add a file to feedback if the feedback has been published. I then re-post the feedback (after marking it as ungraded), and attempt to attach the uploaded file to this feedback. An additional variable is that I have also tried (and removed) the use of a fileName param for the post (it is listed as optional in the APIs).
POST /d2l/api/le/1.4/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID
I am turning on debug now...
ready for POST=/d2l/api/le/1.1/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID/attach
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): d2l.domain.tld
send: b'POST /d2l/api/le/1.1/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID/attach?fileKey=qafSoMUYAm&x_b=_________USERID___________&x_a=_________APP_ID___________&x_d=sw6PXGCvBOsBj07VJCPu4RJ6hN8AN7OIId2ONOj-_CA&x_t=1443053116&x_c=LIrzfSOnb8rz3wkN-ZTbfnckwvrfheUU9kkarOhGpok HTTP/1.1\r\nHost: d2l.domain.tld\r\nConnection: keep-alive\r\nUser-Agent: python-requests/2.7.0 CPython/3.3.6 Darwin/14.5.0\r\nContent-Length: 0\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
DEBUG:requests.packages.urllib3.connectionpool:"POST /d2l/api/le/1.1/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID/attach?fileKey=qafSoMUYAm&x_b=_________USERID___________&x_a=_________APP_ID___________&x_d=sw6PXGCvBOsBj07VJCPu4RJ6hN8AN7OIId2ONOj-_CA&x_t=1443053116&x_c=LIrzfSOnb8rz3wkN-ZTbfnckwvrfheUU9kkarOhGpok HTTP/1.1" 404 0
So, I get a 404 when trying to make this post, and a bunch of errors below from Python that don't really help (but I include them anyway).
header: Server header: X-XSS-Protection header: X-UA-Compatible header: X-Powered-By header: Date header: Content-Length header: Set-Cookie <Response [404]>
Traceback (most recent call last):
File "../valence/sprinkle/production/postgradenoui.py", line 239, in <module>
print(postGrades_handler(dropboxid, gradeddirname,gradeddir_uploadname))
File "../valence/sprinkle/production/postgradenoui.py", line 230, in postGrades_handler
result = sprinkleutil.postRubricFeedback(request, _ac, dropboxid, gradeddirname, _CFG_COURSE['courseorgunit'], gradeddir_uploadname)
File "/Users/sprinkle/work/teaching/ece275-2015F/grading/valence/sprinkle/production/sprinkleutil.py", line 1279, in postRubricFeedback
upload_feedback_files_for_dropbox_feedback(uc,orgunitNeeded,dropboxNeeded,dir,dbfFile,'1.4')
File "/Users/sprinkle/work/teaching/ece275-2015F/grading/valence/sprinkle/production/sprinkleutil.py", line 791, in upload_feedback_files_for_dropbox_feedback
result = attach_uploaded_file(uc,org_unit_id,folder_id,entity_id,fileKey,'temp.zip')
File "/Users/sprinkle/work/teaching/ece275-2015F/grading/valence/sprinkle/production/sprinkleutil.py", line 523, in attach_uploaded_file
return d2lservice._fetch_content(r)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/d2lvalence_util/service.py", line 46, in _fetch_content
r.raise_for_status()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/requests/models.py", line 851, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found
I appreciate any pointers from anyway who has successfully carried out this sequence of operations. My d2l admins don't have bandwidth to support me, and I'm not sure if I can access the logs at all, so I don't know how to improve debugging on my side. Thanks in advance!

For your attach stage, it looks like you're sending in the fileKey as a query parameter. It needs to be a POST form parameter, right?
So the content-type for the POST would be application/x-www-form-urlencoded, and the POST body itself would be
fileKey=qafSoMUYAm&fileName=SomeFileName.zip
For example, see How to mimic an HTML form submission in a POST request.

Related

I am completly lost because I got no idea anymore what I am doing wrong.
I want to make a simple POST request to a certain address. I visited the website using firefox, opened it's terminal, copied the POST request as cURL and executed in the terminal. The recieved response has status 200 but it's body is unreadable like "������q9i".
But when I use Postman->Import->cURL and execute the request it works?! Also status 200 but this time the body contains a properly readable html code just as expected.
So I though maybe it's because Postman is adjusted the request. So I opened the code panel to the right side of the program and exported Postman's request again as cURL, python - http.client and python - request, but none of them are working?! Again I just recieve an unreadable body. How on earth can this happen?
I'm using the same machine for all requests, there is no VPN or something so it cannot be related to IP address. There is no authentification or anything.
There is just maybe one hint I noticed: The response recieved in Postman is exactly one byte shorter then the one recieved in cURL or python. Could this be the problem? Is Postman handling the response's body differently?
I appreciate any help a lot!

cURL is displaying the raw response of the body, while Postman and Firefox process the response. In your case, I suspect that you request a zipped response with a header like Accept-Encoding: gzip, deflate. If you remove that header, you will get the uncompressed response.
If there is no such header in your request, it would be good to see the request you are trying to execute.

I would like to make a GET request to one of my Python CGI files and get a response back, is there a way to handle that with raw CGI? if not, what other tool would you recommend that could potentially solve this issue? (Flask maybe?)
This is needed so I can give the backend a string to sign and return the newly created signed string back to me.
test.html -- This is what's making a GET request to gethandle.py
<script>
$.getJSON('gethandle.py', some_data, () => {alert('done')})
</script>
gethandle.py -- This is my CGI script that should do things with the request and send a response, this is the part where I am not sure what to do.
#get some_data and do things with it
#send back other_data
as it is now; when inspecting the Network tab on the browser Inspector, I see the request made to gethandle.py from test.html with a 200 status but the response is raw gethandle.py file

I'm using api_hour for API web server and trying to response to request with excel file. But it is much more difficult than I thought.
If I use nodejs or django, it is fine and there are many guide for it. But api_hour is not. The following code is mine.
headers = {
'Content-Disposition': 'attachment; filename="excel_file.xlsx"',
'Access-Control-Allow-Headers': 'status,Origin, X-Requested-With, Content-Type, Cookie, Accept, X-PINGOTHER',
'Access-Control-Allow-Methods': '*',
'Accept-Ranges': 'bytes'
}
self.responseHeaders = multidict.MultiDict(headers, )
return Response(content_type='application/vnd.openxmlformatsofficedocument.spreadsheetml.sheet',
status=self.status,
headers=self.responseHeaders,
body=response['excel'])
What I found is that the other frameworks, for example nodejs, django, and ASP.NET Core, respond with file using there own encapsulated functions so do not assign binary data, which is response['excel'] in this code, directly to body.
Is there any way to respond with file ? especially excel ? thanks.

It looks like Response is aiohttp.web.Response.
You may thus want to look into the StreamResponse class instead, which you can stream data into, if that's what you're asking.
Or, if you need to (try to) force the client to download your data as a file, add a Content-Disposition header:
Response(
...,
headers={"Content-Disposition": "attachment; filename=my-excel.xlsx"},
)

I'm trying to send a file via the request library but the receiver requires a part with a designated name (receivers terminology). I have something like this... so far:
filePath = os.path.join( GetDownloadFolder(), fileName )
files = {'upload': open( str( filePath ),'rb')}
response = requests.post( url, headers=header, files=files, verify=False )
GetDownloadFolder() simply gets the location where the file is. Header contains the account info and content type. The code above talks to the server and no longer complains that the file cannot be found. I get an error back from the server that a part with a specific name must exist. I tried using the data=values parameters with:
values = {'upload': ''}
That unfortunately didn't solve the issue. Any ideas would be appreciated.

Oh my... After a bit of debugging I figured it out. The receiver had the wrong error. I had set the content type myself as Content-Type: multipart/form-data.
When I sent file I got back an error stating that I'm missing a named part. I removed the setting of the content type and requests filled out the header like this instead.
Content-Type: multipart/form-data; boundary=3645c8b2b8f74e1a8db8a85c54225964
At that point... the received accepted the data. So the boundary is important. Probably is the size of the content being received or some such detail.

from suds.client import Client
url = r'http://*********?singleWsdl'
c = Client(url)
The requests work fine till here, but when I execute the below statement, I get the error message shown at the end. Please help.
c.service.Method_Name('parameter1', 'parameter2')
The Error message is :
Exception: (415, u'Cannot process the message because the content type
\'text/xml; charset=utf-8\' was not the expected type
\'multipart/related; type="application/xop+xml"\'.')

A Content-Type header of multipart/related; type="application/xop+xml" is the type used by MTOM, a message format used to efficiently send attachments to/from web services.
I'm not sure why the error claims to be expecting it, because the solution I found for my situation was the override the Content-Type header to 'application/soap+xml;charset=UTF-8'.
Example:
soap_client.set_options(headers = {'Content-Type': 'application/soap+xml;charset=UTF-8'})
If you are able, you could also trying checking for MTOM encoding in the web service's configuration and changing it.