We Keep Coding

python assign literal value of a dictionary to key of another dictionary

I am trying to form a web payload for a particular request body but unable to get it right. What I need is to pass my body data as below
data={'file-data':{"key1": "3","key2": "6","key3": "8"}}
My complete payload request looks like this
payload={url,headers, data={'file-data':{"key1": "3","key2": "6","key3": "8"}},files=files}
However, when I pass this, python tries to parse each individual key value and assigns to the 'file-data' key like this
file-data=key1
file-data=key2
file-data=key3
and so on for as many keys I pass within the nested dictionary. The requirement however, is to pass the entire dictionary as a literal content like this(without splitting the values by each key):
file-data={"key1": "3","key2": "6","key3": "8"}
The intended HTTP trace should thus ideally look like this:
POST /sample_URL/ HTTP/1.1
Host: sample_host.com
Authorization: Basic XYZ=
Cache-Control: no-cache
Content-Type: multipart/form-data; boundary=----UVWXXXX
------WebKitFormBoundaryXYZ
Content-Disposition: form-data; name="file-data"
{"key1": "3","key2": "6","key3":"8" }
------WebKitFormBoundaryMANZXC
Content-Disposition: form-data; name="file"; filename=""
Content-Type:
------WebKitFormBoundaryBNM--
As such, I want to use this as part of a payload for a POST request(using python requests library). Any suggestions are appreciated in advance-
Edit1: To provide more clarity, the API definition is this:
Body
Type: multipart/form-data
Form Parameters
file: required (file)
The file to be uploaded
file-data: (string)
Example:
{
"key1": "3",
"key2": "6",
"key3": "8"
}
The python code snippet I used(after checking suggestions) is this:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
url = "https://sample_url/upload"
filepath='mypath'
filename='logo.png'
f=open(filepath+'\\'+filename)
filedata={'file-data':"{'key1': '3','key2': '6','key3': '8'}"}
base64string = encodestring('%s:%s' % ('user', 'password').replace('\n', '')
headers={'Content-type': 'multipart/form-data','Authorization':'Basic %s' % base64string}
r = requests.post(url=url,headers=headers,data=filedata,files={'file':f})
print r.text
The error I get now is still the same as shown below:
{"statusCode":400,"errorMessages":[{"severity":"ERROR","errorMessage":"An exception has occurred"]
It also says that some entries are either missing or incorrect. Note that I have tried passing the file parameter after opening it in binary mode as well but it throws the same error message
I got the HTTP trace printed out via python too and it looks like this:
send: 'POST sample_url HTTP/1.1
Host: abc.com
Connection: keep-alive
Accept-Encoding: gzip,deflate
Accept: */*
python-requests/2.11.1
Content-type: multipart/form-data
Authorization: Basic ABCDXXX=
Content-Length: 342
--CDXXXXYYYYY
Content-Disposition:form-data; name="file-data"
{\'key1\': \'3\',\'key2\': \'6\'
,\'key3\': \'8\'}
--88cdLMNO999999
Content-Disposition: form-data; name="file";
filename="logo.png"\x89PNG\n\r\n--cbCDEXXXNNNN--

If you want to post JSON with python requests, you should NOT use data but json:
r = requests.post('http://httpbin.org/post', json={"key": "value"})
I can only guess that you are using data because of your example
payload={url,headers, data={'file-data':{"key1": "3","key2": "6","key3": "8"}},files=files}
Whis is not valid python syntax btw.

Unicode encoding in email

I have manually created and sent myself an html email in gmail. I want to be able to reuse this html output to programatically send it (using smtplib in python).
In gmail, I view the source, which appears like:
Mime-Version: 1.0 Content-Type: multipart/alternative;
boundary="--==_mimepart_57daadsdas2e101427152ee"; charset=UTF-8
----==_mimepart_57daadsdas2e101427152ee Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi all !
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Venez d=C3=A9couvrir
My problem is that when I then try to send this content as html programatically, it's not displayed correctly. I suspect it's because of unicode conversion. I can't convert back for example the characters "d=C3=A9couvrir" to what it should be: "découvrir".
Could anyone help?

There's are some MIME examples that are probably more suitable, but the simple answer from the headers is that it is UTF8 and quoted-printable encoding, so you can use the quopri module:
>>> quopri.decodestring('Venez d=C3=A9couvrir').decode('utf8')
'Venez découvrir'

parse multipart/form-data, received from requests post

I am writing Web Service Client, using requests library. I am getting data in multipart/form-data that contains a file and text-json. I have no idea how to parse it. Is there a proper library to parse multipart/form-data format in python or should I write parser on my own?
my code:
data = {
"prototypeModel" :('prototypeModel', open(prototypeModel, 'rb'), 'application/octet-stream', {'Expires': '0'}),
"mfcc_1" : ('mfcc', open(mfcc_1, 'rb'), 'application/octet-stream', {'Expires': '0'}),
"mfcc_2" : ('mfcc', open(mfcc_2, 'rb'), 'application/octet-stream', {'Expires': '0'}),
"mfcc_3" : ('mfcc', open(mfcc_3, 'rb'), 'application/octet-stream', {'Expires': '0'}),
}
print( '---------------------- start enroll ----------------------')
testEnrollResponse = requests.post(server+sessionID, files = data, json = declaredParameters)
b'\r\n--c00750d1-8ce4-4d29-8390-b50bf02a92cc\r\nContent-Disposition:
form-data; name="playbackHash"\r\nContent-Type:
application/octet-stream\r\n\r\n\x16\x00\x00\x00\x00\x00\x00\x00serialization::archive\n\x00\x04\x08\x04
....
x00\x00R\x94\x9bp\x8c\x00\r\n--c00750d1-8ce4-4d29-8390-b50bf02a92cc\r\nContent-Disposition:
form-data; name="usersMFCC"\r\nContent-Type:
application/octet-stream\r\n\r\n\x16\x00\x00\x00\x00\x00\x00\x00serialization::archive\n\x00\x04\x08\x04\x08\x01\x00\x00\x00\x00\x00\x00\x00\x00\xf8\x16\x00\x00\x00\x00\x00\x00u\xbd\xb4/\xda1\xea\xbf\x0f\xed\xa2<\xc9\xf8\xe7\xbf?\xd5\xf06u\xe7\xf0\xbf\xd4\x8d\xd4\xa1F\xbe\x03#\x85X!\x19\xd8A\x06#\x8co\xf7\r
.....
x80\xd9\x95Yxn\xd0?\r\n--c00750d1-8ce4-4d29-8390-b50bf02a92cc\r\nContent-Disposition:
form-data; name="scoreAndStatus"\r\nContent-Type: application/json;
charset=utf-8\r\n\r\n{"lexLikelihood":1.544479046897232,"overallScore":-nan,"playbackLikelihood":-inf,"status":{"errorCode":0,"errorMessage":""}}\r\n--c00750d1-8ce4-4d29-8390-b50bf02a92cc--\r\n'
I replaced more binary data with " ..... "

If you're receiving a multipart/form-data response, you can parse it using the requests-toolbelt library like so:
$ pip install requests-toolbelt
After installing it
from requests_toolbelt.multipart import decoder
testEnrollResponse = requests.post(...)
multipart_data = decoder.MultipartDecoder.from_response(testEnrollResponse)
for part in multipart_data.parts:
print(part.content) # Alternatively, part.text if you want unicode
print(part.headers)

Code sample for Flask, uses https://github.com/defnull/multipart
import multipart as mp
from multipart import tob
try:
from io import BytesIO
except ImportError:
from StringIO import StringIO as BytesIO
#app.route('/', methods=["GET","POST"])
def index():
...
elif flask.request.method == "POST":
data = flask.request.data
s = data.split("\r")[0][2:]
p = mp.MultipartParser(BytesIO(tob(data)),s)
blob = p.parts()[0].value
f = open("file.bin","wb")
f.write(blob.encode("latin-1"))
f.close()

A working example of parsing multipart data follows. You can try it out at the interactive python prompt.
import email
msg = email.message_from_string('''\
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=" XXXX"
-- XXXX
Content-Type: text/plain
-- XXXX
Content-Type: text/plain
-- XXXX--
''')
msg.is_multipart()
Once you know its working on your system, you can build your own email message out of the POST data and parse it the same way. If you have the raw post body as a string the rest of the necessary information can be found in the request headers. I added indentation here for clarity, you should not have extraneous indentation in the block string.
epost_data = '''\
MIME-Version: 1.0
Content-Type: %s
%s''' % (self.headers['content-type'], post_data)
msg = email.message_from_string(post_data)
if msg.is_multipart():
for part in msg.get_payload():
name = part.get_param('name', header='content-disposition')
filename = part.get_param('filename', header='content-disposition')
# print 'name %s' % name # "always" there
# print 'filename %s' % filename # only there for files...
payload = part.get_payload(decode=True)
print payload[:100] # output first 100 characters
The first %s will be replaced with the content type, and the second with post_data. You can then write the payload to a file, etc.
Be careful to consider security implications of saving a file. You may not be able to trust the file name posted, it could start with ../../filename.sh for example on some web servers, so if you try to write /my-folder/../../filename.sh the attacker could potentially place a malicious file outside of the location where you are trying to store files. Strong validation of the file being the allowed type before trusting the file itself is also recommended. You do not want to let attackers overwrite any file on your system.

POST request with Multipart/form-data. Content-type not correct

We're trying to write a script with python (using python-requests a.t.m.) to do a POST request to a site where the content has to be MultipartFormData.
When we do this POST request manually (by filling in the form on the site and post), using wireshark, this came up (short version):
Content-Type: multipart/form-data;
Content-Disposition: form-data; name="name"
Data (8 Bytes)
John Doe
When we try to use the python-requests library for achieving the same result, this is sent:
Content-Type: application/x-pandoplugin
Content-Disposition: form-data; name="name"; filename="name"\r\n
Media type: application/x-pandoplugin (12 Bytes)
//and then in this piece is what we posted://
John Doe
The weird thing is that the 'general type' of the packet indeed is multipart/form-data, but the individual item sent (key = 'name', value= 'John Doe') has type application/x-pandoplugin (a random application on my pc I guess).
This is the code used:
response = s.post('http://url.com', files={'name': 'John Doe'})
Is there a way to specify the content-type of the individual items instead of using the headers argument (which only changes the type of the 'whole' packet)?
We think the server doesn't respond correctly due to the fact that it can't understand the content-type we send it.
Little update:
I think the different parts of the multipart content are now identical to the ones sent if I do the POST in the browser, so that's good. Still the server doesn't actually do the changes I send it with the script. The only thing that still is different is the order of the different parts.
For example this is what my browser sends:
Boundary: \r\n------WebKitFormBoundary3eXDYO1lG8Pgxjwj\r\n
Encapsulated multipart part: (text/plain)
Content-Disposition: form-data; name="file"; filename="ex.txt"\r\n
Content-Type: text/plain\r\n\r\n
Line-based text data: text/plain
lore ipsum blabbla
Boundary: \r\n------WebKitFormBoundary3eXDYO1lG8Pgxjwj\r\n
Encapsulated multipart part:
Content-Disposition: form-data; name="seq"\r\n\r\n
Data (2 bytes)
Boundary: \r\n------WebKitFormBoundary3eXDYO1lG8Pgxjwj\r\n
Encapsulated multipart part:
Content-Disposition: form-data; name="name"\r\n\r\n
Data (2 bytes)
And this is what the script (using python-requests) sends:
Boundary: \r\n------WebKitFormBoundary3eXDYO1lG8Pgxjwj\r\n
Encapsulated multipart part:
Content-Disposition: form-data; name="name"\r\n\r\n
Data (2 bytes)
Boundary: \r\n------WebKitFormBoundary3eXDYO1lG8Pgxjwj\r\n
Encapsulated multipart part: (text/plain)
Content-Disposition: form-data; name="file"; filename="ex.txt"\r\n
Content-Type: text/plain\r\n\r\n
Line-based text data: text/plain
lore ipsum blabbla
Boundary: \r\n------WebKitFormBoundary3eXDYO1lG8Pgxjwj\r\n
Encapsulated multipart part:
Content-Disposition: form-data; name="seq"\r\n\r\n
Data (2 bytes)
Could it be possible that the server counts on the order of the parts? According to Multipart upload form: Is order guaranteed?, it apparently is? And if so, is it possible to explicitly force an order using the requests library?
And to make things worse in that case: There is a mixture of a file and just text values.
So forcing an order seems rather difficult. This is the current way I do it:
s.post('http://www.url.com', files=files,data = form_values)
EDIT2:
I did a modification in the requests plugin to make sure the order of the parts is the same as in the original request. This doesn't fix the problem so I guess there is no straightforward solution for my problem. I'll send a mail to the devs of the site and hope they can help me!

your code looks correct.
requests.post('http://url.com', files={'name': 'John Doe'})
... and should send a 'multipart/form-data' Post.
and indeed, I get something like this posted:
Accept-Encoding: gzip, deflate, compress
Connection: close
Accept: */*
Content-Length: 188
Content-Type: multipart/form-data; boundary=032a1ab685934650abbe059cb45d6ff3
User-Agent: python-requests/1.2.3 CPython/2.7.4 Linux/3.8.0-27-generic
--032a1ab685934650abbe059cb45d6ff3
Content-Disposition: form-data; name="name"; filename="name"
Content-Type: application/octet-stream
John Doe
--032a1ab685934650abbe059cb45d6ff3--
I have no idea why you'd get that weird Content-Type header:
Content-Type: application/x-pandoplugin
I would begin by removing Pando Web Plugin from your machine completely, and then try your python-requests code again. (or try from a different machine)

As of today you can do:
response = s.post('http://url.com', files={'name': (filename, contents, content_type)})

Python uses a system-wide configuration file to "guess" the mime-type of a file. If those plugins are registering your file extension with their custom mime-type you'll end up putting that in instead.
The safest approach is make your own mime type guessing that suits the particular server you're sending do, and only use the native python mime type guessing for extensions you didn't think of.
How exactly you specify the content-type manually with python-requests I don't know, but I expect it should be possible.

Can I use curl to test receiving email

I would like an automated way to test how my app handles email, with attachments.
Firstly I modified my app (on App Engine) to log the contents of the request body for a received message (as sent through appspotmail). I copied these contents into a file called test_mail.txt
I figured I could post this file to imitate the inbound mail tester, something like so.
curl --header "Content-Type:message/rfc822" -X POST -d #test_mail.txt http://localhost:8080/_ah/mail/test#example.com
Whenever I do this, the message isn't properly instantiated, and I get an exception when I refer to any of the standard attributes.
Am I missing something in how I am using curl?
I run into the same problem using a simpler email, as posted by _ah/admin/inboundmail
MIME-Version: 1.0
Date: Wed, 25 Apr 2012 15:50:06 +1000
From: test#example.com
To: test#example.com
Subject: Hello
Content-Type: multipart/alternative; boundary=cRtRRiD-6434410
--cRtRRiD-6434410
Content-Type: text/plain; charset=UTF-8
There
--cRtRRiD-6434410
Content-Type: text/html; charset=UTF-8
There
--cRtRRiD-6434410--

Try --data-binary instead of -d as the flag for the input file. When I tried with your flags, it looked like curl stripped the carriage returns out of the input file, which meant the MIME parser choked on the POST data.

Black,
I noticed your program is written in python? Why not use twisted to create a tiny smtp client?
Here are a few examples..
http://twistedmatrix.com/documents/current/mail/tutorial/smtpclient/smtpclient.html