We Keep Coding

Can't send HTTP post request in my wanted format

I want to send a post request but I don't know how to send the data in the way I need it.
The data of the request when I sniff should look like this only(because it looks like this when I sniff while I am using the website):
[username, password]
but I don't know how to send it differently from the format of JSON: username=name&password=pass
Here is what I tried, but it didn't work.
postdata = ["victor", "asdasd"]
resp = requests.post(SERVER_IP, data=postdata)
I expect it to send the data as I wrote but I got an error that said: ValueError: too many values to unpack (expected 2)

try to encode postdata as bytes:
postdata = ["victor", "asdasd"]
resp = requests.post(SERVER_IP, data=str(postdata).encode())
data can be a bytes objects, as per the docstring:
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.

How to use requests to send a PATCH request with headers

I have a Rails 4 application which uses token based authentication for APIs and need to be able to update records through Python 3 script.
My current script looks like this
import requests
import json
url = 'http://0.0.0.0:3000/api/v1/update_experiment.json'
payload = {'expt_name' : 'A60E001', 'status' : 'done' }
r = requests.patch(url, payload)
which works OK if I disable API authentication.
I can't figure out how to add headers to it, requests.patch only takes two parameters according to docs.
I would need to get to the point where the following header info would added
'Authorization:Token token="xxxxxxxxxxxxxxxxxxxxxx"'
This type of header works OK in curl. How can I do this in Python 3 and requests?

patch takes kwargs, just pass headers = {your_header}:
def patch(url, data=None, **kwargs):
"""Sends a PATCH request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request('patch', url, data=data, **kwargs)
So something like this:
head = {"Authorization":"Token token=xxxxxxxxxxxxxxxxxxxxxx"}
url = 'http://0.0.0.0:3000/api/v1/update_experiment.json'
payload = {'expt_name' : 'A60E001', 'status' : 'done' }
r = requests.patch(url, payload, headers=head)

Using requests to send byte-array to webservice using http post

Currently I'm trying to send a byte-array to a webservice, but i get the error message, that a bytearray is not serializeable:
TypeError: bytearray(b'') is not JSON serializable
I'm using the following code
Sending the requests
# Set blob
with open('demo-file.txt') as file:
f = file.read()
b = bytearray(f)
print a.set_data('5cb9bc4d-c0fd-40ab-8b74-4e62b50d8966', b)
Set_Data method:
def set_data(self, path, data):
"""
Save data in
Parameter
--------
path (str): Path as string
data (bytearray): Data as bytearray
"""
result = requests.post(self.url + '/set', json = { 'path': path, 'data': data})
# Check status and token
if result.status_code == 200:
return result.text
What am I doing wrong, do I have to use some other methods for sending bytearrays?
Thank you all a lot!

If you really need json, you have to encode your binary data. See: Base64 encoding in Python 3
An alternative: How to send binary post data via HTTP?

Python Requests, getting back: Unexpected character encountered while parsing value: L. Path

I am attempting to get an auth token from The Trade Desk's (sandbox) api but I get back a 400 response stating:
"Error reading Content-Type 'application/json' as JSON: Unexpected
character encountered while parsing value: L. Path '', line 0,
position 0."
Whole response.json():
{u'ErrorDetails': [{u'Reasons': [u"Error reading Content-Type 'application/json' as JSON: Unexpected character encountered while parsing value: L. Path '', line 0, position 0."], u'Property': u'TokenRequest'}], u'Message': u'The request failed validation. Please check your request and try again.'}
My script (runnable):
import requests
def get_token():
print "Getting token"
url = "https://apisb.thetradedesk.com/v3/authentication"
headers = {'content-type': 'application/json'}
data = {
"Login":"logintest",
"Password":"password",
"TokenExpirationInMinutes":60
}
response = requests.post(url, headers=headers, data=data)
print response.status_code
print response.json()
return
get_token()
Sandbox docs here
I believe this means my headers var is not being serialized correctly by requests, which seems impossible, or not being deserialized correctly by The Trade Desk. I've gotten into the requests lib but I can't seem to crack it and am looking for other input.

You need to do
import json
and convert your dict into json:
response = requests.post(url, headers=headers, data=json.dumps(data))
Another way would be to explicitely use json as parameter:
response = requests.post(url, headers=headers, json=data)
Background: In the prepare_body method of requests a dictionary is explicitely converted to json and a content-header is also automatically set:
if not data and json is not None:
content_type = 'application/json'
body = complexjson.dumps(json)
If you pass data=data then your data will be only form-encoded (see http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests). You will need to explicitely convert it to json, if you want json to be the content-type of your http body.
Your follow-up question was about why headers don't have to be converted to json. Headers can be simply passed as dictionary into the request. There's no need to convert it to json. The reason is implementation specific.

Return a requests.Response object from Flask

I'm trying to build a simple proxy using Flask and requests. The code is as follows:
#app.route('/es/<string:index>/<string:type>/<string:id>',
methods=['GET', 'POST', 'PUT']):
def es(index, type, id):
elasticsearch = find_out_where_elasticsearch_lives()
# also handle some authentication
url = '%s%s%s%s' % (elasticsearch, index, type, id)
esreq = requests.Request(method=request.method, url=url,
headers=request.headers, data=request.data)
resp = requests.Session().send(esreq.prepare())
return resp.text
This works, except that it loses the status code from Elasticsearch. I tried returning resp (a requests.models.Response) directly, but this fails with
TypeError: 'Response' object is not callable
Is there another, simple, way to return a requests.models.Response from Flask?

Ok, found it:
If a tuple is returned the items in the tuple can provide extra information. Such tuples have to be in the form (response, status, headers). The status value will override the status code and headers can be a list or dictionary of additional header values.
(Flask docs.)
So
return (resp.text, resp.status_code, resp.headers.items())
seems to do the trick.

Using text or content property of the Response object will not work if the server returns encoded data (such as content-encoding: gzip) and you return the headers unchanged. This happens because text and content have been decoded, so there will be a mismatch between the header-reported encoding and the actual encoding.
According to the documentation:
In the rare case that you’d like to get the raw socket response from the server, you can access r.raw. If you want to do this, make sure you set stream=True in your initial request.
and
Response.raw is a raw stream of bytes – it does not transform the response content.
So, the following works for gzipped data too:
esreq = requests.Request(method=request.method, url=url,
headers=request.headers, data=request.data)
resp = requests.Session().send(esreq.prepare(), stream=True)
return resp.raw.read(), resp.status_code, resp.headers.items()
If you use a shortcut method such as get, it's just:
resp = requests.get(url, stream=True)
return resp.raw.read(), resp.status_code, resp.headers.items()

Flask can return an object of type flask.wrappers.Response.
You can create one of these from your requests.models.Response object r like this:
from flask import Response
return Response(
response=r.reason,
status=r.status_code,
headers=dict(r.headers)
)

I ran into the same scenario, except that in my case my requests.models.Response contained an attachment. This is how I got it to work:
return send_file(BytesIO(result.content), mimetype=result.headers['Content-Type'], as_attachment=True)

My use case is to call another API in my own Flask API. I'm just propagating unsuccessful requests.get calls through my Flask response. Here's my successful approach:
headers = {
'Authorization': 'Bearer Muh Token'
}
try:
response = requests.get(
'{domain}/users/{id}'\
.format(domain=USERS_API_URL, id=hit['id']),
headers=headers)
response.raise_for_status()
except HTTPError as err:
logging.error(err)
flask.abort(flask.Response(response=response.content, status=response.status_code, headers=response.headers.items()))