We Keep Coding

Python request to a CloudFlare protected API returning 403

I am running mitmproxy with an upstream to remote proxy.
mitmweb --set mode=upstream:http://proxyIp:proxyPort --set ssl_insecure=true
The application flow is:
Make a HTTP request in Python and use mitmproxy server as proxies argument
Intercept the call in mitmproxy, and do an upstream to another proxy.
Return the results.
The HTTP request is made to the external API (I don't have access to it) protected by CloudFlare.
headers = {
'User-Agent': 'PostmanRuntime/7.26.7',
'Accept': 'application/json'
}
r = requests.get("https://api.website.com/",
headers=headers,
verify=False,
proxies={'http': 'http://127.0.0.1:8080', 'https': 'http://127.0.0.1:8080'})
Running this request will result in a 403 response from https://api.website.com/. The result is the same if I skip the mitmproxy part and connect to the end proxy directly from Python. How ever, I tried using Fiddler as a Gateway and it worked good (It's certainly modifying the request in a background).
<span>Error</span><span>1020</span>
If I run the same request with curl the result will be good (200 OK)
curl -x 127.0.0.1:8080 -k 'https://api.website.com/' -H 'user-agent: PostmanRuntime/7.26.7' -H 'accept: application/json'
Those two requests seem identical, yet the Python one returns 403.
Am I missing something in the Python config? Setting some protocol or headers?
NOTE:
I tried running the curl by directly connecting to the end proxy (skipping the mitmproxy), and the request is also failing with a 403 response.
curl -x proxyIp:proxyPort -k 'https://api.website.com/' -H 'user-agent: PostmanRuntime/7.26.7'
Then I tried by using the curl-openssl/bin/curl and it worked, how ever I had to add --tlsv1.3 to it.
/usr/local/opt/curl-openssl/bin/curl --tlsv1.3 -x proxyIp:proxyPort -k 'https://api.website.com/' -H 'user-agent: PostmanRuntime/7.26.7'

How can I post http request instead of using cURL?

I am using anki-connect to communicate with Anki, a spaced repetition software.
In readme.md, it uses following command to get deck name.
curl localhost:8765 -X POST -d "{\"action\": \"deckNames\", \"version\": 5}"
It works right in my Windows system.
How can I use python instead of cURL?
I've tried this but get no luck.
import requests
r = requests.post("http://127.0.0.1:8765", data={'action': 'guiAddCards', 'version': 5})
print(r.text)

When creating request you should:
provide Content-Type header
provide data in format that matches Content-Type header
make sure application supports the format
Both curl and python examples you gave sends request with Content-Type: application/x-www-form-urlencoded, the default one. The difference is curl passes string and python passes an array.
Let's compare curl and requests and what is really posted:
Curl
$ curl localhost -X POST -d "{\"action\": \"deckNames\", \"version\": 5}"
Headers:
Host: localhost
User-Agent: curl/7.52.1
Accept: */*
Content-Length: 37
Content-Type: application/x-www-form-urlencoded
Posted data:
[
'{"action": "deckNames", "version": 5}'
]
Python
import requests
r = requests.post("http://127.0.0.1", data={'action': 'guiAddCards', 'version': 5})
print(r.text)
Headers:
Host: 127.0.0.1
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.10.0
Content-Length: 28
Content-Type: application/x-www-form-urlencoded
Posted data:
[
'action' -> 'guiAddCards',
'version' -> '5',
]
As you can see, incorrect post data format breaks your app.
To be sure, that posted JSON data will be properly read by application you should make requests like that:
Curl
$ curl localhost:8765 -H 'Content-Type: application/json' -d '{"action": "deckNames", "version": 5}'
Python
import requests
r = requests.post("http://127.0.0.1:8765", json={'action': 'guiAddCards', 'version': 5})
print(r.text)

I've tried following after digging and this works.
Can anybody share the reason. Thanks.
import requests
import json
#r = requests.post("http://127.0.0.1:8765", data={'action': 'guiAddCards', 'version': 5})
r = requests.post('http://localhost:8765', data=json.dumps({'action': 'guiAddCards', 'version': 5}))
print(r.text)

This is a reply to user2444791's answer. I can't reply with a comment because I don't have the reputation to comment (I'm new, please forgive a breech of etiquette!)
Without the exact error message, it's hard to be sure, but...
Looking at the Anki Connect API, it expects its POST-ed data to be a single string which contains a JSON object, not a key/value dictionary equivalent to that JSON object.
Every request consists of a JSON-encoded object containing an action, a version, and a set of contextual params.
Their example code (in Javascript): xhr.send(JSON.stringify({action, version, params}));
It might be as simple as sending your data in the wrong format. In the first example, you are sending a dictionary with the key/vale pairs already parsed. In the second example, you're sending a string for them to parse instead.

Python3.6 requests.post results in ConnectionError number 104, but all is fine when using curl

I am trying to rewrite a curl command to python. The idea is the following: we upload an audio file, and get in return what it was said in it in text - e.g. speech to text.
The site has provided us a curl example, which makes a post request and in return receives a simple response.
When I tried to convert the curl request into python one (with little help from https://curl.trillworks.com/ ) I get ConnectionError number 104, connection reset by peer.
I strongly suspect that this happens because when I make the connection vie curl firstly I get the response: < HTTP/1.1 100 Continue, and after short time of waiting another response: < HTTP/1.1 200 OK, with the some sample data. I think that python requests.post just hangs at the first HTTP/1.1 100.
I could not handle or reset the connection using try... except..., neither succeeded in looping it with time.sleep().
Any ideas?
P.S. CODE:
Curl command:
curl "LINK TO SERVER" -H "Content-Type: audio/x-speex;rate=16000" -H "Accept-Language: ENUS" -H "Transfer-Encoding: chunked" -H "Accept: application/xml" -H "Accept-Topic: Dictation" -k --data-binary #audio_16k16bit.pcm
Python equivalent:
#!/bin/python
import requests
import time
headers = {
'Content-Type': 'audio/x-wav;codec=pcm;bit=16;rate=16000',
'Accept-Language': 'ENUS',
'Transfer-Encoding': 'chunked',
'Accept': 'application/xml',
'Accept-Topic': 'Dictation',
}
params = (
('appId', ''),
('appKey',''),
('id', ''),
)
data = open('audio_16k16bit.pcm', 'rb').read()
r = requests.post('LINK TO SERVER', headers=headers, data=data, verify=False)
print(r.content)

SOLUTION
Should remove and let the server determine the type of file, also since we are sending it as one file chunked is not needed:
'Content-Type': 'audio/x-wav;codec=pcm;bit=16;rate=16000',
'Transfer-Encoding': 'chunked'

python variable empty on concat after response.getvalue()

(New to python)
I'm trying to make a simple authenticated put of a file... so I make two curls, the first one to authenticate (which prints the token out as expected) but when I use the same variable (token) to add it to the headers ("Authorization: Bearer %s" % str(token)) token is empty. What am I doing wrong here?
import urllib
import cStringIO
import pycurl
import requests
from urllib import urlencode
import os.path
# declarations
filename = "./profile.jpg"
response = cStringIO.StringIO()
c = pycurl.Curl()
# formdata
post_data = {'username': '...', 'password':'...'}
# Form data must be provided already urlencoded.
postfields = urlencode(post_data)
# Sets request method to POST,
# Content-Type header to application/x-www-form-urlencoded
# and data to send in request body.
print "*****************************************************"
# authenticate
c = pycurl.Curl()
c.setopt(c.POST, 1)
c.setopt(c.URL, "https://.../auth")
c.setopt(c.POSTFIELDS, postfields)
c.setopt(c.SSL_VERIFYPEER, 0)
c.setopt(c.SSL_VERIFYHOST, 0)
c.setopt(c.VERBOSE, 1)
c.perform()
c.close()
token = response.getvalue()
print token
print "*****************************************************"
# upload file
filesize = os.path.getsize(filename)
fin = open(filename, 'rb')
c = pycurl.Curl()
c.setopt(c.PUT, 1)
c.setopt(c.URL, "https://.../avatar")
c.setopt(c.HTTPPOST, [("file", (c.FORM_FILE, filename))])
c.setopt(c.HTTPHEADER, [
"Authorization: Bearer %s" % str(token),
"Content-Type: image/jpeg"
])
c.setopt(c.READFUNCTION, fin.read)
c.setopt(c.POSTFIELDSIZE, filesize)
c.setopt(c.SSL_VERIFYPEER, 0)
c.setopt(c.SSL_VERIFYHOST, 0)
c.setopt(c.VERBOSE, 1)
c.setopt(c.WRITEFUNCTION, response.write),
c.perform()
c.close()
print response.getvalue()
print "*****************************************************"
Request:
> PUT ../avatar HTTP/1.1
User-Agent: PycURL/7.19.3 libcurl/7.35.0 GnuTLS/2.12.23 zlib/1.2.8 libidn/1.28 librtmp/2.3
Host: 127.0.0.1:8080
Accept: */*
Transfer-Encoding: chunked
Authorization: Bearer
Expect: 100-continue
Response:
< HTTP/1.1 100 Continue
< HTTP/1.1 401 Unauthorized
< content-type: application/json; charset=utf-8
< cache-control: no-cache
< content-length: 86
< Date: Tue, 02 Jun 2015 19:09:29 GMT
< Connection: keep-alive
<
* Connection #1 to host 127.0.0.1 left intact
{"statusCode":401,"error":"Unauthorized","message":"Incorrect Token or Token Expired"}

I think there is an encoding problem. The print function is able to output something without caring about the encoding. Looking at the PycURL quickstart documentation it mentions this issue. I would try to manipulate the encoding on this line:
"Authorization: Bearer %s" % str(token)
and try to do something like this instead:
"Authorization: Bearer %s" % token.decode('iso-8859-1')
(I would try .decode("utf-8") also, depending on what the encoding is)
You might need to change response = cStringIO.StringIO() to response = BytesIO(). I cannot give a definitive answer because I'm unsure about your setup.
EDIT: My suspicions about encoding affirmed by this post about cStringIO where it says that Unicode is not supported.

Authenticated HTTP POST with XML payload using Python urllib2

I'm trying to send a POST message with a purely XML payload (I think) using urllib2 in IronPython. However, everytime I send it, it returns Error code 400 (Bad Request).
I'm actually trying to mimick a Boxee remove queue item call for which the actual data packets looks like this (from WireShark):
POST /action/add HTTP/1.1
User-Agent: curl/7.16.3 (Windows build 7600; en-US; beta) boxee/0.9.21.11487
Host: app.boxee.tv
Accept: */*
Accept-Encoding: deflate, gzip
Cookie: boxee_ping_version=9; X-Mapping-oompknoc=76D730BC9E858725098BF13AEFE32EB5; boxee_app=e01e36e85d368d4112fe4d1b6587b1fd
Connection: keep-alive
Content-Type: text/xml
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Keep-Alive: 300
Connection: keep-alive
Content-Length: 53
<message type="dequeue" referral="3102296"></message>
I'm using the following python code to send the POST:
def PostProtectedPage(theurl, username, password, postdata):
req = urllib2.Request(theurl, data=postdata)
req.add_header('Content-Type', 'text/xml')
try:
handle = urllib2.urlopen(req)
except IOError, e: # here we are assuming we fail
pass
else: # If we don't fail then the page isn't protected
print "This page isn't protected by authentication."
sys.exit(1)
if not hasattr(e, 'code') or e.code != 401: # we got an error - but not a 401 error
print "This page isn't protected by authentication."
print 'But we failed for another reason.'
sys.exit(1)
authline = e.headers.get('www-authenticate', '') # this gets the www-authenticat line from the headers - which has the authentication scheme and realm in it
if not authline:
print 'A 401 error without an authentication response header - very weird.'
sys.exit(1)
authobj = re.compile(r'''(?:\s*www-authenticate\s*:)?\s*(\w*)\s+realm=['"](\w+)['"]''', re.IGNORECASE) # this regular expression is used to extract scheme and realm
matchobj = authobj.match(authline)
if not matchobj: # if the authline isn't matched by the regular expression then something is wrong
print 'The authentication line is badly formed.'
sys.exit(1)
scheme = matchobj.group(1)
realm = matchobj.group(2)
if scheme.lower() != 'basic':
print 'This example only works with BASIC authentication.'
sys.exit(1)
base64string = base64.encodestring('%s:%s' % (username, password))[:-1]
authheader = "Basic %s" % base64string
req.add_header("Authorization", authheader)
try:
handle = urllib2.urlopen(req)
except IOError, e: # here we shouldn't fail if the username/password is right
print "It looks like the username or password is wrong."
print e
sys.exit(1)
thepage = handle.read()
return thepage
However, whenever I run this, it returns Error 400 (Bad Request)
I know the authentication is correct because I use it elsewhere to fetch the queue (and I can't imagine it's not used, otherwise how would it now which account to apply the change to?)
Looking at the network capture, could I simply be missing adding some headers to the request? Probably something simple, but I just don't know enough about python or HTTP requests to know what's what.
Edit: BTW, I'm calling the code as follows (it's actually dynamic, but this is the basic idea):
PostProtectedPage("http://app.boxee.tv/action/add", "user", "pass", "<message type=\"dequeue\" referral=\"3102296\"></message>")

This worked fine for me:
curl -v -A 'curl/7.16.3 (Windows build 7600; en-US; beta) boxee/0.9.21.11487' \
-H 'Content-Type: text/xml' -u "USER:PASS" \
--data '<message type="dequeue" referral="12573293"></message>' \
'http://app.boxee.tv/action/add'
But I get 400 Bad Request if I try to remove a referral ID that isn't currently in the queue. If you're using the same referral ID as you detected from Wireshark, that's very likely what's happening for you, too. Use
wget -nv -m -nd --user=USER --password=PASS http://app.boxee.tv/api/get_queue
to make sure what you're trying to remove is actually in the queue.