We Keep Coding

Python request headers. error authenticating with JSON WEB TOKEN

I'm learning python by building a simple trading bot. I receive this error while trying to authenticate using a JWT
{
"error": {
"status": 403,
"message": "Authentication credentials were not provided."
}
}
following the example here https://docs.ledgerx.com/reference/tradedcontracts
import requests
url = "https://api.ledgerx.com/trading/contracts/traded"
headers = {
"Accept": "application/json",
"Authorization": "MY_API_KEY"
}
response = requests.get(url, headers=headers)
print(response.text)
for now im inserting the string literal later i will store this value in an .env
thanks for taking the time to read

Can you try this please
import requests
url = "https://api.ledgerx.com/trading/contracts/traded"
headers = {
"Accept": "application/json",
"Authorization": "JWT MY_API_KEY"
}
response = requests.get(url, headers=headers)
print(response.text)

Scrapy/Python XMLHTTPRequest

I built a scraper that works up to a point: It navigates to a list of records, parses the records to key ones for further crawling, goes to those individual records but is unable to parse tables in the records because they are loaded via JavaScript. JavaScript issues a POST request (xmr) to populate them. So if JavaScript is not enabled it returns something like 'No records found.'
So I read this question: Link
I inspected Request Headers with browser dev tools. Headers include:
fetch("https://example.com/Search/GridQuery?query=foo", {
"headers": {
"accept": "text/plain, */*; q=0.01",
"accept-language": "en-US,en;q=0.9,es;q=0.8",
"cache-control": "no-cache",
"content-type": "application/x-www-form-urlencoded",
"pragma": "no-cache",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-origin",
"x-requested-with": "XMLHttpRequest"
},
"referrer": "https://example.com/SiteSearch/Search?query=bar",
"referrerPolicy": "no-referrer-when-downgrade",
"body": "page=1&size=10&useFilters=false",
"method": "POST",
"mode": "cors",
"credentials": "include"
});
The browser does indicate a cookie although not output by copying fetch...
I then tried this:
url = response.urljoin(response.css('div#Foo a::attr(href)').get())
yield Request(url=url,
method='POST',
body='{"filters": ["page": "1", "size": "10", "useFilters": "False"]}',
headers={'x-requested-with': 'XMLHttpRequest'},
callback=self.parse_table)
I get a response but it still says 'No records found'. So the POST request is not working right.
Do I need to put everything in the request header? How do I determine what must be included? Are cookies required?

I did not test this, since you didn't provide real url, but I see a couple of problems there.
Note that content type is application/x-www-form-urlencoded, and you are sending JSON object in the body (that's for application/json)
Instead, you should be sending FormRequest:
url = "https://example.com/Search/GridQuery?query=foo"
form_data = {"page": "1", "size": "10", "useFilters": "False"}
yield FormRequest(url, formdata=form_data, callback=self.parse_table)
Or simply add parameters as query parameters in the URL (still POST request, just omit the body).
url="https://example.com/Search/GridQuery?query=foo&page=1&size=10&useFilters=False"
Either way, you do not need that "filters":[], just use simple key-value object.

Python requests Json body includes some thai values, needs to be encoded as it is

I have some json containing some thai values. It looks something like
{
"TitleName": "คุณ",
"FirstName": "Simar"
}
I need to make a Http POST request with this json body with the exact thai value.I am using Python 3 requests library to make the request.
I tried this
headers = {
'Content-Type': "application/json",
'Authorization': "xxx",
'cache-control': "no-cache",
'Postman-Token': "xxx"
}
response = requests.request("POST", url, json=request, headers=headers)
It generates json values as
"TitleName": "\\u0e04\\u0e38\\u0e13",
"FirstName": "Simar"
I also tried this
json_request = json.dumps(self.request_data,ensure_ascii=False).encode('utf8')
response = requests.request("POST", url, json=json_request, headers=headers)
It generates json values as
"TitleName": "\xe0\xb8\x84\xe0\xb8\xb8\xe0\xb8\x93",
"FirstName": "Simar"
But I want json values to be generated as
"TitleName": "คุณ",
"FirstName": "Simar"
Help will be appreciated. Thanks in advance.

To preserve non-ascii characters in POST requests you need to serialise to json manually, and explicitly set the content-type header.
data = json.dumps(my_dict, ensure_ascii=False)
r = requests.post(url, headers={'content-type': 'application/json'},
data=data.encode('utf-8'))

curl vs python "requests" when hitting APIs

I am trying to hit the Bitbucket API for my account, and a successful attempt looks like:
curl --user screename:mypassword https://api.bitbucket.org/1.0/user/repositories
in the command line. In python, I try:
import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'
then
r = requests.post(url, data={'username': myscreename, 'password':mypassword})
and
r = requests.post(url, data="myscreename:mypassword")
and
r = requests.post(url, data={"user": "myscreename:mypassword"})
all get 405 error. The API is https://confluence.atlassian.com/bitbucket/rest-apis-222724129.html.
I wonder:
What am I doing wrong in the requests version, they all look similar to my curl attempt
What is the difference between requesting with curl and python requests module? What general pattern can I recognize when reading an API with a curl example and then writing it in python?
Thank you
ANSWER:
it required the right headers
https://answers.atlassian.com/questions/18451025/answers/18451117?flashId=-982194107
UPDATE:
# ===============
# get user
# ===============
import requests
import json
# [BITBUCKET-BASE-URL], i.e.: https://bitbucket.org/
url = '[BITBUCKET-BASE-URL]/api/1.0/user/'
headers = {'Content-Type': 'application/json'}
# get user
# [USERNAME], i.e.: myuser
# [PASSWORD], i.e.: itspassword
r = requests.get(url, auth=('[USERNAME]', '[PASSWORD]'), headers=headers)
print(r.status_code)
print(r.text)
#print(r.content)

Here's a way to do basic HTTP auth with Python's requests module:
requests.post('https://api.bitbucket.org/1.0/user/repositories', auth=('user', 'pass'))
With the other way you're passing the user/pass through the request's payload, which is not desired since HTTP basic auth has its own place in the HTTP protocol.
If you want to "see" what's happening under the hood with your request I recommend using httpbin:
>>> url = "http://httpbin.org/post"
>>> r = requests.post(url, data="myscreename:mypassword")
>>> print r.text
{
"args": {},
"data": "myscreename:mypassword",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "22",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
>>> r = requests.post(url, auth=("myscreename", "mypassword"))
>>> print r.text
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==",
"Content-Length": "0",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
And with curl:
curl -X POST --user myscreename:mypassword http://httpbin.org/post
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==",
"Host": "httpbin.org",
"User-Agent": "curl/7.37.1"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
Notice the resemblance between the last python example and the cURL one.
Now, getting right the API's format is another story, check out this link: https://answers.atlassian.com/questions/94245/can-i-create-a-bitbucket-repository-using-rest-api
The python way should be something like this:
requests.post('https://api.bitbucket.org/1.0/repositories', auth=('user', 'pass'), data = "name=repo_name")

With python3, you can use json={...} instead of data={...}, and it will set the header automatically to application/json:
import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'
data = {
'data1': 'asd',
'data2': 'asd'
}
req = requests.post(url, auth=('user', 'password'), json = data)
data = req.json()
# data['index']

Switching from urllib2 to requests, strangely different results with the same parameters

I'm trying to grab a cookie from a POST request. Previously, I used urllib2, which still works fine but I wanted to switch to the clearer library python-requests. Unfortunately I get an error on the page.
Since the request is HTTPS I can't sniff them to locate the difference.
urllib2 code:
NINTENDO_LOGIN_PAGE = "https://id.nintendo.net/oauth/authorize/"
MIIVERSE_CALLBACK_URL = "https://miiverse.nintendo.net/auth/callback"
parameters = {'client_id': 'ead88d8d450f40ada5682060a8885ec0',
'response_type': 'code',
'redirect_uri': MIIVERSE_CALLBACK_URL,
'username': MIIVERSE_USERNAME,
'password': miiverse_password}
data = urlencode(parameters)
self.logger.debug(data)
req = urllib2.Request(NINTENDO_LOGIN_PAGE, data)
page = urllib2.urlopen(req).read()
self.logger.debug(page)
Result (good):
[...]
<div id="main-body">
<div id="try-miiverse">
<p class="try-miiverse-catch">A glimpse at some of the posts that are currently popular on Miiverse.</p>
<h2 class="headline">Miiverse Sampler</h2>
<div id="slide-post-container" class="list post-list">
[...]
Requests code:
req = requests.post(NINTENDO_LOGIN_PAGE, data=parameters)
self.logger.debug(req.text)
Result (bad):
[...]
<div id="main-body">
<h2 class="headline">Activity Feed</h2>
<div class="activity-feed content-loading-window">
<div>
<img src="https://d13ph7xrk1ee39.cloudfront.net/img/loading-image-green.gif" alt=""></img>
<p class="tleft"><span>Loading activity feed...</span></p>
</div>
</div>
<div class="activity-feed content-load-error-window none"><div>
<p>The activity feed could not be loaded. Check your Internet connection, wait a moment and then try reloading.</p>
<div class="buttons-content">Reload</div>
</div>
</div>
[...]
Thanks in advance for any hints towards solving this.
Update 1: Thank you all for your responses!
As suggested by #abarnert, I checked the redirects.
resp = urllib2.urlopen(req)
print(resp.geturl()) # https://miiverse.nintendo.net/
req = requests.post(NINTENDO_LOGIN_PAGE, data=parameters)
print(req.url) # https://miiverse.nintendo.net/
print(req.history) # (<Response [303]>, <Response [302]>)
It seems they did both follow a redirect, but ended up in the same place.
#sigmavirus24, very useful website, thank you for making me discover it. Here are the results (I edited the order of parameters so they are easily comparable):
urllib2:
{
"args": {},
"data": "",
"files": {},
"form": {
"client_id": "ead88d8d450f40ada5682060a8885ec0",
"response_type": "code",
"redirect_uri": "https://miiverse.nintendo.net/auth/callback",
"username": "Wiwiweb",
"password": "password"
},
"headers": {
"Accept-Encoding": "identity",
"Connection": "close",
"Content-Length": "170",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "Python-urllib/2.7"
},
"json": null,
"origin": "24.85.129.188",
"url": "http://httpbin.org/post"
}
requests:
{
"args": {},
"data": "",
"files": {},
"form": {
"client_id": "ead88d8d450f40ada5682060a8885ec0",
"response_type": "code",
"redirect_uri": "https://miiverse.nintendo.net/auth/callback",
"username": "Wiwiweb",
"password": "password"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, compress",
"Connection": "close",
"Content-Length": "170",
"Content-Type": "application/x-www-form-urlencoded"
"Host": "httpbin.org",
"User-Agent": "python-requests/1.2.3 CPython/2.7.5 Windows/7"
},
"json": null,
"origin": "24.85.129.188"
"url": "http://httpbin.org/post",
}
Looks like some headers are slightly different. I don't have any other idea so I might as well try to completely copy the urllib2 headers. Spoofing the user agent might be it.
Update 2: I have added these headers to the "requests" request:
headers = {'User-Agent': 'Python-urllib/2.7',
'Accept-Encoding': 'identity'}
I am still getting the same results... The only difference between the requests now is the "requests" one has an extra header: "Accept": "*/*". I'm not sure this is the problem.
Could it be coming from the redirect?

Well, I didn't quite solve "why" the redirects are different, but I found out where to get my cookie using requests.
I figured the difference between the two libraries had something to do with the way they handle redirects. So I checked the history of both requests. For 'requests' that's as easy as doing req.history, but for urllib2, I used this bit of code:
class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
def http_error_302(self, req, fp, code, msg, headers):
print("New request:")
print(headers)
return urllib2.HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
opener = urllib2.build_opener(MyHTTPRedirectHandler, urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)
Checking the history allowed me to see that the 'requests' request had the 'set-cookie' header during it's first redirect (so the second request out of three), but not at the end. That's good enough for me because I know where to get it now: req.history[1].cookies['ms']
As a curious note, because of that bit that I added to the urllib2 request, it started returning the same thing as the 'requests' request! Even changing it to that:
class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
pass
opener = urllib2.build_opener(MyHTTPRedirectHandler, urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)
is enough to make it completely change its response to the same thing 'requests' returned all along (That bit I marked as 'bad' in the question).
I'm stumped, but knowing where to find the cookie is good enough for me. Maybe someone curious and bored will be interested in trying to find the cause.
Thank you all for your help :)