Why does requests ignore my custom header field? - python

I have the following Python function which sends a post request using the requests library:
def http_post(self, url: str, headers: dict, data: str, auth: AuthBase):
token = self._xsuaa.get_token(self._service)
headers.update({'Proxy-Authorization': f"Bearer {token}"})
res = requests.post(
url,
headers=headers,
data=data,
proxies={'http': self._proxy},
auth=auth,
verify=False,
timeout=100,
allow_redirects=True)
When printing the headers dict, it looks like this:
{
'Content-Type': 'multipart/mixed;boundary=batch_4724f345-bb46-437d-a970-197a7b82bf41',
'Content-Transfer-Encoding': 'binary',
'sap-cancel-on-close': 'true',
'sap-contextid-accept': 'header',
'Accept': 'application/json',
'Accept-Language': 'de-DE',
'DataServiceVersion': '2.0',
'MaxDataServiceVersion': '2.0',
'Proxy-Authorization': 'Bearer <token>'
}
However, when I take a look at res.request.headers, I get the following:
{
'User-Agent': 'python-requests/2.26.0',
'Accept-Encoding': 'gzip, deflate',
'Accept': 'application/json',
'Connection': 'keep-alive',
'Content-Type': 'multipart/mixed; boundary=batch_4724f345-bb46-437d-a970-197a7b82bf41',
'Content-Transfer-Encoding': 'binary',
'sap-cancel-on-close': 'true',
'sap-contextid-accept': 'header',
'Accept-Language': 'de-DE',
'DataServiceVersion': '2.0',
'MaxDataServiceVersion': '2.0',
'Content-Length': '659',
'Authorization': 'Basic <auth>'
}
For some reason, the proxy-authorization header field is gone and accordingly, I get a 407 error in the response. I have read in the documentation that proxy credentials provided in the URL overwrite proxy-authorization headers, but my URL contains none. I also tried removing the auth=auth line from the request, but the problem still persited. Can someone point me in the right direction as to why this field is seemingly ignored or overwritten by requests?

Related

postman is not passing api_key to flask app

I have setup my postman to use an API Key. I have added it to the authorization section and I see it in the headers of my api call, but when I send the request to my Flask app the API headers are not there. I am printing out all the headers and the "api_key" is not there. What am missing?
all_headers = dict(request.headers)
print(f"api key is {all_headers}")
api key is {'Content-Type': 'application/json', 'User-Agent':
'PostmanRuntime/7.29.2', 'Accept': '/', 'Postman-Token':
'908a9c7f-ca49-481e-9893-1e3a780887bc', 'Host': '127.0.0.1',
'Accept-Encoding': 'gzip, deflate, br', 'Connection': 'keep-alive'}

500 Internal Server error when trying to post using (requests.post()) in python

I am new to python and I am trying to automate post API in which I am using post method as follows:-
headers = {'Content-Type': 'application/json',
'Authorization': 'Basic xxxx=='}
response = requests.post(url=urlsales, data=payload,
headers=headers)
After running the test, getting the following output :-
print("response headers >>>>", response.headers)
response headers >>>> {'Date': 'Fri, 30 Oct 2020 14:11:34 GMT', 'Content-Type': 'application/json', 'Content-Length': '187', 'Connection': 'keep-alive', 'X-Error-Code': '500.45.001', 'Access-Control-Allow-Methods': 'GET,POST,OPTIONS,PATCH,DELETE,PUT', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'Accept, Accept-Encoding, Authorization, Content-Length, Content-Type, Host, User-Agent, X-Forwarded-For, X-Forwarded-Port, X-Forwarded-Proto'}
print("headers >>", headers)
{'Content-Type': 'application/json',
'Authorization': 'Basic xxxx=='}
I am assuming there is issue with headers as when I send only 'Content-Type': 'application/json' in header , then i get 401 unauthorized which is as expected but when I add 'Authorization': 'Basic xxxx==' as mentioned above I start getting 500 internal server error
can someone please help with the possible root cause where it might have went wrong

get specific value from http response

I am using requests library to query a F5 Big IP. I get the list of Virtual Server. I need to do a loop to get each VS name (VS1, VS2, VS3) from the response to use in another request like
https://localhost/mgmt/tm/ltm/virtual/VS1
What code will get each name value from the response? I tried this but could not get it to work.
url = "https://bigipname.domain.local/mgmt/tm/ltm/virtual"
querystring = {"$select":"name"}
headers = {
'Content-Type': "application/json",
'Accept': "*/*",
'Cache-Control': "no-cache",
'Host': "bigipgname.domain.local",
'accept-encoding': "gzip, deflate",
'Connection': "keep-alive",
'cache-control': "no-cache"
}
response = requests.request("GET", url, headers=headers, params=querystring, verify=False)
I get the response in the following json format :
{'kind': 'tm:ltm:virtual:virtualcollectionstate', 'selfLink': 'https://localhost/mgmt/tm/ltm/virtual?$select=name&ver=13.1.1.2', 'items': [{'name': 'VS1'}, {'name': 'VS2'}, {'name': 'VS3'}]}
Any help is appreciated. Thanks
You can use a list comprehension to extract the "items".
new_list = [item["name"] for item in response["items"]]

How to set up a request header for an Authenticated API GET request

I'm trying to make an authenticated GET request to an API. This is one of my first attempts working with Python's request library. I've looked over similar posts to this one, but they're a bit too generic to answer my question, it seems. Their answers work for nearly every other case I've worked with, so it feel a bit stuck.
The request header is fairly lengthy:
':authority': 'api-WEBSITE.com',
':method': 'GET',
':path': 'ENDPOINT',
':scheme': 'https',
'accept': 'application/json',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'authorization': 'AUTH_TOKEN',
'content-type': 'application/json',
'cookie': 'VERY_LONG_COOKIE',
'origin': 'https://WEBSITE.com',
'referer': 'https://WEBSITE.com/app',
'user-agent': 'LIST_OF_BROWSERS'
My code that makes this request:
import requests
requestURL = "https://api-WEBSITE.com/ENDPOINT"
parameters = {
':authority': 'api-WEBSITE.com',
':method': 'GET',
':path': 'ENDPOINT',
':scheme': 'https',
'accept': 'application/json',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'authorization': 'AUTH_TOKEN',
'content-type': 'application/json',
'cookie': 'VERY_LONG_COOKIE',
'origin': 'https://WEBSITE.com',
'referer': 'https://WEBSITE.com/app',
'user-agent': 'LIST_OF_BROWSERS'
}
response = requests.get(requestURL, parameters)
print(response.status_code)
When I run this, I'm getting a 401 status code asking for authentication; however, I can't seem to find out what's throwing this 401 error.
To supply headers for a python request: you must do this
r = requests.get(url, headers = headersDict)
Where headersDict is a valid dictionary of the headers you want added to the request

Authorization Header stripped from request inside Docker Container

I have a python 3.x application that uses requests to submit data to an API. The API requires an Authorization header. Relevant code looks something like this:
the_headers = {'Authorization': auth_header_token}
response = requests.post(
the_url,
json=the_data,
headers=the_headers
)
Works fine when I run it on a server or client machine.
BUT when I run it from a Docker container, I get a bad authorization header error.
In both scenarios, I get the following output:
print(res.request.headers)
{
'User-Agent': 'python-requests/2.18.4',
'Accept-Encoding': 'gzip, deflate',
'Accept': '*/*',
'Connection': 'keep-alive',
'Authorization': 'auth_header_token',
'Content-Length': '2984',
'Content-Type': 'application/json'
}
But the team that supports the API tells me my failing request does not contain the Auth header
Good request:
{
'x-ssl': 'on',
host: 'ip1:port1',
connection: 'close',
'content-length': '2984',
accept: '*/*',
'accept-encoding': 'gzip, deflate',
authorization: 'auth_header_token',
'content-type': 'application/json',
'user-agent': 'python-requests/2.18.4',
'x-forwarded-for': 'ip2',
'x-forwarded-port': 'port2',
'x-forwarded-proto': 'https'
}
Bad request:
{
'x-ssl': 'on',
host: 'ip1:port1',
connection: 'close',
'content-length': '2984',
accept: '*/*',
'accept-encoding': 'gzip, deflate',
'content-type': 'application/json',
'user-agent': 'python-requests/2.18.4',
'x-forwarded-for': 'ip3',
'x-forwarded-port': 'port2',
'x-forwarded-proto': 'https'
}
I had originally thought that this issue was perhaps related to an issue on Github Session's Authorization header isn't sent on redirect, but both requests have x-forwarded-for on the receiving end, so now I'm not so sure.
Same code base, only thing that changes is runtime environment (MacOS vs Docker container running on MacOS).

Categories