We Keep Coding

Why does requests ignore my custom header field?

I have the following Python function which sends a post request using the requests library:
def http_post(self, url: str, headers: dict, data: str, auth: AuthBase):
token = self._xsuaa.get_token(self._service)
headers.update({'Proxy-Authorization': f"Bearer {token}"})
res = requests.post(
url,
headers=headers,
data=data,
proxies={'http': self._proxy},
auth=auth,
verify=False,
timeout=100,
allow_redirects=True)
When printing the headers dict, it looks like this:
{
'Content-Type': 'multipart/mixed;boundary=batch_4724f345-bb46-437d-a970-197a7b82bf41',
'Content-Transfer-Encoding': 'binary',
'sap-cancel-on-close': 'true',
'sap-contextid-accept': 'header',
'Accept': 'application/json',
'Accept-Language': 'de-DE',
'DataServiceVersion': '2.0',
'MaxDataServiceVersion': '2.0',
'Proxy-Authorization': 'Bearer <token>'
}
However, when I take a look at res.request.headers, I get the following:
{
'User-Agent': 'python-requests/2.26.0',
'Accept-Encoding': 'gzip, deflate',
'Accept': 'application/json',
'Connection': 'keep-alive',
'Content-Type': 'multipart/mixed; boundary=batch_4724f345-bb46-437d-a970-197a7b82bf41',
'Content-Transfer-Encoding': 'binary',
'sap-cancel-on-close': 'true',
'sap-contextid-accept': 'header',
'Accept-Language': 'de-DE',
'DataServiceVersion': '2.0',
'MaxDataServiceVersion': '2.0',
'Content-Length': '659',
'Authorization': 'Basic <auth>'
}
For some reason, the proxy-authorization header field is gone and accordingly, I get a 407 error in the response. I have read in the documentation that proxy credentials provided in the URL overwrite proxy-authorization headers, but my URL contains none. I also tried removing the auth=auth line from the request, but the problem still persited. Can someone point me in the right direction as to why this field is seemingly ignored or overwritten by requests?

Unable to login to wordpress site with Python3 Requests Module

I'm writing a python bot to scrape a wordpress site but I need to be logged in to access some of the resources I want. Here is the code:
# username, password, and wp_login is defined above
with requests.Session() as s:
# store any initial cookies in the session
s.get(wp_login)
datas={
'log':username, 'pwd':password, 'wp-submit':'Log In',
'redirect-to':'', 'testcookie':'1'
}
# successful or unsuccessful login returns a 302 so I have to not allow
# redirects or I can't view the response header to check for the cookies I need
response = s.post(wp_login, data=datas, allow_redirects=False)
print(response.status_code)
print()
print(response.request.headers)
print()
print(response.headers)
Here is the output
302
{'User-Agent': 'python-requests/2.22.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': '__cfduid=[REDACTED]', 'Content-Length': '57', 'Content-Type': 'application/x-www-form-urlencoded'}
{'Date': 'Mon, 30 Sep 2019 06:13:43 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'X-Powered-By': 'PHP/5.6.32', 'X-Redirect-By': 'WordPress', 'Location': 'https://________/wp-login.php', 'Server': 'cloudflare', 'CF-RAY': '51e40d72d87c9691-SJC'}
I should be receiving a cookie on successful login and the redirect in the response header should be to the home page. So authentication is failing, and I don't understand why?
Edit: A successful POST request, when viewed in Chrome Developer tools, has the following data string:
log=[username]&pwd=[password]&wp-submit=Log+In&redirect_to=&testcookie=1

get specific value from http response

I am using requests library to query a F5 Big IP. I get the list of Virtual Server. I need to do a loop to get each VS name (VS1, VS2, VS3) from the response to use in another request like
https://localhost/mgmt/tm/ltm/virtual/VS1
What code will get each name value from the response? I tried this but could not get it to work.
url = "https://bigipname.domain.local/mgmt/tm/ltm/virtual"
querystring = {"$select":"name"}
headers = {
'Content-Type': "application/json",
'Accept': "*/*",
'Cache-Control': "no-cache",
'Host': "bigipgname.domain.local",
'accept-encoding': "gzip, deflate",
'Connection': "keep-alive",
'cache-control': "no-cache"
}
response = requests.request("GET", url, headers=headers, params=querystring, verify=False)
I get the response in the following json format :
{'kind': 'tm:ltm:virtual:virtualcollectionstate', 'selfLink': 'https://localhost/mgmt/tm/ltm/virtual?$select=name&ver=13.1.1.2', 'items': [{'name': 'VS1'}, {'name': 'VS2'}, {'name': 'VS3'}]}
Any help is appreciated. Thanks

You can use a list comprehension to extract the "items".
new_list = [item["name"] for item in response["items"]]

How to set up a request header for an Authenticated API GET request

I'm trying to make an authenticated GET request to an API. This is one of my first attempts working with Python's request library. I've looked over similar posts to this one, but they're a bit too generic to answer my question, it seems. Their answers work for nearly every other case I've worked with, so it feel a bit stuck.
The request header is fairly lengthy:
':authority': 'api-WEBSITE.com',
':method': 'GET',
':path': 'ENDPOINT',
':scheme': 'https',
'accept': 'application/json',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'authorization': 'AUTH_TOKEN',
'content-type': 'application/json',
'cookie': 'VERY_LONG_COOKIE',
'origin': 'https://WEBSITE.com',
'referer': 'https://WEBSITE.com/app',
'user-agent': 'LIST_OF_BROWSERS'
My code that makes this request:
import requests
requestURL = "https://api-WEBSITE.com/ENDPOINT"
parameters = {
':authority': 'api-WEBSITE.com',
':method': 'GET',
':path': 'ENDPOINT',
':scheme': 'https',
'accept': 'application/json',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'authorization': 'AUTH_TOKEN',
'content-type': 'application/json',
'cookie': 'VERY_LONG_COOKIE',
'origin': 'https://WEBSITE.com',
'referer': 'https://WEBSITE.com/app',
'user-agent': 'LIST_OF_BROWSERS'
}
response = requests.get(requestURL, parameters)
print(response.status_code)
When I run this, I'm getting a 401 status code asking for authentication; however, I can't seem to find out what's throwing this 401 error.

To supply headers for a python request: you must do this
r = requests.get(url, headers = headersDict)
Where headersDict is a valid dictionary of the headers you want added to the request

Authorization Header stripped from request inside Docker Container

I have a python 3.x application that uses requests to submit data to an API. The API requires an Authorization header. Relevant code looks something like this:
the_headers = {'Authorization': auth_header_token}
response = requests.post(
the_url,
json=the_data,
headers=the_headers
)
Works fine when I run it on a server or client machine.
BUT when I run it from a Docker container, I get a bad authorization header error.
In both scenarios, I get the following output:
print(res.request.headers)
{
'User-Agent': 'python-requests/2.18.4',
'Accept-Encoding': 'gzip, deflate',
'Accept': '*/*',
'Connection': 'keep-alive',
'Authorization': 'auth_header_token',
'Content-Length': '2984',
'Content-Type': 'application/json'
}
But the team that supports the API tells me my failing request does not contain the Auth header
Good request:
{
'x-ssl': 'on',
host: 'ip1:port1',
connection: 'close',
'content-length': '2984',
accept: '*/*',
'accept-encoding': 'gzip, deflate',
authorization: 'auth_header_token',
'content-type': 'application/json',
'user-agent': 'python-requests/2.18.4',
'x-forwarded-for': 'ip2',
'x-forwarded-port': 'port2',
'x-forwarded-proto': 'https'
}
Bad request:
{
'x-ssl': 'on',
host: 'ip1:port1',
connection: 'close',
'content-length': '2984',
accept: '*/*',
'accept-encoding': 'gzip, deflate',
'content-type': 'application/json',
'user-agent': 'python-requests/2.18.4',
'x-forwarded-for': 'ip3',
'x-forwarded-port': 'port2',
'x-forwarded-proto': 'https'
}
I had originally thought that this issue was perhaps related to an issue on Github Session's Authorization header isn't sent on redirect, but both requests have x-forwarded-for on the receiving end, so now I'm not so sure.
Same code base, only thing that changes is runtime environment (MacOS vs Docker container running on MacOS).