We Keep Coding

Python Requests / HTTPX - Authenticate All Redirected URL - How To

I wonder if there is a way to authenticate each redirected URL when working with Python modules such as httpx or requests?
Problem Statement
I am trying to connect to an API endpoint under the company network. Due to the company's cyber security measures, the API endpoint will be randomly masked with a company proxy, which causes the 307 Redirect status code.
my current code snippet looks like the below:
import httpx
api_url = 'https://demo.vizionapi.com/carriers'
head = {
'X-API-Key':'API KEY'
}
response = httpx.get(url=api_url, verify='supporting_files/cacert.pem',
headers=head, auth=('my username', 'my password'),
follow_redirects=True)
With above code, I received the 401 authentication needed error (But auth has been passed). This error will only happen when redirection occurs due to the company proxy.
Question:
My assumption is the authentication is only being passed into the first URL not the redirected URL. Therefore, I wonder if anyone know how I can use the same auth parameter for all URLs (direct & redirect)?
Any suggestion will be deeply appracaited.

I don't know what requests behavior with regards to auth during redirect is, but the first solution to come to mind is to manually follow the redirects yourself. Put your request in a loop that checks for the 3xx response codes, and handle auth however you want to.

Parse Error: The server returned a malformed response - Error: Parse Error: Expected HTTP/ - when trying to access one of my app URLs with postman

I am managing an app built by third parts in python.
I have this url dispatcher
urls += [(r'/path/objectAlpha/(.*)', objectAlphaHandler)] # this was made by third parts, it is expected to work
and this class
class objectAlphaHandler(BaseHandler):
def __init__(self, application, request, **kwargs):
super(objectAlphaHandler, self).__init__(application, request, **kwargs) # do the init of the basehandler
#gen.coroutine
def post(self, action=''):
response = {}
...
response = yield self.my_method(json_data)
...
self.write(json.dumps(response))
def my_method(self, json_data)
...
I want to check that the app correctly receives the request and returns some response.
So I try to access that url with Postman
request type:
POST
URL:
http://<machine_ip>:<machine_port>/path/objectAlpha/
I get this error from Postman response box
Parse Error: The server returned a malformed response
and when I click on "view in console" I see
POST http://machine_ip>:<machine_port>/path/objectAlpha/
Error: Parse Error: Expected HTTP/
Request Headers
Content-Type: application/json
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: d644d7dd-699b-4d77-b32f-46a575ae31fc
Host: xx.xxx.x.xx:22
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Request Body
What does Error: Parse Error: Expected HTTP/ mean?
I checked my app logs but it seems it is not handling any request, even if Postman indicates that the server is returning a (malformed) response.
I also tryed to chenge the target url to:
https...
but it returns
Error: write EPROTO 28427890592840:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:../../third_party/boringssl/src/ssl/tls_record.cc:242:
which I found out it indicates I should stick with HTTP
Then I tried also:
http://<machine_ip>/path/objectAlpha/
and
<machine_ip>/path/objectAlpha/
which generically return:
Error: connect ECONNREFUSED <machine_ip>:80
I also tryed to substitute line
urls += [(r'/path/objectAlpha/(.*)', objectAlphaHandler)]
with
urls += [(r'/path/objectAlpha/', objectAlphaHandler)]
and
urls += [(r'/path/objectAlpha', objectAlphaHandler)]
but none of these worked.
What is wrong? How can I fix it?
UPDATE
Apparently, according to this thread on Postman Github, the problem happens only on Postman Desktop and not on Postman on browser.
So I tryed to send the request form Postman on my browser but I get
Cloud Agent Error: Can not send requests to reserved address. Make sure address is publicly accessible or select a different agent.
because, according to this other thread,
Postman Website cannot send a request to your computer's localhost. It first needs to connect to your PC with the Postman desktop client
and even if I follow the indications in that answer
Run it [ndr. Postman desktop], then go to the Postman workspace in your browser -> send the request and it will work.
I still get the same
Error: Parse Error: Expected HTTP/
on both Postman Desktop and Postman on browser.
UPDATE
Going on debugging, I tryed to cast a curl on that URL from my terminal:
myuser#mymachine-VirtualBox:~$ curl --verbose "http://<target_machine_ip>:<target_machine_port>/path/objectAlpha"
and I got:
* Trying <target_machine_ip>:<target_machine_port>...
* Connected to <target_machine_ip> (<target_machine_ip>) port <target_machine_port> (#0)
> GET /orkpos5/receipt HTTP/1.1
> Host: <target_machine_ip>:<target_machine_port>
> User-Agent: curl/7.74.0
> Accept: */*
>
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed

SOLVED
Since the exposed API has been built by third parts for internal use, it is not exposed to the public.
I did not know that, so in the request URL I had put the well-known port number for HTTP requests, the number 22 (see the list of well-known port numbers.).
To solve the problem, in the request URL, I changed the <target_machine_port> to the port on which the API is actually exposed.

Python OAuth Client Credentials Grant Type Returned 400, [error: unsupported grant] type but Curl Works Fine

I am very new to APIs (still learning) and I encountered a very weird issue with Python requests library when trying to initiate an OAuth Authentication flow with Client Credentials Grant Type.
For some reason, whenever I used my Python script (with the help of requests library) to send the HTTP request to the authentication endpoint, I always get
Response Status Code: 400
Response Body/Data returned: {"error":"unsupported_grant_type"}
However, if I tried using curl command line tool to send the request, I will get a successful response with status code 200 with the access token in the response body like this:
{'access_token': 'some access token',
'expires_in': 'num_of_seconds',
'token_type': 'Bearer'}
As a matter of fact, if I tried sending the request using Curl command line tool WITHIN my Python Script (with subprocess.Popen function), I can get the response with status code 200 and the access token with no problem.
Now, with that said, here's the Python script that I used to send the request to initiate the OAuth authentication flow:
import requests
import os
import base64
clientCredentialEndpoint = "https://base_url/path/token"
client_id = os.environ.get('CLIENT_ID')
client_secret = os.environ.get('CLIENT_SECRET')
# -- Encode the <client_id:client_secret> string to base64 --
auth_value = f'{client_id}:{client_secret}'
auth_value_bytes = auth_value.encode('ascii')
auth_value_b64 = base64.b64encode(auth_value_bytes).decode('ascii')
queryParams ={
'grant_type':'client_credentials',
'scope':'get_listings_data'
}
headers = {
'Authorization':f'Basic {auth_value_b64}',
'Content-Type':'application/x-www-form-urlencoded'
}
# send the post request to Authorisation server
response = requests.post(
clientCredentialEndpoint,
params=queryParams,
headers=headers,
)
print(response.status_code)
print(response.text)
whereas the curl command that I used (and worked) to send the request is:
curl -X POST -u '<client_id>:<client_secret>' \
-H "Content-Type: application/x-www-form-urlencoded" \
-d 'grant_type=client_credentials&scope=get_listings_data' \
'https://base_url/path/token'
Again, like I said, if I execute this curl command inside a Python script, it will successfully return the access token with no issue.
Does anyone know what I did wrong in my Python script which caused my request to always fail?
Thanks in advance!

My goodness me, I just realised that the -d in the curl command does not correspond to query params, it stands for 'data'.
Hence, I just need to change my Python script requests.post() a bit so that it looks like this:
response = requests.post(
clientCredentialEndpoint,
data=queryParams,
headers=headers,
)
Hope this helps others.

Streamlabs API 405 response code

I'm trying to use Streamlabs API. Streamlabs API uses Oauth2 for creating apps. So first I send whoever's using my app to an authorization link containing my app's client id and the scopes I want to use.
(Something like this: streamlabs.com/api/v1.0/authorize?client_id=CLIENT-ID-HERE&redirect_uri=REDIRECT-URI&response_type=code&scope=SOME+SCOPES+HERE)
Once I've done that I receive a code at the redirect uri specified. I then use that code to get the access token for permanent access to the connected user's account. I then receive the access token from a POST request that works perfectly... Now I run into the problem. When getting the temporary code before the access token I specified the scopes: "donations.read +donations.create+alerts.write+alerts.create".
When authorizing, the app asks for permission to the different scopes. The scope in focus is "alerts.write" so that I can send test alerts using POST requests. But this doesn't work for some reason. To send a test alert I have to send a POST request to this url: "https://streamlabs.com/api/alerts/send_test_alert"
I've tried doing that in two different ways.
1:
import requests
url = "https://streamlabs.com/api/alerts/send_test_alert"
data = {
"access_token":"UserAccessTokenHere",
"type":"donation"
}
response = requests.post(url=url, data=data)
print(response.text)
2:
import requests
url = "https://streamlabs.com/api/alerts/send_test_alert?access_token=UserAccessTokenHere&type=donation"
response = requests.post(url=url)
print(response.text)
If I do print(response) it prints "Response [405]".
But if I do print(response.text) I get a long HTML document for this page: Error response page
Any ideas what's going wrong with my Python requests? send_test_alert documentation here: Link

I've contacted support and looks like you've made the same error as me.
You're not actually sending a request to the right URL.
You are a sending a request to: "https://streamlabs.com/api/alerts/send_test_alert"
You should be using the URL: "https://streamlabs.com/api/v1.0/alerts/send_test_alert"

403 when retrieving a WSDL via Python SUDS

I can't seem to get SUDS to download a WSDL that requires Basic auth credentials. My code is simple:
wsdl_url = 'https://example.com/ChangeRequest.do?WSDL'
self.client = Client(wsdl_url, username=username, password=password)
I've also tried:
from suds.transport.https import HttpAuthenticated
wsdl_url = 'https://example.com/ChangeRequest.do?WSDL'
credentials = dict(username=username, password=password)
t = HttpAuthenticated(**credentials)
self.client = Client(url=wsdl_url, transport=t)
In both cases, the service returns a 403 Forbidden error. I can go down into the SUDS code in http.py and add this line to the call:
u2request.add_header('Authorization','Basic xxxxxxxxxxxxxxxxxxxx')
This works. What am I doing wrong to get SUDS to pass my credentials when downloading the WSDL?
Note: I try to connect to the WSDL directly using both Chrome's Postman plugin and SoapUI, and the service works as well. So I know the credentials are correct.

I encountered a similar issue (suds v0.4, wsdl, 403), and found out that it was because the server I'm trying to access blocks any requests with the header User-Agent set like Python-urllib* (suds is using urllib2, hence the default header). Explicitly change the header solves the issue.
Particular to my solution: I overrode the open method of a transport class, and set client options, like the following code snippet. Note that we need to explicitly set for open and subsequent requests separately. Please advice better ways to circumvent this if you know any. And hope this post could help save someone's time in the future.
import urllib2
import suds
from suds.transport.https import HttpAuthenticated
from suds.transport import TransportError
URL = 'https://example.com/ChangeRequest.do?WSDL'
class HttpHeaderModify(HttpAuthenticated):
def open(self, request):
try:
url = request.url
u2request = urllib2.Request(url, headers={'User-Agent': 'Mozilla'})
self.proxy = self.options.proxy
return self.u2open(u2request)
except urllib2.HTTPError, e:
raise TransportError(str(e), e.code, e.fp)
transport = HttpHeaderModify()
client = Client(URL, transport=transport, timeout=10)
# Subsequent requests' header needs to be set again here. The overridden transport
# class only handles opening of the client.
client.set_options(headers={'User-Agent': 'Mozilla'})
P.S. Though my problem may not be the same, searching for "403 suds" pops up this SO question, so I decide just post my solution here.
reference post that gave me the right direction: https://bitbucket.org/jurko/suds/issues/27/client-request-for-wsdl-does-not-use-given

I used to have this issue before and compare with the soap UI header.
Found that suds missing to include the header (Host).
client.set_options(headers={'Host': 'value'})
And issue fixed.