How can I convert this Curl call into Python 3? - python

I have the following curl that successfully logs me into a website. It returns a 302 and a cookie:
curl \
--verbose \
--request POST \
--data '__EVENTTARGET=&body%3Ax%3AtxtLogin=kingkong%40mailinator.com&body%3Ax%3AbtnLogin=&crypted_pass=736015615f9e251692a6a4aa8a7baa14' \
"https://emma.maryland.gov/page.aspx/en/usr/login"
Unfortunately, I have to insert the real username & encrypted password of the account. otherwise this curl it won't work. However, this is a completely dummy account with NO private data in it. So please don't get mad at me.
I want to convert it to a Python3 code using requests.post(). So I made this code:
>>> requests.post(
... url='https://emma.maryland.gov/page.aspx/en/usr/login',
... data={
... '__EVENTTARGET': '',
... 'body:x:txtLogin': 'kingkong#mailinator.com',
... 'body:x:btnLogin': '',
... 'crypted_pass': '736015615f9e251692a6a4aa8a7baa14'
... }
... )
<Response [200]>
But the response I get from the Python3 code (200) doesn't match the response I get from the Curl (302). This means that the target server senses a difference between the two requests.
How can I convert the curl to Python3 that sends the exact same underlying HTTP request?

Your requests code is actually smarter than the cURL command.
HTTP 302 - is a redirect, cURL didn't follow it and gave you the first response it got. You can make cURL follow the redirect with -L: Is there a way to follow redirects with command line cURL?
The requests code followed the redirect and gave you the final response, which happened to be a HTTP 200.
Try your curl command with -L and see if you get HTTP 200 or not.
Alternatively, you can ask requests to not follow redirects with the allow_redirects=False option: Is there an easy way to request a URL in python and NOT follow redirects?

Related

Convert HTTP cURL request to python requests format

I am trying to convert the following cURL request to a python requests request.
I know the format for a GET request but not sure how to format the --output flag.
curl -u usr:pass -X GET "http://url-to-file.com/filefolder/filepath.txt" --output new_file_name.txt
This is what I have so far but it was returning a 405 Method not allowed error.
r = session.get('http://url-to-file.com/filefolder/filepath.txt')

Python OAuth Client Credentials Grant Type Returned 400, [error: unsupported grant] type but Curl Works Fine

I am very new to APIs (still learning) and I encountered a very weird issue with Python requests library when trying to initiate an OAuth Authentication flow with Client Credentials Grant Type.
For some reason, whenever I used my Python script (with the help of requests library) to send the HTTP request to the authentication endpoint, I always get
Response Status Code: 400
Response Body/Data returned: {"error":"unsupported_grant_type"}
However, if I tried using curl command line tool to send the request, I will get a successful response with status code 200 with the access token in the response body like this:
{'access_token': 'some access token',
'expires_in': 'num_of_seconds',
'token_type': 'Bearer'}
As a matter of fact, if I tried sending the request using Curl command line tool WITHIN my Python Script (with subprocess.Popen function), I can get the response with status code 200 and the access token with no problem.
Now, with that said, here's the Python script that I used to send the request to initiate the OAuth authentication flow:
import requests
import os
import base64
clientCredentialEndpoint = "https://base_url/path/token"
client_id = os.environ.get('CLIENT_ID')
client_secret = os.environ.get('CLIENT_SECRET')
# -- Encode the <client_id:client_secret> string to base64 --
auth_value = f'{client_id}:{client_secret}'
auth_value_bytes = auth_value.encode('ascii')
auth_value_b64 = base64.b64encode(auth_value_bytes).decode('ascii')
queryParams ={
'grant_type':'client_credentials',
'scope':'get_listings_data'
}
headers = {
'Authorization':f'Basic {auth_value_b64}',
'Content-Type':'application/x-www-form-urlencoded'
}
# send the post request to Authorisation server
response = requests.post(
clientCredentialEndpoint,
params=queryParams,
headers=headers,
)
print(response.status_code)
print(response.text)
whereas the curl command that I used (and worked) to send the request is:
curl -X POST -u '<client_id>:<client_secret>' \
-H "Content-Type: application/x-www-form-urlencoded" \
-d 'grant_type=client_credentials&scope=get_listings_data' \
'https://base_url/path/token'
Again, like I said, if I execute this curl command inside a Python script, it will successfully return the access token with no issue.
Does anyone know what I did wrong in my Python script which caused my request to always fail?
Thanks in advance!
My goodness me, I just realised that the -d in the curl command does not correspond to query params, it stands for 'data'.
Hence, I just need to change my Python script requests.post() a bit so that it looks like this:
response = requests.post(
clientCredentialEndpoint,
data=queryParams,
headers=headers,
)
Hope this helps others.

Posting a file with a REST API: from a curl example to Python code

I'm trying to code the upload of a file to a web service through a REST API in Python. The service's documentation shows a example using curl as client:
curl -X POST -H \
-H "Content-Type: multipart/form-data" \
-F "file=filename.ext" \
-F "property1=value1" \
-F "property2=value2" \
-F "property3=value3" \
https://domain/api/endpoint
The difficulty for me is that this syntax doesn't match multipart form-data examples I found, including the requests documentation. I tried this, which doesn't work (rejected by the API):
import requests
file_data = [
("file", "filename.ext"),
("property1", "value1"),
("property2", "value2"),
("property3", "value3"),
]
response = requests.post("https://domain/api/endpoint",
headers={"Content-Type": "multipart/form-data"}, files=file_data)
With the error: "org.apache.commons.fileupload.FileUploadException: the request was rejected because no multipart boundary was found"
Can anybody help in transposing that curl example to proper Python code?
Thanks!
R.
OK, looks like the web services documentation is wrong, and metadata simply needs to be sent as parameters. Moreover, I found in another request that you shouldn't set the header. So I was starting from a wrong example.

Django JWT Auth, why one request works and the other not

I was trying Django JWT Auth and noticed that the URL responds well to one type of post but doesn't respond well to another, but i can figure out why.
Basically, if i use the cURL POST referred in the readme.md, everything goes accordingly to planned:
$ curl -X POST -H "Content-Type: application/json" -d '{"username":"admin","password":"abc123"}' http://localhost:8000/api-token-auth/
but if you i use another type of cURL POST with the same info, it doesn't work:
$ curl -d 'username=admin&password=abc123' http://localhost:8000/api-token-auth/
I know that the "Content-Type" is diferent, but shouldn't the request be accepted in the same manner, they are both well formed posts?
Curl's -d option actually sends the request like it's a web browser. My guess is that the URL you're testing against doesn't have a standard web form, so it can't actually process the request.
TL;DR Pretty sure Django JWT Auth doesn't support the application/x-www-form-urlencoded content type.
From curl manual:
-d --data
(HTTP) Sends the specified data in a POST request to the HTTP
server, in the same way that a browser does when a user has
filled in an HTML form and presses the submit button. This will
cause curl to pass the data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F, --form.
Hope this helps!

Spaces in a URL when using requests and python

I hope I can explain myself. with out making an arse of myself.
I am trying to use python 3.4 to send a url to a sparkcore api.
I have managed to use curl direcly from the windows command line:-
curl https://api.spark.io/v1/devices/xxxxxxxxxxxxxxx/led -d access_token=yyyyyyyyyyyyyyyy -d params=l1,HIGH
All works fine. there is a space between the led and -d, but that is not a problem.
I have read that reting to do this within python using libcurl is a big pain and I saw lots of messaged about using Requests, so I though I would give it a go.
So I wrote a small routine:
import requests
r = requests.get('https://api.spark.io/v1/devices/xxxxxxxxxxxxxxxxxx/led -d access_token=yyyyyyyyyyyyyyyyy -d params=l1,HIGH')
print(r.url)
print(r)
I get as return:
<Response [400]>
When I examine the URL which actually got sent out the spaces in the URL are replaced with %20. This seems to be my actual problem, because the %20 being added by requests are confusing the server which fails
"code": 400,
"error": "invalid_request",
"error_description": "The access token was not found"
I have tried reading up on how to inpractice have the spaces with out having a %20 being added by the encoding, but I really could do with a pointer in the right direction.
Thanks
Liam
URLs cannot have spaces. The curl command you are using is actually making a request to the url https://api.spark.io/v1/devices/xxxxxxxxxxxxxxx/led with some command line arguments (using -d)
The curl man (manual) page says this about the -d command line argument
-d, --data
(HTTP) Sends the specified data in a POST request to the HTTP server, in the same way that a browser does when a user has filled in an HTML form and presses the submit button. This will cause curl to pass the data to the server using the content-type application/x-www-form-urlencoded. Compare to -F, --form.
-d, --data is the same as --data-ascii. To post data purely binary, you should instead use the --data-binary option. To URL-encode the value of a form field you may use --data-urlencode.
If any of these options is used more than once on the same command line, the data pieces specified will be merged together with a separating &-symbol. Thus, using '-d name=daniel -d skill=lousy' would generate a post chunk that looks like 'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a file name to read the data from, or - if you want curl to read the data from stdin. Multiple files can also be specified. Posting data from a file named 'foobar' would thus be done with --data #foobar. When --data is told to read from a file like that, carriage returns and newlines will be stripped out.
So that says -d is for sending data to the URL with the POST request using the content-type application/x-www-form-urlencoded
The requests documentation has a good example of how to do that using the requests library: http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests
So for your curl command, I think this should work
import requests
payload = {'access_token': 'yyyyyyyyyyyyyyyy', 'params': 'l1,HIGH'}
r = requests.post("https://api.spark.io/v1/devices/xxxxxxxxxxxxxxx/led", data=payload)
print(r.text)

Categories