We Keep Coding

Python requests - download image and write to file not working due to nature of URI and authentification procedure

I am writing a script that downloads Sentinel 2 products (satellite imagery) using sentinelsat Python API.
A product's description is structured as JSON and contains the parameter quicklook_url.
Example:
https://apihub.copernicus.eu/apihub/odata/v1/Products('862619d6-9b82-4fe0-b2bf-4e1c78296990')/Products('Quicklook')/$value
Any Sentinel API calls require credentials. So does retrieving a product and also opening the link stored inside quicklook_url. When I call the example in my browser I get asked to enter username and password in order to get
with the name S2A_MSIL2A_20210625T065621_N0300_R063_T39NTJ_20210625T093748-ql.jpg.
Needless to say I am just starting with the API so I am probably missing something but
requests.post(product_description['quicklook_url'], verify=False, auth=HTTPBasicAuth(username, password)).content
yields 0KB damaged file and
requests.get(product_description['quicklook_url']).content
yields 1KB damaged file.
I have looked into requests.Session
session = requests.Session()
session.auth = (username, password)
auth = session.post('URL_FOR_LOGING_IN')
img = session.get(product_description['quicklook_url']).content
The problem is I am unable to find the URL I need to post my session authentification. I am somewhat sure that the sentinelsat API does that but my looks have not yielded any successful result.
I am currently looking into the SentinelAPI class. It has the download_quicklook() function, which I am using right now but I am still curious how to do this without the function.

I guess you don't need to sent a post request. Basic authentication works by sending a header along with each request. The following should work
session = requests.Session()
session.auth = (username, password)
img = session.get(product_description['quicklook_url']).content
Your first attempt is failed because of using POST I think.
requests.gett(product_description['quicklook_url'], verify=False, auth=HTTPBasicAuth(username, password)).content
should also work.

Receiving 'BadRequestKeyError' when attempting to GET data from endpoint url

I was trying to implement an API using Python, and flask to help myself learn and practice REST.
The idea was to receive a HTTP POST with data that looks like as such:
{"startDate":"2015-07-01","endDate":2015-07-08","within":{"value":9000,"units":miles}} and send some of the data to a NASA API(endpoint).
I was able to create a POST method , and I am able to receive the data (both in POSTMAN and in the browser). Here is the relevant code :
#neows.route('/UserInput',methods=['GET','POST'])
def UserInput():
startDate = request.args.get('startDate')
endDate = request.args.get('endDate')
#print (type(startDate))
#print (type(endDate))
getAsteroids(startDate,endDate)
return jsonify(request.args)
But when I extract some data from the POST above to send to a NASA API (GET), I am receiving this error:
werkzeug.exceptions.BadRequestKeyError
Here is the url I am trying to hit : (https://api.nasa.gov/neo/rest/v1/feed?start_date=START_DATE&end_date=END_DATE&api_key=API_KEY)
I am able to hit the url both on POSTMAN and browser, outside of my code.
The relevant piece of code with the error is posted below and the line that seems to be throwing the error is in Italics (marked with *).
def getAsteroids(startDate,endDate):
API_KEY='xxx'
print (startDate)
print (endDate)
*result=request.args["https://api.nasa.gov/neo/rest/v1/feed?
start_date="+startDate+"&end_date="+endDate+"&api_key="+API_KEY+""]*
I would really appreciate if some one could help me understand and resolve this issue.

If you want to do a request against the NASA's API you can use requests module. (Or any other module to send HTTP requests)
import requests
# ...
def getAsteroids(startDate, endDate):
API_KEY='xxx'
payload = {'start_date': startDate, 'end_date': endDate, 'api_key': API_KEY}
result = requests.get('https://api.nasa.gov/neo/rest/v1/feed', params=payload)
request.args is something different used to get the parameters of the incoming request.

Streamlabs API 405 response code

I'm trying to use Streamlabs API. Streamlabs API uses Oauth2 for creating apps. So first I send whoever's using my app to an authorization link containing my app's client id and the scopes I want to use.
(Something like this: streamlabs.com/api/v1.0/authorize?client_id=CLIENT-ID-HERE&redirect_uri=REDIRECT-URI&response_type=code&scope=SOME+SCOPES+HERE)
Once I've done that I receive a code at the redirect uri specified. I then use that code to get the access token for permanent access to the connected user's account. I then receive the access token from a POST request that works perfectly... Now I run into the problem. When getting the temporary code before the access token I specified the scopes: "donations.read +donations.create+alerts.write+alerts.create".
When authorizing, the app asks for permission to the different scopes. The scope in focus is "alerts.write" so that I can send test alerts using POST requests. But this doesn't work for some reason. To send a test alert I have to send a POST request to this url: "https://streamlabs.com/api/alerts/send_test_alert"
I've tried doing that in two different ways.
1:
import requests
url = "https://streamlabs.com/api/alerts/send_test_alert"
data = {
"access_token":"UserAccessTokenHere",
"type":"donation"
}
response = requests.post(url=url, data=data)
print(response.text)
2:
import requests
url = "https://streamlabs.com/api/alerts/send_test_alert?access_token=UserAccessTokenHere&type=donation"
response = requests.post(url=url)
print(response.text)
If I do print(response) it prints "Response [405]".
But if I do print(response.text) I get a long HTML document for this page: Error response page
Any ideas what's going wrong with my Python requests? send_test_alert documentation here: Link

I've contacted support and looks like you've made the same error as me.
You're not actually sending a request to the right URL.
You are a sending a request to: "https://streamlabs.com/api/alerts/send_test_alert"
You should be using the URL: "https://streamlabs.com/api/v1.0/alerts/send_test_alert"

Amazon Kinesis Video GetMedia/PutMedia

I used python 3.6 and I want to post video stream to aws kinesis with API.
I used python aws client to create stream and GetDataEndPoint but when I want to post my data with my custom request (PutMedia doesn't include in python client actually), I get an error Unable to determine service/operation name to be authorized.
I've follow the api doc of aws kinesis video media PutMedia and GetMedia.
So I start by getting endpoint with GetDataEndPoint with client method:
response = client.get_data_endpoint( # aws client method
StreamName=STREAM_NAME,
APIName='PUT_MEDIA'
)
end_point = response['DataEndpoint'] # https://s-EXAMPLE.kinesisvideo.eu-west-1.amazonaws.com
and I post my data at this url:
headers = {
"x-amzn-stream-arn": STREAM_ARN,
"x-amzn-fragment-timecode-type": "ABSOLUTE",
"x-amzn-producer-start-timestamp": start_tmstp
}
# Sign header...
response = requests.post(end_point, data=data, headers=headers) # 403 - Unable to determine service/operation name to be authorized
So I don't understand why I get this error... I've found this troubleshooting on aws doc. But they say we must specify ApiName parameter. What I do...
This error might occur if the endpoint is not properly specified. When you are getting the endpoint, be sure to include the following parameter in the GetDataEndpoint call, depending on the API to be called:
I'm also wondering if the GetMedia method is actually implemented in client as they say here because when I debug this method, client don't call GetDataEndPoint and so make request at https://kinesisvideo.region.amazonaws.com insteed of https://ID_EXAMPLE.kinesisvideo.region.amazonaws.com. So method get error Unable to determine service/operation name to be authorized as explained in troubleshooting

The error you're getting is because you're probably providing the endpoint without the "action" that in your case would be putMedia.
Try to append /putMedia to your endpoint and don't forget to specify the "content-type": "application/json" header.
Btw you have also to generate the v4 signatures for your request. You can use a lib or follow this python guide to do it.

How to use oauth2 to access StackExchange API?

I'm following the instructions mentioned here: https://api.stackexchange.com/docs/authentication
But since there is no code provided, I'm not able to understand the flow correctly.
I've been trying to get the authentication part done using two methods below but I have hit a deadend.
1)
import requests
from pprint import pprint
resp = requests.get('https://stackexchange.com/oauth/dialog?client_id=6667&scope=private_info&redirect_uri=https://stackexchange.com/oauth/login_success/')
pprint(vars(resp))
2)
import oauth2 as oauth
from pprint import pprint
url = 'https://www.stackexchange.com'
request_token_url = '%s/oauth/' % url
access_token_url = '%s/' % url
consumer = oauth.Consumer(key='mykey',
secret='mysecret')
client = oauth.Client(consumer)
response, content = client.request(request_token_url, 'GET')
print(response, content)
I'm not sure how to go forward from here? I need to use the access token that is returned and use it to query the API. A sample code would really really help! Thanks.
EDIT: This is the code I'm using currently:
from requests_oauthlib import OAuth2Session
from pprint import pprint
client_id = 'x'
client_secret = 'x'
redirect_uri = 'https://stackexchange.com/oauth/login_success'
scope = 'no_expiry'
oauth = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scope)
pprint(vars(oauth))
authorization_url, state = oauth.authorization_url('https://stackexchange.com/oauth/dialog')
print(authorization_url)
Instead of having to click on the authorization_url and get the token, is there a way I can directly fetch the token within the script itself?

Of the two methods you used, the first is the recommended method for desktop applications. It is probably correct.
OAuth is intended to force the user to go to a specific webpage and acknowledge that they are giving permission (usually through clicking a button) for an application to access their data. The HTTP responses you print are merely the webpage where a user needs to click accept.
To get a feeling for the flow, put the first address (https://stackexchange.com/oauth/dialog?client_id=6667&scope=&redirect_uri=https://stackexchange.com/oauth/login_success/) in the address bar and click accept on the loaded page. The access_token will be in the URL right after that.
If you are making the application only for yourself, the access_token can be copied into your Python script. The token expires in one day; if that is too short add no_expiry to scope to make it last forever. DO NOT share the token with anyone else, since it gives them access to details of your account! Each user of the script must generate their own token.
Test the access_token by inserting in your app's key and the access_token you just obtained into the url: https://api.stackexchange.com/2.2/me?key=key&site=stackoverflow&order=desc&sort=reputation&access_token=&filter=default
If you need a more automated, integrated, user-friendly solution, I would look at selenium webdriver to open a browser window and get the resulting credentials.

Just one minor correction on Marc's answer. If you want the access token to last forever, you should add no_expiry instead of no_expire.