I'm trying to execute my query using Athena boto client.
self.athena_client = boto3.client('athena')
response = self.athena_client.start_query_execution(
QueryString=sql.format(**query_params) if query_params else sql,
ResultConfiguration={
'OutputLocation': '...'
}
)
But I'm getting the error bellow:
Invalid length for parameter QueryString, value: 0, valid range: 1-inf
I couldn't figure out what is the root cause.
You should certify you are correctly sending the QueryString param to boto method start_query_execution.
It's a required parameter and it'll throw this error when you, for example, request send QueryString with an empty string.
Check more here at boto3 docs
Related
I am trying to get a report I created on SalesForce via simple_salesforce library in python. I am able to connect successfully. However, I get invalid session id error, because the link is wrong, that is created by simple_salesforce. The url I am trying to get is different from what simple_salesforce is searching (which is given in the error below).
The link I am trying to get is : "https://gkg-mfsa.lightning.force.com/lightning/r/Report/00O9N000000JwK2UAK/view?queryScope=userFolders"
But the link simple_salesforce is searching is : "https://gkg-mfsa.my.salesforce.com/services/data/v42.0/lightning/r/Report/00O9N000000JwK2UAK/view?queryScope=userFolders" (as given in the error)
How can I get simple_salesforce library to search for the link I am trying to get instead of what it looks for.
from simple_salesforce import Salesforce
sf = Salesforce(username='myUserName',
password='myPassword',
security_token='mySecurityToken',
instance_url = "")
report_id = 'myreportId'
sf.restful("lightning/r/Report/ + reportId + /view?queryScope=userFolders")
output
SalesforceExpiredSession: Expired session for https://gkg-mfsa.my.salesforce.com/services/data/v42.0/lightning/r/Report/00O9N000000JwK2UAK/view?queryScope=userFolders. Response content: [{'message': 'This session is not valid for use with the REST API', 'errorCode': 'INVALID_SESSION_ID'}]
Grab the session id and base endpoint from successful login call
session_id, instance = SalesforceLogin(
username='myemail#example.com',
password='password',
security_token='token')
And then run a REST request manually. But you'll have to pass the session id as a cookie, not as a "Authorisation Bearer <session_id>" http header.
There's an example in
https://github.com/simple-salesforce/simple-salesforce/issues/584
And one of my old answers (showing raw http but still, should give you an idea) https://stackoverflow.com/a/56162619/313628, https://stackoverflow.com/a/57745683/313628
I used python 3.6 and I want to post video stream to aws kinesis with API.
I used python aws client to create stream and GetDataEndPoint but when I want to post my data with my custom request (PutMedia doesn't include in python client actually), I get an error Unable to determine service/operation name to be authorized.
I've follow the api doc of aws kinesis video media PutMedia and GetMedia.
So I start by getting endpoint with GetDataEndPoint with client method:
response = client.get_data_endpoint( # aws client method
StreamName=STREAM_NAME,
APIName='PUT_MEDIA'
)
end_point = response['DataEndpoint'] # https://s-EXAMPLE.kinesisvideo.eu-west-1.amazonaws.com
and I post my data at this url:
headers = {
"x-amzn-stream-arn": STREAM_ARN,
"x-amzn-fragment-timecode-type": "ABSOLUTE",
"x-amzn-producer-start-timestamp": start_tmstp
}
# Sign header...
response = requests.post(end_point, data=data, headers=headers) # 403 - Unable to determine service/operation name to be authorized
So I don't understand why I get this error... I've found this troubleshooting on aws doc. But they say we must specify ApiName parameter. What I do...
This error might occur if the endpoint is not properly specified. When you are getting the endpoint, be sure to include the following parameter in the GetDataEndpoint call, depending on the API to be called:
I'm also wondering if the GetMedia method is actually implemented in client as they say here because when I debug this method, client don't call GetDataEndPoint and so make request at https://kinesisvideo.region.amazonaws.com insteed of https://ID_EXAMPLE.kinesisvideo.region.amazonaws.com. So method get error Unable to determine service/operation name to be authorized as explained in troubleshooting
The error you're getting is because you're probably providing the endpoint without the "action" that in your case would be putMedia.
Try to append /putMedia to your endpoint and don't forget to specify the "content-type": "application/json" header.
Btw you have also to generate the v4 signatures for your request. You can use a lib or follow this python guide to do it.
I am using a simple boto3 script to retrieve a parameter from SSM param store in my aws account. The python script looks like below:
client = get_boto3_client('ssm', 'us-east-1')
try:
response = client.get_parameter(Name='my_param_name',WithDecryption=True)
except Exception as e:
logging.error("retrieve param error: {0}".format(e))
raise e
return response
If the given parameter is not available, I get a generic error in the response like below:
An error occurred (ParameterNotFound) when calling the GetParameter operation: Parameter my_param_name not found.
I have verified method signature from boto3 ssm docs. Related AWS API Docs confirms to return a 400 response when parameter does not exist in the param store.
My question is that how do I verify if the exception caught in the response is actually a 400 status code so that I can handle it accordingly.
You can try catching client.exceptions.ParameterNotFound:
client = get_boto3_client('ssm', 'us-east-1')
try:
response = client.get_parameter(Name='my_param_name',WithDecryption=True)
except client.exceptions.ParameterNotFound:
logging.error("not found")
You can look at the status via response['Error']['Code'], but since there are multiple reasons for a 400, I would recommend a better approach:
response = client.get_parameter(Name='my_param_name',WithDecryption=True)
if 'Parameters' not in response:
raise ValueError('Response did not contain parameters key')
else:
return response
I'm trying to create simple Lambda function using Python 3.6.
The function should get a userId (my primary key in DynamoDB) in the request query string params and returns 200 if item exist in DB, here is my lambda function
import boto3
import os
from boto3.dynamodb.conditions import Key, Attr
def lambda_handler(event, context):
userId = event["userId"]
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(os.environ['Customers'])
items = table.query(
KeyConditionExpression=Key('userId').eq(userId)
)
return items["Items"]
When i am doing tests in Lambda interface it works and return the correct user however, when trying from Postman or using API Gateway it returns the following error
{
"errorMessage": "'userId'",
"errorType": "KeyError",
"stackTrace": [
[
"/var/task/index.py",
7,
"lambda_handler",
"userId = event["userId"]"
]
]
}
What am i missing here ?
Struggling to understand "event" , documentation states its a python
dictionary but how can i print the result of it and actually debug the lambda
when called from Postman or API Gateway?
You are using event["userId"], this means that sending the request payload for example
GET API : api/users/
Request Body payload:
{
"userId":"1234"
}
then above code works, Suppose you want to send userId as path parameter
GET API :api/user/{userId}
then you can access in lambda function
userId = (event['pathparameters']['userId'])
better add the print statement
print(event) and check the logs in cloudwatch logs
This solved it for me on post requests
import json
def lambda_handler(event, context):
data = json.loads(event["body"])
email = data['email']
in case you are using the serverless framework you can also add the following code under your http event. but i dont think it is that necessary.
request:
parameters:
application/json: '{"email":"$input.params(''email'')"}'
Make sure you hadn't selected "Lambda Proxy" while creating the HTTP method. Proxy will not convert/modify the request and hence "event" will be null
In my case my Python Lambda required a key called exclude. To resolve the issue of getting this response when calling via API Gateway, I needed to update the integration request with a mapping template:
I am working on a project and part of it involves inserting rows in to a Google Fusion Table for the Project from a python script. I have spent the last couple days trying to figure out just how to do that and I am officially confused.
My research seems to indicate that I need to use Oauth 2.0 to access the API. In doing so I can successfully get an access token but I can't seem to successfully get a refresh token. I'm not sure if this is going to hamper my ability to successfully integrate access to my Fusion Table with my Python code.
The second problem I am having is that I don't really understand how exactly to code inserting a row in my table. Most of the material I have found on it is from the deprecated Fusion Tables SQL API and I don't fully understand the new way of doing it.
I'm a beginner at this sort of thing and any direction to help me is very much appreciated!
Edit:
So the code I have working so far looks like this:
client_id = "<client_i>"
client_secret = "<client_secret>"
table_id = "<table_id>"
access_token = ""
refresh_token = "<refresh_token>"
# the refresh token is used to request a new access token
data = urllib.urlencode({
'client_id': client_id,
'client_secret': client_secret,
'refresh_token': refresh_token,
'grant_type': 'refresh_token'})
request = urllib2.Request(
url='https://accounts.google.com/o/oauth2/token',
data=data)
request_open = urllib2.urlopen(request)
response = request_open.read()
request_open.close()
tokens = json.loads(response)
access_token = tokens['access_token']
# Read the table
request_read = urllib2.Request(
url='https://www.google.com/fusiontables/api/query?%s' % \
(urllib.urlencode({'access_token': access_token,
'sql': 'SELECT * FROM table_id'})))
request_open = urllib2.urlopen(request_read)
response = request_open.read()
request_open.close()
print response
And my code for trying to insert a new row into my table:
date = str(datetime.now().date())
time = str(datetime.now().time())
query = 'INSERT INTO table_id (Date,Time,Saskatoon,Regina,MeadowLake)VALUES(date,time,60.01,60.02,59.99)'
data = urllib2.Request(
url='https://www.google.com/fusiontables/api/query?%s' % \
(urllib.urlencode({'access_token': access_token,
'sql': query})))
request_open = urllib2.urlopen(data)
When I run this i get
HTTP Error 400: HTTP GET can only be used for select queries.
I am know I'm supposed to be making a POST not a GET for the INSERT, I'm just not sure what needs to change in my code for that to happen.
Sorry for being a noob.
2ND EDIT:
Sorry for making this longer but I feel it is pertinent to show where I've gotten so far. I switched to the library requests and things have gotten somewhat easier however I still haven't successfully made a POST. My new code for importing rows is as follows:
def importRows(self):
print 'IMPORT ROWS'
date = str(datetime.now().date())
time = str(datetime.now().time())
data = {'Date': date,
'Time': time,
'Saskatoon': '60.01',
'Regina': '59.95'}
url = 'https://www.googleapis.com/upload/fusiontables/v1/tables/%s/import/%s' % \
(tableid, self.params) # self.params is access token
importRow = requests.post(url, params=data)
print importRow.status_code
print importRow.text
Which gives me
400
{
"error": {
"errors": [
{
"domain": "fusiontables",
"reason": "badImportInputEmpty",
"message": "Content is empty."
}
],
"code": 400,
"message": "Content is empty."
}
}
If your application needs offline access to a Google API, then the request for an authorization code should include the access_type parameter, where the value of that parameter is offline.
https://developers.google.com/accounts/docs/OAuth2WebServer#offline
Then, to obtain an access token using the refresh token you send a POST request including grant_type with value refresh_token.
Basically, the way SQL works is you send POST requests using a subset of SQL statements https://www.googleapis.com/fusiontables/v1/query?sql=STATEMENT_HERE
Refer to
https://developers.google.com/fusiontables/docs/v1/reference/query
https://developers.google.com/fusiontables/docs/v1/sql-reference
Edit:
Since you are using urllib2 without a data parameter, it defaults to GET. To fix this you should either use another HTTP library that allows for explicitly specifying method (like requests or httplib) or do something like this:
query = "INSERT INTO %s(EXAMPLE_COL1,EXAMPLE_COL2) VALUES"\
"('EXAMPLE_INFO1','EXAMPLE_INFO2')" % table_id # Single quotes
opener = urllib2.build_opener(urllib2.HTTPHandler)
request = urllib2.Request('https://www.google.com/fusiontables/api/query?%s' % \
(urllib.urlencode({'access_token': access_token,
'sql': query})),
headers={'Content-Length':0}) # Manually set length to avoid 411 error
request.get_method = lambda: 'POST' # Change HTTP request method
response = opener.open(request).read()
print response
Important to notice:
Monkey patch the method to do what we want (POST with an empty body) otherwise we would receive HTTP Error 400: HTTP GET can only be used for SELECT queries.
Manually specify that we do not have a body (Content-Length is 0) otherwise we would receive HTTP Error 411: Length Required.
Must use double quotes with single quotes inside or escape the inner quotes to submit strings via the query. In other words, "INSERT INTO %s(EXAMPLE_COL1,EXAMPLE_COL2) VALUES(EXAMPLE_INFO1,EXAMPLE_INFO2)" % table_id does not work.
If we tried to use the previous line we would get something like HTTP Error 400: Parse error near 'SOME_STRING' (line X, position Y)
See for info on changing method with urllib2:
Is there any way to do HTTP PUT in python