I've got some code that looks like this
from elasticsearch import Elasticsearch
client = Elasticsearch(hosts = [myhost])
try:
results = es_client.search(
body = {
'query' : {
'bool' : {
'must' : {
'term' : {
'foo' : 'bar',
'hello' : 'world'
}
}
}
}
},
index = 'index_A,index_B',
size = 10,
from_ = 0
)
except Exception as e:
## my code stops here, as there is an exception
import pdb
pdb.set_trace()
Examining the exception
SearchPhaseExecutionException[Failed to execute phase [query], all shards failed;
And further down
Parse Failure [Failed to parse source [{"query": {"bool": {"must": {"term": {"foo": "bar", "hello": "world"}}}}}]]]; nested: QueryParsingException[[index_A] [bool] query does not support [must]];
The stack trace was huge so I just grabbed snippets of it, but the main error appears to be that "must" is not supported, at least the way I have constructed my query.
I was using this and this for guidance on constructing the query.
I can post a more complete stack trace, but I was hoping someone is able to see a very obvious error that I have made inside the "body" parameter inside the "search" method.
Can anyone see anything that I have clearly done wrong as far as constructing the query body for the python API?
The syntax of the query doesn't look correct to me. Try this:
results = es_client.search(
body = {
"query": {
"bool": {
"must": [
{
"term": {
"foo": {
"value": "bar"
}
}
},
{
"term": {
"hello": {
"value": "world"
}
}
}
]
}
}
},
index = 'index_A,index_B',
size = 10,
from_ = 0
)
Related
When executing a select query against an athena table via boto3, the response object given is in the syntax:
{
"UpdateCount":0,
"ResultSet":{
"Rows":[
{
"Data":[
{
"VarCharValue":"site_name"
},
{
"VarCharValue":"volume_out_capacity"
},
{
"VarCharValue":"region"
},
{
"VarCharValue":"site_ref"
}
]
},
{
"Data":[
{
"VarCharValue":"ASSET 12"
},
{
"VarCharValue":"10"
},
{
"VarCharValue":"NORTH"
},
{
"VarCharValue":"RHW007777000138"
}
]
}
]
}
Is there an additional argument that can be passed so that the response object will contain columns that do not contain values? Something like:
{
"VarCharValue":"xyz"
}
]
},
{
"Data":[
{
"VarCharValue":None
}
I have looked through the documentation extensively but cannot find arguments that can describe how to format the response in get_query_results() or start_query_execution()
I do not see a option to get that data back directly from Athena. Alternatively if you use S3 Select instead of Athena, you'll get back a json object with all the columns whether they have data or are empty.
Sample Data:
name,number,city,state
coin,1,driggs,
michelle,,chicago,
shaniqua,2,,
marcos,3,stlouis,
S3 Select Result:
{"name":"coin","number":"1","city":"driggs","state":""}
{"name":"michelle","number":"","city":"chicago","state":""}
{"name":"shaniqua","number":"2","city":"","state":""}
{"name":"marcos","number":"3","city":"stlouis","state":""}
Code:
import boto3
session = boto3.session.Session(profile_name="<my-profile>")
s3 = session.client('s3')
resp = s3.select_object_content(
Bucket='<my-bucket>',
Key='<my-file>',
ExpressionType='SQL',
Expression="SELECT * FROM s3object",
InputSerialization={'CSV': {"FileHeaderInfo": "Use", "RecordDelimiter": '\r\n'}, 'CompressionType': 'NONE'},
OutputSerialization={'JSON': {}},
)
for event in resp['Payload']:
if 'Records' in event:
records = event['Records']['Payload'].decode('utf-8')
print(records)
elif 'Stats' in event:
statsDetails = event['Stats']['Details']
print("Stats details bytesScanned: ")
print(statsDetails['BytesScanned'])
print("Stats details bytesProcessed: ")
print(statsDetails['BytesProcessed'])
print("Stats details bytesReturned: ")
print(statsDetails['BytesReturned'])
I am trying to update a value of an array stored in a mongodb collection
any_collection: {
{
"_id": "asdw231231"
"values": [
{
"item" : "a"
},
{
"item" : "b"
}
],
"role": "role_one"
},
...many similar
}
the idea is that I want to access values and edit a value with the following code that I found in the mongodb documentation
conn.any_collection.find_one_and_update(
{
"_id": any_id,
"values.item": "b"
},
{
"$set": {
"values.$.item": "new_value" # here the error, ".$."
}
}
)
This should work, but I can't understand what the error is or what is the correct syntax for pymongo. The error is generated when adding "$";
It works fine with my fastAPI.
#app.get("/find/{id}")
async def root(id: int):
db = get_database()
q = {'_id': 'asdw231231','values.item': 'b'}
u = {'$set': {'values.$.item': 'new_value' }}
c = db['any'].find_one_and_update(q, u)
return {"message": c}
mongoplayground
I tried to using Pinpoint services in my Lambda Function using Python Language. But I am facing time out error once Executed my function. Here is my Lambda Function source code.
import json
import boto3
client = boto3.client('pinpoint')
def trigger(event, context):
print("---------1----------------")
payload = {
'TraceId': 'sadfasdf',
'Context': {
'string': 'string'
},
'MessageConfiguration': {
'GCMMessage': {
'RawContent': ""
}
},
'TemplateConfiguration': {
'PushTemplate': {
'Name': 'coll-1-en'
}
},
'Users': {
"id": {
"Substitutions":{
"subject": ["string"],
"id": ["120913"]
}
}
}
}
try:
print("---------2----------------")
response = client.send_users_messages(
ApplicationId='my_aws_pinpoint_services_project.id',
SendUsersMessageRequest=payload
)
print("---------3----------------")
data = response.read()
print(data.decode("utf-8"))
return response
except Exception:
return False
I am not sure what is the issue from my code. So I have inserted print("-------") to find out the error lines.
The codes didn't executed since response = client.send_users_messages( line. So I was sure the client.send_users_messages had an issue. But I am not sure what is wrong in this line. Pls help me to fix this issue. I attached the screenshot of the Test result.
I tried to using pinpoint services in my Lambda Function using Python Language.
But I am facing time out error once Executed my function.
Here is my lambda function source code.
import json
import boto3
client = boto3.client('pinpoint')
def trigger(event, context):
print("---------1----------------")
payload = {
'TraceId': 'sadfasdf',
'Context': {
'string': 'string'
},
'MessageConfiguration': {
'GCMMessage': {
'RawContent': ""
}
},
'TemplateConfiguration': {
'PushTemplate': {
'Name': 'coll-1-en'
}
},
'Users': {
"id": {
"Substitutions":{
"subject": ["string"],
"id": ["120913"]
}
}
}
}
try:
print("---------2----------------")
response = client.send_users_messages(
ApplicationId='my_aws_pinpoint_services_project.id',
SendUsersMessageRequest=payload
)
print("---------3----------------")
data = response.read()
print(data.decode("utf-8"))
return response
except Exception:
return False
I am not sure what is the issue from my code.
So I have inserted print("string") to find out the error lines.
The codes didn't executed since response = client.send_users_messages( line.
So I was sure the client.send_users_messages had an issue.
But I am not sure what is wrong in this line.
Pls help me to fix this issue.
I attached the screenshot of the Test result.
The query I would like to replicate in DSL is as below:
GET /_search
{
"query":{
"bool":{
"must":[
{
"term":{
"destination":"singapore"
}
},
{
"terms":{
"tag_ids":[
"tag_luxury"
]
}
}
]
}
},
"aggs":{
"max_price":{
"max":{
"field":"price_range_from.SGD"
}
},
"min_price":{
"min":{
"field":"price_range_from.SGD"
}
}
},
"post_filter":{
"range":{
"price_range_from.SGD":{
"gte":0.0,
"lte":100.0
}
}
}
}
The above query
Matches terms - destination and tags_ids
Aggregates to result to find the max price from field price_range_from.SGD
Applies another post_filter to subset the result set within price limits
It works perfectly well in the Elastic/Kibana console.
I replicated the above query in elasticsearch-dsl as below:
es_query = []
es_query.append(Q("term", destination="singapore"))
es_query.append(Q("terms", tag_ids=["tag_luxury"]))
final_query = Q("bool", must=es_query)
es_conn = ElasticSearch.instance().get_client()
dsl_client = DSLSearch(using=es_conn, index=index).get_dsl_client()
dsl_client.query = final_query
dsl_client.aggs.metric("min_price", "min", field="price_range_from.SGD")
dsl_client.aggs.metric("max_price", "max", field="price_range_from.SGD")
q = Q("range", **{"price_range_from.SGD":{"gte": 0.0, "lte": 100.0}})
dsl_client.post_filter(q)
print(dsl_client.to_dict())
response = dsl_client.execute()
print(response.to_dict().get("hits", {}))
Although the aggregations are correct, products beyond the price range are also being returned. There is no error returned but it seems like the post_filter query is not applied.
I dived in the dsl_client object to see whether my query is being captured correctly. I see only the query and aggs but don't see the post_filter part in the object. The query when converted to a dictionary using dsl_client.to_dict() is as below -
{
"query":{
"bool":{
"must":[
{
"term":{
"destination":"singapore"
}
},
{
"terms":{
"tag_ids":[
"tag_luxury"
]
}
}
]
}
},
"aggs":{
"min_price":{
"min":{
"field":"price_range_from.SGD"
}
},
"max_price":{
"max":{
"field":"price_range_from.SGD"
}
}
}
}
Please help. Thanks!
You have to re-assign the dsl_client like:
dsl_client = dsl_client.post_filter(q)