I have the following code:
client = Elasticsearch(hosts=['host'], port=9200)
scan_arguments = {'query': {'slice': {'max': 1, 'id': 0}}, 'preference': '_shards:0', 'index': u'my_index'}
for hit in scan(client, **scan_args):
# do something with hit
and I get the following error
RequestError: TransportError(400, u'parsing_exception', u'[slice] failed to parse field [max]')
How should the slice parameter be passed in the scan function?
"max" needs to be >1 in my experience. I saw the same error before when using "max":1.
The raw error from the HTTP API says max must be greater than 1.
{
"error": {
"root_cause": [
{
"type": "x_content_parse_exception",
"reason": "[3:20] [slice] failed to parse field [max]"
}
],
"type": "x_content_parse_exception",
"reason": "[3:20] [slice] failed to parse field [max]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "max must be greater than 1"
}
},
"status": 400
}
Related
I am trying to extract data from Netsuite and load it into Azure Databricks, by scripting a JSON config and running it through Azure Data Factory pipeline. I get the error that is mentioned above:
ERROR ScalaDriverLocal: User Code Stack Trace:
java.lang.Exception: max value not updated
Could this be related to an error checkpoint table updation?
I am providing the JSON script I used below. I hope someone can help me figure out the error. Thanks.
{
"parallelism": 1,
"onJobFailure": "Fail",
"onEmptyDF": "Fail",
"ignoreInvalidRows": true,
"cleanColumnNames": true,
"jobs": [
{
"name": "GenericPassThroughBatchJob.CURRENCY_EXCHANGE_RATE_L1",
"description": "Extract CURRENCY_EXCHANGE_RATE_L1 data from NetSuite",
"ignoreInvalidRows": true,
"cleanColumnNames": true,
"jdbcInputs": [
{
"dataFrameName": "CURRENCY_EXCHANGE_RATE_L1",
"driver": "com.netsuite.jdbc.openaccess.OpenAccessDriver",
"flavor": "oracle",
"url": "${spark.wsgc.jdbcUrl}",
"keyVaultAuth": {
"keyVaultParams": {
"clientId": "${spark.wsgc.clientId}",
"usernameKey": "${spark.wsgc.usernamekey}",
"passwordKey": "${spark.wsgc.passwordkey}",
"clientKey": "${spark.wsgc.clientkey}",
"vaultBaseUrl": "${spark.wsgc.vaultbaseurl}"
}
},
"incrementalParams": {
"checkpointTablePath": "dbfs:/mnt/data/governed/l1/audit/log/checkpoint_log/",
"extractId": "NETSUITE_CURRENCY_EXCHANGE_RATE",
"incrementalSql": "(select b.NAME as BASE_CURRENCY_CD, c.NAME as CURRENCY_CD, a.EXCHANGE_RATE, a.DATE_EFFECTIVE from Administrator.CURRENCY_EXCHANGE_RATES a left join Administrator.CURRENCIES b on a.BASE_CURRENCY_ID = b.CURRENCY_ID left join Administrator.CURRENCIES c on a.CURRENCY_ID = c.CURRENCY_ID) a1",
"maxCheckPoint1": "(select to_char(max(DATE_EFFECTIVE), 'DD-MM-YYYY HH24:MI:SS') from Administrator.CURRENCY_EXCHANGE_RATES where DATE_EFFECTIVE > to_date('%%{CHECKPOINT_VALUE_1}', 'YYYY-MM-DD HH24:MI:SS'))"
}
}
],
"fileOutputs": [
{
"dataFrameName": "CURRENCY_EXCHANGE_RATE_L1",
"format": "PARQUET",
"path": "dbfs:/mnt/data/governed/l1/global_netsuite/CurrencyExchangeRate/table/inbound/All_Currency_Exchange_Rate/",
"saveMode": "Overwrite"
},
{
"dataFrameName": "CURRENCY_EXCHANGE_RATE_L1",
"format": "DELTA",
"path": "dbfs:/mnt/data/governed/l1/global_netsuite/CurrencyExchangeRate/table/inbound_archive/All_Currency_Exchange_Rate/",
"saveMode": "Append"
}
]
}
]
}
This was an issue with Checkpoint table updation. Once I rectified the checkpoint value, this error was resolved. I had to set the checkpoint value to min(DATE_LAST_MODIFIED) from the records in the table.
Using flask i am hitting the razorpay api in this way given below :
virtualAccUrl = 'https://'+str(current_app.config.get('RAZORPAY_ID'))+':'+str(
current_app.config.get('RAZORPAY_SECRET_ID'))+'#api.razorpay.com/v1/virtual_accounts'
virtual_payload = {
"receivers": {
"types": [
"qr_code"
],
"amount": 200,
"qr_code": {
"name": "Store_1",
"fixed_amount": 'true',
"payment_amount": 30000,
"customer_id": "CUS158022319",
"notes": {
"purpose": "Test UPI QR code notes"
}
},
"description": "Test UPI QR description",
"notes": {
"purpose": "Test UPI QR notes"
}
},
"amount_expected": 100
}
virtual_response = requests.post(url=virtualAccUrl, params=virtual_payload)
virtual_response = json.loads(virtual_response.content)
print('111111111111111')
print(virtual_response)
Biut it is returning me the error like :
{'error': {'code': 'BAD_REQUEST_ERROR', 'description': 'The receivers
must be an array.', 'source': 'business', 'step':
'payment_initiation', 'reason': 'input_validation_failed', 'metadata':
{}, 'field': 'receivers'}}
Can anyone please suggest me ?? how we can send the data using flask in api in array form . i am stuck here . i dont know how to solve this error . i am a newbe in flask thanks in advance.
As per the response "receivers" key's value must be an array whwereas you are passing dictionary. Try enclosing that value in list as
"receivers":[your_previous_value]
It would be better if you can share the api documentation you are reffering.
So I'm pretty new to implementing flask-restplus and I have encountered this road block.
I have read the restplus docs over and over again and followed several exampled. But the behavior that I'm facing is very much different from what is supposed to be.
So I have a model that is supposed to be a list of objects of another model (returned from the function drone_model()).
drones_list = api.model('drones_list', {
'items': fields.List(fields.Nested(drone_model())),
'message':fields.String(''),
'code': fields.Integer('')
})
Everything works fine, no errors. But when I try the API (http://127.0.0.1:5000/datamine/v2/drones), as a response I get the Marshalling model back instead of the data itself. If I print the data, it gets printed, but for some reason in the web, the restplus model is returned.
Below I have the code that I had written. If I take the marshal_with decorator off, then the data is returned just fine.
#api.route('/')
class DronesList(Resource):
#api.marshal_with(drones_list, envelope='data')
#api.response(200, 'All drones successfully fetched!')
def get(self):
"""
Get all drones!.
"""
from app.utils.common import get_start_end_date_from_request
start_date, end_date = get_start_end_date_from_request(request)
drones = []
for drone in Drone.objects:
drones.append({
'id': str(drone.id),
'serial_id': drone.serial_id,
'maintenances': [],
'status': get_dynamic_status(drone, start_date, end_date),
'picture_url': drone.asset.picture_url,
'manufacturer': drone.asset.manufacturer,
'model_name': drone.asset.model_name,
'drone_type': drone.asset.drone_type,
'payload_type': drone.asset.payload_type,
'asset_url': drone.get_url(drone.id)
})
success = ClientSuccessFunctionClass('All drones successfully fetched!', 200, drones)
return (success.to_dict())
These are the outputs on the browser:
1. Without the marshal decorator:
{
"data": {
"items": [
{
"id": "5aeafcb93a33683f73827e91",
"serial_id": "Drone 1",
"maintenances": [],
"status": "Decommissioned",
"picture_url": "some img url",
"manufacturer": "DJI",
"model_name": "Phantom 4 Pro",
"drone_type": "Quadcopter",
"payload_type": "RGB Camera",
"asset_url": "http://127.0.0.1:5000/datamine/v1/drones/5aeafcb93a33683f73827e91"
},
{
"id": "5aeaff374f85747f90df2714",
"serial_id": "Drone 2",
"maintenances": [],
"status": "Available",
"picture_url": "sime url",
"manufacturer": "DJI",
"model_name": "Phantom 4",
"drone_type": "Quadcopter",
"payload_type": "RGB Camera",
"asset_url": "http://127.0.0.1:5000/datamine/v1/drones/5aeaff374f85747f90df2714"
}
],
"message": "All drones successfully fetched!",
"code":200
}
}
2. With the marshal decorator:
{
"data": {
"items": [
{
"id": "Id of Drone",
"serial_id": "Name of Drone",
"status": "Status of Drone",
"maintenances": null,
"picture_url": "Picture URL",
"manufacturer": "Manufacturer of Drone",
"model_name": "Model name of Drone",
"drone_type": "Type of Drone",
"payload_type": "Payload type of Drone",
"asset_url": "Asset URL of Drone"
}
],
"message": "",
"code": ""
}
}
It would be really helpful if someone could tell me what I'm doing wrong as I need to recive the output as the one shown in snippet of the output without the decorator.
Thank you.
Here is a diagram showing invocation order from top to bottom to help make sense of what is happening:
get()
→ api.response(200, 'All drones successfully fetched!') # documents the response
→ api.marshal_with(drones_list, envelope='data')` # returns marshalled dict
The result from invoking get is passed to the api.response decorator function whose result is passed on to api.marshal_with decorator function.
Looking at the shape of the dictionary returned from invoking get()
{
data {
items [
{
id,
serial_id,
maintenances,
status,
picture_url,
manufacturer,
model_name,
drone_type,
payload_type,
asset_url
}
],
message,
code
}
}
The message and code in the response are nested inside of the data.
You need to model the data appropriately, to be able to marshal it. This can be done by passing an argument for what field to look up in the marshal dictionary.
drones_list = api.model('drones_list', {
'items': fields.List(fields.Nested(drone_model()), attribute='data.items'),
'message':fields.String(attribute='data.message'),
'code': fields.Integer(attribute='data.code')
})
As you can see, it's pretty redundant applying the api.marshal_with decorator function on the view given that it's only unnests then nests the result in data field.
I've got some code that looks like this
from elasticsearch import Elasticsearch
client = Elasticsearch(hosts = [myhost])
try:
results = es_client.search(
body = {
'query' : {
'bool' : {
'must' : {
'term' : {
'foo' : 'bar',
'hello' : 'world'
}
}
}
}
},
index = 'index_A,index_B',
size = 10,
from_ = 0
)
except Exception as e:
## my code stops here, as there is an exception
import pdb
pdb.set_trace()
Examining the exception
SearchPhaseExecutionException[Failed to execute phase [query], all shards failed;
And further down
Parse Failure [Failed to parse source [{"query": {"bool": {"must": {"term": {"foo": "bar", "hello": "world"}}}}}]]]; nested: QueryParsingException[[index_A] [bool] query does not support [must]];
The stack trace was huge so I just grabbed snippets of it, but the main error appears to be that "must" is not supported, at least the way I have constructed my query.
I was using this and this for guidance on constructing the query.
I can post a more complete stack trace, but I was hoping someone is able to see a very obvious error that I have made inside the "body" parameter inside the "search" method.
Can anyone see anything that I have clearly done wrong as far as constructing the query body for the python API?
The syntax of the query doesn't look correct to me. Try this:
results = es_client.search(
body = {
"query": {
"bool": {
"must": [
{
"term": {
"foo": {
"value": "bar"
}
}
},
{
"term": {
"hello": {
"value": "world"
}
}
}
]
}
}
},
index = 'index_A,index_B',
size = 10,
from_ = 0
)
I am trying to update a document in elasticsearch using the default python interface for Elasticsearch using the below command.
res = es.update(index='its', doc_type='vents', id=txid, body={"doc":{"f_vent" :{"b_vent":rx_buffer}}})
The updated document is shown below.
{
"_index": "its",
"_type": "vents",
"_id": "4752956038",
"_score": null,
"_source": {
"ResponseTime": 0,
"Session": "None",
"Severity": "warn",
"StatusCode": 0,
"Subject": "Reporting Page Load Time",
"Time": "Fri Jun 05 2015 12:23:46 GMT+1200 (NZST)",
"Timestamp": "1433463826535",
"TransactionId": "4752956038",
"msgType": "0",
"tid": "1",
"f_vent": {
"b_vent": "{\"ActiveTransactions\": 6, \"AppName\": \"undefined\", \"TransactionId\": \"4752956038\", \"UserInfo\": \"Unknown\"}"
}
},
"fields": {
"_timestamp": 1433818222372
},
"sort": [
1433818222372
]
}
I copied this from Kibana4 discover tab by expanding the document.The 'transaction Id' inside b_vent has to be accessed as f_vent.b_vent.TransactionId. I suspect this is putting some restricions on me plotting a graph on transaction Id. I tried using
res = es.update(index='its', doc_type='vents', id=txid, body={"doc":{"b_vent":rx_buffer}})
so that I could use b_vent.TransactionId but I am getting the following error when calling es.update().
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'MapperParsingException[object mapping for [events] tried to parse field [be_event] as object, but got EOF, has a concrete value been provided to it?]')
What am I doing wrong? How can I fix this problem?
This is the almost full strucuture of b_vent.
"{
\"ActiveTr\": 6,
\"ErrorM\": \"None\",
\"HError\": \"false\",
\"HMPct\": 62,
\"NHMPct\": 57,
\"Parameter\": \"1433195852706\",
\"ParameterD\": \"false\",
\"ProcessCPU\": 1,
\"Proxies\": \"None\",
\"RStatusCode\": \"34500\",
\"Severity\": \"info\",
\"ThrWtTi\": -1,
\"ThrWai\": 16,
\"Timestamp\": \"TueJun0209: 58: 16NZST2015\",
\"TxId\": \"316029416\",
\"UserInfo\": \"Unknown\"
}"
It does seem to have some strange escape sequences. I am not sure why they are there. But json.loads() does seem to parse the file. I don't know how to fix this issue?