I am trying to use elastic search libraries like pyelasticsearch and elasticsearch I am not getting any method where i can pass dataframe and that method will load data frame data to elastic search.
I am trying a this code:
for i,df in enumerate(csvfile):
print (i)
records=df.where(pd.notnull(df), None).T.to_dict()
list_records=[records[it] for it in records]
print(list_records)
try :
es.bulk_index("call_demo_data1","tweet",list_records)
except :
print ("error!, skiping some tweets sorry")
pass
where csvfile is my dataframe where my all data is present. but I am getting following error
'str' object has no attribute 'where'
I have used recommendation in comments
Now that problem is solved I am getting this error while bulk loading
I am using above method to load data elastic search I was facing problem so here is the link to the question I posted earlier
Here is the code I am using now :
records= csvfile.T.to_dict()
list_records=[records[it] for it in records]
#print(list_records)
es.bulk_index("call_demo_data1","tweet",list_records)
Error I am getting is :
too many values to unpack (expected 2)
this error is coming while bulk indexing. csvfile in above code is a data frame. I am using this liabrary pyelasticsearch
This is the error traceback
Related
I have been trying to do a bulk index in the elasticsearch 8.x using this:
from elasticsearch.helpers import bulk as bulk_indexer
success, failed = bulk_indexer(self.es_client, actions, stats_only=True, chunk_size=900)
Apparently i am seeing :
RuntimeError: elasticsearch.BadRequestError: BadRequestError(400, 'illegal_argument_exception', 'Action/metadata line [1] contains an unknown parameter [_type]') [while running 'bulk index']
I came to know about https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html
and i think this is the reason for it , but can someone help me to know why i am seeing this or what could be the issue?
I tried to migrate from es 5 to es 8.x
There must be a _type in the action command line that shouldn't be there anymore. If those records have been stored in Kafka in earlier versions of ES and you're replaying them, or if the client application still produces those action records with the same logic, then you either need to change that producing logic and/or post-process those records in your bulk indexer to remove the _type field.
Problem: I want to pick a field of index in elasticsearch and look for all the values against it. Like if I give a key I should get the value for that key and if that key exists more than once, so each whats the each value. Or even if I get one of the values would work for me.
How I am trying to work through it: Query the elasticsearch
I am trying to query my data from Elasticsearch;
r = es.search(index="test",body = {'query': {'wildcard': {'task_name': "*"}}})
I thought to load the data to a python object ( dictionary) to read a key values. However, when I try json.loads(r.json) it gives me an error : AttributeError: 'dict' object has no attribute 'json'.
I even tried with json.load(r) but the error remains the same.
I have someone's code who is not in the company and I want to get the data stored in mongo database which is using AsyncIOMotorClient. I have no experience with motor.
Problem:
I am able to get this:
AsyncIOMotorCursor(<pymongo.cursor.Cursor object at 0x7fbd87b45ca0>)
I know how to iterate over <pymongo.cursor.Cursor object at 0x7fbd87b45ca0>, but when I try to iterate over the above AsyncIOMotorCursor, I get the following error:
TypeError: 'AsyncIOMotorCursor' object is not iterable
Kindly help.
I have written the following query in mysql using python
self.sql.get_value("delete from devices where asset_id = '{}'".format(VARIABLE))
When I run this I am getting an error:
Unable to retrieve data : 'NoneType' object has no attribute 'getitem'
Things I have tried:
Select query in same format works fine.
Checked if VARIABLE is Null, it is not
Tried this - delete from devices where asset_id = 'T1234'
Nothing worked. Any ideas on this would be helpful
How to fixed error "how to fix invalid character 'x' in string escape code by py mongo"? My error like this
Check your BSON packet where there is X in it, you can query for the value x using the mongo find command if you already have in other mongo instance, it clearly shows it imported 14k records but some records can't be imported because of incompatible format. The solution query the db and find the error documents and remove it.