We Keep Coding

Is there a way to make a response from Python requests a list within a Python dictionary?

I am using the Python requests library to get some data from an API. However, within the output dictionary, I would like to have a list. I want to have a list within the data_providers key where the first entry for name, url and api_url to be hard coded and the second entry to be gotten from the response.
Currently I get the error TypeError: unhashable type: 'dict', and my code look like this
output_dict = {
'data_providers' : {
{
'name': 'NBN Atlas',
'url': 'https://registry.nbnatlas.org/datasets',
'api_url': 'https://registry.nbnatlas.org/ws/dataResource',
},
{
'name': owner_name,
'url': owner_url,
'api_url': owner_api_url,
},
}
}
After running the code and dumping it into a JSON file my desired response should look something like this:
[
{
"data_providers": [
{
"name": "NBN Atlas",
"url": "https://registry.nbnatlas.org/datasets",
"api_url": "https://registry.nbnatlas.org/ws/dataResource"
},
{
"name": "Marine Biological Association",
"url": "https://registry.nbnatlas.org/public/show/dp80",
"api_url": "https://registry.nbnatlas.org/ws/dataProvider/dp80"
}
]
}
]
Any way to go about this?

If I understood your question correctly, you wish to fix the unhashable type error and display valid JSON. If so, you can correct the error by changing data_providers dictionary to a list since it contains multiple nested dictionaries.
output_dict = {
'data_providers': [
{
'name': 'NBN Atlas',
'url': 'https://registry.nbnatlas.org/datasets',
'api_url': 'https://registry.nbnatlas.org/ws/dataResource'
},
{
'name': owner_name,
'url': owner_url,
'api_url': owner_api_url
}
]
}
You can now use a statement similar to print(json.dumps(output_dict)) to display the result as JSON.

RequestError: RequestError(400, search_phase_execution_exception failed to create query For input string in elasticsearch

Below is my dictionary
abc = [
{'id':"1", 'name': 'cristiano ronaldo', 'description': 'portugal#fifa.com'},
{'id':"2", 'name': 'lionel messi', 'description': 'argentina#fifa.com'},
{'id':"3", 'name': 'Lionel Jr', 'description': 'brazil#fifa.com'}
]
Ingested the players into elasticsearch
for i in abc:
es.index(index="players", body=i, id=i['id'])
Below is the dsl query
resp = es.search(index="players",body={
"query": {
"query_string": {
"fields": ["id^12","description^2", "name^2"],
"query": "brazil#fifa.com"
}
}})
resp
Issue 1:
if "fields": ["id^12","description^2", "name^2"] then i am getting error RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to create query: For input string: "brazil#fifa.com"'
Issue 2:
if my fields are ["description^2", "name^2"] I am expecting one document which contain brazil#fifa.com but returning all 3 documents
Edited: From the comment of sagar my setting id was long which i changed now . mapping is below. and issue1 is resolved
{'players': {'mappings': {'properties': {'description': {'type': 'text',
'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}},
'id': {'type': 'text',
'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}},
'name': {'type': 'text',
'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}}}}}

Issue 1: if "fields": ["id^12","description^2", "name^2"] then i am
getting error RequestError: RequestError(400,
'search_phase_execution_exception', 'failed to create query: For input
string: "brazil#fifa.com"'
Above issue you are getting because your id field is define as integer or flot type of field (other then text type of field). You need to provide "lenient": true in your query and it will not return any exception.
Issue 2: if my fields are ["description^2", "name^2"] I am expecting
one document which contain brazil#fifa.com but returning all 3
documents
Above issue is happening because you are searching on text type of field which applied default standard analyzer when you do search.
So when you search brazil#fifa.com then it will create two token brazil and fifa.com. Here, fifa.com is matching in your all 3 documents so it is returning in result. To resolved this issue, you can use description.keyword field.
Below query will resolved your both issue:
{
"query": {
"query_string": {
"lenient": true,
"fields": [
"id^12",
"description.keyword^2",
"name^2"
],
"query": "brazil#fifa.com"
}
}
}
Updated:
Based on comment if you want to search fifa as well then you need to provide description as field but when you search brazil#fifa.com then you need to provide it in double quotes for exact match. Please see below example:
{
"query": {
"query_string": {
"lenient": true,
"fields": [
"id^12",
"description^2",
"name^2"
],
"query": "\"brazil#fifa.com\""
}
}
}

Eve: how to use different endpoints to access the same collection with different filters

I have an Eve app publishing a simple read-only (GET) interface. It is interfacing a MongoDB collection called centroids, which has documents like:
[
{
"name":"kachina chasmata",
"location":{
"type":"Point",
"coordinates":[-116.65,-32.6]
},
"body":"ariel"
},
{
"name":"hokusai",
"location":{
"type":"Point",
"coordinates":[16.65,57.84]
},
"body":"mercury"
},
{
"name":"cañas",
"location":{
"type":"Point",
"coordinates":[89.86,-31.188]
},
"body":"mars"
},
{
"name":"anseris cavus",
"location":{
"type":"Point",
"coordinates":[95.5,-29.708]
},
"body":"mars"
}
]
Currently, (Eve) settings declare a DOMAIN as follows:
crater = {
'hateoas': False,
'item_title': 'crater centroid',
'url': 'centroid/<regex("[\w]+"):body>/<regex("[\w ]+"):name>',
'datasource': {
'projection': {'name': 1, 'body': 1, 'location.coordinates': 1}
}
}
DOMAIN = {
'centroids': crater,
}
Which will successfully answer to requests of the form http://hostname/centroid/<body>/<name>. Inside MongoDB this represents a query like: db.centroids.find({body:<body>, name:<name>}).
What I would like to do also is to offer an endpoint for all the documents of a given body. I.e., a request to http://hostname/centroids/<body> would answer the list of all documents with body==<body>: db.centroids.find({body:<body>}).
How do I do that?
I gave a shot by including a list of rules to the DOMAIN key centroids (the name of the database collection) like below,
crater = {
...
}
body = {
'item_title': 'body craters',
'url': 'centroids/<regex("[\w]+"):body>'
}
DOMAIN = {
'centroids': [crater, body],
}
but didn't work...
AttributeError: 'list' object has no attribute 'setdefault'

Got it!
I was assuming the keys in the DOMAIN structure was directly related to the collection Eve was querying. That is true for the default settings, but it can be adjusted inside the resources datasource.
I figured that out while handling an analogous situation as that of the question: I wanted to have an endpoint hostname/bodies listing all the (unique) values for body in the centroids collection. To that, I needed to set an aggregation to it.
The following settings give me exactly that ;)
centroids = {
'item_title': 'centroid',
'url': 'centroid/<regex("[\w]+"):body>/<regex("[\w ]+"):name>',
'datasource': {
'source': 'centroids',
'projection': {'name': 1, 'body': 1, 'location.coordinates': 1}
}
}
bodies = {
'datasource': {
'source': 'centroids',
'aggregation': {
'pipeline': [
{"$group": {"_id": "$body"}},
]
},
}
}
DOMAIN = {
'centroids': centroids,
'bodies': bodies
}
The endpoint, for example, http://127.0.0.1:5000/centroid/mercury/hokusai give me the name, body, and coordinates of mercury/hokusai.
And the endpoint http://127.0.0.1:5000/bodies, the list of unique values for body in centroids.
Beautiful. Thumbs up to Eve!

python eve - data_relation on DELETE - preserve/error if referenced

currently when I DELETE an item which is being referenced by. I can still delete it. I would like to prohibit this giving an error. I couldn't find anything on http://docs.python-eve.org or google.
say I have the following two schemas:
foos = {
...
'schema' : {
...,
'bar': {
'type': 'string',
'required': true,
'data_relation': {
'resource': 'bars',
'field': 'name',
'embeddable': True
}
}
}
}
bars = {
...
'schema' : {
...,
'name': {
'type': 'string',
'required': true,
}
}
}
if I delete now an item from bars which is still being referenced by >=1 items in foos. How could I return an error instead of deleting the bars item?
Would i need to write a
on_delete_item
def event(resource_name, item)
and how? Any help is appreciated.

elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'No handler for type [string] declared on field [texts]')

I use elasticsearch python api to create mappings, but it went some wrong:
es = Elasticsearch("localhost:9200")
request_body = {
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1
},
'mappings': {
'examplecase': {
'properties': {
'tbl_id': {'index': 'not_analyzed', 'type': 'string'},
'texts': {'index': 'analyzed', 'type': 'string'},
}
}
}
}
es.indices.create(index='example_index', body=request_body)
it shows elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'No handler for type [string] declared on field [texts]'), and I find some solution that they say: use text instead of string in the field type, but it also went wrong: elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'Failed to parse mapping [examplecase]: Could not convert [texts.index] to boolean'). The elasticsearch version iselasticsearch-6.5.4. How can I deal with it?

this
'index': 'analyzed' OR 'index': 'not_analyzed'
is an older elasticsearch version mapping and not needed.
All you need to do is use 'text' for analyzed string fields and 'keyword' for not_analyzed text fields, like this:
es = Elasticsearch("localhost:9200")
request_body = {
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1
},
'mappings': {
'examplecase': {
'properties': {
'tbl_id': {'type': 'keyword'},
'texts': {'type': 'text'},
}
}
}
}
es.indices.create(index='example_index', body=request_body)
see reference in Elastic docs here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

See this. The index setting in your mapping is incorrectly configured. It is a mapping parameter and can be set to true or false only. You cannot set this within the properties parameter.