I've been searching for a pretty long time but I can't figure out how to update a field in a document using the Firestore REST API. I've looked on other questions but they haven't helped me since I'm getting a different error:
{'error': {'code': 400, 'message': 'Request contains an invalid argument.', 'status': 'INVALID_ARGUMENT', 'details': [{'#type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'oil', 'description': "Error expanding 'fields' parameter. Cannot find matching fields for path 'oil'."}]}]}}
I'm getting this error even though I know that the "oil" field exists in the document. I'm writing this in Python.
My request body (field is the field in a document and value is the value to set that field to, both strings received from user input):
{
"fields": {
field: {
"integerValue": value
}
}
}
My request (authorizationToken is from a different request, dir is also a string from user input which controls the directory):
requests.patch("https://firestore.googleapis.com/v1beta1/projects/aethia-resource-management/databases/(default)/documents/" + dir + "?updateMask.fieldPaths=" + field, data = body, headers = {"Authorization": "Bearer " + authorizationToken}).json()
Based on the the official docs (1,2, and 3), GitHub and a nice article, for the example you have provided you should use the following:
requests.patch("https://firestore.googleapis.com/v1beta1/projects{projectId}/databases/{databaseId}/documents/{document_path}?updateMask.fieldPaths=field")
Your request body should be:
{
"fields": {
"field": {
"integerValue": Value
}
}
}
Also keep in mind that if you want to update multiple fields and values you should specify each one separately.
Example:
https://firestore.googleapis.com/v1beta1/projects/{projectId}/databases/{databaseId}/documents/{document_path}?updateMask.fieldPaths=[Field1]&updateMask.fieldPaths=[Field2]
and the request body would have been:
{
"fields": {
"field": {
"integerValue": Value
},
"Field2": {
"stringValue": "Value2"
}
}
}
EDIT:
Here is a way I have tested which allows you to update some fields of a document without affecting the rest.
This sample code creates a document under collection users with 4 fields, then tries to update 3 out of 4 fields (which leaves the one not mentioned unaffected)
from google.cloud import firestore
db = firestore.Client()
#Creating a sample new Document “aturing” under collection “users”
doc_ref = db.collection(u'users').document(u'aturing')
doc_ref.set({
u'first': u'Alan',
u'middle': u'Mathison',
u'last': u'Turing',
u'born': 1912
})
#updating 3 out of 4 fields (so the last should remain unaffected)
doc_ref = db.collection(u'users').document(u'aturing')
doc_ref.update({
u'first': u'Alan',
u'middle': u'Mathison',
u'born': 2000
})
#printing the content of all docs under users
users_ref = db.collection(u'users')
docs = users_ref.stream()
for doc in docs:
print(u'{} => {}'.format(doc.id, doc.to_dict()))
EDIT: 10/12/2019
PATCH with REST API
I have reproduced your issue and it seems like you are not converting your request body to a json format properly.
You need to use json.dumps() to convert your request body to a valid json format.
A working example is the following:
import requests
import json
endpoint = "https://firestore.googleapis.com/v1/projects/[PROJECT_ID]/databases/(default)/documents/[COLLECTION]/[DOCUMENT_ID]?currentDocument.exists=true&updateMask.fieldPaths=[FIELD_1]"
body = {
"fields" : {
"[FIELD_1]" : {
"stringValue" : "random new value"
}
}
}
data = json.dumps(body)
headers = {"Authorization": "Bearer [AUTH_TOKEN]"}
print(requests.patch(endpoint, data=data, headers=headers).json())
I found the official documentation to not to be of much use since there was no example mentioned. This is the API end-point for your firestore database
PATCH https://firestore.googleapis.com/v1beta1/projects/{YOUR_PROJECT_ID}/databases/(default)/documents/{COLLECTION_NAME}/{DOCUMENT_NAME}
the following code is the body of your API request
{
"fields": {
"first_name": {
"stringValue":"Kurt"
},
"name": {
"stringValue":"Cobain"
},
"band": {
"stringValue":"Nirvana"
}
}
}
The response you should get upon successful update of the database should look like
{
"name": "projects/{YOUR_PROJECT_ID}/databases/(default)/documents/{COLLECTION_ID/{DOC_ID}",
{
"fields": {
"first_name": {
"stringValue":"Kurt"
},
"name": {
"stringValue":"Cobain"
},
"band": {
"stringValue":"Nirvana"
}
}
"createTime": "{CREATE_TIME}",
"updateTime": "{UPDATE_TIME}"
Note that performing the above action would result in a new document being created, meaning that any fields that existed previously but have NOT been mentioned in the "fields" body will be deleted. In order to preserve fields, you'll have to add
?updateMask.fieldPaths={FIELD_NAME} --> to the end of your API call (for each individual field that you want to preserve).
For example:
PATCH https://firestore.googleapis.com/v1beta1/projects/{YOUR_PROJECT_ID}/databases/(default)/documents/{COLLECTION_NAME}/{DOCUMENT_NAME}?updateMask.fieldPaths=name&updateMask.fieldPaths=band&updateMask.fieldPaths=age. --> and so on
Related
I have a index named "twitter_profile_response_tms" in which I have doc type named as "posts"
In want to query all data in posts, i have total of 2636 data in posts but after running the query I only get 10 number of match data.
following is my query:
res = self.es.search(
index="twitter_profile_response_tms",
doc_type='posts',
body={
"query": {
"match": {
'username': 'wasimakramlive'
}
}
},
)
How should I resolve this?
you can use hits.total.values key from the elasticsearch result payload to get the number of matching records.
You can control it using size parameter. For e.g. to fetch 50 records set "size": 50.
res = self.es.search(
index="twitter_profile_response_tms",
doc_type='posts',
body={
"query": {
"match": {
'username': 'wasimakramlive'
},
"size": 50
}
},
)
You can fetch data page wise as well using from and size parameters. Read here more.
I have an Eve app publishing a simple read-only (GET) interface. It is interfacing a MongoDB collection called centroids, which has documents like:
[
{
"name":"kachina chasmata",
"location":{
"type":"Point",
"coordinates":[-116.65,-32.6]
},
"body":"ariel"
},
{
"name":"hokusai",
"location":{
"type":"Point",
"coordinates":[16.65,57.84]
},
"body":"mercury"
},
{
"name":"cañas",
"location":{
"type":"Point",
"coordinates":[89.86,-31.188]
},
"body":"mars"
},
{
"name":"anseris cavus",
"location":{
"type":"Point",
"coordinates":[95.5,-29.708]
},
"body":"mars"
}
]
Currently, (Eve) settings declare a DOMAIN as follows:
crater = {
'hateoas': False,
'item_title': 'crater centroid',
'url': 'centroid/<regex("[\w]+"):body>/<regex("[\w ]+"):name>',
'datasource': {
'projection': {'name': 1, 'body': 1, 'location.coordinates': 1}
}
}
DOMAIN = {
'centroids': crater,
}
Which will successfully answer to requests of the form http://hostname/centroid/<body>/<name>. Inside MongoDB this represents a query like: db.centroids.find({body:<body>, name:<name>}).
What I would like to do also is to offer an endpoint for all the documents of a given body. I.e., a request to http://hostname/centroids/<body> would answer the list of all documents with body==<body>: db.centroids.find({body:<body>}).
How do I do that?
I gave a shot by including a list of rules to the DOMAIN key centroids (the name of the database collection) like below,
crater = {
...
}
body = {
'item_title': 'body craters',
'url': 'centroids/<regex("[\w]+"):body>'
}
DOMAIN = {
'centroids': [crater, body],
}
but didn't work...
AttributeError: 'list' object has no attribute 'setdefault'
Got it!
I was assuming the keys in the DOMAIN structure was directly related to the collection Eve was querying. That is true for the default settings, but it can be adjusted inside the resources datasource.
I figured that out while handling an analogous situation as that of the question: I wanted to have an endpoint hostname/bodies listing all the (unique) values for body in the centroids collection. To that, I needed to set an aggregation to it.
The following settings give me exactly that ;)
centroids = {
'item_title': 'centroid',
'url': 'centroid/<regex("[\w]+"):body>/<regex("[\w ]+"):name>',
'datasource': {
'source': 'centroids',
'projection': {'name': 1, 'body': 1, 'location.coordinates': 1}
}
}
bodies = {
'datasource': {
'source': 'centroids',
'aggregation': {
'pipeline': [
{"$group": {"_id": "$body"}},
]
},
}
}
DOMAIN = {
'centroids': centroids,
'bodies': bodies
}
The endpoint, for example, http://127.0.0.1:5000/centroid/mercury/hokusai give me the name, body, and coordinates of mercury/hokusai.
And the endpoint http://127.0.0.1:5000/bodies, the list of unique values for body in centroids.
Beautiful. Thumbs up to Eve!
So I'm pretty new to implementing flask-restplus and I have encountered this road block.
I have read the restplus docs over and over again and followed several exampled. But the behavior that I'm facing is very much different from what is supposed to be.
So I have a model that is supposed to be a list of objects of another model (returned from the function drone_model()).
drones_list = api.model('drones_list', {
'items': fields.List(fields.Nested(drone_model())),
'message':fields.String(''),
'code': fields.Integer('')
})
Everything works fine, no errors. But when I try the API (http://127.0.0.1:5000/datamine/v2/drones), as a response I get the Marshalling model back instead of the data itself. If I print the data, it gets printed, but for some reason in the web, the restplus model is returned.
Below I have the code that I had written. If I take the marshal_with decorator off, then the data is returned just fine.
#api.route('/')
class DronesList(Resource):
#api.marshal_with(drones_list, envelope='data')
#api.response(200, 'All drones successfully fetched!')
def get(self):
"""
Get all drones!.
"""
from app.utils.common import get_start_end_date_from_request
start_date, end_date = get_start_end_date_from_request(request)
drones = []
for drone in Drone.objects:
drones.append({
'id': str(drone.id),
'serial_id': drone.serial_id,
'maintenances': [],
'status': get_dynamic_status(drone, start_date, end_date),
'picture_url': drone.asset.picture_url,
'manufacturer': drone.asset.manufacturer,
'model_name': drone.asset.model_name,
'drone_type': drone.asset.drone_type,
'payload_type': drone.asset.payload_type,
'asset_url': drone.get_url(drone.id)
})
success = ClientSuccessFunctionClass('All drones successfully fetched!', 200, drones)
return (success.to_dict())
These are the outputs on the browser:
1. Without the marshal decorator:
{
"data": {
"items": [
{
"id": "5aeafcb93a33683f73827e91",
"serial_id": "Drone 1",
"maintenances": [],
"status": "Decommissioned",
"picture_url": "some img url",
"manufacturer": "DJI",
"model_name": "Phantom 4 Pro",
"drone_type": "Quadcopter",
"payload_type": "RGB Camera",
"asset_url": "http://127.0.0.1:5000/datamine/v1/drones/5aeafcb93a33683f73827e91"
},
{
"id": "5aeaff374f85747f90df2714",
"serial_id": "Drone 2",
"maintenances": [],
"status": "Available",
"picture_url": "sime url",
"manufacturer": "DJI",
"model_name": "Phantom 4",
"drone_type": "Quadcopter",
"payload_type": "RGB Camera",
"asset_url": "http://127.0.0.1:5000/datamine/v1/drones/5aeaff374f85747f90df2714"
}
],
"message": "All drones successfully fetched!",
"code":200
}
}
2. With the marshal decorator:
{
"data": {
"items": [
{
"id": "Id of Drone",
"serial_id": "Name of Drone",
"status": "Status of Drone",
"maintenances": null,
"picture_url": "Picture URL",
"manufacturer": "Manufacturer of Drone",
"model_name": "Model name of Drone",
"drone_type": "Type of Drone",
"payload_type": "Payload type of Drone",
"asset_url": "Asset URL of Drone"
}
],
"message": "",
"code": ""
}
}
It would be really helpful if someone could tell me what I'm doing wrong as I need to recive the output as the one shown in snippet of the output without the decorator.
Thank you.
Here is a diagram showing invocation order from top to bottom to help make sense of what is happening:
get()
→ api.response(200, 'All drones successfully fetched!') # documents the response
→ api.marshal_with(drones_list, envelope='data')` # returns marshalled dict
The result from invoking get is passed to the api.response decorator function whose result is passed on to api.marshal_with decorator function.
Looking at the shape of the dictionary returned from invoking get()
{
data {
items [
{
id,
serial_id,
maintenances,
status,
picture_url,
manufacturer,
model_name,
drone_type,
payload_type,
asset_url
}
],
message,
code
}
}
The message and code in the response are nested inside of the data.
You need to model the data appropriately, to be able to marshal it. This can be done by passing an argument for what field to look up in the marshal dictionary.
drones_list = api.model('drones_list', {
'items': fields.List(fields.Nested(drone_model()), attribute='data.items'),
'message':fields.String(attribute='data.message'),
'code': fields.Integer(attribute='data.code')
})
As you can see, it's pretty redundant applying the api.marshal_with decorator function on the view given that it's only unnests then nests the result in data field.
I'm having trouble posting mutations with GraphQL and Python Requests.
My function looks like:
def create(request):
access_token = 'REDACTED'
headers = {
"X-Shopify-Storefront-Access-Token": access_token
}
mutation = """
{
checkoutCreate(input: {
lineItems: [{ variantId: "Z2lkOi8vc2hvcGlmeS9Qcm9kdWN0VmFyaWFudC80", quantity: 1 }]
}) {
checkout {
id
webUrl
lineItems(first: 5) {
edges {
node {
title
quantity
}
}
}
}
}
}
"""
data = (requests.post('https://catsinuniform.myshopify.com/api/graphql', json={'mutation': mutation}, headers=headers).json())
print(data)
return render(request, 'Stock/create.html', { 'create': data })
I'm getting errors saying I have a bad request "bad_request - Parameter Missing or Invalid" in my json response.
Even though you're sending a mutation, your request body should still include a query property, the value of which should be the string representing your operation. It's a bit confusing, but informally both queries and mutations are called "queries" (you're still "querying" the server either way). Change your request to:
requests.post('https://catsinuniform.myshopify.com/api/graphql', json={'query': mutation}, headers=headers)
i am using this python script to feed my data to elasticsearch 6.0. How can i store the variable Value with type float in Elasticsearch?
I can't use the metric options for the visualization in Kibana, because all the data is stored automatically as string
from elasticsearch import Elasticsearch
Device=""
Value=""
for key, value in row.items():
Device = key
Value = value
print("Dev",Device, "Val:", Value)
doc = {'Device':Device, 'Measure':Value , 'Sourcefile':filename}
print(' doc: ', doc)
es.index(index=name, doc_type='trends', body=doc)
Thanks
EDIT:
After the advice of #Saul, i could fix this problem with the following code:
import os,csv
import time
from elasticsearch import Elasticsearch
#import pandas as pd
import requests
Datum = time.strftime("%Y-%m-%d_")
path = '/home/pi/Desktop/Data'
os.chdir(path)
name = 'test'
es = Elasticsearch()
#'Time': time ,
#url = 'http://localhost:9200/index_name?pretty'
doc = {
"mappings": {
"doc": {
"properties": {
"device": { "type": "text" },
"measure": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
#headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
#r = requests.post(url, data=json.dumps(data), headers=headers)
r= es.index(index=name, doc_type='trends', body=doc)
print(r)
You need to send a HTTP Post request using python request, as follows:
url = "http://localhost:9200/index_name?pretty”
data = {
"mappings": {
"doc": {
"properties": {
"title": { "type": "text" },
"name": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(data), headers=headers)
Please replace index_name in the URL with the name of the index you are defining in to elasticsearch engine.
If you want to delete the index before creating it again, please do as follows:
url = "http://localhost:9200/index_name”
data = { }
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.delete(url, data=json.dumps(data), headers=headers)
please replace index_name in the URL with your actual index name. After deleting the index, create it again with the first code example above including the mappings that you would need. Enjoy.
Elasticsearch defines field types in the index mapping. It looks like you probably have dynamic mapping enabled, so when you send data to Elasticsearch for the first time, it makes an educated guess about the shape of your data and the field types.
Once those types are set, they are fixed for that index, and Elasticsearch will continue to interpret your data according to those types no matter what you do in your python script.
To fix this you need to either:
Define the index mapping before you load any data. This is the better option as it gives you complete control over how your data is interpreted. https://www.elastic.co/guide/en/elasticsearch/reference/6.0/mapping.html
Make sure that, the first time you send data into the index, you use the correct data types. This will rely dynamic mapping generation, but it will typically do the right thing.
Defining the index mapping is the best option. It's common to do that once off, in Kibana or with curl, or if you create a lot of indices, with a template.
However if you want to use python, you should look at the create or put_mapping functions on IndicesClient