I need to fetch data from Elastic database every 4 minutes, but I am facing problems in how to modify the #timestamp variable in the below mentioned query so as I can push the appropriate query to fetch the data from the URL.
I am using Python as the language.
Curl:
curl -XGET "URL" -H 'Content-Type: application/json' -k -u u_name:XXX -d'
{
"query": {
"query_string": {
"query": "#timestamp:[2018-06-29T06:47:40.000Z TO *]"
}
},
"size": 1000
}
'|json_pp )
I can use CRON to run the script scheduled every 7 minutes, but I can't understand how can I modify the #timestamp variable in the above query so as I can get every new data since the last run.
Any inputs are valuable.
You can use command date in Bash to format timestamp.
current date and time
date +%Y-%m-%dT%H:%M:%S
# 2018-07-14T03:00:58
minus 7 minutes
date --date '-7 min' +%Y-%m-%dT%H:%M:%S
# 2018-07-14T02:53:58
Using `` (ticks/backticks) you can try to put it in other command in Bash (but you many need to use " " instead of ' ' in -d)
curl -XGET "URL" -H 'Content-Type: application/json' -k -u u_name:XXX -d'
{
"query": {
"query_string": {
"query": "#timestamp:[`date --date \'-7 min\' +%Y-%m-%dT%H:%M:%S`.000Z TO *]"
}
},
"size": 1000
}
'|json_pp )
If you need it as Python code then you can use page https://curl.trillworks.com/ to convert curl to requests and later you can make modifications.
import requests
import datetime
import pprint # pretty print
#dt = datetime.datetime(2018, 6, 29, 6, 47, 40)
dt = datetime.datetime.now()
td_7mins = datetime.timedelta(minutes=7)
dt = dt - td_7mins # now - 7 minutes
#timestamp = "#timestamp:[{}.000Z TO *]".format(now.strftime("%Y-%m-%dT%H:%M:%S"))
timestamp = dt.strftime("#timestamp:[%Y-%m-%dT%H:%M:%S.000Z TO *]")
data = {
"query": {
"query_string": {
"query": timestamp
}
},
"size": 1000
}
print(data)
url = "https://httpbin.org/get" # good for tests
r = requests.get(url, json=data, headers=headers, verify=False, auth=('u_name', 'XXX'))
pprint.pprint(r.json())
Related
I try to do this query with elasticsearch python client :
curl -X GET "localhost:9200/articles/_knn_search" -H 'Content-Type: application/json' -d '
{
"knn": {
"field": "title_vector",
"query_vector": [-0.01807806, 0.024579186,...],
"k": 10,
"num_candidates": 100
},
"_source": ["title", "category"]
}
'
If anyone can help me with thanks.
EDIT : with elasticsearch python client > 8.0 there is a new function named knn_search so we can run knn_search very easy:
query = {
"field": "title_vector",
"query_vector": [-0.01807806, 0.024579186,...],
"k": 10,
"num_candidates": 100
}
es = Elasticsearch(request_timeout=600, hosts='http://localhost:9200')
res = es.knn_search(index="index_name", knn=query, source=["filed1", "field2"])
$ curl https://api.goclimate.com/v1/flight_footprint \
-u YOUR_API_KEY: \
-d 'segments[0][origin]=ARN' \
-d 'segments[0][destination]=BCN' \
-d 'segments[1][origin]=BCN' \
-d 'segments[1][destination]=ARN' \
-d 'cabin_class=economy' \
-d 'currencies[]=SEK' \
-d 'currencies[]=USD' \
-G
I have the following input, provided as an example by the creators of the API. This input is meant to be used in the terminal and give output in form of a dictionary. How would it be possible to write the input above in a list or dictionary to use it as part of an Python script? I tried it like below but the response from the API is solely b' '
payload = {
"segments" : [
{
"origin" : "ARN",
"destination" : "BCN"
},
{
"origin" : "BCN",
"destination" : "ARN"
}
],
"cabin_class" : "economy",
"currencies" : [
"SEK", "USD"
]
}
r = requests.get('https://api.goclimate.com/v1/flight_footprint', auth=('my_API_key', ''), data=payload)
print(r.content)
You are making a GET request with requests, but you are trying to pass data, which would be appropriate for making a POST request. Here you want to use params instead:
response = requests.get(
"https://api.goclimate.com/v1/flight_footprint",
auth=("my_API_key", ""),
params=payload,
)
print(response.content)
Now, what should payload be? It can be a dictionary, but it can't be nested in the way you had it, since it needs to be encoded into the URL as parameters (N.B. this is what your -G option was doing in the curl request).
Looking at the docs and your curl example, I think it should be:
payload = {
"segments[0][origin]": "ARN",
"segments[0][destination]": "BCN",
"segments[1][origin]": "BCN",
"segments[1][destination]": "ARN",
"cabin_class": "economy",
"currencies[]": "SEK", # this will actually be overwritten
"currencies[]": "USD", # since this key is a duplicate (see below)
}
response = requests.get(
"https://api.goclimate.com/v1/flight_footprint",
auth=("my_API_key", ""),
params=payload,
)
print(response.content)
Thinking of how we might parse your original dictionary into this structure:
data = {
"segments" : [
{
"origin" : "ARN",
"destination" : "BCN"
},
{
"origin" : "BCN",
"destination" : "ARN"
}
],
"cabin_class" : "economy",
"currencies" : [
"SEK", "USD"
]
}
payload = {}
for index, segment in enumerate(data["segments"]):
origin = segment["origin"]
destination = segment["destination"]
# python 3.6+ needed:
payload[f"segments[{index}][origin]"] = origin
payload[f"segments[{index}][destination]"] = destination
payload["cabin_class"] = data["cabin_class"]
# requests can handle repeated parameters with the same name this way:
payload["currencies[]"] = data["currencies"]
... should do it.
Below are two curl command.
I need to add the below content in curl in index of elasticsearch through python api, How to achieve this
curl -XPUT 'http://localhost:9200/my_country_index_5/country/1' -d '
{
"name": "Afginastan"
}'
curl -XPUT 'http://localhost:9200/my_country_index_5/state/1?parent=3' -d '
{
"name": "Andra Pradesh",
"country": "India"
}'
curl -XPUT 'http://localhost:9200/my_country_index_5/city/1?parent=5' -d '
{
"name": "Kolhapur",
"state": "Maharashtra"
}'
I have created index in python below is the code
from elasticsearch import Elasticsearch
es = Elasticsearch()
es.indices.create(index='my_country_index_5', ignore=400)
How to put in to same index(my_country_index_5) but different document country, state, city
doc = {
"name": "Afginastan"
}
res = es.index(index="my_country_index_5", id=1, body=doc)
print(res['result'])
If I understand well, you want to have type which is shown in python as doc_type. type is omitted from version 7. In elasticsearch version lower than 7, you can have this by sending doc_type in the index arguments.
res = es.index(index="my_country_index_5", id=1, doc_type="state", body=doc)
res = es.index(index="my_country_index_5", id=1, doc_type="city", body=doc)
res = es.index(index="my_country_index_5", id=1, doc_type="country", body=doc)
I want to convert this json :
{
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
to this:
"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}".
So that i can send it as part of payload in request in Python.
And i have tried multiple methods for the same. json.dumps doesnt work cause it doesnt escape characters in this case & .replace(""",r"\"") doesnt work cause it creates the string like this :
{\\"rate_limit_by\\": [{\\"type\\": \\"IP\\", \\"extract_from_header\\": \\"X-Forwarded-For\\"}]}
(Below is the example of curl but i want to send the data in specific format using python request.)
My upstream expects data in certain format, as of now am sending data to upstream as below:
curl -i --request POST --data "rule_name=only_ip" \
--data-binary "#data.txt" \
--url http://localhost:8001/plugin/rules
Where data.txt looks like this:
rule={
"rate_limit_by": [
{ "type":"IP", "extract_from_header": "X-Forwarded-For" }
]
}
Am trying to convert it to :
curl -i --request POST -H 'Content-Type: application/json' --data-binary #data.json http://localhost:8001/plugin/rules
Where data.json should like this
{
"rule_name" : "test_ip",
"rule":"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}"
}
Now the value of "rule" is string with character escape.
This am trying to achieve & am doing post using python.
And below is the code for same:-
import requests
import json
import re
url = 'http://localhost:8001/plugin/rules'
rule = {
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
rule = json.dumps(json.dumps(rule))
print(rule) #this output the data in correct format
obj = {
"rule_name" : "test_ip",
"rule": rule #but when used it here its get wrapped in two \\
}
headers = {'Content-Type': 'application/json', 'Accept': 'text/plain'}
print(obj)
r = requests.post(url, data=obj, headers=headers)
print(r.text)
desired is what you say you need in your something.json file. The following prints True. See https://repl.it/repls/DistantTeemingProtocol.
import json
desired = r'''{
"rule_name" : "test_ip",
"rule":"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}"
}'''
d = {
"rate_limit_by": [{
"type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
s = json.dumps(d)
xxx = json.dumps({"rule_name": "test_ip", "rule": s}, indent=4)
o = json.loads(desired)
yyy = json.dumps(o, indent=4)
print(xxx == yyy)
If you're going to POST using requests, then you should not be posting the string but should instead be posting the dictionary.
I.e.,
r = requests.post(url, json={"rule_name": "test_ip", "rule": s})
Do you mean you want to access the items inside somehow?
You should drop the "[]" because that part doesn't really make sense.
import json
x = str({
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
})
x = x.replace("[","")
x = x.replace("]","")
x = eval(x)
d = json.dumps(x)
l = json.loads(d)
l['rate_limit_by']['type']
This outputs "IP". Now you have a dictionary of all you need called l.
params_for_update_cat = {
'api_key': API_KEY,
'api_username':"ausername",
"name": name_and_ticker,
"color": bg_colour,
"text_color": text_color,
"permission": {'admins': 1,'staff': 2}
}
headers = {"Content-Type": "multipart/form-data"}
r = rq.post(url=create_cat_url, data=params, headers=headers)
So what I want is and the above certainly doesn't do this:
permissions[admin]=1&permissions[staff]=2
as parts of the params list. this should be in the post body of course.
Some caveats:
The api doesn't support content-type: application/json request headers.
The api is expecting the request to be made as data was coming from a form.
If I were to do this in curl and this is successfully test btw - it would be:
!curl -X POST "http://<mydomaingoeshere>/categories.json" /
-H "Content-Type: multipart/form-data;" /
-F "api_key=xxxxxxxxx_xxxxx" /
-F "api_username=ausername" /
-F "name=checkcheckcheck" /
-F "text_color=FFFFFF" /
-F "color=49d9e9" /
-F "permissions[admins]=1" /
-F "permissions[staff]=2"
3 days on this - and no one has been able to help. Maybe something like this is a bad idea? I don't know.
How about being explicit in the dictionary:
params_for_update_cat = {
'api_key': API_KEY,
'api_username':"ausername",
"name": name_and_ticker,
"color": bg_colour,
"text_color": text_color,
"permissions[admins]": 1,
"permissions[staff]": 2
}
This creates a request with the URL looking like:
"http://example.com?...&permissions%5Badmins%5D=1&permissions%5Bstaff%5D=2"