$ curl https://api.goclimate.com/v1/flight_footprint \
-u YOUR_API_KEY: \
-d 'segments[0][origin]=ARN' \
-d 'segments[0][destination]=BCN' \
-d 'segments[1][origin]=BCN' \
-d 'segments[1][destination]=ARN' \
-d 'cabin_class=economy' \
-d 'currencies[]=SEK' \
-d 'currencies[]=USD' \
-G
I have the following input, provided as an example by the creators of the API. This input is meant to be used in the terminal and give output in form of a dictionary. How would it be possible to write the input above in a list or dictionary to use it as part of an Python script? I tried it like below but the response from the API is solely b' '
payload = {
"segments" : [
{
"origin" : "ARN",
"destination" : "BCN"
},
{
"origin" : "BCN",
"destination" : "ARN"
}
],
"cabin_class" : "economy",
"currencies" : [
"SEK", "USD"
]
}
r = requests.get('https://api.goclimate.com/v1/flight_footprint', auth=('my_API_key', ''), data=payload)
print(r.content)
You are making a GET request with requests, but you are trying to pass data, which would be appropriate for making a POST request. Here you want to use params instead:
response = requests.get(
"https://api.goclimate.com/v1/flight_footprint",
auth=("my_API_key", ""),
params=payload,
)
print(response.content)
Now, what should payload be? It can be a dictionary, but it can't be nested in the way you had it, since it needs to be encoded into the URL as parameters (N.B. this is what your -G option was doing in the curl request).
Looking at the docs and your curl example, I think it should be:
payload = {
"segments[0][origin]": "ARN",
"segments[0][destination]": "BCN",
"segments[1][origin]": "BCN",
"segments[1][destination]": "ARN",
"cabin_class": "economy",
"currencies[]": "SEK", # this will actually be overwritten
"currencies[]": "USD", # since this key is a duplicate (see below)
}
response = requests.get(
"https://api.goclimate.com/v1/flight_footprint",
auth=("my_API_key", ""),
params=payload,
)
print(response.content)
Thinking of how we might parse your original dictionary into this structure:
data = {
"segments" : [
{
"origin" : "ARN",
"destination" : "BCN"
},
{
"origin" : "BCN",
"destination" : "ARN"
}
],
"cabin_class" : "economy",
"currencies" : [
"SEK", "USD"
]
}
payload = {}
for index, segment in enumerate(data["segments"]):
origin = segment["origin"]
destination = segment["destination"]
# python 3.6+ needed:
payload[f"segments[{index}][origin]"] = origin
payload[f"segments[{index}][destination]"] = destination
payload["cabin_class"] = data["cabin_class"]
# requests can handle repeated parameters with the same name this way:
payload["currencies[]"] = data["currencies"]
... should do it.
Related
Query using curl.
curl -XGET http://localhost:9200/users/_search?pretty=true -H 'Content-Type: application/json' -d '
{
"from" : 0, "size" : 5,
"query" : {
"query_string" : {
"query" : "Port"
}
}
}
'
Same thing i am trying to do in python requests:
url = f'{URL}/{INDEX}/_search/'
es_query = {
"query" : {
"query_string" : {
"query" : "Port"
}
}
}
res = requests.get(url, data=es_query)
print(res.json())
I am getting below error while doing this.
`{'error': 'Content-Type header [application/x-www-form-urlencoded] is not supported', 'status': 406}`
Please take a look what can be the issue
You can use curlconverter.com to easily convert cURL commands to Python requests. Here's the output for your cURL command:
import requests
params = {
'pretty': 'true',
}
json_data = {
'from': 0,
'size': 5,
'query': {
'query_string': {
'query': 'Port',
},
},
}
response = requests.get('http://localhost:9200/users/_search', params=params, headers=headers, json=json_data)
Your URL is wrong
url = f'{URL}/{INDEX}/_bulk/'
Should be
url = f'{URL}/{INDEX}/_search/'
Also you need to specify the following request header
Content-type: application/json
Like this:
res = requests.get(url, json=es_query, headers={"Content-type": "application/json"})
Below are two curl command.
I need to add the below content in curl in index of elasticsearch through python api, How to achieve this
curl -XPUT 'http://localhost:9200/my_country_index_5/country/1' -d '
{
"name": "Afginastan"
}'
curl -XPUT 'http://localhost:9200/my_country_index_5/state/1?parent=3' -d '
{
"name": "Andra Pradesh",
"country": "India"
}'
curl -XPUT 'http://localhost:9200/my_country_index_5/city/1?parent=5' -d '
{
"name": "Kolhapur",
"state": "Maharashtra"
}'
I have created index in python below is the code
from elasticsearch import Elasticsearch
es = Elasticsearch()
es.indices.create(index='my_country_index_5', ignore=400)
How to put in to same index(my_country_index_5) but different document country, state, city
doc = {
"name": "Afginastan"
}
res = es.index(index="my_country_index_5", id=1, body=doc)
print(res['result'])
If I understand well, you want to have type which is shown in python as doc_type. type is omitted from version 7. In elasticsearch version lower than 7, you can have this by sending doc_type in the index arguments.
res = es.index(index="my_country_index_5", id=1, doc_type="state", body=doc)
res = es.index(index="my_country_index_5", id=1, doc_type="city", body=doc)
res = es.index(index="my_country_index_5", id=1, doc_type="country", body=doc)
I want to convert this json :
{
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
to this:
"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}".
So that i can send it as part of payload in request in Python.
And i have tried multiple methods for the same. json.dumps doesnt work cause it doesnt escape characters in this case & .replace(""",r"\"") doesnt work cause it creates the string like this :
{\\"rate_limit_by\\": [{\\"type\\": \\"IP\\", \\"extract_from_header\\": \\"X-Forwarded-For\\"}]}
(Below is the example of curl but i want to send the data in specific format using python request.)
My upstream expects data in certain format, as of now am sending data to upstream as below:
curl -i --request POST --data "rule_name=only_ip" \
--data-binary "#data.txt" \
--url http://localhost:8001/plugin/rules
Where data.txt looks like this:
rule={
"rate_limit_by": [
{ "type":"IP", "extract_from_header": "X-Forwarded-For" }
]
}
Am trying to convert it to :
curl -i --request POST -H 'Content-Type: application/json' --data-binary #data.json http://localhost:8001/plugin/rules
Where data.json should like this
{
"rule_name" : "test_ip",
"rule":"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}"
}
Now the value of "rule" is string with character escape.
This am trying to achieve & am doing post using python.
And below is the code for same:-
import requests
import json
import re
url = 'http://localhost:8001/plugin/rules'
rule = {
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
rule = json.dumps(json.dumps(rule))
print(rule) #this output the data in correct format
obj = {
"rule_name" : "test_ip",
"rule": rule #but when used it here its get wrapped in two \\
}
headers = {'Content-Type': 'application/json', 'Accept': 'text/plain'}
print(obj)
r = requests.post(url, data=obj, headers=headers)
print(r.text)
desired is what you say you need in your something.json file. The following prints True. See https://repl.it/repls/DistantTeemingProtocol.
import json
desired = r'''{
"rule_name" : "test_ip",
"rule":"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}"
}'''
d = {
"rate_limit_by": [{
"type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
s = json.dumps(d)
xxx = json.dumps({"rule_name": "test_ip", "rule": s}, indent=4)
o = json.loads(desired)
yyy = json.dumps(o, indent=4)
print(xxx == yyy)
If you're going to POST using requests, then you should not be posting the string but should instead be posting the dictionary.
I.e.,
r = requests.post(url, json={"rule_name": "test_ip", "rule": s})
Do you mean you want to access the items inside somehow?
You should drop the "[]" because that part doesn't really make sense.
import json
x = str({
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
})
x = x.replace("[","")
x = x.replace("]","")
x = eval(x)
d = json.dumps(x)
l = json.loads(d)
l['rate_limit_by']['type']
This outputs "IP". Now you have a dictionary of all you need called l.
I need to fetch data from Elastic database every 4 minutes, but I am facing problems in how to modify the #timestamp variable in the below mentioned query so as I can push the appropriate query to fetch the data from the URL.
I am using Python as the language.
Curl:
curl -XGET "URL" -H 'Content-Type: application/json' -k -u u_name:XXX -d'
{
"query": {
"query_string": {
"query": "#timestamp:[2018-06-29T06:47:40.000Z TO *]"
}
},
"size": 1000
}
'|json_pp )
I can use CRON to run the script scheduled every 7 minutes, but I can't understand how can I modify the #timestamp variable in the above query so as I can get every new data since the last run.
Any inputs are valuable.
You can use command date in Bash to format timestamp.
current date and time
date +%Y-%m-%dT%H:%M:%S
# 2018-07-14T03:00:58
minus 7 minutes
date --date '-7 min' +%Y-%m-%dT%H:%M:%S
# 2018-07-14T02:53:58
Using `` (ticks/backticks) you can try to put it in other command in Bash (but you many need to use " " instead of ' ' in -d)
curl -XGET "URL" -H 'Content-Type: application/json' -k -u u_name:XXX -d'
{
"query": {
"query_string": {
"query": "#timestamp:[`date --date \'-7 min\' +%Y-%m-%dT%H:%M:%S`.000Z TO *]"
}
},
"size": 1000
}
'|json_pp )
If you need it as Python code then you can use page https://curl.trillworks.com/ to convert curl to requests and later you can make modifications.
import requests
import datetime
import pprint # pretty print
#dt = datetime.datetime(2018, 6, 29, 6, 47, 40)
dt = datetime.datetime.now()
td_7mins = datetime.timedelta(minutes=7)
dt = dt - td_7mins # now - 7 minutes
#timestamp = "#timestamp:[{}.000Z TO *]".format(now.strftime("%Y-%m-%dT%H:%M:%S"))
timestamp = dt.strftime("#timestamp:[%Y-%m-%dT%H:%M:%S.000Z TO *]")
data = {
"query": {
"query_string": {
"query": timestamp
}
},
"size": 1000
}
print(data)
url = "https://httpbin.org/get" # good for tests
r = requests.get(url, json=data, headers=headers, verify=False, auth=('u_name', 'XXX'))
pprint.pprint(r.json())
Problem description
I want to run a query string like this for example :
{"query": {
"query_string" : {
"fields" : ["description"],
"query" : "illegal~"
}
}
}
I have a side synonyms.txt file that contains synonyms :
illegal, banned, criminal, illegitimate, illicit, irregular, outlawed, prohibited
otherWord, synonym1, synonym2...
I want to find all elements having any one of these synonyms.
What I tried
First I want to index those synonyms in my ES database.
I tried to run this query with curl :
curl -X PUT "https://instanceAdress.europe-west1.gcp.cloud.es.io:9243/app/kibana#/dev_tools/console/sources" -H 'Content-Type: application/json' -d' {
"settings": {
"index" : {
"analysis" : {
"analyzer" : {
"synonym" : {
"tokenizer" : "whitespace",
"filter" : ["synonym"]
}
},
"filter" : {
"synonym" : {
"type" : "synonym",
"synonyms_path" : "synonyms.txt"
}
}
}
}
}
}
'
but it doesn't work {"statusCode":404,"error":"Not Found"}
I then need to change my query so that it takes into account the synonyms but I have no idea how.
So my questions are :
How can I index my synonyms ?
How can I change my query so that it does the query for all synonyms ?
Is there any way to index them in Python ?
example of a get query using Python Elasticsearch
es = Elasticsearch(
['fullAdress.europe-west1.gcp.cloud.es.io'],
http_auth=('login', 'password'),
scheme="https",
port=9243,
)
es.get(index="sources", doc_type='rcp', id="301495")
You can index using synonyms with Python by:
First, create a token filter:
synonyms_token_filter = token_filter(
'synonyms_token_filter', # Any name for the filter
'synonym', # Synonym filter type
synonyms=your_synonyms # Synonyms mapping will be inlined
)
And then create an analyzer:
custom_analyzer = analyzer(
'custom_analyzer',
tokenizer='standard',
filter=[
'lowercase',
synonyms_token_filter
])
There's also a package for this: https://github.com/agora-team/elasticsearch-synonyms