We Keep Coding

Post data (dictionary list) with python requests

I'd like to post my dictionary list below via python's http requests.
my_data=[
{
'kl':'ngt',
'schemas':
[
{
'date':'14-12-2022',
'name':'kolo'
}
],
},
{
'kl':'mlk',
'schemas':
[
{
'date':'23-10-2022',
'name':'maka'
}
]
}
]
trying to do
url='http://myapi.com/product
x=requests.post(url,json=my_data)
after execution it does not post the products on the database

I think when you want to send json payload in post request you should add headers argument:
headers = {'Content-Type': 'application/json', 'Accept':'application/json'}
r = requests.post(url = 'http://myapi.com/product', data = my_data, headers=headers)
response_result = r.text
then check response status code of the post request, If it isn't 200 then post request doesn't completed successfully.

Convert terminal input to python dictionary for use with API

$ curl https://api.goclimate.com/v1/flight_footprint \
-u YOUR_API_KEY: \
-d 'segments[0][origin]=ARN' \
-d 'segments[0][destination]=BCN' \
-d 'segments[1][origin]=BCN' \
-d 'segments[1][destination]=ARN' \
-d 'cabin_class=economy' \
-d 'currencies[]=SEK' \
-d 'currencies[]=USD' \
-G
I have the following input, provided as an example by the creators of the API. This input is meant to be used in the terminal and give output in form of a dictionary. How would it be possible to write the input above in a list or dictionary to use it as part of an Python script? I tried it like below but the response from the API is solely b' '
payload = {
"segments" : [
{
"origin" : "ARN",
"destination" : "BCN"
},
{
"origin" : "BCN",
"destination" : "ARN"
}
],
"cabin_class" : "economy",
"currencies" : [
"SEK", "USD"
]
}
r = requests.get('https://api.goclimate.com/v1/flight_footprint', auth=('my_API_key', ''), data=payload)
print(r.content)

You are making a GET request with requests, but you are trying to pass data, which would be appropriate for making a POST request. Here you want to use params instead:
response = requests.get(
"https://api.goclimate.com/v1/flight_footprint",
auth=("my_API_key", ""),
params=payload,
)
print(response.content)
Now, what should payload be? It can be a dictionary, but it can't be nested in the way you had it, since it needs to be encoded into the URL as parameters (N.B. this is what your -G option was doing in the curl request).
Looking at the docs and your curl example, I think it should be:
payload = {
"segments[0][origin]": "ARN",
"segments[0][destination]": "BCN",
"segments[1][origin]": "BCN",
"segments[1][destination]": "ARN",
"cabin_class": "economy",
"currencies[]": "SEK", # this will actually be overwritten
"currencies[]": "USD", # since this key is a duplicate (see below)
}
response = requests.get(
"https://api.goclimate.com/v1/flight_footprint",
auth=("my_API_key", ""),
params=payload,
)
print(response.content)
Thinking of how we might parse your original dictionary into this structure:
data = {
"segments" : [
{
"origin" : "ARN",
"destination" : "BCN"
},
{
"origin" : "BCN",
"destination" : "ARN"
}
],
"cabin_class" : "economy",
"currencies" : [
"SEK", "USD"
]
}
payload = {}
for index, segment in enumerate(data["segments"]):
origin = segment["origin"]
destination = segment["destination"]
# python 3.6+ needed:
payload[f"segments[{index}][origin]"] = origin
payload[f"segments[{index}][destination]"] = destination
payload["cabin_class"] = data["cabin_class"]
# requests can handle repeated parameters with the same name this way:
payload["currencies[]"] = data["currencies"]
... should do it.

Convert json to formatted string

I want to convert this json :
{
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
to this:
"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}".
So that i can send it as part of payload in request in Python.
And i have tried multiple methods for the same. json.dumps doesnt work cause it doesnt escape characters in this case & .replace(""",r"\"") doesnt work cause it creates the string like this :
{\\"rate_limit_by\\": [{\\"type\\": \\"IP\\", \\"extract_from_header\\": \\"X-Forwarded-For\\"}]}
(Below is the example of curl but i want to send the data in specific format using python request.)
My upstream expects data in certain format, as of now am sending data to upstream as below:
curl -i --request POST --data "rule_name=only_ip" \
--data-binary "#data.txt" \
--url http://localhost:8001/plugin/rules
Where data.txt looks like this:
rule={
"rate_limit_by": [
{ "type":"IP", "extract_from_header": "X-Forwarded-For" }
]
}
Am trying to convert it to :
curl -i --request POST -H 'Content-Type: application/json' --data-binary #data.json http://localhost:8001/plugin/rules
Where data.json should like this
{
"rule_name" : "test_ip",
"rule":"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}"
}
Now the value of "rule" is string with character escape.
This am trying to achieve & am doing post using python.
And below is the code for same:-
import requests
import json
import re
url = 'http://localhost:8001/plugin/rules'
rule = {
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
rule = json.dumps(json.dumps(rule))
print(rule) #this output the data in correct format
obj = {
"rule_name" : "test_ip",
"rule": rule #but when used it here its get wrapped in two \\
}
headers = {'Content-Type': 'application/json', 'Accept': 'text/plain'}
print(obj)
r = requests.post(url, data=obj, headers=headers)
print(r.text)

desired is what you say you need in your something.json file. The following prints True. See https://repl.it/repls/DistantTeemingProtocol.
import json
desired = r'''{
"rule_name" : "test_ip",
"rule":"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}"
}'''
d = {
"rate_limit_by": [{
"type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
s = json.dumps(d)
xxx = json.dumps({"rule_name": "test_ip", "rule": s}, indent=4)
o = json.loads(desired)
yyy = json.dumps(o, indent=4)
print(xxx == yyy)
If you're going to POST using requests, then you should not be posting the string but should instead be posting the dictionary.
I.e.,
r = requests.post(url, json={"rule_name": "test_ip", "rule": s})

Do you mean you want to access the items inside somehow?
You should drop the "[]" because that part doesn't really make sense.
import json
x = str({
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
})
x = x.replace("[","")
x = x.replace("]","")
x = eval(x)
d = json.dumps(x)
l = json.loads(d)
l['rate_limit_by']['type']
This outputs "IP". Now you have a dictionary of all you need called l.

Converting Curl command line to python

I am using the Fiware Orion Context Broker and I want to POST some data using a python script. The command line (which works fine) looks like this:
curl -X POST -H "Accept: application/json" -H "Fiware-ServicePath: /orion" -H "Fiware-Service: orion" -H "Content-Type: application/json" -d '{"id": "JetsonTX1", "type": "sensor", "title": {"type": "Text","value": "Init"}, "percentage": { "type": "Text", "value": "0%"}}' "http://141.39.159.63:1026/v2/entities/"
My Python script:
import requests
import json
url = 'http://141.39.159.63:1026/v2/entities/'
data = '''{
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}'''
data_json = json.dumps(data)
headers = {"Accept": "application/json", "Fiware-ServicePath": "/bonseyes", "Fiware-Service": "bonseyes", "Content-Type": "application/json"}
response = requests.post(url, data=data_json, headers=headers)
print(response.json())
This is what response.json() returns:
{u'description': u'Errors found in incoming JSON buffer', u'error': u'ParseError'}
Any ideas how to fix this?
Thank you!

You probably should not pass data as string like so:
data = '''{
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}'''
Pass it as normal dict:
data = {
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}}
The request library will automatically convert this dictionary for you. Also make sure you want to use the data parameter and not the json. Below expert from the documentation should clear out why.
def post(url, data=None, json=None, **kwargs):
r"""Sends a POST request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
:param json: (optional) json data to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
From your comment it seems you should pass your data like so:
response = requests.post(url, json=data_json, headers=headers)
Because your endpoint requires json and not form-encoded bytes
And there is also missing curly brace at the end.

In my opinion you should try using keyValues option for OCB. It will make your payload way shorter. I use similar python program for updating values, therefore PATCH request in my approach:
#Sorting out url and payload for request
data = '{"' + attribute + '":' + value + '}'
headers = {'Content-type': 'application/json'}
url = urlOfOCB + '/v2/entities/' + entityId + '/attrs?options=keyValues'
r = requests.patch(url, (data), headers=headers)
You can read about this option here. As I can see you are not defining any new type for your attributes, therefore it will be "Text" by default, while using keyValues.
Attribute/metadata type may be omitted in requests. When omitted in attribute/metadata creation or in update operations, a default is used for the type depending on the value:
If value is a string, then type Text is used
If value is a number, then type Number is used.
If value is a boolean, then type Boolean is used.
If value is an object or array, then StructuredValue is used.
If value is null, then None is used.
More on those things you can find here.

How to fetch data from Elastic Database periodically using python?

I need to fetch data from Elastic database every 4 minutes, but I am facing problems in how to modify the #timestamp variable in the below mentioned query so as I can push the appropriate query to fetch the data from the URL.
I am using Python as the language.
Curl:
curl -XGET "URL" -H 'Content-Type: application/json' -k -u u_name:XXX -d'
{
"query": {
"query_string": {
"query": "#timestamp:[2018-06-29T06:47:40.000Z TO *]"
}
},
"size": 1000
}
'|json_pp )
I can use CRON to run the script scheduled every 7 minutes, but I can't understand how can I modify the #timestamp variable in the above query so as I can get every new data since the last run.
Any inputs are valuable.

You can use command date in Bash to format timestamp.
current date and time
date +%Y-%m-%dT%H:%M:%S
# 2018-07-14T03:00:58
minus 7 minutes
date --date '-7 min' +%Y-%m-%dT%H:%M:%S
# 2018-07-14T02:53:58
Using `` (ticks/backticks) you can try to put it in other command in Bash (but you many need to use " " instead of ' ' in -d)
curl -XGET "URL" -H 'Content-Type: application/json' -k -u u_name:XXX -d'
{
"query": {
"query_string": {
"query": "#timestamp:[`date --date \'-7 min\' +%Y-%m-%dT%H:%M:%S`.000Z TO *]"
}
},
"size": 1000
}
'|json_pp )
If you need it as Python code then you can use page https://curl.trillworks.com/ to convert curl to requests and later you can make modifications.
import requests
import datetime
import pprint # pretty print
#dt = datetime.datetime(2018, 6, 29, 6, 47, 40)
dt = datetime.datetime.now()
td_7mins = datetime.timedelta(minutes=7)
dt = dt - td_7mins # now - 7 minutes
#timestamp = "#timestamp:[{}.000Z TO *]".format(now.strftime("%Y-%m-%dT%H:%M:%S"))
timestamp = dt.strftime("#timestamp:[%Y-%m-%dT%H:%M:%S.000Z TO *]")
data = {
"query": {
"query_string": {
"query": timestamp
}
},
"size": 1000
}
print(data)
url = "https://httpbin.org/get" # good for tests
r = requests.get(url, json=data, headers=headers, verify=False, auth=('u_name', 'XXX'))
pprint.pprint(r.json())