Trying to use urllib3 to post JSON-encoded data.
Just want my POST payload to be raw JSON string, with content type application/json.
I just cannot see how to do this.
The urllib3 documentation describes posting data in "fields", i.e. dicts with (key,value) pairs, like how HTML forms are URL-encoded with the URL. But I don't want to do that.
The closest I've been able to get is this (I just guessed where to put the data, as it's not documented anywhere that I can find):
http = urllib3.PoolManager()
headers = urllib3.util.make_headers(basic_auth=key+":")
r = http.request_encode_body('POST', path, json.dumps(payload), headers=headers)
which causes this urllib3 error:
File "C:\Python27\lib\site-packages\urllib3-1.7.1-py2.7.egg\urllib3\filepost.py", line 44, in iter_field_objects
yield RequestField.from_tuples(*field)
TypeError: from_tuples() takes exactly 3 arguments (2 given)
Thanks for any pointers!
you can't use PoolManager.request for that, it tries to concoct the body iself, use the lower level urlopen:
In [16]: pool = urllib3.PoolManager()
In [17]: print pool.urlopen('POST', 'http://httpbin.org/post', headers={'Content-Type':'application/json'}, body='{"sup":"son"}').data
{
"data": "{\"sup\":\"son\"}",
"form": {},
"json": {
"sup": "son"
},
"origin": "50.74.23.243",
"args": {},
"url": "http://httpbin.org/post",
"files": {},
"headers": {
"Host": "httpbin.org",
"Content-Length": "13",
"Content-Type": "application/json",
"Accept-Encoding": "identity",
"Connection": "close"
}
}
Related
I have the following code
import requests
import json
import sys
credentials_User=sys.argv[1]
credentials_Password=sys.argv[2]
email=sys.argv[3]
def auth_api(login_User,login_Password,):
gooddata_user=login_User
gooddata_password=login_Password
body = json.dumps({
"postUserLogin":{
"login": gooddata_user,
"password": gooddata_password,
"remember":1,
"verify_level":0
}
})
headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'
}
url="https://reports.domain.com/gdc/account/login"
response = requests.request(
"POST",
url,
headers=headers,
data=body
)
sst=response.headers.get('Set-Cookie')
return sst
def query_api(cookie,email):
url="https://reports.domain.com/gdc/account/domains/domain/users?login="+email
body={}
headers={
'Content-Type': 'application/json',
'Accept': 'application/json',
'Cookie': cookie
}
response = requests.request(
"GET",
url,
headers=headers,
data=body
)
jsonContent=[]
jsonContent.append({response.text})
accountSettings=jsonContent[0]
print(accountSettings)
cookie=auth_api(credentials_User,credentials_Password)
profilehash=query_api(cookie,email)
The code itself works and sends a request to the Gooddata API.
The query_api() function returns JSON similar to below
{
"accountSettings": {
"items": [
{
"accountSetting": {
"login": "user#example.com",
"email": "user#example.com",
"firstName": "First Name",
"lastName": "Last Name",
"companyName": "Company Name",
"position": "Data Analyst",
"created": "2020-01-08 15:44:23",
"updated": "2020-01-08 15:44:23",
"timezone": null,
"country": "United States",
"phoneNumber": "(425) 555-1111",
"old_password": "secret$123",
"password": "secret$234",
"verifyPassword": "secret$234",
"authenticationModes": [
"SSO"
],
"ssoProvider": "sso-domain.com",
"language": "en-US",
"ipWhitelist": [
"127.0.0.1"
],
"links": {
"projects": "/gdc/account/profile/{profile_id}/projects",
"self": "/gdc/account/profile/{profile_id}",
"domain": "/gdc/domains/default",
"auditEvents": "/gdc/account/profile/{profile_id}/auditEvents"
},
"effectiveIpWhitelist": "[ 127.0.0.1 ]"
}
}
],
"paging": {
"offset": 20,
"count": 100,
"next": "/gdc/uri?offset=100"
}
}
}
The issue I am having is reading specific keys from this JSON Dict, I can use accountSettings=jsonContent[0] but that just returns the same JSON.
What I want to do is read the value of the project key within links
How would I do this with a dict?
Thanks
Based on your description, uyou have your value inside a list, (not a set. Foergt about set: sets are not used with JSON). Inside your list, you either your content as a single string, which then you'd have to parse with json.loads, or it is simply a well behaved nested data structure already extracted from JSON, but which is inside a single element list. This seems the most likely.
So, you should be able to do:
accountlink = jsonContent[0]["items"][0]["accountSetting"]["login"]
otherwise, if it is encoded as a a json string, you have to parse it first:
import json
accountlink = json.loads(jsonContent[0])["items"][0]["accountSetting"]["login"]
Now, given your question, I'd say your are on a begginer level as a programmer, or a casual user, just using Python to automatize something either way, I'd recommend you do try some exercising before proceeding: it will save you time (a lot of time). I am not trying to bully or mock anything here: this is the best advice I can offer you. Seek for tutorials that play around on the interactive mode, rather than trying entire programs at once that you'd just copy and paste.
Using the below code fixed the issue
jsonContent=json.loads(response.text)
print(type(jsonContent))
test=jsonContent["accountSettings"]["items"][0]
test2=test["accountSetting"]["links"]["self"]
print(test)
print(test2)
I believe this works because for some reason I didn't notice I was using .append for my jsonContent. This resulted in the data type being something other than it should have been.
Thanks to everyone who tried helping me.
I use the below function to get data from the web , but failed. I wonder whether urllib.quote use incorrect
i have used urllib.urlencode(xx) but it show not a valid non-string sequence or mapping object
and my request data is:
[{"Keys": "SV_cY1tKhYiocNluHb", "Details": [{"id2": "PK_2gl9xtYKX7TJi29"}], "language": "EN", "id": "535985"}]
Anyone can help. Thanks a lot !!!
###This Funcation call API Post Data
def CallApi(apilink, indata):
token = gettoken()
data = json.dumps(indata, ensure_ascii=False)
print(data)
headers = {'content-type': 'application/json', 'Accept': 'application/json', 'Authorization': 'Bearer %s' % (token)}
proxy = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
DataForGet=urllib.quote(data)
NewUrl= apilink + "?" + DataForGet
request = urllib2.Request(NewUrl, headers=headers)
response = urllib2.urlopen(request, timeout=300)
message = response.read()
print(message)
Error:
the err message below: File "/opt/freeware/lib/python2.7/urllib2.py", line 1198, in do_open raise URLError(err)
You could see the comment of urlencode
Encode a dict or sequence of two-element tuples into a URL query string
So you could choose removing the outer []
{"Keys": "SV_cY1tKhYiocNluHb", "Details": [{"id2": "PK_2gl9xtYKX7TJi29"}], "language": "EN", "id": "535985"}
Or use a two-element tuple
(('Keys', 'SV_cY1tKhYiocNluHb'), ('Details', [{...}]...)
Demo
>>> import urllib
>>> s = {"Keys": "SV_cY1tKhYiocNluHb", "Details": [{"id2": "PK_2gl9xtYKX7TJi29"}], "language": "EN", "id": "535985"}
>>> urllib.urlencode(s)
'Keys=SV_cY1tKhYiocNluHb&Details=%5B%7B%27id2%27%3A+%27PK_2gl9xtYKX7TJi29%27%7D%5D&language=EN&id=535985'
>>>
import requests #package required to handle API requests
import json #built in package to handle JSON
import os #accesing variables from .bash_profile
i= 0 #const required to iteration of loop
f = open("/mnt/c/_KOD_/project/JSON/output.txt", "a", encoding="UTF-8" )
fraza=str
id_category=int
url = os.environ.get("PROD_LB")
print("Provide search phrase :")
input(fraza)
print("provide category:")
input(id_category)
payload = "\r\n{\r\n \"criteria\": {\r\n \"product_list.show\": true,\r\n \"product_list.show_if_below\": -1,\r\n \"product_list.limit\": -1,\r\n \"product_sum.show\": true,\r\n \"query.phrase\": \"", fraza ,"\",\r\n \"category.id_or_deeper\": [\"", id_category,"diod \"]\r\n }\r\n}"
headers = {
'Content-Type': "application/json",
'User-Agent': "PostmanRuntime/7.18.0",
'Accept': "*/*",
'Cache-Control': "no-cache",
'Postman-Token': "75924d05-2bd0-4133-aed7-515aa644a535,bfdc06e9-6142-4822-a919-81390ba871e4",
'Host': "search.tme.eu:8443",
'Accept-Encoding': "gzip, deflate",
'Content-Length': "230",
'Connection': "keep-alive",
'cache-control': "no-cache"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
ress = json.loads(response.text)
list = ress["product_list"]
a = list.__len__() #no longer needed in for loop.
# this part below is quite junky and i have no clue how to revive it...
**For list, value in list.items() :
# while i < a:
f.write(list["symbol"])
f.write("\r\n")
# i +=1
f.close**
Payload above genere JSON response that look like
{
"request_id": 4454370058,
"product_list": [
{
"score": 19993,
"symbol": "M22-ES-MS2",
"id": 346733
},
.
.
.
{
"score": 19989,
"symbol": "M22-D-R-X0/KC11/I",
"id": 94432
}
],
"do_show": true,
"do_show_list": {
"do_show_products": true,
"do_show_parameters": false,
"do_show_parameter_values": false,
"do_show_flags": false
}
}
My goal is to save part of response to txt file that look like:
M22-ES-MS2
M22-D-R-X0/KC11/I
I was able to do this by saving respones to JSON file and from that to save to final format... however that was not optimal..
Currently im stuck with either of 2 errors
When i try to do this via While loop im receving :
in _encode_params
for k, vs in to_key_val_list(data):
ValueError: too many values to unpack (expected 2)
or after reading THIS
im stuck with
File "/mnt/c/_KOD_/project/JSON/request.py", line 39
For list, value in list.items() :
^
SyntaxError: invalid syntax
As I have no clue how this should work.
This is my most advanced python code I have ever created in my 3 weeks of programming.
you can load the json data using the json library and then just iterate over the products list printing the symbols.
data = """{
"request_id": 4454370058,
"product_list": [
{
"score": 19993,
"symbol": "M22-ES-MS2",
"id": 346733
},
{
"score": 19989,
"symbol": "M22-D-R-X0/KC11/I",
"id": 94432
}
],
"do_show": true,
"do_show_list": {
"do_show_products": true,
"do_show_parameters": false,
"do_show_parameter_values": false,
"do_show_flags": false
}
}"""
import json
json_data = json.loads(data)
for product in json_data['product_list']:
print(product['symbol'])
OUTPUT
M22-ES-MS2
M22-D-R-X0/KC11/I
list is a reserved name in python, try changing the name list to something else.
You can put key, value instead of list, value.
Also the For should be lower case for.
I am using the Fiware Orion Context Broker and I want to POST some data using a python script. The command line (which works fine) looks like this:
curl -X POST -H "Accept: application/json" -H "Fiware-ServicePath: /orion" -H "Fiware-Service: orion" -H "Content-Type: application/json" -d '{"id": "JetsonTX1", "type": "sensor", "title": {"type": "Text","value": "Init"}, "percentage": { "type": "Text", "value": "0%"}}' "http://141.39.159.63:1026/v2/entities/"
My Python script:
import requests
import json
url = 'http://141.39.159.63:1026/v2/entities/'
data = '''{
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}'''
data_json = json.dumps(data)
headers = {"Accept": "application/json", "Fiware-ServicePath": "/bonseyes", "Fiware-Service": "bonseyes", "Content-Type": "application/json"}
response = requests.post(url, data=data_json, headers=headers)
print(response.json())
This is what response.json() returns:
{u'description': u'Errors found in incoming JSON buffer', u'error': u'ParseError'}
Any ideas how to fix this?
Thank you!
You probably should not pass data as string like so:
data = '''{
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}'''
Pass it as normal dict:
data = {
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}}
The request library will automatically convert this dictionary for you. Also make sure you want to use the data parameter and not the json. Below expert from the documentation should clear out why.
def post(url, data=None, json=None, **kwargs):
r"""Sends a POST request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
:param json: (optional) json data to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
From your comment it seems you should pass your data like so:
response = requests.post(url, json=data_json, headers=headers)
Because your endpoint requires json and not form-encoded bytes
And there is also missing curly brace at the end.
In my opinion you should try using keyValues option for OCB. It will make your payload way shorter. I use similar python program for updating values, therefore PATCH request in my approach:
#Sorting out url and payload for request
data = '{"' + attribute + '":' + value + '}'
headers = {'Content-type': 'application/json'}
url = urlOfOCB + '/v2/entities/' + entityId + '/attrs?options=keyValues'
r = requests.patch(url, (data), headers=headers)
You can read about this option here. As I can see you are not defining any new type for your attributes, therefore it will be "Text" by default, while using keyValues.
Attribute/metadata type may be omitted in requests. When omitted in attribute/metadata creation or in update operations, a default is used for the type depending on the value:
If value is a string, then type Text is used
If value is a number, then type Number is used.
If value is a boolean, then type Boolean is used.
If value is an object or array, then StructuredValue is used.
If value is null, then None is used.
More on those things you can find here.
i am using this python script to feed my data to elasticsearch 6.0. How can i store the variable Value with type float in Elasticsearch?
I can't use the metric options for the visualization in Kibana, because all the data is stored automatically as string
from elasticsearch import Elasticsearch
Device=""
Value=""
for key, value in row.items():
Device = key
Value = value
print("Dev",Device, "Val:", Value)
doc = {'Device':Device, 'Measure':Value , 'Sourcefile':filename}
print(' doc: ', doc)
es.index(index=name, doc_type='trends', body=doc)
Thanks
EDIT:
After the advice of #Saul, i could fix this problem with the following code:
import os,csv
import time
from elasticsearch import Elasticsearch
#import pandas as pd
import requests
Datum = time.strftime("%Y-%m-%d_")
path = '/home/pi/Desktop/Data'
os.chdir(path)
name = 'test'
es = Elasticsearch()
#'Time': time ,
#url = 'http://localhost:9200/index_name?pretty'
doc = {
"mappings": {
"doc": {
"properties": {
"device": { "type": "text" },
"measure": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
#headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
#r = requests.post(url, data=json.dumps(data), headers=headers)
r= es.index(index=name, doc_type='trends', body=doc)
print(r)
You need to send a HTTP Post request using python request, as follows:
url = "http://localhost:9200/index_name?pretty”
data = {
"mappings": {
"doc": {
"properties": {
"title": { "type": "text" },
"name": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(data), headers=headers)
Please replace index_name in the URL with the name of the index you are defining in to elasticsearch engine.
If you want to delete the index before creating it again, please do as follows:
url = "http://localhost:9200/index_name”
data = { }
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.delete(url, data=json.dumps(data), headers=headers)
please replace index_name in the URL with your actual index name. After deleting the index, create it again with the first code example above including the mappings that you would need. Enjoy.
Elasticsearch defines field types in the index mapping. It looks like you probably have dynamic mapping enabled, so when you send data to Elasticsearch for the first time, it makes an educated guess about the shape of your data and the field types.
Once those types are set, they are fixed for that index, and Elasticsearch will continue to interpret your data according to those types no matter what you do in your python script.
To fix this you need to either:
Define the index mapping before you load any data. This is the better option as it gives you complete control over how your data is interpreted. https://www.elastic.co/guide/en/elasticsearch/reference/6.0/mapping.html
Make sure that, the first time you send data into the index, you use the correct data types. This will rely dynamic mapping generation, but it will typically do the right thing.
Defining the index mapping is the best option. It's common to do that once off, in Kibana or with curl, or if you create a lot of indices, with a template.
However if you want to use python, you should look at the create or put_mapping functions on IndicesClient