i'm working with python and requests. My api data looks like as shown below :
[
{
'id_list' = [{'id':1,'name' = 's1'},{'id':2,'name' = 's2'}], 'username'='oiyio'
}
{
'id_list' = [{'id':3,'name' = 's3'},{'id':4,'name' = 's4'}], 'username'='oiyio2'
}
]
I can make request as the following :
http://api.example.com/users?username=oiyio
However, i need to check key "id" and retrieve the one with id=1 for example. How to do that?
I tried below one but it did not work as i have expected.
http://api.example.com/users?id_list?id=1
Thanks in advance.
Since it is an API response, you need to modify the API code to return results as per the parameters requested by the client. If you cannot alter the API code, you have to change the code on the client side before consuming it. So, in your case:
for item in response:
for data in item.getlist("id_list"):
if data.get("id") == 1:
print data
And if you have control over the API, you can always modify the API. You need to read the documentation of the API to know what all requests it caters to.
Related
I am working on a script that will get data from a Website (Cisco Patches site) and based on the data received, I need to post it to another site (ServiceNow Event Management). The POST needs to be REST/JSON with specific keys for this to work.
I have enough code to GET the data and I have the code to POST working.
I am having a hard time with converting the data I get from GET to map it into valid JSON key value pairs to POST.
I am using the following code to get a list of new patches from Cisco website. I am getting the correct data but the format if the data is not how I can use it to post to another tool in JSON format (using different keys but values from the returned information.
This works -
def getjson(ciscourl):
response = urllib.request.urlopen(ciscourl)
ciscodata = response.read().decode("utf-8")
return json.loads(ciscodata)
The data I get back looks like below (this query resulted in 2 patches):
[{"identifier":"cisco-sa-20180521-cpusidechannel","title":"CPU Side-Channel Information Disclosure Vulnerabilities: May 2018","version":"1.5","firstPublished":"2018-05-22T01:00:00.000+0000","lastPublished":"2018-05-31T20:44:16.123+0000","workflowStatus":null,"id":1,"name":"Cisco Security Advisory","url":"https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180521-cpusidechannel","severity":"Medium","workarounds":"No","cwe":null,"cve":"CVE-2018-3639,CVE-2018-3640","ciscoBugId":"","status":"Updated","summary":"On May 21, 2018, researchers disclosed two vulnerabilities that take advantage of the implementation of speculative execution of instructions on many modern microprocessor architectures to perform side-channel information disclosure attacks. These vulnerabilities could allow an unprivileged, ","totalCount":6,"relatedResource":[]},{"identifier":"cisco-sa-20180516-firepwr-pb","title":"Cisco Firepower Threat Defense Software Policy Bypass Vulnerability","version":"1.0","firstPublished":"2018-05-16T16:00:00.000+0000","lastPublished":"2018-05-16T16:00:00.000+0000","workflowStatus":null,"id":1,"name":"Cisco Security Advisory","url":"https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180516-firepwr-pb","severity":"Medium","workarounds":"No","cwe":"CWE-693","cve":"CVE-2018-0297","ciscoBugId":"CSCvg09316","status":"New","summary":"A vulnerability in the detection engine of Cisco Firepower Threat Defense software could allow an unauthenticated, remote attacker to bypass a configured Secure Sockets Layer (SSL) Access Control (AC) policy to block SSL traffic.The vulnerability is due to the incorrect handling ","totalCount":6,"relatedResource":[]}]
I can extract values from this, as such print(jarray.get('identifier')) but I am having a hard time being able to map these values into my own JSON map with keys I define. So the value from the key identifier I got back, needs to map to a key called "node" in my JSON map.
I have tried json.loads, json.load, json.dump, json.dumps. Each time the error is Attribute Type error.
This is the code where I am confused:
def createJson(l):
#try:
jsonarray = l
o_source = "CiscoUpdatePatchChecker"
o_node = (jsonarray.get('identifier')) #this does not work
o_metric_name = ("Critical")
o_type = ("test")
o_resource = ("test_resource")
o_description = jsonarray #this works
o_event_class = ("test event class")
o_additional_info = jsonarray
print ("-" * 50)
print (o_source, o_node, o_metric_name, o_type, o_resource, o_description, o_event_class, o_additional_info)
print ("-" * 50)
data = {"source": o_source, "node": o_node, "metric_name": o_metric_name, "type": o_type, "resource": o_resource, "event_class": o_event_class, "description": o_description, "additional_info": o_additional_info}
return json.dumps(data)
# except:
#pass
Beyond this, the rest of the code just posts the data to ITSM which is working. -
def postjson(data):
# try:
url = posturl
auth = HTTPBasicAuth(username, password)
head = {'Content-type': 'application/json',
'Accept': 'application/json'}
payld = data
ret = requests.post(url, auth=auth , data=payld, headers=head)
# sys.stdout.write(ret.text)
returned_data = ret.json()
print(returned_data)
So my issue is to map data I am getting back to my keys:value pairs in a JSON map, & I will need to loop the code for as many times as the number of patches are retrieved. I am currently planning to loop in my main function for number of JSON maps that need to POST.
For now, I am just take all the data I get and mapping all the data I get to the "description" and "additional_info" field. This works and I can post the data fine.
It will help me tremendously if someone can point me to examples of how to manipulate the data I am getting from my GET request.
json.loads(ciscodata) returns an array of dictionaries.
The statement o_node = (jsonarray.get('identifier')) fails because it is trying to get a dictionary key when jsonarray is a list.
Try o_node = jsonarray[0].get('identifier')
I am not sure why you have parens around jsonarray -- those shouldn't be there.
In general, you can do something like:
for elem in jsonarray:
identifier = elem.get('identifier')
title = elem.get('title')
... etc
do_something(identifier, title, ...)
This makes it easier to debug because you can test the values you get back from the api call.
If you are always getting the same keys in your dictionary, you can skip this assignment and just do
for elem in jsonarray:
do_something(elem.get('key1'), elem.get('key2'), ...)`
Hope that helps
I'm using the pydocumentdb library to query document-db.
I get errors when trying to execute cross-partitions queries, which are... considering I use a primary key as partition-key... every query I do.
The error message mentions the parameter x-ms-documentdb-query-enablecrosspartition. Which is indeed a rest http header constant, but I don't find any way to add parameters to the headers neither in the document_client module.
Finally found it :
client_documentDB = document_client.DocumentClient(
config['ENDPOINT'],
{
'masterKey': config['MASTERKEY']
}
)
client_documentDB.default_headers['x-ms-documentdb-query-enablecrosspartition'] = True
So, I am using API calls to get some json data, and I am using a for loop to execute the call using multiple ids which will return different data based on the user's id. So basically we need to update the permissions for some of our users so the json data looks somewhat like this for the users without permissions:
{"meta":{"httpStatus":"401 - Unauthorized","error":{"errorMessage":"Insufficient
permissions.","errorCode":"ARH_8.5"},"requestId":null}}
and this for users with the permissions:
{"result":{"elements":[{"id":"dslkjafsadl","name":"Test","ownerId"
:"sdfadfsfsda","lastModified":"2016","isActive":true},
{"id":"dsafsad","name":"Test","ownerId":"sdfasdfa","lastModified":"2016","isActive":true}],"nextPage":null},"meta":{"httpStatus":"200 - OK","requestId":"3242343"}}
so my question is if I have a big response with many results both with and without permissions how could I only pull the ids from the users with permissions, and exclude the ones without the permissions.
Say the response is stored in the variable "data" and the ids that I am looking for will be stored in "requestId" this is the code I tried:
requestId = request_id = [element['requestId'] for element in data['result']['elements']]
from this code I receive the a TypeError, and it doesn't recognize 'result' because the first response is a user without the correct permissions.
Best way is to check the http status code from the original request. Say you use the requests library to perform the GET to whatever resource, status_code of the requests object will contain 401 for permission denied.
Just relying on the code you already have, you can include a conditional statement as:
requestId = request_id = [element['requestId'] for element in data['result']['elements'] if element.get('meta').get('httpStatus') == '200 - OK']
which will basically only take elements where 'meta'.'httpStatus' is the desired one.
For the requests library, take a look here: http://docs.python-requests.org/
If you have control over your JSON:
One potential solution would be to consturct your json such that there's a parent node which describes the permissions of the returned result. For example:
{"permissions": false,
{"meta":{"httpStatus":"401 - Unauthorized","error":
{"errorMessage":"Insufficient permissions.","errorCode":"ARH_8.5"},
"requestId":null}
}
}
And
{"permissions": true,
{"result":{"elements":[{"id":"dslkjafsadl","name":"Test","ownerId"
:"sdfadfsfsda","lastModified":"2016","isActive":true},{"
id":"dsafsad","name":"Test","ownerId":"sdfasdfa","la
stModified":"2016","isActive":true}],"nextPage":null},"meta":{"h
ttpStatus":"200 - OK","requestId":"3242343"}}}
With this setup, just read the first value, and then structure your json-parsing code accordingly.
If your don't have control over the JSON:
In this case, you have to figure out which request you're receiving before you can really process it. One way to do this would be to read the names of the top keys.
In the case of the correct response, you could do something like this:
parsed_json = json.loads(json_string)
# The key "result" is only in the sucess response, not in the error response
if "result" in parsed_json:
# Success
else:
# Error
When sending data through python-requests a GET request, I have a need to specifically add something at the beginning of the query string. I have tried passing the data in through dicts and json strings with no luck.
The request as it appears when produced by requests:
/apply/.../explain?%7B%22......
The request as it appears when produced by their interactive API documentation (Swagger):
/apply/.../explain?record=%7B%22....
Where the key-value pairs of my data follow the excerpt above.
Ultimately, I think the missing piece is the record= that gets produced by their documentation. It is the only piece that is different from what is produced by Requests.
At the moment I've got it set up something like this:
import requests
s = requests.Session()
s.auth = requests.auth.HTTPBasicAuth(username,password)
s.verify = certificate_path
# with data below being a dictionary of the values I need to pass.
r = s.get(url,data=data)
I am trying to include an image of the documentation below, but don't yet have enough reputation to do so:
apply/model/explain documentation
'GET' requests don't have data, that's for 'POST' and friends.
You can send the query string arguments using params kwarg instead:
>>> params = {'record': '{"'}
>>> response = requests.get('http://www.example.com/explain', params=params)
>>> response.request.url
'http://www.example.com/explain?record=%7B%22'
From the comments i felt the need to explain this.
http://example.com/sth?key=value&anotherkey=anothervalue
Let's assume you have a url like the above in order to call with python requests you only have to write
response = requests.get('http://example.com/sth', params={
'key':'value',
'anotherkey':'anothervalue'
})
Have in mind that if your value or your keys have any special character in them they will be escaped thats the reason for the %7B%2 part of url in your question.
I'm trying to figure out what's the correct URL format for the Intersango API (which is poorly documented). I'm programming my client in C#, but I'm looking at the Python example and I'm a little confused as to what is actually being placed in the body of the request:
def make_request(self,call_name,params):
params.append(('api_key',self.api_key)) // <-- How does this get serialized?
body = urllib.urlencode(params)
self.connect()
try:
self.connection.putrequest('POST','/api/authenticated/v'+self.version+'/'+call_name+'.php')
self.connection.putheader('Connection','Keep-Alive')
self.connection.putheader('Keep-Alive','30')
self.connection.putheader('Content-type','application/x-www-form-urlencoded')
self.connection.putheader('Content-length',len(body))
self.connection.endheaders()
self.connection.send(body)
response = self.connection.getresponse()
return json.load(response)
//...
I can't figure out this piece of code: params.append(('api_key',self.api_key))
Is it some kind of a dictionary, something that gets serialized to JSON, comma delimited, or exactly how does it get serialized? What would the body look like when the parameters are encoded and assigned to it?
P.S. I don't have anything that I can run the code with so I can debug it, but I'm just hoping that this is simple enough to understand for somebody that knows Python and they would be able to tell me what's happening on that line of code.
params is a list of 2-element lists. The list would look like ((key1, value1), (key2, value2), ...)
params.append(('api_key',self.api_key)) adds another 2-element list to the existing params list.
Finally, urllib.urlencode takes this list and converts it into a propert urlencoded string. In this case, it will return a string key1=value1&key2=value2&api_key=23423. If there are any special characters in your keys or values, urlencode will %encode them. See documentation for urlencode
I tried to get the C# code working, and it kept failing with exception {"The remote server returned an error: (417) Expectation Failed."}. I finally found what the problem is. You could read about it in depth here
In short, the way to make C# access Intersango API is to add following code:
System.Net.ServicePointManager.Expect100Continue = false;
This code needs to only run once. This is a global setting, so it affects your full application, so beware that something else could break as a result.
Here's a sample code:
System.Net.ServicePointManager.Expect100Continue = false;
var address = "https://intersango.com/api/authenticated/v0.1/listAccounts.php";
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
var postBytes = Encoding.UTF8.GetBytes("api_key=aa75***************fd65785");
request.ContentLength = postBytes.Length;
var dataStream = request.GetRequestStream();
dataStream.Write(postBytes, 0, postBytes.Length);
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Piece of cake
instead of params.append(('api_key',self.api_key))
just write:
params['api_key']=self.api_key