I am trying to convert a raw curl command to use Python Request Module with no luck. This is a simple request to query JBoss Mgmt interface but it does not parse my JSON correctly.
16:34:26,868 DEBUG [org.jboss.as.domain.http.api] (HttpManagementService-threads - 15) Unable to construct ModelNode 'Invalid character: o'
Python version
Python 2.7.6
Working raw cURL command:
/usr/bin/curl --digest -v -L -D - 'http://brenn:!12rori#localhost:9990/management' --header Content-Type:application/json '-d {"operation":"read-attribute","name":"server-state","json.pretty":1}'
In python code i read in my REST/cURL payload like so
import requests
----
def readconfigfile():
with open('jboss_modification.cfg') as f:
lines = f.readlines()
return lines
The config file looks like so
{"operation":"read-attribute","name":"server-state","json.pretty":1}
I convert the str format from readconfigfile() to a dictionary as follows
def converttodictionary(incominglines):
commands = []
for lin in incominglines:
#dumps = json.dumps(lin)
obj = json.loads(lin)
commands.append(obj)
return commands
The python code to execute this request is as follows
def applyconfig(lines):
url="http://localhost:9990/management"
auth=HTTPBasicAuth('brenn', '!12rori')
s = requests.Session()
re=s.get(url, auth=HTTPDigestAuth('brenn', '!12rori')) ##200 RESP
s.headers.update({'Content-Type': 'application/json'})
for line in lines:
payload=line
r=s.post(url,payload)
print(r.text)
Any help much appreciated?
Note: This question has been updated a few times as I resolved other issues....
The issues was...
Initial JSON requestfailed because when i read it from file python interpreted as a str.
Converted to dictionary using json.loads and server accepted request but could not parse the JSON with illegal character error
Converted this json back to a str using json.dumps -- which to my mind looks like what i was trying to do in the first place -- and this now works
read JSON file as per def readconfigfile(): above
convert to json/dictionary as per def converttodictionary above: json.loads(lin)
Convert this json "back" to a string using json.dumps and POST as follows
payload = json.dumps(command)
r = session.post(url, payload,auth=HTTPDigestAuth('brenn', '!12rori')
)
Related
I am trying to get value of id i.e. "id": 59 which is in the curl output in form of json.
Below is the curl output in json:
[{"id":59,"description":"This is a demo project","name":"Demo_Project","name_with_namespace":"sam / Demo_Project","path":"demo_project","path_with_namespace":"sam/demo_project","created_at":"2020-03-02T08:43:13.664Z","default_branch":"master","tag_list":[],"ssh_url_to_repo":"ssh://git#od-test.od.com:2222/sam/demo_project.git","http_url_to_repo":"https://od-test.od.com/gitlab/sam/demo_project.git","web_url":"https://od-test.od.com/gitlab/sam/demo_project","readme_url":"https://od-test.od.com/gitlab/sam/demo_project/blob/master/README.md","avatar_url":null,"star_count":0,"forks_count":0,"last_activity_at":"2020-04-09T09:28:09.860Z","namespace":{"id":2259,"name":"sam","path":"sam","kind":"user","full_path":"sam","parent_id":null,"avatar_url":"https://secure.gravatar.com/avatar/755db8ssqaq50dcc9d189c53523b?s=80\u0026d=identicon","web_url":"https://od-test.od.com/gitlab/sam"}}]
I am using python to parse the json and get the value of id.
I have tried the following command to do the same but got an error.
curl --header "PRIVATE-TOKEN: 9999ayayayar66" "https://od-test.od.com/gitlab/api/v4/search?scope=projects&search=demo_project" | python -c 'import sys, json; print(json.load(sys.stdin)["id"])'
Error:
Can anyone help me the correct python command to get the value of id.
Thanks in advance.
The JSON contains an array of objects but you are treating it like it is a single object:
import sys, json; print(json.load(sys.stdin)["id"])
You're basically saying "here is a collection of objects, give me the ID of the object". It doesn't make sense.
If you assume you only ever want the ID of the first object in the array, you can use this:
import sys, json; print(json.load(sys.stdin)[0]["id"])
The root of you JSON object is actually an array, so you should get the first item out of it first:
curl --header "PRIVATE-TOKEN: 9999ayayayar66" "https://od-test.od.com/gitlab/api/v4/search?scope=projects&search=demo_project" | python -c 'import sys, json; print(json.load(sys.stdin)[0]["id"])'
You have to write print(json.load(sys.stdin)[0]["id"]) because json response is a list of dictionaries, not a dictionary.
Since you also tagged Python Requests, your CURL command could be translated to:
import requests
headers = {
'PRIVATE-TOKEN': '9999ayayayar66',
}
params = (
('scope', 'projects'),
('search', 'demo_project'),
)
response = requests.get('https://od-test.od.com/gitlab/api/v4/search', headers=headers, params=params)
Then you can get the id value from the JSON Response Content with response.json()[0]["id"]. This uses the built-in JSON decoder to convert the JSON response to a list of dictionaries.
I used python2 to make a request to RNAcentral database, and I read the response as JSON format by the use of this command: response.json().
This let me read the data as a dictionary data type, so I used the corresponding syntax to obtain the data from cross references, which contained some links to other databases, but when I try to make the request for each link using the command mentioned above, I can't read it as JSON, because I can only obtain the response content as HTML.
So I need to know how to read make a request to each link from cross references and read it as JSON using python language.
Here is the code:
direcc = 'http://rnacentral.org/api/v1/rna/'+code+'/?flat=true.json'
resp = requests.get(direcc)
datos=resp.json()
d={}
links = []
for diccionario in datos['xrefs']['results']:
if diccionario['taxid']==9606:
base_datos=diccionario['database']
for llave,valor in diccionario['accession'].iteritems():
d[base_datos]={'url':diccionario['accession']['url'],
'expert_db_url':diccionario['accession']['expert_db_url'],
'source_url':diccionario['accession']['source_url']}
for key,value in d.iteritems():
links.append(d[key]['expert_db_url'])
for item in links:
response = requests.get(item)
r = response.json()
And this is the error I get: ValueError: No JSON object could be decoded.
Thank you.
The following code in utils.py
manager = PoolManager()
data = json.dumps(dict) #takes in a python dictionary of json
manager.request("POST", "https://myurlthattakesjson", data)
Gives me ValueError: need more than 1 value to unpack when the server is run. Does this most likely mean that the JSON is incorrect or something else?
Your Json data needs to be URLencoded for it to be POST (or GET) safe.
# import parser
import urllib.parse
manager = PoolManager()
# stringify your data
data = json.dumps(dict) #takes in a python dictionary of json
# base64 encode your data string
encdata = urllib.parse.urlencode(data)
manager.request("POST", "https://myurlthattakesjson", encdata)
I believe in python3 they made some changes that the data needs to be binary. See unable to Post data to a login form using urllib python v3.2.1
I don't know a single thing of perl but from a big perl script, I managed to get the relevant parts and make a HTTP request. So, this perl code works perfectly.
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
use HTTP::Request::Common;
my $ua = new LWP::UserAgent;
my $request = "X-CTCH-PVer: 0000001\r\n";
my $method_url = "http://localhost:8088/ctasd/GetStatus";
my $response = $ua->request (POST $method_url,Content => $request);
my $data = $response->status_line . "\n";
print $data;
print $response->content;
The above code outputs:
200 OK
X-CTCH-PVer: 0000001
From my understanding, it's doing a POST to a URL with the specified data. With that base, my python code looks like:
#!/usr/bin/python
import urllib
url = "http://localhost:8088/ctasd/GetStatus"
data = urllib.urlencode([("X-CTCH-PVer", "0000001")])
print urllib.urlopen(url, data).read()
But, this returns response as:
X-CTCH-Error: Missing protocol header "X-CTCH-PVer"
Please help me in making a Python equivalent of perl code.
So, the actual thing was, the $request in Perl was literally been sent as POST data without any change. Now I get why is the name content in Perl.
#!/usr/bin/python
import urllib
url = "http://localhost:8088/ctasd/GetStatus"
print urllib.urlopen(url, "X-CTCH-PVer: 0000001").read()
Worked. I actually found out about this after capturing the traffic in both cases and analysing it in wireshark.
The error is because you are not sending the header, you are making/sending a urlencoded string, hence the function urllib.urlencode
Try setting the request with actual headers:
#!/usr/bin/python
import urllib2
request = urllib2.Request("http://localhost:8088/ctasd/GetStatus", headers={"X-CTCH-PVer" : "0000001"})
contents = urllib2.urlopen(request).read()
I have a file, and I need to send its contents to a function. What I am currently doing is as follows:
$ curl --data-urlencode "$( cat my_file.xml)" MyIP
# in function
def receiver(request):
print request.POST
This returns something like:
<QueryDict: {u'\ufeff<?xml version ': [u' "1.0" encoding = "UTF-8"?>\n<Orders>\n ...</Orders>']}>
In other words, the curl command is correctly sending the data to the django function; however, it is not easily retrievable since the POST data is split up between the request.POST key (`u'\ufeff
In addition the xml file is not properly url-encoded, so I don't think I can use curl -d #file.
It seems to be taking '\ufeff<?xml version ' as the QueryDict key. How would I pass the file contents here and retrieve it as a single string so I can parse the xml?
There is a very easy (hack) solution for this, use string concatenation on the request.POST dict to re-combine the information:
contents = request.POST.keys()[0] + "=" + request.POST.values()[0]