So I have been trying to use base64 to decode a value and then be able to use those decoded strings to print out each for themselves.
Basically my decoded base64 is:
{
"trailerColor": "FF0017",
"complete": 59,
"bounds": [
25,
65,
62,
5
],
"Stamina": 0,
"cardId": "d4fc5458-3481-4ce6-be32-acd03c2cfd16",
}
Im using this code which gets the metadata that I wish and then convert it into a UTF-8 basically with the code below.
resp = requests.get(url, headers=headers, json=json, timeout=6)
getmetadata = resp.json()['objects'][1]['metadata']
newdata = base64.b64decode(getmetadata).decode('UTF-8')
print(newdata)
However usually if I did newdata['trailerColor'] it should be able to print out only trailerColor if I do that but what i'm getting for error is:
TypeError: string indices must be integers
How can I solve this by printing whatever I want out through that json?
EDIT:
Process Process-1:
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 249, in _bootstrap
self.run()
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\TEST.py", line 194, in script
print(newdata['complete'])
TypeError: string indices must be integers
resp = requests.get(url, headers=headers, json=json, timeout=6)
getmetadata = resp.json()['objects'][1]['metadata']
newdata = base64.b64decode(getmetadata).decode('UTF-8')
data = json.loads(newdata)
print(data['complete'])
base64.b64decode(str).decode(str) returns a string str. If that string should be json then you can use json.loads to transform the string str to a dict where you can get a single value by it's key.
Related
this is my code:
prettyPicture(clf, features_test, labels_test)
output_image("F:/test.png", "png", open("F:/test.png", "rb").read())
def output_image(name, format, bytes):
image_start = "BEGIN_IMAGE_f9825uweof8jw9fj4r8"
image_end = "END_IMAGE_0238jfw08fjsiufhw8frs"
data = {}
data['name'] = name
data['format'] = format
data['bytes'] = base64.encodestring(bytes)
print(image_start + json.dumps(data) + image_end)
this errors is:
Traceback (most recent call last):
File "studentMain.py", line 41, in <module>
output_image("F:/test.png", "png", open("F:/test.png", "rb").read())
File "F:\Demo\class_vis.py", line 69, in output_image
print(image_start + json.dumps(data) + image_end)
File "C:\Users\Tony\AppData\Local\Programs\Python\Python36-
32\lib\json\__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "C:\Users\Tony\AppData\Local\Programs\Python\Python36-
32\lib\json\encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "C:\Users\Tony\AppData\Local\Programs\Python\Python36-
32\lib\json\encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "C:\Users\Tony\AppData\Local\Programs\Python\Python36-
32\lib\json\encoder.py", line 180, in default
o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable
The issue here is that base64.encodestring() returns a bytes object, not a string.
Try:
data['bytes'] = base64.encodestring(bytes).decode('ascii')
Check out this question and answer for a good explanation of why this is:
Why does base64.b64encode() return a bytes object?
Also see: How to encode bytes in JSON? json.dumps() throwing a TypeError
You're only missing one aspect here: When you use .encodestring, you have a bytes object as result, and bytes are not json serializable in python 3.
You can solve it just encoding your data["bytes"]:
data['bytes'] = base64.encodestring(bytes).decode("utf-8")
I'm assuming you'll always receive a bytes object at the "bytes" variable, otherwise you should add a checker for the type of the object, and not encoding when it's already a string.
I am trying to bruteforce a session via sending random cookies until the correct cookie gives me an admin session. I am using python 3.6 on Windows 10.
The cookie I want to use is PHPSESSID and I have set it to a hex encoded string consisting of "#-admin". The website gives a random PHPSESSID that is hex encoded, but only the number changes ('-admin' is consistent after every refresh). The source code maxes out the number to 640 hence the range.
The code is below:
for x in range(1,641):
if x % 10 == 0:
print (str(x) + ' Sessions Tested')
cookies = dict(PHPSESSID=(binascii.hexlify(str(x).encode('ascii')+b'-admin')))
r = requests.get(target, cookies=cookies)
if r.text.find(trueStr) != -1:
print ('Got it!')
I receive the following error after running the script on windows:
Traceback (most recent call last):
File "natas19.py", line 14, in <module>
r = requests.get(target, cookies=cookies)
File "C:\Users\e403sa\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\e403sa\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\e403sa\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\sessions.py", line 494, in request
prep = self.prepare_request(req)
File "C:\Users\e403sa\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\sessions.py", line 415, in prepare_request
cookies = cookiejar_from_dict(cookies)
File "C:\Users\e403sa\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\cookies.py", line 518, in cookiejar_from_dict
cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
File "C:\Users\e403sa\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\cookies.py", line 345, in set_cookie
if hasattr(cookie.value, 'startswith') and cookie.value.startswith('"') and cookie.value.endswith('"'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
I have no idea where to start. I followed the documentation for python requests. Any suggestions on where to look would be greatly appreciated.
Cookie values must be str objects, but binascii.hexlify() returns a bytes object:
>>> import binascii
>>> x = 1
>>> binascii.hexlify(str(x).encode('ascii')+b'-admin')
b'312d61646d696e'
Decode that first:
cookies = {
'PHPSESSID': binascii.hexlify(b'%d-admin' % x).decode('ascii')
}
In your example, cookies is a dict set by:
dict(PHPSESSID=(binascii.hexlify(str(x).encode('ascii') + b'-admin')))
If you break up the steps of that one-liner, you'll see the problem:
>>> binascii.hexlify(str(x).encode('ascii') + b'-admin')
b'312d61646d696e'
>>> b'312d61646d696e'.startswith('3')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
You're performing a bytes operation with a str first arg. Since it's the requests package managing your cookies, convert the value to a str before setting PHPSESSID.
for x in range(1,641):
if x % 10 == 0:
print (str(x) + ' Sessions Tested')
b_sess_id = binascii.hexlify(str(x).encode('ascii')+b'-admin'))
cookies = dict(PHPSESSID=b_sess_id.decode())
r = requests.get(target, cookies=cookies)
if r.text.find(trueStr) != -1:
print ('Got it!')
# This code searches the query from googlecustomsearch api and returns data in json format
import pprint
import json
from googleapiclient.discovery import build
my_api_key = "**************************************"
my_cse_id = "*************************************"
def google_search(search_term, api_key, cse_id, **kwargs):
service = build("customsearch", "v1", developerKey=api_key)
res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
return res['items']
results = google_search(
'Roshan Patel', my_api_key, my_cse_id, num=10)
for result in results:
pprint.pprint(result)
result_dict = json.loads(result)
print result_dict['formattedUrl']
This is the output i am getting:
I want to get only the url parts for eg :u'formattedUrl' and store it in a list , how to do it?
New error
Traceback (most recent call last):
File "<ipython-input-38-eb898c8de239>", line 1, in <module>
runfile('C:/Users/abc/untitled9.py', wdir='C:/Users/abc')
File "C:\Users\abc\Anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 880, in runfile
execfile(filename, namespace)
File "C:\Users\abc\Anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 87, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)
File "C:/Users/abc/untitled9.py", line 28, in <module>
result_dict = json.loads(result)
File "C:\Users\abc\Anaconda2\lib\json\__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\Users\abc\Anaconda2\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
Do you need first convert the results to a Python object, like a dict, to do that uses the json module:
result_dict = json.loads(result)
Now you can filter your dict:
result_dict['formattedUrl']
result is not JSON, but an actual Python dictionary, so just:
print result[u'formattedUrl']
>>> import json
>>> d2 = json.loads(open("t.json").read())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
obj, end = self._scanner.iterscan(s, **kw).next()
File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
rval, next_pos = action(m, context)
File "/usr/lib64/python2.6/json/decoder.py", line 185, in JSONObject
raise ValueError(errmsg("Expecting object", s, end))
ValueError: Expecting object: line 1 column 11 (char 11)
[ RHEL - ~/testing ]$ cat t.json
{"us": u"OFF", "val": u"5"}
Here is what I have in my JSON file and when I try to read it using open and json.load and json.loads it fails.
After using json.load
>>> import json
>>> d2 = json.load(open("t.json"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/json/__init__.py", line 267, in load
parse_constant=parse_constant, **kw)
File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
obj, end = self._scanner.iterscan(s, **kw).next()
File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
rval, next_pos = action(m, context)
File "/usr/lib64/python2.6/json/decoder.py", line 185, in JSONObject
raise ValueError(errmsg("Expecting object", s, end))
ValueError: Expecting object: line 1 column 11 (char 11)
>>>
You are using the wrong function. Use json.load() (no s!) to load data from an open file object:
d2 = json.load(open("t.json"))
The json.loads() function expects you to pass in a string, not a file object. You'd have to read your file in that case, returning the read data:
d2 = json.loads(open("t.json").read())
Next, you have invalid JSON in that file:
{"us": u"OFF", "val": u"5"}
# ^ ^
JSON is not Python; those u prefixes are not supported nor needed. You'll need to remove those from the file before it'll load.
If you have an API producing that format, it is not giving you JSON. It could be that it is producing a (strange form of) Python syntax instead; Python itself would produce {'us': u'OFF', 'val': u'5'} (single quotes). You can have Python interpret that as Python literals with ast.literal_eval():
import ast
with open('t.json') as fileobj:
d2 = ast.literal_eval(fileobj.read())
but it could be that the format is broken in other ways we cannot determine from a single isolated sample. It could be using true and false for boolean values, like in JSON, for example.
Better to have the API fixed rather that try and work around this broken-ness.
You are using the json.loads method. More documentation here. This method is used for string arguments only. Luckily, there is a similarly named json.load method documented here. This one can be used directly on a file object.
d2 = json.load(open("t.json"))
Your issue is that the JSON is not valid.
It looks like it is a python dictionnary. u'string' is a python 2 unicode string.
If you remove the u from your strings, it works fine.
>>> import json
>>> json.load(open('i.json'))
{u'val': u'5', u'us': u'OFF'}
Here is the json file:
$ cat i.json
{"us": "OFF", "val": "5"}
I am trying to query DBpedia using SPARQLWrapper in Python (v3.3). This is my query:
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?slot WHERE {
<http://dbpedia.org/resource/Week> <http://www.w3.org/2002/07/owl#sameAs> ?slot
}
It results in an error from the SPARQLWrapper package:
ValueError: Invalid \escape: line 118 column 74 (char 11126)
Code:
query = "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> SELECT ?slot WHERE{{ {subject} {predicate} {object} }} "
query = query.format(subject=subject, predicate=predicate, object= objectfield)
self.sparql.setQuery(query)
self.sparql.setReturnFormat(JSON)
results = self.sparql.query().convert() # Error thrown at this line
Error :
Traceback (most recent call last):
File "getUriLiteralAgainstPredicate.py", line 84, in <module>
sys.exit(main())
File "getUriLiteralAgainstPredicate.py", line 61, in main
entity,predicateURI,result = p.getObject(dataAtURI,predicates, each["entity"])
File "getUriLiteralAgainstPredicate.py", line 30, in getObject
result = self.run_sparql("<"+subjectURI+">","<"+predicateURI+">","?slot")
File "getUriLiteralAgainstPredicate.py", line 24, in run_sparql
results = self.sparql.query().convert()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/SPARQLWrapper-1.5.2-py3.3.egg/SPARQLWrapper/Wrapper.py", line 539, in convert
return self._convertJSON()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/SPARQLWrapper-1.5.2-py3.3.egg/SPARQLWrapper/Wrapper.py", line 476, in _convertJSON
return jsonlayer.decode(self.response.read().decode("utf-8"))
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/SPARQLWrapper-1.5.2-py3.3.egg/SPARQLWrapper/jsonlayer.py", line 76, in decode
return _decode(string)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/SPARQLWrapper-1.5.2-py3.3.egg/SPARQLWrapper/jsonlayer.py", line 147, in <lambda>
_decode = lambda string, loads=json.loads: loads(string)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/decoder.py", line 352, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/decoder.py", line 368, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Invalid \escape: line 118 column 74 (char 11126)
The problem is, that dbpedia output has this line:
{ "slot": { "type": "uri", "value": "http://got.dbpedia.org/resource/\U00010345\U00010339\U0001033A\U00010349" }},
Notice literals which start with \U (capital U). This is not valid JSON and python doesn't know how to handle it. So, problem is on DBPedia side and it can't be handled on SPARQLWrapper side.
But… You can handle it yourself like this:
results = self.sparql.query()
body = results.response.read()
fixed_body = body.decode("unicode_escape")
from SPARQLWrapper.Wrapper import jsonlayer
results = jsonlayer.decode(fixed_body)
try python-cjson
so the above thing can also be tried as below
import cjson
results = self.sparql.query()
body = results.response.read()
results = cjson.decode(body)