I am using requests module to run a curl command, not sure why its prefixing ')]}\' in the front of the output, it makes r.json() fail as shown below.
When I paste the URL in a browser and execute, it downloads a .json file, and that file has same characters in the front, Am I missing some option? I need to process the output as json.
>>> r = requests.get('https://gerrit-review-server/a/changes/?q=status:open%20project:myproj/test/a_proj%20change:1510&o=CURRENT_REVISION', verify=False, auth=HTTPBasicAuth('user','pass'),
>>>
>>> r.text
')]}\'\n[{"id":"myproj%2Ftest%2Fa_proj~master~I15790ba05690e0a9984cb05bce06574645274966","project":"myproj/test/a_proj","branch":"master","hashtags":[],"change_id":"I15790ba05690e0a9984cb05bce06574645274966","subject":"Test","status":"NEW","created":"2021-01-27 19:38:57.000000000","updated":"2021-03-21 14:19:42.000000000","submit_type":"MERGE_IF_NECESSARY","mergeable":true,"insertions":1,"deletions":0,"total_comment_count":0,"unresolved_comment_count":0,"has_review_started":true,"_number":1510,"owner":{"_account_id":10008339},"current_revision":"fe3cc60cc66ad6f20c631ee818ccd91955c69d37",<..deleted..>,"description":"Rebase"}},"requirements":[]}]\n'
>>> r.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>>>
As Carcigenicate says, it seems that the json is wrongly formatted. You can try the loads, and if it fails try to correct it:
r = requests.get(some_url)
try:
data = r.json()
except json.JSONDecodeError:
# cut everything in front of the first "\n"
raw_data = r.text.split("\n", maxsplit=1)[1]
# cut everything behind the last "\n"
raw_data = raw_data.rsplit("\n", maxsplit=1)[0]
# try to load again the json
# If it fails it will raise the exception again
data = json.loads(raw_data)
Related
import json
string = "'massage':'testing'"
json.loads(string)
But I get this error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/data/data/com.termux/files/usr/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/data/data/com.termux/files/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/data/data/com.termux/files/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
That is not valid JSON. This works:
import json
string = '{"message": "testing"}'
print(json.loads(string))
Not sure what this has to do with converting lists to dictionaries though.
according to the document
If the data being deserialized is not a valid JSON document, a
JSONDecodeError will be raised.
you have 2 mistake
you miss the brackets in your string
JSON specification - RFC7159 states that a string begins and ends with quotation mark.
That mean single quoute ' has no semantic meaning in JSON and is allowed only inside a string.
result:
import json
string = '{"massage":"testing"}'
json.loads(string)
here is the full error message for when i run the code that is under this error message:
PS C:\Users\Admin> & C:/Users/Admin/AppData/Local/Programs/Python/Python39/python.exe c:/Users/Admin/Desktop/dadjoke.py
Traceback (most recent call last):
File "c:\Users\Admin\Desktop\dadjoke.py", line 9, in <module>
data = json.loads(str(question.text ))
File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
PS C:\Users\Admin>
here is my code:
import requests
import json
params = {"q":"joke"}
url = "https://icanhazdadjoke.com"
question = requests.get(url, params)
if question.status_code == 200:
data = json.loads(str(question.text ))
print(data)
I've just take a look at the API docs of the website.
Since you didn't specify your Accept header, the website will return a HTML response as default
Solution:
headers = {
"Accept": "application/json"
}
url = "https://icanhazdadjoke.com"
r = requests.get(url, headers = headers)
I'm new to python and APIs and I'm using this tutorial to learn with last-fm. I'm getting these errors:
Traceback (most recent call last):
File "/Users/vikram03/Desktop/Python_learning/fm.py", line 89, in <module>
total_pages = int(response.json()['artists']['#attr']['totalPages'])
File "/Applications/anaconda3/lib/python3.8/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/Applications/anaconda3/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/Applications/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Applications/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Here's the code in question:
#initializes reponses list
responses = []
page = 1
total_pages = 70
while page <= total_pages:
payload = {
'method': 'chart.gettopartists',
'limit': 500,
'page': page
}
#print
print("Requesting page {}/{}".format(page, total_pages))
#clear clear_output
clear_output(wait=True)
#make the API call
response = lastfm_get(payload)
#error check
if response.status_code != 200:
print(reponse.text)
break
#extract pagination info
page = int(response.json()['artists']['#attr']['page'])
total_pages = int(response.json()['artists']['#attr']['totalPages'])
# append
responses.append(response)
#if its not cached, sleep
if not getattr(response, 'from_cache', False):
time.sleep(0.25)
#increment
page += 1
This function: lastfm_get just does requests.get with the appropriate params and this url 'http://ws.audioscrobbler.com/2.0/'
From other posts I get the impression the program is getting non-JSON objects, but when I print
total_pages = int(response.json()['artists']['#attr']['totalPages']) I get the correct number.
edit:
this is the function:M
def lastfm_get(payload):
#define headers and url
headers = {'user-agent': USER_AGENT}
url = 'http://ws.audioscrobbler.com/2.0/'
#add API key and format
payload['api_key'] = API_KEY
payload['format'] = 'json'
response = requests.get(url, headers = headers, params = payload)
return response
edit2:
I added this code to check if I was getting a JSON object, and am getting a type error after about 200 pages:
#check object
try:
json_test = json.loads(lastfm_get(payload))
except TypeError:
print("not a json object")
Error:
Requesting page 256/7743
Requesting page 259/7743
Requesting page 275/7743
Requesting page 294/7743
Traceback (most recent call last):
File "/Users/vikram03/Desktop/Python_learning/fm.py", line 94, in <module>
page = int(response.json()['artists']['#attr']['page'])
File "/Applications/anaconda3/lib/python3.8/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/Applications/anaconda3/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/Applications/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Applications/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I got a type error when I tried with except ValueError:
Traceback (most recent call last):
File "/Users/vikram03/Desktop/Python_learning/fm.py", line 84, in <module>
json_test = json.loads(lastfm_get(payload))
File "/Applications/anaconda3/lib/python3.8/json/__init__.py", line 341, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not Response
EDIT3: I get this error when testing the output of lastfm_get:
TypeError: Object of type Response is not JSON serializable
Unsure how to resolve this.
I'm not sure why the page numbers are skipping either
I am creating a website using Django .
Below is my code:
views.py:
for i in range(0,len(userdata)):
json_hall = requests.get(
"https://www.onebookingsystem.com/API/Admin/booking_list_id.php?id=%s" % userdata[i]['bookid'])
r = json_hall.json()
hall_data = json.loads(json_hall.text)
id_data[i]['bookid'] = hall_data[0]['bookid']
When i run i am getting the error like this.
File "D:\KarthikWorkSpace\Projects\Python\obs_admin\Application\obs_app\views.py", line 1968, in bookings_owner
hall_data = json.loads(json_hall.text)
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Any help would be appreciated.
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
This usually means that the string you're trying to decode is empty, eg:
>>> import json
>>> json.loads('')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Please, make sure the request you are sending is correct.
You could print or inspect response.text to confirm that.
When using requests, the .json() method on a response object already gives you a Python object (decoded from the literal response)
You don't need to parse again the data with json.loads. By the way, here's a more Pythonic way to iterate over your data:
for (idx, user) in enumerate(userdata):
response = requests.get(
"https://www.onebookingsystem.com/API/Admin/booking_list_id.php?id=%s" % user['bookid'])
hall_data = response.json()
id_data[idx]['bookid'] = hall_data[0]['bookid']
I have this piece of Python code:
homedir = '/home/cloudera/'
baseurl = 'http://ergast.com/api/f1/'
apilist = ['seasons', 'results', 'qualifying', 'driverStandings', 'constructorStandings', 'drivers', 'constructors',
'circuits', 'status', 'laps', 'pitstops']
for dir in apilist:
target = homedir + str(dir)
try:
shutil.rmtree(target, ignore_errors=True)
os.makedirs(target)
except:
pass
parent_url = baseurl + "{y}/{api}.json?limit=100000"
parent_fname = homedir + "{api}/{yf}.json"
urls = [parent_url.format(y=y, api=a) for y, a in itertools.product(range(1950, 2016), apilist)]
files = [parent_fname.format(yf=y, api=a) for y, a in itertools.product(range(1950, 2016), apilist)]
for url, file in zip(urls, files):
print(url, file)
response = requests.get(url).json()
with open(file, 'w') as f:
json.dump(response, f)
When I run this, this is the output I get:
/home/cloudera/PycharmProjects/ErgastF1/ErgastF1/bin/python /home/cloudera/PycharmProjects/ErgastF1/F1Demo.py
http://ergast.com/api/f1/1950/seasons.json?limit=100000 /home/cloudera/seasons/1950.json
http://ergast.com/api/f1/1950/results.json?limit=100000 /home/cloudera/results/1950.json
http://ergast.com/api/f1/1950/qualifying.json?limit=100000 /home/cloudera/qualifying/1950.json
http://ergast.com/api/f1/1950/driverStandings.json?limit=100000 /home/cloudera/driverStandings/1950.json
http://ergast.com/api/f1/1950/constructorStandings.json?limit=100000 /home/cloudera/constructorStandings/1950.json
http://ergast.com/api/f1/1950/drivers.json?limit=100000 /home/cloudera/drivers/1950.json
http://ergast.com/api/f1/1950/constructors.json?limit=100000 /home/cloudera/constructors/1950.json
http://ergast.com/api/f1/1950/circuits.json?limit=100000 /home/cloudera/circuits/1950.json
http://ergast.com/api/f1/1950/status.json?limit=100000 /home/cloudera/status/1950.json
http://ergast.com/api/f1/1950/laps.json?limit=100000 /home/cloudera/laps/1950.json
Traceback (most recent call last):
File "/home/cloudera/PycharmProjects/ErgastF1/F1Demo.py", line 74, in <module>
response = requests.get(url).json()
File "/home/cloudera/PycharmProjects/ErgastF1/ErgastF1/lib/python3.5/site-packages/requests/models.py", line 812, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Process finished with exit code 1
Basically, it creates the JSON files for the year 1950, but the next time the loop runs for the year 1951, it exits. I'm new to Python and newer still to JSON. Can someone figure out what's going on?
Note: A less complex version of this program, with all the years from 1950 to 2015, but only with qualifying had worked. I was trying to go large scale and get all attributes together. That version merely used a range to iterate through the years and generate file names and urls.
This problem are raised because the url: http://ergast.com/api/f1/1950/laps.json?limit=100000 returns a error. Look the url:http://ergast.com/api/f1/1950/status.json?limit=100000 (Image Above)
This Image have a working json format, and the source this page is:
The url:http://ergast.com/api/f1/1950/laps.json?limit=100000 return a html error:
And the source of this url is a html table and not is a json format:
This error can be suppresed using a error treatment, but cant resolve the format. Maybe need some other argument in the url to return the corret values.
I this link: http://ergast.com/mrd/ have the documentation of this api, and in the section Overview says:
If you try to put a numeric value into a url after year, like : http://ergast.com/api/f1/1950/1/laps.json?limit=100000 you return some valid results, like this image: