We Keep Coding

Python - requests is prefixing extra characters in the output

I am using requests module to run a curl command, not sure why its prefixing ')]}\' in the front of the output, it makes r.json() fail as shown below.
When I paste the URL in a browser and execute, it downloads a .json file, and that file has same characters in the front, Am I missing some option? I need to process the output as json.
>>> r = requests.get('https://gerrit-review-server/a/changes/?q=status:open%20project:myproj/test/a_proj%20change:1510&o=CURRENT_REVISION', verify=False, auth=HTTPBasicAuth('user','pass'),
>>>
>>> r.text
')]}\'\n[{"id":"myproj%2Ftest%2Fa_proj~master~I15790ba05690e0a9984cb05bce06574645274966","project":"myproj/test/a_proj","branch":"master","hashtags":[],"change_id":"I15790ba05690e0a9984cb05bce06574645274966","subject":"Test","status":"NEW","created":"2021-01-27 19:38:57.000000000","updated":"2021-03-21 14:19:42.000000000","submit_type":"MERGE_IF_NECESSARY","mergeable":true,"insertions":1,"deletions":0,"total_comment_count":0,"unresolved_comment_count":0,"has_review_started":true,"_number":1510,"owner":{"_account_id":10008339},"current_revision":"fe3cc60cc66ad6f20c631ee818ccd91955c69d37",<..deleted..>,"description":"Rebase"}},"requirements":[]}]\n'
>>> r.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>>>

As Carcigenicate says, it seems that the json is wrongly formatted. You can try the loads, and if it fails try to correct it:
r = requests.get(some_url)
try:
data = r.json()
except json.JSONDecodeError:
# cut everything in front of the first "\n"
raw_data = r.text.split("\n", maxsplit=1)[1]
# cut everything behind the last "\n"
raw_data = raw_data.rsplit("\n", maxsplit=1)[0]
# try to load again the json
# If it fails it will raise the exception again
data = json.loads(raw_data)

How to read a JSON root array into a list?

I have a string containing a JSON array:
s = "['GY2_CAMP1', 'GY2_CAMP2', 'GY2_CAMP3', 'GY2_CAMP4', 'GY2_CAMP5']"
I tried to parse it as a JSON:
import json
l = json.loads(s)
I except to get a List, but an Exception has been raised:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/xx/anaconda3/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/home/xx/anaconda3/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/xx/anaconda3/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
I am not sure what happens here.

Use ast.literal_eval:
>>> from ast import literal_eval
>>> s = "['GY2_CAMP1', 'GY2_CAMP2', 'GY2_CAMP3', 'GY2_CAMP4', 'GY2_CAMP5']"
>>> literal_eval(s)
['GY2_CAMP1', 'GY2_CAMP2', 'GY2_CAMP3', 'GY2_CAMP4', 'GY2_CAMP5']

Json standard requires double quotes, it does not support single quotes.
That's why you get the error
Thus, the string should be
s = '["GY2_CAMP1", "GY2_CAMP2", "GY2_CAMP3", "GY2_CAMP4", "GY2_CAMP5"]'
Then you get the expected behaviour
import json
l = json.loads(s)
print(l)
>> ['GY2_CAMP1', 'GY2_CAMP2', 'GY2_CAMP3', 'GY2_CAMP4', 'GY2_CAMP5']

First replace the ' with " and then load.
import json
s = "['GY2_CAMP1', 'GY2_CAMP2', 'GY2_CAMP3', 'GY2_CAMP4', 'GY2_CAMP5']"
s.replace("'", "\"")
l = json.loads(s1)
print(l)
will get
['GY2_CAMP1', 'GY2_CAMP2', 'GY2_CAMP3', 'GY2_CAMP4', 'GY2_CAMP5']

google translate sudently stopped working i didint change the code any way

Google translate libary stopped working.
I didn't change anything about my code and it worked like an hour ago. I don't understand what is happening.
Here is my code:
from googletrans import Translator
import time
K = Translator()
def anna_sanat(x):
K = Translator()
a = K.translate(x,dest="sv",src="en").text
b = K.translate(x,dest="fi",src="en").text
return [x,a,b]
f = open("sanat.txt",mode="r",encoding="utf-8")
g = f.read()
f.close()
g = g.split(" ")
h = []
for i in g:
h += i.split("\n")
for i in h:
print(i)
print(anna_sanat(i.replace("\"","")))
time.sleep(1)
and here is the error message:
Magnus
Traceback (most recent call last):
File "C:\Users\Taavi\Desktop\Koodaus\uusin Python\ruotsi\ruotsipeli.py", line 21, in <module>
print(anna_sanat(i.replace("\"","")))
File "C:\Users\Taavi\Desktop\Koodaus\uusin Python\ruotsi\ruotsipeli.py", line 7, in anna_sanat
a = K.translate(x,dest="sv",src="en").text
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\googletrans\client.py", line 172, in translate
data = self._translate(text, dest, src)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\googletrans\client.py", line 81, in _translate
data = utils.format_json(r.text)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\googletrans\utils.py", line 62, in format_json
converted = legacy_format_json(original)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\googletrans\utils.py", line 54, in legacy_format_json
converted = json.loads(text)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Oh I know this one, I think this happens because googletrans makes requests to google translate directly, and after a while, google translate starts returning nothing, the only way that I know to get around this is to add a time.sleep(*Large number here*) and test again after that.

Can't create JSON doc from dict with string containing line feed chars

I'm creating a JSON structure which I ultimately need to save to a file but am having problems with embedded line feed characters.
I first create a dictionary:
changes = {
"20161101": "Added logging",
"20161027": "Fixed scrolling bug",
"20161024": "Added summary functionality"
}
and then convert it to a single line-feed separated string:
changes_str = '\n'.join([ "{0} - {1}".format(x, y) for x, y in changes.items() ])
print changes_str
'20161101 - Added logging\n20161027 - Fixed scrolling bug\n20161024 - Added summary functionality'
So far, so good. Now I add it into string (which in reality would come from a text template):
changes_str_json_str = '{ "version": 1.1, "changes": "' + changes_str + '" }'
print changes_str_json_str
'{ "version": 1.1, "changes": 20161101 - Added logging\n20161027 - Fixed scrolling bug\n20161024 - Added summary functionality }'
but when I come to create / encode a JSON object from this using loads, I hit problems:
json_obj = json.loads(changes_str_json_str)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/opt/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/python2.7/json/decoder.py", line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 1 column 55 (char 54)
Changing the line feed to another character does fix the problem so clearly that's where the problem lies, however, I do need the character to be a line feed as ultimately the data in the file needs to be formatted like this (the file is passed on to another system over which I have no control. Also, as far as I know, line feed is a supported character in JSON strings.
What exactly is the problem here and how can I work around it?

In JSON you need to properly escape the control characters including \n. Here's example on what's currently happening:
>>> import json
>>> json.loads('"foo\nbar"')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\python35\lib\json\__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "C:\python35\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\python35\lib\json\decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 5 (char 4)
If you properly escape the newline character with backslash it will work as expected:
>>> json.loads('"foo\\nbar"')
'foo\nbar'
So you could fix your code by doing following:
changes_str = '\\n'.join([ "{0} - {1}".format(x, y) for x, y in changes.items() ])
The better alternative would be to first construct the object you want to output and then use dumps so you wouldn't have to worry about escaping at all:
obj = {
'version': 1.1,
'changes': changes_str
}
changes_str_json_str = json.dumps(obj)

To convert it to a single line-feed separated string:
import json
changes_str = json.dumps(changes)
To load a string JSON in dict python:
dict_changes = json.loads(changes_str)

Issue in reading JSON file in python

>>> import json
>>> d2 = json.loads(open("t.json").read())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
obj, end = self._scanner.iterscan(s, **kw).next()
File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
rval, next_pos = action(m, context)
File "/usr/lib64/python2.6/json/decoder.py", line 185, in JSONObject
raise ValueError(errmsg("Expecting object", s, end))
ValueError: Expecting object: line 1 column 11 (char 11)
[ RHEL - ~/testing ]$ cat t.json
{"us": u"OFF", "val": u"5"}
Here is what I have in my JSON file and when I try to read it using open and json.load and json.loads it fails.
After using json.load
>>> import json
>>> d2 = json.load(open("t.json"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/json/__init__.py", line 267, in load
parse_constant=parse_constant, **kw)
File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
obj, end = self._scanner.iterscan(s, **kw).next()
File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
rval, next_pos = action(m, context)
File "/usr/lib64/python2.6/json/decoder.py", line 185, in JSONObject
raise ValueError(errmsg("Expecting object", s, end))
ValueError: Expecting object: line 1 column 11 (char 11)
>>>

You are using the wrong function. Use json.load() (no s!) to load data from an open file object:
d2 = json.load(open("t.json"))
The json.loads() function expects you to pass in a string, not a file object. You'd have to read your file in that case, returning the read data:
d2 = json.loads(open("t.json").read())
Next, you have invalid JSON in that file:
{"us": u"OFF", "val": u"5"}
# ^ ^
JSON is not Python; those u prefixes are not supported nor needed. You'll need to remove those from the file before it'll load.
If you have an API producing that format, it is not giving you JSON. It could be that it is producing a (strange form of) Python syntax instead; Python itself would produce {'us': u'OFF', 'val': u'5'} (single quotes). You can have Python interpret that as Python literals with ast.literal_eval():
import ast
with open('t.json') as fileobj:
d2 = ast.literal_eval(fileobj.read())
but it could be that the format is broken in other ways we cannot determine from a single isolated sample. It could be using true and false for boolean values, like in JSON, for example.
Better to have the API fixed rather that try and work around this broken-ness.

You are using the json.loads method. More documentation here. This method is used for string arguments only. Luckily, there is a similarly named json.load method documented here. This one can be used directly on a file object.
d2 = json.load(open("t.json"))

Your issue is that the JSON is not valid.
It looks like it is a python dictionnary. u'string' is a python 2 unicode string.
If you remove the u from your strings, it works fine.
>>> import json
>>> json.load(open('i.json'))
{u'val': u'5', u'us': u'OFF'}
Here is the json file:
$ cat i.json
{"us": "OFF", "val": "5"}