Convert google search results into json in python 3.1 - python

I am writing a Python program that feeds a search term to google using the google search API and downloads the first 10 results. I was able to do this in Python 2.6 as follows:
query = urllib.parse.urlencode({'q' : 'searchterm','start' : k},doseq=false)
url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&%s' \
% (query)
results = urllib.urlopen(url)
resultsjson = json.loads(results.read())
betterResults += resultsjson["responseData"]["results"]
Google's search API returns the results as a json, so I used the above code to download the results into a json of my and parse them into a list (betterResults).
When I switched over to Python 3, my program began throwing exceptions. Apparently, in Python 2.6 the object returned by urlopen() is a file-like object that can be loaded into a json. In Python 3.1, the object returned is an HTTPResponse object, which does contain a read() method, as required by the json specifications, but is a byte object. I was therefore unable to access the information as I had in 2.6.
Is there any way to access the json returned by google? How can I get the results in Python 3 and be able to select which fields I want, as I was able to do with the json?
Thank you very much,
bsg


You'll need to decode the byte object if you want to use it with json.loads
resultjson = json.loads(results.read().decode())
docs also suggest to pass encoding parameter to the loads function:
json.loads(results.read(), encoding=<encoding-type>)
I think Lennart has an explanation how to get the encoding-type.


The object returned by urlopen is file like, you are wrong there. But you use json.loads(), which expects a string. json.load() expects a file like object.
However, json.load() expects the result of the read() method to be a string, while of course the read you get will be bytes, so you need to decode it from bytes to a string first.
So, something like this:
query = urllib.parse.urlencode({'q' : 'searchterm','start' : k},doseq=false)
url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&%s' \
% (query)
results = urllib.urlopen(url)
encoding = input.getheader('content-type').split('=')[-1]
resultsjson = json.loads(results.read().decode(encoding))
betterResults += resultsjson["responseData"]["results"]
Might work. (I didn't test it).

Related

Is there a way to search for a string and copy text in front until it reaches a comma?

I am new to python and wanted to store the recentAveragePrice inside a variable (from a string like this one)
{"assetStock":null,"sales":250694,"numberRemaining":null,"recentAveragePrice":731,"originalPrice":null,"priceDataPoints":[{"value":661,"date":"2022-08-11T05:00:00Z"},{"value":592,"date":"2022-08-10T05:00:00Z"},{"value":443,"date":"2022-08-09T05:00:00Z"}],"volumeDataPoints":[{"value":155,"date":"2022-08-11T05:00:00Z"},{"value":4595,"date":"2022-08-10T05:00:00Z"},{"value":12675,"date":"2022-08-09T05:00:00Z"},{"value":22179,"date":"2022-08-08T05:00:00Z"},{"value":15181,"date":"2022-08-07T05:00:00Z"},{"value":14541,"date":"2022-08-06T05:00:00Z"},{"value":15310,"date":"2022-08-05T05:00:00Z"},{"value":14146,"date":"2022-08-04T05:00:00Z"},{"value":13083,"date":"2022-08-03T05:00:00Z"},{"value":14460,"date":"2022-08-02T05:00:00Z"},{"value":16809,"date":"2022-08-01T05:00:00Z"},{"value":17571,"date":"2022-07-31T05:00:00Z"},{"value":23907,"date":"2022-07-30T05:00:00Z"},{"value":39007,"date":"2022-07-29T05:00:00Z"},{"value":38823,"date":"2022-07-28T05:00:00Z"}]}
My current solution is this:
var = sampleStr[78] + sampleStr[79] + sampleStr[80]
It works for the current string but if the recentAveragePrice was above 999 it would stop working and i was wondering if instead of getting a fixed number i could search for it inside the string.

Your replit code shows that you're acquiring JSON data from some website. Here's an example based on the URL that you're using. It shows how you check the response status, acquire the JSON data as a Python dictionary then print a value associated with a particular key. If the key is missing, it will print None:
import requests
(r := requests.get('https://economy.roblox.com/v1/assets/10159617728/resale-data')).raise_for_status()
jdata = r.json()
print(jdata.get('recentAveragePrice'))
Output:
640

Since this is json you should just be able to parse it and access recentAveragePrice:
import json
sample_string = '''{"assetStock":null,"sales":250694,"numberRemaining":null,"recentAveragePrice":731,"originalPrice":null,"priceDataPoints":[{"value":661,"date":"2022-08-11T05:00:00Z"},{"value":592,"date":"2022-08-10T05:00:00Z"},{"value":443,"date":"2022-08-09T05:00:00Z"}],"volumeDataPoints":[{"value":155,"date":"2022-08-11T05:00:00Z"},{"value":4595,"date":"2022-08-10T05:00:00Z"},{"value":12675,"date":"2022-08-09T05:00:00Z"},{"value":22179,"date":"2022-08-08T05:00:00Z"},{"value":15181,"date":"2022-08-07T05:00:00Z"},{"value":14541,"date":"2022-08-06T05:00:00Z"},{"value":15310,"date":"2022-08-05T05:00:00Z"},{"value":14146,"date":"2022-08-04T05:00:00Z"},{"value":13083,"date":"2022-08-03T05:00:00Z"},{"value":14460,"date":"2022-08-02T05:00:00Z"},{"value":16809,"date":"2022-08-01T05:00:00Z"},{"value":17571,"date":"2022-07-31T05:00:00Z"},{"value":23907,"date":"2022-07-30T05:00:00Z"},{"value":39007,"date":"2022-07-29T05:00:00Z"},{"value":38823,"date":"2022-07-28T05:00:00Z"}]}'''
data = json.loads(sample_string)
recent_price = data['recentAveragePrice']
print(recent_price)
outputs:
731

Your data is in a popular format called JSON (JavaScript Object Notation). It's commonly used to exchange data between different systems like a server and a client, or a Python program and JavaScript program.
Now Python doesn't use JSON per-se, but it has a data type called a dictionary that behaves very similarly to JSON. You can access elements of a dictionary as simply as:
print(my_dictionary["recentAveragePrice"])
Python has a built-in library meant specifically to handle JSON data, and it includes a function called loads() that can convert a string into a Python dictionary. We'll use that.
Finally, putting all that together, here is a more robust program to help parse your string and pick out the data you need. Dictionaries can do a lot more cool stuff, so make sure you take a look at the links above.
# import the JSON library
# specifically, we import the `loads()` function, which will convert a JSON string into a Python object
from json import loads
# let's store your string in a variable
original_string = """
{"assetStock":null,"sales":250694,"numberRemaining":null,"recentAveragePrice":731,"originalPrice":null,"priceDataPoints":[{"value":661,"date":"2022-08-11T05:00:00Z"},{"value":592,"date":"2022-08-10T05:00:00Z"},{"value":443,"date":"2022-08-09T05:00:00Z"}],"volumeDataPoints":[{"value":155,"date":"2022-08-11T05:00:00Z"},{"value":4595,"date":"2022-08-10T05:00:00Z"},{"value":12675,"date":"2022-08-09T05:00:00Z"},{"value":22179,"date":"2022-08-08T05:00:00Z"},{"value":15181,"date":"2022-08-07T05:00:00Z"},{"value":14541,"date":"2022-08-06T05:00:00Z"},{"value":15310,"date":"2022-08-05T05:00:00Z"},{"value":14146,"date":"2022-08-04T05:00:00Z"},{"value":13083,"date":"2022-08-03T05:00:00Z"},{"value":14460,"date":"2022-08-02T05:00:00Z"},{"value":16809,"date":"2022-08-01T05:00:00Z"},{"value":17571,"date":"2022-07-31T05:00:00Z"},{"value":23907,"date":"2022-07-30T05:00:00Z"},{"value":39007,"date":"2022-07-29T05:00:00Z"},{"value":38823,"date":"2022-07-28T05:00:00Z"}]}
"""
# convert the string into a dictionary object
dictionary_object = loads(original_string)
# access the element you need
print(dictionary_object["recentAveragePrice"])
Output upon running this program:
$ python exp.py
731

Overflow error when reading json file

I am trying to read a json which includes a number of tweets, but I get the following error.
OverflowError: int too large to convert
The script filters multiple json files to get specific tweets, and it crashes when reaching to a specific json.
The line that creates the error is this one :
df_temp = pd.read_json(path_or_buf=json_path, lines=True)
Here is the error in the cmd

Just store the user id as a String, and treat it like it is one (this is actually what you should do when dealing with this kind of ids). If you can't change the json input format, you can always parse it like a string before parsing it like a json object, and add the quotes to the id code, using for instance regexes: Regex in python.
I don't know with which library you are parsing the json, but maybe also implicit casting will work: either try the "getString" method on the number instead of the "getInt" method, or force python to treat the object like a string, with something like x = "" + json.getId()
Python is pretty loose on typing and may let you do it.

Decode text response from API in Python 3.6

I am trying to extract data from mailchimp export api, which returns responses based on the following specifications:
Returns:
Parameter - text
Description:
a plain text dump of JSON objects. The first row is a header row. Each additional row returned is an individual JSON object. Rows are delimited using a newline (\n) marker, so implementations can read in a single line at a time, handle it, and move on.
To get the data I am using:
response = requests.get(urldetails).text
If I use .json() it errors out with a JSON decode error. The output of the above is something along the lines of:
{data..}
{data...}
I am unsure whether each dict is on a separate row, however I am under the impression it's actually just one continuous string as many of my attempts to decode it ended up with an error 'str' object cannot be...etc. . I don't see the '\n' separators anywhere when I am using the .text method.
What's the best way of going about and make each dict a separate item in a list or a row in a dataframe (which I can unpack later).
Thanks

You can get all the data from the MailChimp export api using a simple approach. Please note that I am using f-strings, only available in Python 3.6+.
import requests
import json
apikey = '<your-api-key>'
id = "<list-id>"
URL = f"https://us10.api.mailchimp.com/export/1.0/campaignSubscriberActivity/?apikey={apikey}&id={id}"
json_data = [json.loads(s) for s in requests.get(URL).text.strip().split("\n")]
print(json_data[0]['<some-subscriber-email>'][0]['action'])

Provided that the text response isn't insanely badly formed json, you can use the json library. In particular, the loads() function.
import json
json_response = json.loads(response)
loads() loads JSON into a python dict from a string.
EDIT:
The Mailchimp API states that each JSON object is separated by a newline character. We can create a list of dicts with the following code:
# get response from GET request and load as a string
import json
json_resp = [json.loads(line) for line in response.split('\n')]

Flask - Get_JSON changing delimiter

I post the following JSON to my flask server:
'{"on":false}'
I then use the following line of code to return the JSON to HTTP PUT it onto another device on the network:
content = ("'" + str(request.get_json()) + "'").lower()
However, instead of returning the expected:
'{"on":false}'
It returns:
'{'on':false}'
Thus meaning the JSON is invalid and does not work. Is there something I can change in the request.get_json() or is there a different method?

You are doing some strange things here.
get_json() automatically parses the incoming JSON string into a Python data structure. You then call str on it, converting it back not into JSON but into a representation of the Python structure.
Now, you could call json.dumps instead of str, but it would be better to avoid converting it from JSON in the first place. Instead of using request.get_json, use request.get_data; now Flask won't parse the content from JSON, and your quotes will be preserved.

What is the difference between json.JSONDecoder().decode() and json.loads()

I am using urllib2 to grab the html of a url and then a regex to extract a JSON that I need from there. I want to get the usual "dictionary of dictionaries" Python object and both of the following work:
my_json #a correctly formatted json string
json_dict1 = json.JSONDecoder().decode(my_json)
json_dict2 = json.loads(my_json)
What is the difference and which is better in what circumstances (besides mine, but that one in particular)?

json.loads() essentially creates a json.JSONDecoder() instance and calls decode on it. As such your first line is exactly the same thing as the second line. See the json.loads() source code.
The module offers you flexibility; a simple function API or a full OO API that you can subclass if needed.

Categories