I have used an API using python, to call out to "NewsAPI" to get all the latest news that I need and I have actually save it into a text file called "NewsAPI.txt".
My code is:
import json
import requests
def newsAPI():
url = ('https://newsapi.org/v2/everything?' #API URL
'q=procurement AND tender&' #keywords on procurement AND tender
'sortBy=popularity&' #Sort them by popularity
'apiKey=***') #Personal API key
# GET
response = requests.get(url)
#storing the output into variable "results"
results = response.json()
# save the JSON output into a txt file for future usage
with open("NewsAPI.txt", "w") as text_file:
json.dump(results, text_file)
After calling json.dump, it gets saved into the "NewsAPI.txt" file as I have mentioned. But I'm having trouble putting it into a treeview in Tkinter, or am I using the wrong widget to display them?
Output data:
Related
The goal is to open a json file or websites so that I can view earthquake data. I create a json function that use dictionary and a list but within the terminal an error appears as a invalid argument. What is the best way to open a json file using python?
import requests
`def earthquake_daily_summary():
req = requests.get("https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson")
data = req.json() # The .json() function will convert the json data from the server to a dictionary
# Open json file
f = open('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson')
# returns Json oject as a dictionary
data = json.load(f)
# Iterating through the json
# list
for i in data['emp_details']:
print(i)
f.close()
print("\n=========== PROBLEM 5 TESTS ===========")
earthquake_daily_summary()`
You can immediately convert the response to json and read the data you need.
I didn't find the 'emp_details' key, so I replaced it with 'features'.
import requests
def earthquake_daily_summary():
data = requests.get("https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson").json()
for row in data['features']:
print(row)
print("\n=========== PROBLEM 5 TESTS ===========")
earthquake_daily_summary()
I am using requests.get in python for a URL to get the data as below:
import requests
username = 'ROAND'
password = dbutils.secrets.get("lab-secrets","ROSecret")
response = requests.get('https://pit.service.com/api/table', auth=(username,password))
The count for this is
print(response.headers)
X-Total-Count': '799434'
Im trying to load this into a json file as below:
data = response.content
with open('/path/file.json', 'wb') as f:
f.write(data)
But the file contains only 1439 records.
The json file content looks like the below image:
Ive tried multiple ways, but not successful.
I just want to exactly bring all my contents from requests.get into a json file.
Kindly help.
I am trying to read a JSON file (BioRelEx dataset: https://github.com/YerevaNN/BioRelEx/releases/tag/1.0alpha7) in Python. The JSON file is a list of objects, one per sentence.
This is how I try to do it:
def _read(self, file_path):
with open(cached_path(file_path), "r") as data_file:
for line in data_file.readlines():
if not line:
continue
items = json.loads(lines)
text = items["text"]
label = items.get("label")
My code is failing on items = json.loads(line). It looks like the data is not formatted as the code expects it to be, but how can I change it?
Thanks in advance for your time!
Best,
Julia
With json.load() you don't need to read each line, you can do either of these:
import json
def open_json(path):
with open(path, 'r') as file:
return json.load(file)
data = open_json('./1.0alpha7.dev.json')
Or, even cooler, you can GET request the json from GitHub
import json
import requests
url = 'https://github.com/YerevaNN/BioRelEx/releases/download/1.0alpha7/1.0alpha7.dev.json'
response = requests.get(url)
data = response.json()
These will both give the same output. data variable will be a list of dictionaries that you can iterate over in a for loop and do your further processing.
Your code is reading one line at a time and parsing each line individually as JSON. Unless the creator of the file created the file in this format (which given it has a .json extension is unlikely) then that won't work, as JSON does not use line breaks to indicate end of an object.
Load the whole file content as JSON instead, then process the resulting items in the array.
def _read(self, file_path):
with open(cached_path(file_path), "r") as data_file:
data = json.load(data_file)
for item in data:
text = item["text"]
label appears to be buried in item["interaction"]
I am trying to bulk download movie information from The Movie Database. The preferred method mentioned on their website is to loop through movie IDs from 1 until the most recent movie ID. When I pull individual movies using their ID, I get the entire set of information. However, when I pull it into a loop, I receive an error 34, resource cannot be found. For my example, I picked specifically a movie ID that I have grabbed individual (Skyfall, 37724), which returns the resource cannot be found error.
import requests
dataset = []
for i in range(37724, 37725):
url = 'https://api.themoviedb.org/3/movie/x?api_key=*****&language=en-US'
movieurl = url[:35] + str(i) + url[36:]
payload = "{}"
response = requests.request("GET", url, data=payload)
data = response.json()
dataset.append(data)
print(movieurl)
dataset
[ANSWERED] 1) Is there a reason for why the loop cannot pull the information? Is this a programming question or specific to the API?
2) Is the way my code set up the best to pull the information and store it in bulk? My ultimate goal is to create a CSV file with the data.
Your request uses url, while your actual url is in the movieurl variable.
To write your data to csv, I would recommend the python csv DictWriter, as your data are dicts (response.json() produces a dict).
BONUS: If you want to format a string, use the string.format method:
url = 'https://api.themoviedb.org/3/movie/{id}?api_key=*****&language=en-US'.format(id=i)
this is much more robust.
The working, improved version of your code, with writing to csv would be:
import csv
import requests
with open('output.csv', 'w') as csvfile:
writer = csv.DictWriter(csvfile)
for i in range(37724, 37725):
url = 'https://api.themoviedb.org/3/movie/{id}?api_key=*****&language=en-US'.format(id=i)
payload = "{}"
response = requests.request("GET", url, data=payload)
writer.writerow(response.json())
I am trying to send data from a text file to a server looking for a match to the sent data in order to get that matched data returned back to me that I store in an existing text file. If I send a list of names to the server within the script, I am fine. I however want to repeat the request and insert a text file as the names to be matched and returned. Here is my text so far:
import json
import urllib2
values = 'E:\names.txt'
url = 'https://myurl.com/get?name=values&key=##########'
response = json.load(urllib2.urlopen(url))
with open('E:\data.txt', 'w') as outfile:
json.dump(response, outfile, sort_keys = True, indent = 4,ensure_ascii=False);
This code just send back a one line file showing nothing has matched. I am assuming that it is just looking at the values as the name instead of the data in the values text file.
Update Trial 1: I updated my code as per suggested below to include the urllib.urlencode suggestion. Here is my updated code:
import json
import urllib
import urllib2
file = 'E:\names.txt'
url = 'https://myurl.com/get'
values = {'name' : file,
'key' : '##########'}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = json.load(urllib2.urlopen(req))
with open('E:\data.txt', 'w') as outfile:
json.dump(response, outfile, sort_keys = True, indent = 4,ensure_ascii=False);
fixed traceback errors by editing url. However it is just passing "e:\names.txt" as name in the JSON request. So it seems my issue now is just trying to send the data in the names.txt file to the tuple 'names' properly. Any thoughts?
Make sure when sending parameters to server, they're encoded -- see urllib.urlencode()