Ok, so what I need is just CVS file with basic stocks data with additional sector column for each company. Getting sectors and tickers from wikipedia works fine.
But Google/Yahoo data doesn't want to work anymore, so I've tried to use Quandl.I've recently made a few technical indicators based on it so I thought it would be great.But there is following error.
I am still in doubt if I can format my request like that so I assume that there is a problem but I can't figure a way to make it work.
Thank you for any advice and sorry for misspelings.
from bs4 import BeautifulSoup
import requests
import pandas as pd
import lxml
import quandl as qdl
def get_ticker_and_sector(url='https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'):
r = requests.get(url)
data = r.text
soup = BeautifulSoup(data, 'lxml')
table = soup.find('table')
sp500 = {}
for tr in table.find_all('tr')[1:]:
tds = tr.find_all('td')
ticker = tds[0].text
sector = tds[3].text
sp500[ticker] = sector
return sp500
if __name__ == '__main__':
sp500 = get_ticker_and_sector()
for i, (ticker, sector) in enumerate(sp500.items()):
stock_df = qdl.get('WIKI/%s', start_date="2010-12-11", end_date="2011-12-31")%(ticker)
stock_df['Name'] = ticker
stock_df['Sector'] = sector
if i == 0:
all_df = stock_df
else:
all_df = all_df.append(stock_df)
all_df.to_csv('all_sp500_data_2.csv')
Error.
Traceback (most recent call last):
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/quandl/connection.py", line 55, in parse
return response.json()
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/requests/models.py", line 886, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tomek/PycharmProjects/untitled4/get_file.py", line 30, in <module>
stock_df = qdl.get('WIKI/%s', start_date="2010-12-11", end_date="2011-12-31")%('ticker')
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/quandl/get.py", line 48, in get
data = Dataset(dataset_args['code']).data(params=kwargs, handle_column_not_found=True)
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/quandl/model/dataset.py", line 47, in data
return Data.all(**updated_options)
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/quandl/operations/list.py", line 14, in all
r = Connection.request('get', path, **options)
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/quandl/connection.py", line 36, in request
return cls.execute_request(http_verb, abs_url, **options)
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/quandl/connection.py", line 44, in execute_request
cls.handle_api_error(response)
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/quandl/connection.py", line 61, in handle_api_error
error_body = cls.parse(resp)
File "/home/tomek/tomek-workspace/pythons/udemy/lib/python3.5/site-packages/quandl/connection.py", line 57, in parse
raise QuandlError(http_status=response.status_code, http_body=response.text)
quandl.errors.quandl_error.QuandlError: (Status 400) Something went wrong. Please try again. If you continue to have problems, please contact us at connect#quandl.com.
The traceback is giving you an 400 status code which means something is wrong with the request to the API.
More specifically, Quandl's documentation says the following regarding a 400 status code:
We could not recognize your API key. Please check your API key and try again. You can find your API key under your account settings.
As a good starting point, have you entered your API key after installing the Quandl module? Steps are here for more information on how to do it.
Please note, I only glanced at the rest of your code, and didn't test it out fully to see if there are other errors!
Related
I have tested around 1000 bitcoin addresses and there were no problems, however after trying to test around 40,000 addresses it gives me this error
from requests import get
import pandas as pd
import json
def make_api_url():
data = pd.read_csv('bitcoinaddr.csv')
Wallet_Address = (data.loc[:,"Address"])
BASE_URL = "https://blockchain.info/balance"
for addresses in Wallet_Address:
addresses = '|'.join(Wallet_Address)
url = BASE_URL + f"?active={addresses}" #error
return url
get_balance_url = make_api_url()
response = get(get_balance_url)
j_data = response.json()
finalbalance=[]
tx=[]
totalreceived=[]
new_dataset = pd.DataFrame.from_dict(j_data, orient='index')
print(new_dataset)
finalbalance=new_dataset['final_balance'].to_list()
tx=new_dataset['n_tx'].to_list()
totalreceived=new_dataset['total_received'].to_list()
data['Final Balance']=finalbalance
data['n_tx']=tx
data['Total Received']=totalreceived
new_dataset.to_csv('CheckedBTCAddress.csv',index=True,header=True)
# https://blockchain.info/balance?active=$address
The error code shows
Traceback (most recent call last): File "C:\Users\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py",
line 971, in json
return complexjson.loads(self.text, **kwargs) File "C:\Users\AppData\Local\Programs\Python\Python310\lib\json_init_.py",
line 346, in loads
return _default_decoder.decode(s) File "C:\Users\AppData\Local\Programs\Python\Python310\lib\json\decoder.py",
line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Users\Forensic\AppData\Local\Programs\Python\Python310\lib\json\decoder.py",
line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"c:\Users
Scripting\BTCAddressBulkChecker.py", line 21, in
data = response.json() File "C:\UsersPython\Python310\lib\site-packages\requests\models.py",
line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1
(char 0)
I am using requests module to run a curl command, not sure why its prefixing ')]}\' in the front of the output, it makes r.json() fail as shown below.
When I paste the URL in a browser and execute, it downloads a .json file, and that file has same characters in the front, Am I missing some option? I need to process the output as json.
>>> r = requests.get('https://gerrit-review-server/a/changes/?q=status:open%20project:myproj/test/a_proj%20change:1510&o=CURRENT_REVISION', verify=False, auth=HTTPBasicAuth('user','pass'),
>>>
>>> r.text
')]}\'\n[{"id":"myproj%2Ftest%2Fa_proj~master~I15790ba05690e0a9984cb05bce06574645274966","project":"myproj/test/a_proj","branch":"master","hashtags":[],"change_id":"I15790ba05690e0a9984cb05bce06574645274966","subject":"Test","status":"NEW","created":"2021-01-27 19:38:57.000000000","updated":"2021-03-21 14:19:42.000000000","submit_type":"MERGE_IF_NECESSARY","mergeable":true,"insertions":1,"deletions":0,"total_comment_count":0,"unresolved_comment_count":0,"has_review_started":true,"_number":1510,"owner":{"_account_id":10008339},"current_revision":"fe3cc60cc66ad6f20c631ee818ccd91955c69d37",<..deleted..>,"description":"Rebase"}},"requirements":[]}]\n'
>>> r.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>>>
As Carcigenicate says, it seems that the json is wrongly formatted. You can try the loads, and if it fails try to correct it:
r = requests.get(some_url)
try:
data = r.json()
except json.JSONDecodeError:
# cut everything in front of the first "\n"
raw_data = r.text.split("\n", maxsplit=1)[1]
# cut everything behind the last "\n"
raw_data = raw_data.rsplit("\n", maxsplit=1)[0]
# try to load again the json
# If it fails it will raise the exception again
data = json.loads(raw_data)
I'm new to python and APIs and I'm using this tutorial to learn with last-fm. I'm getting these errors:
Traceback (most recent call last):
File "/Users/vikram03/Desktop/Python_learning/fm.py", line 89, in <module>
total_pages = int(response.json()['artists']['#attr']['totalPages'])
File "/Applications/anaconda3/lib/python3.8/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/Applications/anaconda3/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/Applications/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Applications/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Here's the code in question:
#initializes reponses list
responses = []
page = 1
total_pages = 70
while page <= total_pages:
payload = {
'method': 'chart.gettopartists',
'limit': 500,
'page': page
}
#print
print("Requesting page {}/{}".format(page, total_pages))
#clear clear_output
clear_output(wait=True)
#make the API call
response = lastfm_get(payload)
#error check
if response.status_code != 200:
print(reponse.text)
break
#extract pagination info
page = int(response.json()['artists']['#attr']['page'])
total_pages = int(response.json()['artists']['#attr']['totalPages'])
# append
responses.append(response)
#if its not cached, sleep
if not getattr(response, 'from_cache', False):
time.sleep(0.25)
#increment
page += 1
This function: lastfm_get just does requests.get with the appropriate params and this url 'http://ws.audioscrobbler.com/2.0/'
From other posts I get the impression the program is getting non-JSON objects, but when I print
total_pages = int(response.json()['artists']['#attr']['totalPages']) I get the correct number.
edit:
this is the function:M
def lastfm_get(payload):
#define headers and url
headers = {'user-agent': USER_AGENT}
url = 'http://ws.audioscrobbler.com/2.0/'
#add API key and format
payload['api_key'] = API_KEY
payload['format'] = 'json'
response = requests.get(url, headers = headers, params = payload)
return response
edit2:
I added this code to check if I was getting a JSON object, and am getting a type error after about 200 pages:
#check object
try:
json_test = json.loads(lastfm_get(payload))
except TypeError:
print("not a json object")
Error:
Requesting page 256/7743
Requesting page 259/7743
Requesting page 275/7743
Requesting page 294/7743
Traceback (most recent call last):
File "/Users/vikram03/Desktop/Python_learning/fm.py", line 94, in <module>
page = int(response.json()['artists']['#attr']['page'])
File "/Applications/anaconda3/lib/python3.8/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/Applications/anaconda3/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/Applications/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Applications/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I got a type error when I tried with except ValueError:
Traceback (most recent call last):
File "/Users/vikram03/Desktop/Python_learning/fm.py", line 84, in <module>
json_test = json.loads(lastfm_get(payload))
File "/Applications/anaconda3/lib/python3.8/json/__init__.py", line 341, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not Response
EDIT3: I get this error when testing the output of lastfm_get:
TypeError: Object of type Response is not JSON serializable
Unsure how to resolve this.
I'm not sure why the page numbers are skipping either
Google translate libary stopped working.
I didn't change anything about my code and it worked like an hour ago. I don't understand what is happening.
Here is my code:
from googletrans import Translator
import time
K = Translator()
def anna_sanat(x):
K = Translator()
a = K.translate(x,dest="sv",src="en").text
b = K.translate(x,dest="fi",src="en").text
return [x,a,b]
f = open("sanat.txt",mode="r",encoding="utf-8")
g = f.read()
f.close()
g = g.split(" ")
h = []
for i in g:
h += i.split("\n")
for i in h:
print(i)
print(anna_sanat(i.replace("\"","")))
time.sleep(1)
and here is the error message:
Magnus
Traceback (most recent call last):
File "C:\Users\Taavi\Desktop\Koodaus\uusin Python\ruotsi\ruotsipeli.py", line 21, in <module>
print(anna_sanat(i.replace("\"","")))
File "C:\Users\Taavi\Desktop\Koodaus\uusin Python\ruotsi\ruotsipeli.py", line 7, in anna_sanat
a = K.translate(x,dest="sv",src="en").text
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\googletrans\client.py", line 172, in translate
data = self._translate(text, dest, src)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\googletrans\client.py", line 81, in _translate
data = utils.format_json(r.text)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\googletrans\utils.py", line 62, in format_json
converted = legacy_format_json(original)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\googletrans\utils.py", line 54, in legacy_format_json
converted = json.loads(text)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Taavi\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Oh I know this one, I think this happens because googletrans makes requests to google translate directly, and after a while, google translate starts returning nothing, the only way that I know to get around this is to add a time.sleep(*Large number here*) and test again after that.
I have this piece of Python code:
homedir = '/home/cloudera/'
baseurl = 'http://ergast.com/api/f1/'
apilist = ['seasons', 'results', 'qualifying', 'driverStandings', 'constructorStandings', 'drivers', 'constructors',
'circuits', 'status', 'laps', 'pitstops']
for dir in apilist:
target = homedir + str(dir)
try:
shutil.rmtree(target, ignore_errors=True)
os.makedirs(target)
except:
pass
parent_url = baseurl + "{y}/{api}.json?limit=100000"
parent_fname = homedir + "{api}/{yf}.json"
urls = [parent_url.format(y=y, api=a) for y, a in itertools.product(range(1950, 2016), apilist)]
files = [parent_fname.format(yf=y, api=a) for y, a in itertools.product(range(1950, 2016), apilist)]
for url, file in zip(urls, files):
print(url, file)
response = requests.get(url).json()
with open(file, 'w') as f:
json.dump(response, f)
When I run this, this is the output I get:
/home/cloudera/PycharmProjects/ErgastF1/ErgastF1/bin/python /home/cloudera/PycharmProjects/ErgastF1/F1Demo.py
http://ergast.com/api/f1/1950/seasons.json?limit=100000 /home/cloudera/seasons/1950.json
http://ergast.com/api/f1/1950/results.json?limit=100000 /home/cloudera/results/1950.json
http://ergast.com/api/f1/1950/qualifying.json?limit=100000 /home/cloudera/qualifying/1950.json
http://ergast.com/api/f1/1950/driverStandings.json?limit=100000 /home/cloudera/driverStandings/1950.json
http://ergast.com/api/f1/1950/constructorStandings.json?limit=100000 /home/cloudera/constructorStandings/1950.json
http://ergast.com/api/f1/1950/drivers.json?limit=100000 /home/cloudera/drivers/1950.json
http://ergast.com/api/f1/1950/constructors.json?limit=100000 /home/cloudera/constructors/1950.json
http://ergast.com/api/f1/1950/circuits.json?limit=100000 /home/cloudera/circuits/1950.json
http://ergast.com/api/f1/1950/status.json?limit=100000 /home/cloudera/status/1950.json
http://ergast.com/api/f1/1950/laps.json?limit=100000 /home/cloudera/laps/1950.json
Traceback (most recent call last):
File "/home/cloudera/PycharmProjects/ErgastF1/F1Demo.py", line 74, in <module>
response = requests.get(url).json()
File "/home/cloudera/PycharmProjects/ErgastF1/ErgastF1/lib/python3.5/site-packages/requests/models.py", line 812, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Process finished with exit code 1
Basically, it creates the JSON files for the year 1950, but the next time the loop runs for the year 1951, it exits. I'm new to Python and newer still to JSON. Can someone figure out what's going on?
Note: A less complex version of this program, with all the years from 1950 to 2015, but only with qualifying had worked. I was trying to go large scale and get all attributes together. That version merely used a range to iterate through the years and generate file names and urls.
This problem are raised because the url: http://ergast.com/api/f1/1950/laps.json?limit=100000 returns a error. Look the url:http://ergast.com/api/f1/1950/status.json?limit=100000 (Image Above)
This Image have a working json format, and the source this page is:
The url:http://ergast.com/api/f1/1950/laps.json?limit=100000 return a html error:
And the source of this url is a html table and not is a json format:
This error can be suppresed using a error treatment, but cant resolve the format. Maybe need some other argument in the url to return the corret values.
I this link: http://ergast.com/mrd/ have the documentation of this api, and in the section Overview says:
If you try to put a numeric value into a url after year, like : http://ergast.com/api/f1/1950/1/laps.json?limit=100000 you return some valid results, like this image: