We Keep Coding

Downloading Shares Gives "JSONDecodeError: Expecting value: line 1 column 1 (char 0)"

I am downloading shared from Finance Yahoo. There is this error prompts:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I have checked all the similar questions and applied but all in vain, that's why asking this question again specifying my problem.
I am downloading shares details of the top 100 current stocks of Nasdaq using this repo, that is basically for the prediction of stock shares based on financial analysis. there is this stage of downloading shares and saving them as Pandas DataFrame. The code is:
shares = []
tickers_done = []
for ticker in tqdm(tickers):
if ticker in tickers_done:
continue
d = requests.get(f"https://query1.finance.yahoo.com/ws/fundamentals-timeseries/v1/finance/timeseries/{ticker}?symbol={ticker}&padTimeSeries=true&type=annualPreferredSharesNumber,annualOrdinarySharesNumber&merge=false&period1=0&period2=2013490868")
if not d.ok:
time.sleep(300)
d = requests.get(f"https://query1.finance.yahoo.com/ws/fundamentals-timeseries/v1/finance/timeseries/{ticker}?symbol={ticker}&padTimeSeries=true&type=annualPreferredSharesNumber,annualOrdinarySharesNumber&merge=false&period1=0&period2=2013490868")
ctn = d.json()['timeseries']['result']
dct = dict()
for n in ctn:
type = n['meta']['type'][0]
dct[type] = dict()
if type in n:
for o in n[type]:
if o is not None:
dct[type][o['asOfDate']] = o['reportedValue']['raw']
df = pd.DataFrame.from_dict(dct)
df['symbol'] = ticker
shares.append(df)
tickers_done.append(ticker)
time.sleep(1)
# save dataframe
df = pd.concat(shares)
df['date'] = df.index
df.to_csv(f"data/{project}/shares.csv", index=False)
And the error screen shot is:
And I have checked each ticker requests status_code as:
for x in tickers:
d = requests.get(f"https://query1.finance.yahoo.com/ws/fundamentals-timeseries/v1/finance/timeseries/{x}?symbol={x}&padTimeSeries=true&type=annualPreferredSharesNumber,annualOrdinarySharesNumber&merge=false&period1=0&period2=2013490868")
print(d.status_code)
The results are all 403. Your kind help will be highly appreciable. Thanks!
But searching the link https://query1.finance.yahoo.com/ws/fundamentals-timeseries/v1/finance/timeseries/AAPL?symbol=AAPL&padTimeSeries=true&type=annualPreferredSharesNumber,annualOrdinarySharesNumber&merge=false&period1=0&period2=2013490868in chrome by putting one of the tickers, like, AAPL, it gives some data, like,

The error
Per your error trace, the below line is throwing an error:
ctn = d.json()['timeseries']['result']
The error is trying to tell you that the data in d is not JSON-formatted:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
So basically the very first character (line 1, column 1) is not the expected { that starts JSON.
Add a print/breakpoint prior to the line above and see for yourself what data is in d (and whether or not it contains valid JSON).
The cause
As you point out towards the end:
The results are all 403. Your kind help will be highly appreciable. Thanks!
Since the requests.get calls are being rejected by the server, the responses don't contain valid JSON.
Find out why the server is rejecting your calls (403 suggests you don't have access to the requested resource); start with the most basic call that returns [any] data and then add bits until you either get a working query, or an error.

Python For Loop giving RANDOM errors on each run, IndexError or ValueError

I'm new to python. I put together this code to pull daily Options data from yfinance for multiple stock symbols for all possible expiration dates for each stock symbol (each symbol can have different expirations dates). So I created two For Loops, first loop picks the stock, second loop picks the expiration date for that selected symbol. The code looks like this
pip install yfinance --upgrade --no-cache-dir
import yfinance as yf
import pandas as pd
# List of tickers
tickers = ["AAPL","ABBV","ABT","ACN","ADBE","ADI","ADP","AEP","AGG","ALL","AMAT","AMD","AMGN","AMT","AMZN","APD","ARKF","ARKG","ARKK","ARKQ","ARKW","ARKX","XOP"]
# Loop to call put and call values for all expirations for all tickers in the list
put_call_combined = []
for X in tickers:
ticker = X
DateArray = yf.Ticker(ticker).options
for Y in DateArray:
strikeChoice = Y
opt = yf.Ticker(ticker).option_chain(strikeChoice)
calls = opt.calls
puts = opt.puts
put_call_combined.append([calls.lastTradeDate.max().date(),ticker,strikeChoice,puts['openInterest'].sum(),puts['volume'].sum(),calls['openInterest'].sum(),calls['volume'].sum()])
ArrayStore = None
#Final Output
df = pd.DataFrame(data=put_call_combined, columns=["dataset_day","ticker", "expiry","putOI","putVolume","callOI","callVolume"])
df
My problem is; on every run I'm getting random errors, when I look at the final DF, I can see the loop was broken at different symbols. Sometimes it fails due to an IndexError
IndexError Traceback (most recent call last)
<ipython-input-26-de7016fd3a37> in <module>
6
7 ticker = X
----> 8 DateArray = yf.Ticker(ticker).options
9
10 for X in DateArray:
~\anaconda3\lib\site-packages\yfinance\ticker.py in options(self)
193 def options(self):
194 if not self._expirations:
--> 195 self._download_options()
196 return tuple(self._expirations.keys())
~\anaconda3\lib\site-packages\yfinance\ticker.py in _download_options(self, date, proxy)
59 self._expirations[_datetime.datetime.utcfromtimestamp(
60 exp).strftime('%Y-%m-%d')] = exp
---> 61 return r['optionChain']['result'][0]['options'][0]
62 return {}
63
IndexError: list index out of range
And sometime it's a ValueError;
ValueError Traceback (most recent call last)
<ipython-input-25-0a07edf80d9a> in <module>
9 for x in DateArray:
10 strikeChoice = x
---> 11 opt = yf.Ticker(ticker).option_chain(strikeChoice)
12 calls = opt.calls
13 puts = opt.puts
~\anaconda3\lib\site-packages\yfinance\ticker.py in option_chain(self, date, proxy, tz)
92 self._download_options()
93 if date not in self._expirations:
---> 94 raise ValueError(
95 "Expiration `%s` cannot be found. "
96 "Available expiration are: [%s]" % (
ValueError: Expiration `2021-10-15` cannot be found. Available expiration are: [2021-09-17, 2022-01-21, 2023-01-20]
If I reduce the number of symbols, for instance 4 or 5 symbols at a time, I can get the output for all of these symbols, but when the list becomes too many, then the errors start kicking in RANDOMLY. Anyone has any idea why this might be happening? Am I pushing the limits of Yahoo Finance API?
Thanks
MT

I'm getting a similar issue when I try to download a large number (100 or more) of historical stock prices using yf.download. I thought I might try using a for loop with yf.Ticker to get the prices one at a time, and put a try-except block inside the for loop to deal with exceptions. Hopefully that will at least allow the for loop to complete and I'll get the rest of the data.

Index out of bounds error for raster data extraction code

I am using a code written by Victor Velasquez to extract data from raster files which contain dayly precipitation data since 1981.
When I run the code, I get this error that some index is out of bounds. I did a little research and found that this is common and there are a lot of similar questions here, but I haven´t been able to find the specific solution for this case.
The error:
IndexError Traceback (most recent call last)
<ipython-input-8-eff66ef74d73> in <module>
1 Pisco = Extract_Pisco()
----> 2 Pisco.DataPre()
3 Pisco.ExportExcel()
<ipython-input-7-6cf99336b9e1> in DataPre(self)
23 Band = Data.read(1)
24 X,Y = Data.index(self.x,self.y) #extraigo
---> 25 Pre = Band[X,Y]
26 self.ListPre.append(Pre) #agrego a lista
27
IndexError: index 158116290 is out of bounds for axis 0 with size 198
The part of the code pointed by the traceback is:
def DataPre(self):
os.chdir(path)
fileDir= path
fileExt = r".tif"
Lis = [_ for _ in os.listdir(fileDir) if _.endswith(fileExt)]
Lis.sort() #ordeno archivos .tif
Inicio = '1981-01-01.tif'
Fin = '2018-07-31.tif'
Rini = Lis.index(Inicio)
Rend = Lis.index(Fin)
self.Lis = Lis[Rini:Rend+1]
self.ListPre = []
for i in tnrange (0,len(self.Lis),desc = "!! Extrayendo Datos !!"):
with rasterio.open(self.Lis[i]) as Data:
Band = Data.read(1)
X,Y = Data.index(self.x,self.y)
Pre = Band[X,Y]
self.ListPre.append(Pre)
Thank you very much!

It looks like the file you are reading does not contain the geospatial point you are trying to find data for. (If this is incorrect please let me know).
You can add a statement to catch if a point is contained in the data:
Band = Data.read(1)
X,Y = Data.index(self.x,self.y)
if 0 <= X < Band.height and 0 <= Y <= Band.width:
Pre = Band[X,Y]
self.ListPre.append(Pre)

I am using the Foursquare API and keep getting an error where it does not have a zip/postal code and no results are returned

My code is this...
results = requests.get(url).json()['response']['groups'][0]['items']
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-225-110fa1855079> in <module>
1 london_venues = getNearbyVenues(names=df2['Postcode'],
2 latitudes=df2['Latitude'],
----> 3 longitudes=df2['Longitude']
4 )
<ipython-input-223-c23495b2f972> in getNearbyVenues(names, latitudes, longitudes, radius)
16
17 # make the GET request
---> 18 results = requests.get(url).json()['response']['groups'][0]['items']
19
20 # return only relevant information for each nearby venue
KeyError: 'groups'
I think it is because there is no data returned in some cases - is there a way I can just return no data?

If in some cases you do not have group, you can simply change your line to:
results = requests.get(url).json()['response'].get('groups',[{}])[0].get('items', [])
it will returns None if you miss groups or items in your response.

KeyError: 'begins_at'

TICKERS = ['^GSPC','^IXIC','^GDAXI', '^FTSE']
ind_data = pd.DataFrame()
for f in TICKERS:
ind_data [f] = wb.DataReader(f,data_source = 'robinhood',start = '1998-01-01') ['close_price'].values
KeyError
Traceback (most recent call last)
<ipython-input-813-4985fcf38db5> in <module>()
2 ind_data = pd.DataFrame()
3 for f in TICKERS:
----> 4 ind_data [f] = wb.DataReader(f,data_source = 'robinhood',start = '1998-01-01') ['close_price'].values
KeyError: 'begins_at'

I was using dash and I encountered the same problem.
this is a very helpful document of dataframe
https://media.readthedocs.org/pdf/pandas-datareader/latest/pandas-datareader.pdf
I still couldn't get stock price from robinhood to work, but I successfully displayed my chart with other API in my case, It was quandl.
Few things I got from reading the doc given above.
each broker offers different length of data. eg. robinhood only offers 1 years. ( this doesn't seem to be a problem tho. even if you asked for more than 1 year, i will just give you 1 year data)
Symbols used in each broker is different.
eg. morningstar (no longer available) used aapl for Apple stock. quandl used WIKI/AAPL.
I am sure that there are more exception like those I give above, but that's all I know for now because I only took a look at the doc less than 10 mins.

I was having the same problem with all the ticker symbols that contained dashes and caret.
Also if the ticker symbol is not supported by Robinhood, than you will get the same 'begins_at' massage.
Here is an example the ticker symbol IMIO is not supported by Robinhood and will give the same massage.
import pandas_datareader.data as web
web.DataReader("IMIO", 'robinhood')