Thank you in advanced for any insights you can offer to my current challenge. I am running Python 3.9.1 and the latest versions of PANDAS and TIINGO libraries.
The problem is a reliable API call has started failing for certain tickers. I assume the problem is with the API provider but find it odd that no one else is reporting the problems.
My (simplified) code:
# Loads historical stock data into files
import datetime
#from pandas_datareader import data
import pandas as pd
import io
import os
#import requests
from tiingo import TiingoClient
os.environ["TIINGO_API_KEY"] = 'MY API KEY'
client = TiingoClient()
ticker = 'PG'
last_load_date = '2020-08-19'
today = '2021-02-12'
#idx = client.get_dataframe(ticker, frequency='daily', startDate=last_load_date, endDate=today)
##history = client.get_dataframe(['GOOGL',],
## frequency='daily',
## metric_name='adjClose',
## startDate='2020-01-01',
## endDate='2021-05-31')
prices = client.get_ticker_price(ticker,
fmt='json',
startDate=last_load_date,
endDate=today,
frequency='daily')
The failed output:
Traceback (most recent call last):
File "V:/woodstock/jobs/tiingo_test.py", line 22, in <module>
prices = client.get_ticker_price(ticker,
File "C:\Users\wood\AppData\Local\Programs\Python\Python39\lib\site-packages\tiingo\api.py", line 241, in get_ticker_price
return response.json()
File "C:\Users\wood\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\wood\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\wood\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\wood\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I have tried both the direct JSON method (above) and the leveraging dataframes (preferred but commented out above). This works for a ticker like GE but fails for PG, TSLA and others.
I have contacted Tiingo support but appreciate any insights from the community.
THANK YOU!
Tiingo support answered my question very quickly and professionally. As a "free" customer, I am incredibly impressed by their service.
The answer, for anyone else facing this is that Tiingo has a constraint on the number of calls per hour AND total number of unique symbols requested. The API is failing due to the fact that I have reached my 500 ticker limit and am requesting a ticker outside of that list. I will be upgrading my service to get around this issue.
Related
I'm trying to do a simple correlation between a few stock symbols. I've run the same code before and it worked. Now, it's returning an exception that references a bunch of files and contains other messages that I don't understand. Also, the command has to be manually stopped in the console. I'm very new to python and programming in general. I'm using Spyder 5.0.3 and anaconda3. I've tried reinstalling Spyder and updated the packages used, neither resolved the issue.
The problem appears to be with the yfinance package. When I tried to download price data for a single ticker a similar exception was returned. It doesn't appear to be pandas or numpy because I was able to make a small dataframe and run corrcoeff() without any problem. For what it's worth this is the code I used:
import pandas as pd
import numpy as np
d = {'col1': [1, 2], 'col2': [3, 4]}
df = pd.DataFrame(data=d)
r = np.corrcoef(df)
The following is the code that used to work but is no longer working:
import pandas as pd
import numpy as np
import yfinance as yf
AAPL = yf.download('AAPL', period='32D', interval='1D')
MSFT = yf.download('MSFT', period='32D', interval='1D')
NVDA = yf.download('NVDA', period='32D', interval='1D')
corr3 = pd.DataFrame([AAPL['Close'],
MSFT['Close'],
NVDA['Close']])
r = np.corrcoef(corr3)
When I run the file I receive the following exception in the console:
Exception in thread Thread-10:
Traceback (most recent call last):
File "C:\Users\jdejo\anaconda3\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\jdejo\anaconda3\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\jdejo\anaconda3\lib\site-packages\multitasking\__init__.py", line 102, in _run_via_pool
return callee(*args, **kwargs)
File "C:\Users\jdejo\anaconda3\lib\site-packages\yfinance\multi.py", line 167, in _download_one_threaded
data = _download_one(ticker, start, end, auto_adjust, back_adjust,
File "C:\Users\jdejo\anaconda3\lib\site-packages\yfinance\multi.py", line 179, in _download_one
return Ticker(ticker).history(period=period, interval=interval,
File "C:\Users\jdejo\anaconda3\lib\site-packages\yfinance\base.py", line 157, in history
data = data.json()
File "C:\Users\jdejo\anaconda3\lib\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\jdejo\anaconda3\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\Users\jdejo\anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\jdejo\anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I am wanting to know what went wrong with my code. It used to work, now it doesn't. Any guidance would be greatly appreciated.
I have solved the problem. I needed to update yfinance through the console in spyder. I had been using the command prompt to update packages.
I am currently trying to use folium library in python to create webmaps. I have a file world.json which contains geo_data. I have provided a link to the file at the end of this post. I tried the following code:
data = [json.loads(line) for line in open('world.json', 'r')]
and received the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <listcomp>
File "C:\Users\name\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\Users\name\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\name\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
How can I load this file?
What I want to achieve is essentially obtain the population data and create a Choropleth and overlay it on my webmap.
Edit: Forgot the link:
https://1drv.ms/u/s!Army95vqcKXpaooVAZU_g-VCAVw?e=vwTknq
Edit: Previous link to skydrive stopped working due to "high traffic". Below is link to dropbox, hopefully this works:
https://www.dropbox.com/s/gmm8db0g03rc7cv/world.json?dl=0
Good news/bad news:
It turns out that this file was encoded in a locale that we are not accustomed to, and json/ascii cannot make sense of some of the character encoding. I tried this, and it seems to be working for me -- with a major caveat:
with open("world.json", "r") as fh:
contents = fh.read()
asciiContents = contents.encode("ascii", errors="ignore")
data = json.loads(asciiContents)
The major caveat is that only 3 countries come through with no encoding errors:
>>> len(data["features"])
3
Maybe there another source for this data that is closer to a native english locale, or maybe someone else can provide wisdom in encoding foreign data in a more friendly way...
The open command will return a file handle, not string lines. I would do:
with open('world.json', 'r') as fh:
data = json.load(fh)
data will then be your contents converted to python (list or dictionary, etc)
I have been using pandas_datareader to retrieve stock prices from the IEX and Robinhood APIs for some time without any hitches. However, recently (early this week), my code started failing.
Here's the code and the error:
IEX API
import pandas_datareader.data as web
import datetime as dt
end = dt.datetime.today()
start = end - relativedelta(months=20)
data = web.DataReader('MSFT', 'iex', start, end)
And the error:
web.DataReader('MSFT', 'iex', start, end)
Traceback (most recent call last):
File "<ipython-input-88-2781d4515d5c>", line 1, in <module>
web.DataReader('MSFT', 'iex', start, end)
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/data.py", line 322, in DataReader
session=session).read()
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/iex/daily.py", line 91, in read
self._get_params(self.symbols))
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/base.py", line 84, in _read_one_data
out = self._read_url_as_StringIO(url, params=params)
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/base.py", line 95, in _read_url_as_StringIO
response = self._get_response(url, params=params)
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/base.py", line 155, in _get_response
raise RemoteDataError(msg)
RemoteDataError: Unable to read URL: https://api.iextrading.com/1.0/stock/market/batch?symbols=MSFT&types=chart&range=2y
Response Text:
b'Forbidden'
Robinhood API
import pandas_datareader.data as web
import datetime as dt
end = dt.datetime.today()
start = end - relativedelta(months=20)
data = web.DataReader('MSFT', 'robinhood', start, end)
And the error:
web.DataReader('MSFT', 'robinhood', start, end)
Traceback (most recent call last):
File "<ipython-input-90-0a0fc1b0d4ce>", line 1, in <module>
web.DataReader('MSFT', 'robinhood', start, end)
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/data.py", line 391, in DataReader
session=session).read()
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/base.py", line 77, in read
return self._read_one_data(self.url, self.params)
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/base.py", line 86, in _read_one_data
out = self._get_response(url, params=params).json()
File "/anaconda3/lib/python3.7/site-packages/pandas_datareader/base.py", line 136, in _get_response
last_response_text = response.text.encode(response.encoding)
TypeError: encode() argument 1 must be str, not None
The exact code was working fine until recently. I wonder what changed, and what I can do to correct it.
Most APIs listed on the pandas_datareader documentation have been deprecated and are no longer effective through the pandas_datareader framework (Yahoo, Google, etc).
However, Tiingo works just fine, at least for now.
To use Tiingo API, you'll first need to sign up for a free account (or paid if you want premium features). You can do that here. After activating your Tiingo account, you'll receive an API key which you can use to retrieve stock historical data with pandas_datareader (find details here).
From https://iextrading.com/developer/docs/#stocks
"On June 1, 2019, IEX Group removed all non-IEX data, and certain functionality, according to the schedule. IEX Cloud, a non-Exchange platform, will continue to provide access to third-party data sources."
Looks like you'll need access and secret keys to call their API. I'm looking into this for my python app stock_quote that hit this issue.
You can use the efficient Python library for Yahoo Finance, yfinance (GitHub link).
You can also find example codes here and here.
I am trying to get started writing a Groupme bot using Groupy.
There have been a few related posts to this one, but they seemed a bit more technically challenging of issues, the solutions didn't seem efficient, and none of them solved the issues I'm having.
Here is the code I have so far:
import groupy
token = "{My Token}"
from groupy.client import Client
client = Client.from_token(token)
groups = list(client.groups.list_all())
for group in groups:
print(group)
Firstly, I don't believe this is related but, in eclipse it is giving a warning that groupy is an unused import, even though it seems to me that that is not so.
More importantly, the strange part about the issues I'm having is that the error messages change without any edits to the above code.
Here is an error message that I've gotten:
In each of these error messages, I've edited the beginning of the file locations... because I'm paranoid
Traceback (most recent call last):
File "\Projects\Eclipse\eclipse-workspace\Gort's Cousin\groupme_bot.py", line 15, in <module>
<Group(name='One of my group's names')>
<Group(name='Another one of my group's names')>
print(group.name)
File "\Python\Python37-32\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 27-28: character maps to <undefined>
So, this time the system actually returns a couple groups I am in. But then calls an exception because of a unicode error of sorts.
Without changing any code, I also get this error:
Traceback (most recent call last):
File "\Python\Python37-32\lib\site-packages\groupy\session.py", line 49, in data
return self.json()['response']
File "\Python\Python37-32\lib\site-packages\requests\models.py", line 808, in json
return complexjson.loads(self.text, **kwargs)
File "\Python\Python37-32\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "\Python\Python37-32\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 60865 (char 60864)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "\Eclipse\eclipse-workspace\Gort's Cousin\groupme_bot.py", line 10, in <module>
groups = list(client.groups.list_all())
File "\Python\Python37-32\lib\site-packages\groupy\pagers.py", line 61, in autopage
self.items = self.fetch_next()
File "\Python\Python37-32\lib\site-packages\groupy\pagers.py", line 51, in fetch_next
return self.fetch()
File "\Python\Python37-32\lib\site-packages\groupy\pagers.py", line 42, in fetch
return self.endpoint(**self.params)
File "\Python\Python37-32\lib\site-packages\groupy\api\groups.py", line 21, in _raw_list
return [Group(self, **group) for group in response.data]
File "\Python\Python37-32\lib\site-packages\groupy\session.py", line 51, in data
raise exceptions.InvalidJsonError(self._resp) from e
groupy.exceptions.InvalidJsonError: The JSON was incomplete/invalid
I am most likely incorrect, but it seems to me that there is some issue with the way groupy is aggregating the groups, and/or how I've set up my project to handle those calls.
Any help with this issue would be awesome.
I can also add some code from the groupy library if that becomes necessary.
TLDR: Getting a "UnicodeEncodeError" error using Groupy to write a groupme bot in Python
Solved:
It was an issue where group names with emojis were not playing nice with the Cp1252 encoding I have as a global setting in eclipse. I changed the run configuration of the file to encode to UTF-16BE. The emojis don't print correctly to the console, but all of the groups are printed with no errors.
I am trying to parse some text files containing JSON objects in Python using the json.load() method. It's working for one set of them, but for this one it will not:
{
"mapinfolist":{
"mapinfo":[
{"sku":"00028-0059","price":"38.35","percent":"50","basepercent":"50","exact":0,"match":0,"roundup":0}
,{"sku":"77826-7230","price":"4.18","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-1310","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-2020","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-3360","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-4060","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-4510","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
,{"sku":"77827-7230","price":"2.36","percent":"60","basepercent":"60","exact":1,"match":0,"roundup":0}
],
"count":2
}
}
It is in a file called 'map.txt' - I open it using open('map.txt') and then call json.load(). When I run my test program (test.py), the following error trace is generated:
Traceback (most recent call last):
File "test.py", line 28, in <module>
main()
File "test.py", line 23, in main
map_list = json.load(f1)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
The JSON object is valid - when I put it into https://www.jsoneditoronline.org/ it is parsed and displayed correctly, so I am having trouble identifying what could be stopping it from working when I try to do it in Python. Any advice would be much appreciated. Thanks!
EDIT: Here's my code.
import json
def main():
with open('map.txt') as f1:
map_list = json.load(f1)
Trying map_list = json.loads(f1.read()) also does not work and gives me an almost identical error trace.
EDIT - RESOLVED:
I just copied and pasted FROM map.txt into a new TextEdit file map2.txt and used the new file instead, and it works now. I copied directly from the old file and made no changes - the only difference is that it is a different file. I can't make heads or tails of why that would be - any ideas? I would like to understand what may have happened so I can avoid the problem in the future.
Does the following solution work for you?
import json
f = open("map.txt")
map = json.loads(f.read())
Python Docs
maybe try to read all the file to string and then use json.loads
def yourfunc():
file = open('map.txt')
json_string = file.read()
map = json.loads(json_string)