I am trying to get EIA data using its API, however I encountered json errors when it's calling the series. I recalled it was working fine about 6 months ago, not sure if it is something changed in EIA's API. Could anyone shed some light how to fix this?
Here's the code:
import pandas as pd
import eia
def retrieve_data():
# Create EIA API using your specific API key
api_key = "YOUR_API_KEY"
api = eia.API(api_key)
# Retrieve Data By Series ID
series_ID='STEO.PASC_OECD_T3.M'
series_search = api.data_by_series(series=series_ID)
df = pd.DataFrame(series_search)
df.index.names = ['Date']
df.columns=[ "Price"]
df.index = df.index.str.replace('^([\d]{4})\s([\d]{2})([\d]
{2})\s[\d] {2}', r'\1-\2-\3',regex=True)
df.index = pd.to_datetime(df.index)
return df
data = retrieve_data()
print(data)
and the error message is as the following:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/tmp/ipykernel_942387/4124051913.py in <module>
17 return df
18
---> 19 data = retrieve_data()
20 print(data)
21 #data.to_csv('OK_WTI_Spot_Price_FOB.csv',index=True)
/tmp/ipykernel_942387/4124051913.py in retrieve_data()
9 # Retrieve Data By Series ID
10 series_ID='STEO.PASC_OECD_T3.M'
---> 11 series_search = api.data_by_series(series=series_ID)
12 df = pd.DataFrame(series_search)
13 df.index.names = ['Date']
~/miniconda3/lib/python3.7/site-packages/eia/api.py in data_by_series(self, series)
422 else:
423 lst_dates = [x[0][0:4] + " " + x[0][4:] + " " + x[0][6:8]
--> 424 for x in search.json()['series'][0]['data']]
425 lst_values = [x[1] for x in
426 search.json()['series'][0]['data']]
KeyError: 'series'
Related
I try to get data from google trends in a g sheet. First time it runned smoothly, second time not so much. I got an error called:
ValueError: No objects to concatenate
I searched this error on Stack Overflow before but couldn't find any solutions. I use the code displayed below:
!pip install Pytrends
!pip install pandas
!pip install pytrends --upgrade <---------Note: this solved a different error.
from pytrends.request import TrendReq
import pandas as pd
import time
startTime = time.time()
pytrend = TrendReq(hl='nl-NL', tz=360)
df = wb = gc.open_by_url('https://docs.google.com/spreadsheets/d/1QE1QilM-GDdQle6eVunepqG5RNWv39xO0By84C19Ehc/edit?usp=sharing')
sheet = wb.sheet1
df2 = sheet.col_values(5)
d_from = sheet.acell('B7').value
d_to = sheet.acell('B8').value
geo1 = sheet.acell('B10').value
dataset = []
for x in range(1,len(df2)):
keywords = [df2[x]]
pytrend.build_payload(
kw_list=keywords,
cat=0,
timeframe= str(d_from + " " + d_to),
geo= str(geo1))
data = pytrend.interest_over_time()
if not data.empty:
data = data.drop(labels=['isPartial'],axis='columns')
dataset.append(data)
result = pd.concat(dataset, axis=1)
result.to_csv('search_trends_DOWNLOAD_ME.csv')
!cp search_trends_DOWNLOAD_ME.csv "/content/drive/My Drive/Colab Notebooks/Output"
executionTime = (time.time() - startTime)
print('Execution time in sec.: ' + str(executionTime))
The error I got:
ValueError Traceback (most recent call last)
<ipython-input-5-b86c7b4df727> in <module>()
25 data = data.drop(labels=['isPartial'],axis='columns')
26 dataset.append(data)
---> 27 result = pd.concat(dataset, axis=1)
28 result.to_csv('search_trends_DOWNLOAD_ME.csv')
29 get_ipython().system('cp search_trends_DOWNLOAD_ME.csv "/content/drive/My Drive/Colab Notebooks/Output"')
1 frames
/usr/local/lib/python3.6/dist-packages/pandas/core/reshape/concat.py in __init__(self, objs, axis, join, keys, levels, names, ignore_index, verify_integrity, copy, sort)
327
328 if len(objs) == 0:
--> 329 raise ValueError("No objects to concatenate")
330
331 if keys is None:
ValueError: No objects to concatenate
The keywords I use are located in df = wb = gc.open_by_url. It is a g-sheet with the location, language and the keywords.
this happened to me earlier, it was just miss typing path\url of the file.
check the path again.
i'm trying to use pytrend to get several keywords from different countries over weekly periods
i don't know why but a much simpler code works, I'm not sure if it's the for loop or something else.
this is the code:
from pytrends.request import TrendReq
import pandas as pd
data = pd.read_csv('data.csv')
pytrends = TrendReq(hl='en-US', tz=360, )
contr = ['US', 'UK', 'IE', 'AU','CA','NZ',
'PE', 'MX','VE', 'CO','AR', 'CL']
en_keyw = [['key1', 'key2', 'key3'],
['key4', 'key5', 'key6'],
['key7', 'key8', 'key9']]
weeks = pd.date_range('2018-12-1',
'2020-06-20',
freq='W'
).strftime('%Y-%m-%d')
for i in range(len(contr)):
for ii in range(9):
for iii in range(len(weeks)):
pytrends.build_payload(kw_list=en_keyw[ii],
cat=0,
timeframe='{} {}'.format(weeks[iii], weeks[iii + 1]),
geo=contr[i],
data = pytrends.interest_over_time(),
gprop='')
data1 = data.drop(labels=['isPartial'],axis='columns')
dataset.append(data1)
result = pd.concat(dataset, axis=1)
result.to_csv('trends.csv')
but I'm getting a 'request' KeyError:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
27 timeframe='{} {}'.format(weeks[iii], weeks[iii + 1]),
28 geo=contr[i],
----> 29 data = pytrends.interest_over_time(),
30 gprop='')
31 data = data.drop(labels=['isPartial'],axis='columns')
in interest_over_time(self)
204 over_time_payload = {
205 # convert to string as requests will mangle
--> 206 'req': json.dumps(self.interest_over_time_widget['request']),
207 'token': self.interest_over_time_widget['token'],
208 'tz': self.tz
KeyError: 'request'
I guess that pytrends.interest_over_time() should be outside of build_payload, i.e.:
pytrends.build_payload(kw_list=en_keyw[ii],
cat=0,
timeframe='{} {}'.format(weeks[iii], weeks[iii + 1]),
geo=contr[i],
data=pytrends.interest_over_time(),
gprop='')
data1 = data.drop(labels=['isPartial'], axis='columns')
dataset.append(data1)
should be something like:
pytrends.build_payload(kw_list=en_keyw[ii],
cat=0,
timeframe='{} {}'.format(weeks[iii], weeks[iii + 1]),
geo=contr[i],
gprop='')
data = pytrends.interest_over_time()
if not data.empty:
data1 = data.drop(labels=['isPartial'], axis='columns')
dataset.append(data1)
The following code tries to extract 10 instagram comments from 10 instagram posts using an Instagram scraper (https://github.com/realsirjoe/instagram-scraper).
The error encountered is a TypeError (NoneType object is not subscriptable).
from igramscraper.instagram import Instagram
from time import sleep
import pandas as pd
import requests
instagram = Instagram()
pepsi = instagram.get_account('pepsi')
pepsi.media = instagram.get_medias("pepsi", 10)
p = [pp.__dict__ for pp in pepsi.media]
df = pd.DataFrame(p)
df['link']
comments_df=pd.DataFrame(columns = ["link","post_id", "comment"])
x1=10;
def get_media_id(url):
req = requests.get('https://api.instagram.com/oembed/?url={}'.format(url))
media_id = req.json()['media_id']
return media_id
def get_posts(link):
id=get_media_id(link)
comment0 = instagram.get_media_comments_by_id(id, 10)
cdf=pd.DataFrame(columns = ["link","post_id", "comment"])
for comment in comment0['comments']:
cdf=cdf.append({"link":link,"post_id":id,"comment":comment.text},ignore_index = True)
return cdf;
for index, row in df.iterrows():
comments_df=comments_df.append(get_posts(row["link"]), ignore_index = True)
comments_df
The problem I encounter is as follows:
TypeError Traceback (most recent call last)
<ipython-input-27-8e72850a113a> in <module>
56
57 for index, row in df.iterrows():
---> 58 comments_df=comments_df.append(get_posts(row["link"]), ignore_index = True)
59
60
<ipython-input-27-8e72850a113a> in get_posts(link)
41 id=get_media_id(link)
42
---> 43 comment0 = instagram.get_media_comments_by_id(id, 10)
44
45 cdf=pd.DataFrame(columns = ["link","post_id", "comment"])
TypeError: 'NoneType' object is not subscriptable
Try printing 'df' before accessing df['link'] to see if 'link' has any information inside.
I am trying to access Salesforce through their API. I am trying to run the preliminary code but I get a NameError which doesn't make any sense to me at the moment.
This is my first time taking the previous developers code and trying to run it. I keep getting an error after error.
l=[]
for p in products:
query = 'SELECT '
for c in columns[:-1]:
query += c+','
if (p=='Sigma Upgrade to Insight'):
query += columns[-1] + ' FROM Implementation__c WHERE Imp_Type__c = \''+p+'\''
else:
query += columns[-1] + ' FROM Implementation__c WHERE Imp_Type__c INCLUDES (\''+p+'\') AND Implementation_Status__c != \'PE Trial Ended\''
df_temp = pd.DataFrame(sf.query(query))
l.append(df_temp)
runthrough = ('nextRecordsUrl' in df_temp.columns)
while runthrough:
df_temp = pd.DataFrame(sf.query_more(nextRecord,True))
l.append(df_temp)
runthrough = ('nextRecordsUrl' in df_temp.columns)
I am expecting to run this without any errors but I keep getting a NameError
NameError Traceback (most recent call last)
<ipython-input-4-7b44b14c79ff> in <module>
14 runthrough = ('nextRecordsUrl' in df_temp.columns)
15 while runthrough:
---> 16 df_temp = pd.DataFrame(sf.query_more(nextRecord,True))
17 l.append(df_temp)
18 runthrough = ('nextRecordsUrl' in df_temp.columns)
NameError: name 'nextRecord' is not defined
You need to set nextRecord to the value of the nextRecordsUrl element of df_temp.
df_temp = pd.DataFrame(sf.query(query))
l.append(df_temp)
while 'nextRecordsUrl' in df_temp.columns:
nextRecord = df_temp['nextRecordsUrl']
df_temp = pd.DataFrame(sf.query_more(nextRecord,True))
l.append(df_temp)
I am using this function to pull data from the Cryptocompare website into a pandas dataframe:
def daily_price_historical(symbol, comparison_symbol='USD', limit=1, aggregate=1, exchange='', allData='true'):
url = 'https://min-api.cryptocompare.com/data/histoday?fsym={}&tsym={}&limit={}&aggregate={}&allData={}'\
.format(symbol.upper(), comparison_symbol.upper(), limit, aggregate, allData)
if exchange:
url += '&e={}'.format(exchange)
page = requests.get(url)
data = page.json()['Data']
df = pd.DataFrame(data)
df['timestamp'] = [datetime.datetime.fromtimestamp(d) for d in df.time]
df.set_index('timestamp', inplace=True)
df['symbol'] = symbol
df['1dret'] = 100* df['close'].pct_change()
return df
This works fine for most symbols I pass in, but when I loop over a longer list of symbols I get the error: AttributeError: 'DataFrame' object has no attribute 'time'
I assume this is due to the API returning an error for certain symbols, e.g.:
https://min-api.cryptocompare.com/data/histoday?fsym=FAKE&tsym=USD
returns "Response":"Error" with no further data
I'm afraid I'm not very experienced with url requests/APIs. Is there code I can add to the function to skip the symbols that are causing the issue?
Thanks for your help!
Additional information:
Code used to loop over coins (which is a list of 130 symbols):
price_columns = ['close', 'high', 'low', 'open', 'time',
'volumefrom','volumeto', 'symbol', '1dret']
top_coin_prices = pd.DataFrame(columns=price_columns)
for coin in coins:
output = daily_price_historical(coin)
top_coin_prices = top_coin_prices.append(output)
Full Traceback:
AttributeError Traceback (most recent call last)
<ipython-input-277-126f5d1686b2> in <module>()
8 # populate df with data for all coins
9 for coin in coins:
---> 10 output = daily_price_historical(coin)
11 top_coin_prices = top_coin_prices.append(output)
12
<ipython-input-111-65b3fa76b4ab> in daily_price_historical(symbol, comparison_symbol, limit, aggregate, exchange, allData)
7 data = page.json()['Data']
8 df = pd.DataFrame(data)
----> 9 df['timestamp'] = [datetime.datetime.fromtimestamp(d) for d in df.time]
10 df.set_index('timestamp', inplace=True)
11 df['symbol'] = symbol
/anaconda/lib/python3.6/site-packages/pandas/core/generic.py in __getattr__(self, name)
2968 if name in self._info_axis:
2969 return self[name]
-> 2970 return object.__getattribute__(self, name)
2971
2972 def __setattr__(self, name, value):
AttributeError: 'DataFrame' object has no attribute 'time'