I trying to grab one-year's worth of currency data from the TraderMade Python SDK and store that data in a SQLite database. I have created the DB already with the column headings I need (date, open, high, low, close).
I have written the following code to get and store the data:
conn = db.connect("MarketData.db")
c = conn.cursor()
def data():
tm.set_rest_api_key([MY API KEY])
request = tm.timeseries(
currency='EURUSD',
start="2022-01-01",
end="2022-12-31",
interval="daily",
fields=["open", "high", "low", "close"]
)
print(request)
stmt = '''INSERT INTO eurusd VALUES (?,?,?,?,?), (date, open, high, low, close)'''
c.executemany(stmt, request)
return
data()
conn.commit()
print('complete')
The data is coming from TraderMade fine as 'request' is printing fine:
enter image description here
However, I am getting the following error when I run the code:
enter image description here
I have a date column in my DB. Does anyone know what is wrong with my code? My only thought is that the data is not being pulled in the correct format to store - but I'm not sure why as when 'request' prints it has 5 columns all with the correct column headings.
The column names should come before the values clause:
stmt = '''INSERT INTO eurusd (date, open, high, low, close) VALUES (?,?,?,?,?)'''
Related
I am trying to write result into Oracle Database using executemany command. However, I am neither getting any error message nor the database table is getting updated.
Using same connection object and cursor I am able to connect to database, extract data and insert the data into new table. However, I am NOT able to update the result to a existing table
I am using Oracle 19c database , python 3.8.8 and cx_oracle 8.0.0.
Reading the file from Oracle database table 'bench_project_d'. To reproduce the error I have created a csv file and reading the data from csv file.
The data has 7 fields
ROW_ID type: NUMBER(19,0),
GROUP_ID type: NUMBER(19,0),
PLANNED_UNITS type: NUMBER,
IQR_PTU(Calculated on the fly),
Q1_PTU (Calculated on the fly) ,
Q2_PTU (Calculated on the fly) ,
ANOMALY type: NUMBER
All the fields are having data except the new column "ANOMALY".
In this field , all are null values. This is a field where we want to store the results.
While importing the data to python, we are taking first 6 feature, calculate the anomaly field and push the anomaly result to database
'''python
#connecting to Database
username="**********"
password="********"
tsn="*********"
conn = cx_Oracle.connect(username, password, tsn)
cur = conn.cursor()
global_df = pd.read_csv("project_data.csv")
#Filtering the group having more than 3 projects
grouped = global_df.groupby("GROUP_ID")
filtered_df = grouped.filter(lambda x: x["GROUP_ID"].count()>3)
filtered_df
#Handling zero Interquartile Range
x = filtered_df[filtered_df['IQR_PTU'] == 0]['GROUP_ID'].unique()
for i in x:
filtered_df.loc[filtered_df['GROUP_ID'] == i,'IQR_PTU'] = (filtered_df[filtered_df['GROUP_ID']==i]['PLANNED_UNITS'].quantile(0.95)) - (filtered_df[filtered_df['GROUP_ID']==i]['PLANNED_UNITS'].quantile(0.05))
#Creating the output 'Anomaly' field
filtered_df['ANOMALY'] =0
filtered_df.loc[filtered_df['PLANNED_UNITS'] > (filtered_df['Q2_PTU']+(1.5*filtered_df['IQR_PTU'])),'ANOMALY']=1
filtered_df.loc[filtered_df['PLANNED_UNITS'] < (filtered_df['Q1_PTU']-(1.5*filtered_df['IQR_PTU'])),'ANOMALY']=-1
#Formating the Dataframe
result_df = df.loc[:,['ROW_ID','GROUP_ID', 'ANOMALY']]
result_df = result_df.round(2)
result_df=result_df.astype('object')
value_to_stre_db=result_df.values.tolist()
#Pushing the result to Database table bench_project_d
statement = 'update bench_project_d set GROUP_ID = :2, ANOMALY= :3 where ROW_ID = :1'
cur.executemany(statement, value_to_stre_db)
conn.commit()
EDIT 1:
I have tried to convert list of array to list of tuples and executed the same code again. But still no luck.
rows = [tuple(x) for x in value_to_stre_db]
#Pushing the result to Database table bench_project_d
statement = ''update bench_project_d set GROUP_ID = :2, ANOMALY= :3 where ROW_ID = :1''
cur.executemany(statement, rows)
conn.commit()
I am trying to create a database with three columns, URL which is the location of the data I am aiming to scrape, STATUS which is the ticker symbol of the stock, and STATUS which will be used to inform whether the data has been acquired yet, or not.
import sqlite3
conn = sqlite3.connect('tickers.db')
conn.execute('''CREATE TABLE TAB(URL, TICKER, STATUS default "Not started");''')
for i in url_list:
conn.execute("INSERT INTO TAB(URL) VALUES(?)",(i,))
for j in ticklist:
conn.execute("INSERT INTO TAB(TICKER) VALUES(?)",(j,))
for row in conn.execute("SELECT URL, TICKER, STATUS from TAB"):
print('URL={i}'.format(i=row[0]))
print('TICKER={i}'.format(i=row[1]))
print('STATUS={i}'.format(i=row[2]))
To populate the URL column I have used a list of URL's, similarly I am trying to the same thing with TICKER, however when I run the code, the column is only populated with 'none' for all rows.
Output
URL=https://api.pushshift.io/reddit/search/submission/?q=$AACG&subreddit=wallstreetbets&metadata=true&size=0&after=1610928000&before=1613088000
TICKER=None
STATUS=Not started
URL=https://api.pushshift.io/reddit/search/submission/?q=$AACIU&subreddit=wallstreetbets&metadata=true&size=0&after=1610928000&before=1613088000
TICKER=None
STATUS=Not started
Instead of trying to populate the columns, why not insert as rows directly?
Assuming url_list and ticklist are of equal length (and even if not) you can Try this:
for i, j in zip(url_list,ticklist):
conn.execute("INSERT INTO TAB(URL, TICKER) VALUES(?,?)",(i,j))
That way you are adding the values as expected and not creating new rows with every insert
i am a newbees for programming, i have an db file with some date, open, high, low , close data in it, and name with 0001.HK; 0002.HK; 0003.HK
then i try to build a loop to take out some data in the database.
conn = sqlite3.connect(os.path.join('data', "hkprice.db"))
def read_price(stock_id):
connect = 'select Date, Open, High, Low, Close, Volume from ' + stock_id
df = pd.read_sql(connect, conn,index_col=['Date'], parse_dates=['Date'])
for y in range(1 ,2):
read_price(str(y).zfill(4) + '.HK')
when it output it show: Execution failed on sql 'select Date, Open, High, Low, Close, Volume from 0001.HK': unrecognized token: "0001.HK"
but i should have the 0001.HK table in the database
what should i do?
If you want to use variables with a query, you need to put a placeholder ?. So in your particular case:
connect = 'select Date, Open, High, Low, Close, Volume from ?'
After that in read_sql you can provide a list of your variables to the params kwarg like so:
df = pd.read_sql(connect, conn, params=[stock_id], index_col=['Date'], parse_dates=['Date'])
If you have multiple parameters and, hence, multiple ? placeholders then when you supply the list of variables to params they need to be in exactly the same order as your ?.
EDIT:
For example if I had a query where I wanted to get data between some dates, this is how I would do it:
start = ['list of dates']
end = ['another list of dates']
query = """select *
from table
where start_date >= ? and
end_date < ?
"""
df = pd.read_sql_query(query, conn, params=[start, end])
Here interpreter will see the first ? and grab the first item from the first list, then when it gets to the second ? it will grab the first item from the second list. If there's a mismatch between the number of ? and the number of supplied params then it will throw an error.
I am trying to insert stock market csv data that I downloaded from Yahoo finance into a mysql table named 'TEST' that is in
a database named 'stocks' but I am getting an error code from python:
InternalError: (1292, "incorrect date value: 'Date for column 'date at
row 1")
the data that I am trying to insert has hundreads of rows that look something like this:
1995-03-31,0.141150,0.141150,0.141150,0.141150,0.105375,10000
the table that i am trying to insert this data into contains the following columns:
date DATE NOT NULL PRIMARY KEY,
open DECIMAL(10,6),
high DECIMAL(10,6),
low DECIMAL(10,6),
close DECIMAL(10,6),
adj_close DECIMAL(10,6),
volume INT,
this is the python code that i have used to insert the data into the table
with open('/home/matt/Desktop/python_projects/csv_files/CH8_SG.csv',
'r') as f:
reader = csv.reader(f)
data = next(reader)
query = 'insert into TEST values (%s,%s,%s,%s,%s, %s, %s)'
query = query.format(','.split('%s' * len(data)))
cursor = connection.cursor()
cursor.execute(query, data)
for data in reader:
cursor.execute(query, data)
cursor.commit()
when i run the code pictured above I get the following error
InternalError: (1292, "incorrect date value: 'Date for column 'date at row 1")
I really think that I am close but I do not know what is going on with that error.
Can anyone help me?
When you write the query to insert date time in Python, you should use convert operation in MySQL. In your case:
query = 'insert into TEST values (%s,%s,%s,%s,%s, %s, %s)'
=> 'insert into TEST values (STR_TO_DATE(%s),%s,%s,%s,%s, %s, %s)'
And make sure that there are no duplicate in "date" field.
I am new to databases. I am running following query in python which is successfully creating and inserting values (using data_entry method) into sqlite database from csv files. But in year column there are no values but 'BLOB' is showing up. I have tried the data type with INTEGER and TEXT both but still 'BLOB' is showing up. What I am doing wrong here ? Thanks.
cursor.execute('CREATE TABLE IF NOT EXISTS GII(year TEXT, series_index TEXT, series TEXT, rank INTEGER, country TEXT, score REAL, value TEXT )')
def data_entry(i):
year_i = year[i]
index_i = index[i]
series_i = series[i]
rank_i = rank[i]
country_i = country[i]
score_i = score[i]
value_i = value[i]
cursor.execute('INSERT INTO GII (year, series_index, series, rank, country, score, value) VALUES (?,?,?,?,?,?,?)',
(year_i,index_i,series_i,rank_i, country_i, score_i, value_i))
conn.commit()