Python & SQLite Filling Database Function Crashing - python

I created a function to fill a database with API requests values but since I added new columns to the database, that are modified through a web page, I changed INSERT OR REPLACE by INSERT OR UPDATE(Also tried UPSERT) but the function doesn't run anymore.
Since I'm using Django and VSCode, I have no idea how to print things or see an error (this runs in a separate thread so doesn't crash the webpage).
There might be something wrong with the function but I can't see what so I'm hoping someone can help me out.
def fill_resources():
r = pip._vendor.requests.get(BASE_URL+'/api/resources?&maxResults=500',auth=(USER,PASS))
data0 = json.loads(r.text)
conn = sqlite3.connect('/app/docaret.sqlite3')
c = conn.cursor()
for res in data0['data']:
resId = res['id']
r1 = pip._vendor.requests.get(BASE_URL+'/api/resources/'+str(resId)+'/administrative?&maxResults=500',auth=(USER,PASS))
admin = json.loads(r1.text)
r2 = pip._vendor.requests.get(BASE_URL+'/api/resources/'+str(resId)+'/information?&maxResults=500',auth=(USER,PASS))
info = json.loads(r2.text)
r3 = pip._vendor.requests.get(BASE_URL+'/api/resources/'+str(resId)+'/technical-data?&maxResults=500',auth=(USER,PASS))
tech = json.loads(r3.text)
if res['relationships']['mainManager']['data'] == None:
mainManager = 0
else:
mainManager = res['relationships']['mainManager']['data']['id']
if tech['data']['attributes']['diplomas'] != None:
diplomas = tech['data']['attributes']['diplomas']
else:
diplomas = ''
dipText = ''
for dip in diplomas:
newdip = dip + '#µ§'
dipText += newdip
collab = {
"BoondID": resId,
"lastName": encrypt(res['attributes']['lastName']),
"firstName": encrypt(res['attributes']['firstName']),
"dateOfBirth": encrypt(admin['data']['attributes']['dateOfBirth']),
"placeOfBirth": encrypt(admin['data']['attributes']['placeOfBirth']),
"address": encrypt(info['data']['attributes']['address']),
"postcode": encrypt(info['data']['attributes']['postcode']),
"town": encrypt(info['data']['attributes']['town']),
"country": encrypt(info['data']['attributes']['country']),
"email1": encrypt(res['attributes']['email1']),
"email2": encrypt(info['data']['attributes']['email2']),
"phone1": encrypt(res['attributes']['phone1']),
"phone2": encrypt(res['attributes']['phone2']),
"administrativeComments": encrypt(admin['data']['attributes']['administrativeComments']),
"title": encrypt(res['attributes']['title']),
"diplomas": dipText,
"mainManager": mainManager,
"agency": res['relationships']['agency']['data']['id'],
"healthCareNumber": encrypt(admin['data']['attributes']['healthCareNumber']),
"state": info['data']['attributes']['state']
}
values = (collab['BoondID'],collab['lastName'],collab['firstName'],collab['dateOfBirth'],collab['placeOfBirth'],collab['address'],collab['postcode'],collab['town'],collab['country'],collab['email1'],collab['email2'],collab['phone1'],collab['phone2'],collab['administrativeComments'],collab['title'],collab['mainManager'],collab['agency'],collab['healthCareNumber'],collab['diplomas'],collab['state'])
#collabList.append(collab)
c.execute("INSERT OR UPDATE INTO RESOURCES (BoondID,lastName,firstName,dateOfBirth,placeOfBirth,address,postcode,town,country,email1,email2,phone1,phone2,administrativeComments,title,mainManager,agency,healthCareNumber,diplomas,state) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", values)
conn.commit()
conn.close()
I get the following error when running the function independently:
Traceback (most recent call last):
File "<ipython-input-12-87021414c9f9>", line 1, in <module>
fill_resources()
File "C:/Users/valen/OneDrive/Bureau/DOCARET/tous les tests/testcontracts.py", line 63, in fill_resources
c.execute("INSERT OR UPDATE INTO RESOURCES (BoondID,lastName,firstName,dateOfBirth,placeOfBirth,address,postcode,town,country,email1,email2,phone1,phone2,administrativeComments,title,mainManager,agency,healthCareNumber,diplomas,state) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", values)
OperationalError: near "UPDATE": syntax error

SQLite does not support INSERT OR UPDATE INTO....
You can use UPSERT if your version of SQLite is 3.24.0+.
Assuming that BoondID is the PRIMARY KEY of the table or there is a unique constraint defined for it:
sql = """
INSERT INTO RESOURCES (
BoondID, lastName, firstName, dateOfBirth, placeOfBirth, address,
postcode, town, country, email1, email2, phone1, phone2,
administrativeComments, title, mainManager, agency, healthCareNumber,
diplomas, state
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(BoondID) DO UPDATE SET
(lastName, firstName, dateOfBirth, placeOfBirth, address, postcode, town,
country, email1, email2, phone1, phone2, administrativeComments, title,
mainManager, agency, healthCareNumber, diplomas, state) =
(EXCLUDED.lastName, EXCLUDED.firstName, EXCLUDED.dateOfBirth,
EXCLUDED.placeOfBirth, EXCLUDED.address, EXCLUDED.postcode,
EXCLUDED.town, EXCLUDED.country, EXCLUDED.email1, EXCLUDED.email2,
EXCLUDED.phone1, EXCLUDED.phone2, EXCLUDED.administrativeComments,
EXCLUDED.title, EXCLUDED.mainManager, EXCLUDED.agency,
EXCLUDED.healthCareNumber, EXCLUDED.diplomas, EXCLUDED.state);
"""
c.execute(sql, values)

Related

sqlite3 in python doesn't save more than row

i am using sqlite3 in python and whenever i insert data it replace it with the data in the DB instead of adding rows
i couldn't have more than one row..
--
def success(self):
global cursor, conn
wallet = random.randint(1000000000, 9999999999)
if self.checkID:
query = [self.fNEntry.get(), self.lNEntry.get(), self.idEntry.get(), self.pwEntry.get(), self.emEntry.get(),
self.pNEntry.get(), wallet, 1000]
cursor = conn.execute('INSERT INTO STUDENTS(FNAME, LNAME, ID, PASSWORD, EMAIL, PHONE, WALLET, BALANCE) \
VALUES(?, ?, ?, ?, ?, ?, ?, ?);', query)
conn.commit()

Problem while trying to insert multiple values to Sqlite database

I have to make a request to a Brazil ZIPCODES API to get JSON data and insert it on a sqlite database using python. I'm currenctly using pycharm but I need to insert a lot of columns, but somehow the code don't insert the values. Here's the code
import requests
import sqlite3
import json
CEPC = input("Please type the zipcode:")
print("Identifying the ZIP CODE")
Requisicao = requests.get(f"https://viacep.com.br/ws/{CEPC}/json")
if Requisicao.status_code == 200:
data = Requisicao.json()
# Database
con = sqlite3.connect("Banco de dados/CEPS.db")
cur = con.cursor()
cur.execute("DROP TABLE IF EXISTS Requisicao")
cur.execute("CREATE TABLE Requisicao (cep, logradouro, bairro, uf, ddd, siafi,
validation, created json)")
cur.executemany("insert into Requisicao values (?, ?, ?, ?, ?, ?, ?, ?)", (data["cep"],
json.dumps(data)))
con.commit()
con.close()
else:
print(f"Request failed with status code {Requisicao.status_code} ")
The outpout of the zipcode is:
{
"cep": "05565-000",
"logradouro": "Avenida General Asdrúbal da Cunha",
"complemento": "",
"bairro": "Jardim Arpoador",
"localidade": "São Paulo",
"uf": "SP",
"ibge": "3550308",
"gia": "1004",
"ddd": "11",
"siafi": "7107"
}
I need to insert all of these columns: "cep, logadouro, complemento, bairro, localidade, uf, ibge, gia, ddd, siafi".When I try to run the code, It gives me the error:
Traceback (most recent call last):
File "C:\Users\Gui\PycharmProjects\pythonProject\main.py", line 19, in <module>
cur.executemany("insert into Requisicao values (?, ?, ?, ?, ?, ?, ?, ?)", (data["cep"],
json.dumps(data)))
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement
uses 8, and there are 9 supplied
When I try to put the exact same value of columns with the "?", the errors says that "uses 8, and there are 7 supplied.
This code will insert all 10 values from the JSON into the table Requisicao and 0 for both validation and created, though that can be changed.
import requests
import sqlite3
import json
CEPC = input("Please type the zipcode:")
print("Identifying the ZIP CODE")
Requisicao = requests.get(f"https://viacep.com.br/ws/{CEPC}/json")
if Requisicao.status_code == 200:
data = Requisicao.json()
# Database
con = sqlite3.connect("CEPS.db")
cur = con.cursor()
cur.execute("DROP TABLE IF EXISTS Requisicao")
cur.execute("CREATE TABLE Requisicao (cep,logradouro,complemento,bairro,localidade,uf,ibge,gia,ddd,siafi, validation, created)")
cur.execute("insert into Requisicao values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",tuple(data.values())+(0, 0))
con.commit()
con.close()
else:
print(f"Request failed with status code {Requisicao.status_code} ")

Returning "KeyError: " in Python when trying to populate a SQlite database

I am currently getting the following error:
Traceback (most recent call last):
File "/Users/Stephen/Desktop/projects/Option_History/snapshot.py", line 44, in <module>
data = (current_timestamp, option['underlying'], option['symbol'], option['description'], option['strike'], option['bid'], option['ask'], option['volume'], option['greeks']['delta'], option['greeks']['gamma'], option['greeks']['theta'], option['greeks']['vega'], option['greeks']['rho'], option['greeks']['phi'], option['greeks']['mid_iv'], option['greeks']['smv_vol'])
KeyError: 'greeks'
My code is shown below:
import config, requests, pprint, sqlite3
from datetime import datetime
connection = sqlite3.connect('option_history.db')
cursor = connection.cursor()
try:
cursor.execute("""
CREATE TABLE option_history (
timestamp text,
underlying text,
symbol text,
description text,
strike real,
bid real,
ask real,
volume real,
delta real,
gamma real,
theta real,
vega real,
rho real,
phi real,
mid_iv real,
smv_vol real
)
""")
except:
pass
response = requests.get(config.OPTION_CHAIN_URL,
params={'symbol': 'SPY', 'expiration': '2020-12-04', 'greeks': 'true'},
headers=config.HEADERS
)
json_response = response.json()
options = json_response['options']['option']
current_timestamp = datetime.now().replace(second=0, microsecond=0).isoformat()
print(options)
for option in options:
data = (current_timestamp, option['underlying'], option['symbol'], option['description'], option['strike'], option['bid'], option['ask'], option['volume'], option['greeks']['delta'], option['greeks']['gamma'], option['greeks']['theta'], option['greeks']['vega'], option['greeks']['rho'], option['greeks']['phi'], option['greeks']['mid_iv'], option['greeks']['smv_vol'])
print(",".join(map(str, data)))
cursor.execute("""
INSERT INTO option_history (
timestamp, underlying, symbol, description, strike, bid, ask, volume, delta, gamma, theta, vega, rho, phi, mid_iv, smv_vol
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", data)
connection.commit()
connection.close()
I was running this code for an entire day every 5 minutes using cron and it properly inserted all the data I needed with no errors. Now when attempting to run it again today using crontab, the DB was not populated with any new data.
The interesting part is that even though I am getting the KeyError message, the data variable is still populated properly because the
print(",".join(map(str, data))
line is updating properly with a new API call of data and printing to the console without issue.

Writing Csv file to already existing table in SQL Server database using Python

I was trying to insert the CSV file to an already existing table in the SSMS database table. I have a data column in my data. But I keep getting this error when I try to insert data. Please tell me where am I doing it wrong because server connection and extracting data from the database are fine. Below is the code.
with open("combine.csv", encoding="utf8") as f:
csvreader = csv.reader(f)
csvdata = []
for row in csvreader:
csvdata.append(row)
print(csvdata)
for row in csvdata:
# Insert a row of data
print(row)
if len(row)>=8:
data = [row[0],row[1],row[2],row[3],row[4],row[5],row[6],row[7]]
cursor.execute("INSERT INTO BILLING_COPY (DATE, DEPARTMENT_NUMBER, DEPARTMENT_NAME, DIVISION_CODE, DIVISION_NAME, O_T_AMT, R_AMT, U_AMT ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", data)
Error:
File "", line 7, in
cursor.execute("INSERT INTO BILLING_COPY (DATE, DEPARTMENT_NUMBER, DEPARTMENT_NAME, DIVISION_CODE, DIVISION_NAME, O_T_AMT, R_AMT, U_AMT ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", data)
DataError: ('22007', '[22007] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Conversion failed when converting date and/or time from character string. (241) (SQLExecDirectW)')
File "", line 7, in
cursor.execute("INSERT INTO BILLING_COPY (DATE, DEPARTMENT_NUMBER, DEPARTMENT_NAME, DIVISION_CODE, DIVISION_NAME, O_T_AMT, R_AMT, U_AMT ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", data)
I think the data type you mentioned in VALUES(?,?,? etc) is not right valid data type, try using it as %d or %s
Here is some example:
mySql_insert_query = """INSERT INTO Laptop (Id, Name, Price, Purchase_date)
VALUES
(10, 'ProductValues SP99', 6459, '2019-12-27') """
cursor = connection.cursor()
cursor.execute(mySql_insert_query)
connection.commit()
My two cents: Better to assign insert query to a variable just like data variable.

SQLite3 help, saving variables into a databse

I have 2 things I needed help with:
1) I am unsure as to how I can check if a table exists in python using the sqlite3 library.
2) I am unsure as to how I can save variables from the program to a database. I want to be able to check if UserDetails exists before making the database.
I've been reading around and everyone is doing stuff differently,
Here is the section of my code that is responsible for saving the variables:
connection = sqlite3.connect("UserDetails.db")
crsr = connection.cursor()
#create table
sql_command = table_creator
crsr.execute(sql_command)
#insert values into table
data_to_insert = (username, first_name, surname, age, user_salt, password_hash, date_today)
sql_command = """INSERT INTO UserDetails VALUES ((?, ?, ?, ?, ?, ?, ?), data_to_insert);"""
crsr.execute(sql_command)
connection.commit() #save changes
connection.close() #terminate connection
and in case you want to see table_creator it looks like this:
table_creator = '''CREATE TABLE `UserDetails` (
`Username` VARCHAR(8) NOT NULL,
`Firstname` VARCHAR(10) NOT NULL,
`Surname` VARCHAR(20) NOT NULL,
`Age` INT(2) NOT NULL,
`Salt` VARCHAR(10) NOT NULL,
`Hash` VARCHAR(64) NOT NULL,
`Date` DATE NOT NULL,
PRIMARY KEY (`UserName`)
);'''
I will appreciate and feedback or support.
I am still learning to code, and my CompSci teacher doesnt teach us Python specifically, so what I know is self taught.
Oh and this is the error message I get:
Traceback (most recent call)
File "c:/Users/Arslan/Project A2/login.py", line 99, in <module>
save_details()
File "c:/Users/Arslan/Project A2/login.py", line 93, in save_details
crsr.execute(sql_command)
sqlite3.OperationalError: no such column: data_to_insert
How to check if a table exists or no :
The first way :
Use this query:
SELECT name FROM sqlite_master WHERE type='table' AND name='{table_name}';
Modify {table_name} with your table to check
There are two cases :
. If the cursor equal to 0 ==> the table does not exist
Else, the table exists
The second way:
Use :
PRAGMA table_info(table_name)
example:
The third way :
Use this query :
select 1 from table
It will return the constant 1 for every row of the table if the table exists, or nothing if not.
There are many other ways, but I listed the best in my opinion.
How to save variables from the program to a database:
To insert data into sqlite3, you can use :
cursor.execute("insert into UserDetails values (?, ?, ?, ?, ?, ?, ?)", (username, firstname, surname, age, salt, hash, date))
DON'T USE (SQL injection):
cursor.execute("insert into UserDetails values ('{1}', '{2}', '{3}', '{4}', '{5}', '{6}', '{7}')".format(username, firstname, surname, age, salt, hash, date))
Don't forget :
conn.commit()
Or you can use instead of it the connection as a context manager:
with conn:
# then cursor.execute..
1) I am unsure as to how I can check if a table exists in python using the sqlite3 library.
Use CREATE TABLE IF NOT EXISTS:
table_creator = '''CREATE TABLE IF NOT EXISTS `UserDetails` (
`Username` VARCHAR(8) NOT NULL,
`Firstname` VARCHAR(10) NOT NULL,
...
);'''
2) I am unsure as to how I can save variables from the program to a database.
You can pass variables for insert with the following syntax:
data_to_insert = (username, first_name, surname, age, user_salt, password_hash, date_today)
sql_command = '''INSERT INTO UserDetails VALUES (?, ?, ?, ?, ?, ?, ?)''';
crsr.execute(sql_command, data_to_insert )

Categories