We Keep Coding

How to retrieve data from SQL Server using pyodbc and Python?

I have a SQL statement that retrieve data from SQL Server using pyodbc package.
The problem is that the query doesn't return all the data instead it return several records.
Example:
CREATE TABLE [dbo].[t1]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[first] [nvarchar](50) NULL,
[last] [nchar](50) NULL,
[Rating] [int] NULL,
CONSTRAINT [PK_t1]
PRIMARY KEY CLUSTERED ([ID] ASC)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
Data:
SQL statement:
advanced_search_term_list =[]
sql = (
r"select ID, first, last, Rating "
r"from dbo.t3 "
r"where "
r"(first LIKE CONCAT('%', ?, '%') OR ? IS NULL) AND "
r"(last LIKE CONCAT('%', ?, '%') OR ? IS NULL) AND "
r"(Rating = ? OR ? IS NULL)"
)
param1 = advanced_search_term_list[0]
param2 = advanced_search_term_list[1]
param3 = advanced_search_term_list[2]
rows = cursor.execute(sql, [param1, param1, param2, param2, param3, param3]).fetchall()
for row in rows:
print(row.ID)
print(row.first)
print(row.last)
print(row.Rating)
If the user runs this SQL statement without any user input, the expected output is all the rows.
But instead, it returns these rows only:

based on the comments of #Larnu i edit the sql statement by adding for the Rating field parameter = 0 like the below line :
r"(Rating = ? OR ? IS NULL OR ? = 0)"

incorrect integer value mySQL

Im receiving an error where I am using an incorrect integer value for userID_fk and target. The error comes up for values which have an integer as their data type and if its changed to text or varchar it will state a site has been created and the siteID will increase but no other data will be included. I want the user to input their username so its matched with its userID and inserted into userID_fk through python with Tkinter.
Below is the structure for my users and sites table
users:
CREATE TABLE `users` (
`userID` int(255) NOT NULL AUTO_INCREMENT,
`userName` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_cs NOT NULL,
`userPassword` varchar(225) CHARACTER SET latin1 COLLATE latin1_general_cs NOT NULL,
`Name` varchar(255) NOT NULL,
`phoneNum` text NOT NULL,
`email` varchar(230) NOT NULL,
`region` text NOT NULL,
`accessLevel` int(10) NOT NULL,
PRIMARY KEY (`userID`)
) ENGINE=InnoDB AUTO_INCREMENT=10002 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT
sites:
CREATE TABLE `sites` (
`siteID` int(225) NOT NULL AUTO_INCREMENT,
`siteName` text CHARACTER SET latin1 COLLATE latin1_general_cs NOT NULL,
`userID_fk` int(255) NOT NULL,
`region` text NOT NULL,
`risklevel` text NOT NULL,
`siteType` text NOT NULL,
`target` int(225) NOT NULL,
PRIMARY KEY (`siteID`),
KEY `userID_fk` (`userID_fk`),
CONSTRAINT `sites_ibfk_1` FOREIGN KEY (`userID_fk`) REFERENCES `users` (`userID`) ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT
Python code to insert a site into the sites table:
def register_site():
sitename_info = sitename2.get()
username2_info = username2.get()
region_info = region.get()
risklevel_info = risklevel.get()
sitetype_info = sitetype.get()
targetpercent_info = targetpercent.get()
# Sql code for writing the data that was written in the regsitering page.
cursor = cnn.cursor()
sitequery = "INSERT INTO `sites`(`siteID`, `siteName`, `userID_fk`, `region`, `risklevel`, `siteType`, `target`) VALUES (NULL,%s,%s,%s,%s,%s,%s)"
sitequery_vals = (sitename_info, username2_info, region_info, risklevel_info, sitetype_info, targetpercent_info)
cursor.execute(sitequery, sitequery_vals)
cnn.commit()
cursor.close()
cnn.close()
# removes the values in the entrys once the user selects that the registration was successful
sitename2_entry.delete(0, END)
region_entry.delete(0, END)
risklevel_entry.delete(0, END)
sitetype_entry.delete(0, END)
targetpercent_entry.delete(0, END)
Label(screen10, text = "Site Created", fg = "green", font = ("calibri", 11)).pack()

If username2_info is the userName, you need to get the userID from the users table:
sitequery = ("INSERT INTO `sites` (`siteName`, `userID_fk`, `region`, `risklevel`, `siteType`, `target`) "
"SELECT %s, `userID`, %s, %s, %s, %s FROM `users` WHERE `userName` = %s")
sitequery_vals = (sitename_info, region_info, risklevel_info, sitetype_info, targetpercent_info, username2_info)
cursor.execute(sitequery, sitequery_vals)
cnn.commit()

How to upsert pandas DataFrame to MySQL with SQLAlchemy

I'm pushing data from a data-frame into MySQL, right now it is only adding new data to the table if the data does not exists(appending). This works perfect, however I also want my code to check if the record already exists then it needs to update. So I need it to append + update. I really don't know how to start fixing this as I got stuck....someone tried this before?
This is my code:
engine = create_engine("mysql+pymysql://{user}:{pw}#localhost/{db}"
.format(user="root",
pw="*****",
db="my_db"))
my_df.to_sql('my_table', con = engine, if_exists = 'append')

You can use next solution on DB side:
First: create table for insert data from Pandas (let call it test):
CREATE TABLE `test` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(100) NOT NULL,
`capacity` INT(11) NOT NULL,
PRIMARY KEY (`id`)
);
Second: Create table for resulting data (let call it cumulative_test) exactly same structure as test:
CREATE TABLE `cumulative_test` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(100) NOT NULL,
`capacity` INT(11) NOT NULL,
PRIMARY KEY (`id`)
);
Third: set trigger on each insert into the test table will insert ore update record in the second table like:
DELIMITER $$
CREATE
/*!50017 DEFINER = 'root'#'localhost' */
TRIGGER `before_test_insert` BEFORE INSERT ON `test`
FOR EACH ROW BEGIN
DECLARE _id INT;
SELECT id INTO _id
FROM `cumulative_test` WHERE `cumulative_test`.`name` = new.name;
IF _id IS NOT NULL THEN
UPDATE cumulative_test
SET `cumulative_test`.`capacity` = `cumulative_test`.`capacity` + new.capacity;
ELSE
INSERT INTO `cumulative_test` (`name`, `capacity`)
VALUES (NEW.name, NEW.capacity);
END IF;
END;
$$
DELIMITER ;
So you will already insert values into the test table and get calculated results in the second table. The logic inside the trigger can be matched for your needs.

Similar to the approach used for PostgreSQL here, you can use INSERT … ON DUPLICATE KEY in MySQL:
with engine.begin() as conn:
# step 0.0 - create test environment
conn.execute(sa.text("DROP TABLE IF EXISTS main_table"))
conn.execute(
sa.text(
"CREATE TABLE main_table (id int primary key, txt varchar(50))"
)
)
conn.execute(
sa.text(
"INSERT INTO main_table (id, txt) VALUES (1, 'row 1 old text')"
)
)
# step 0.1 - create DataFrame to UPSERT
df = pd.DataFrame(
[(2, "new row 2 text"), (1, "row 1 new text")], columns=["id", "txt"]
)
# step 1 - create temporary table and upload DataFrame
conn.execute(
sa.text(
"CREATE TEMPORARY TABLE temp_table (id int primary key, txt varchar(50))"
)
)
df.to_sql("temp_table", conn, index=False, if_exists="append")
# step 2 - merge temp_table into main_table
conn.execute(
sa.text(
"""\
INSERT INTO main_table (id, txt)
SELECT id, txt FROM temp_table
ON DUPLICATE KEY UPDATE txt = VALUES(txt)
"""
)
)
# step 3 - confirm results
result = conn.execute(
sa.text("SELECT * FROM main_table ORDER BY id")
).fetchall()
print(result) # [(1, 'row 1 new text'), (2, 'new row 2 text')]

Inserting date string into SQLite from Python giving unexpected results

As the title states, I'm trying to insert a date (formatted as a string) into a SQLite database. This works, however the date is not showing up correctly in SQLite.
Here is a subset of my code:
print("Connecting to the database...")
sqlite_file = './sqlite_db/cfr_changes.db'
conn = sqlite3.connect(sqlite_file)
c = conn.cursor()
today_date = datetime.now().strftime("%Y-%m-%d")
print(today_date)
print("Inserting tracker into database...")
c.execute("INSERT INTO DATE (`date_id`) VALUES(" + today_date + ")")
c.executemany("INSERT INTO TRACKER (`cfr_status`, `comment`, `mdu`, `iwb`, `obsolete`, `date_id`) VALUES(?,?,?,?,?, " + today_date + ")", list(tracker_df.to_records(index=False)))
#print(c.fetchall())
conn.commit()
conn.close()
Printing 'today_date' returns what I'd expect:
2018-10-24
However when I check the records in SQLite through the terminal, the date is shown as:
1984
Note that 'date_id' is a VARCHAR(255), and date formatting should not be an issue. I would think that is simply a string being stored into a string (or close enough).
Can anyone inform me why this doesn't work as expected?
For reference, here is how the 'TRACKER' and 'DATE' tables were created:
CREATE TABLE `DATE` (
`date_id` VARCHAR(255) NOT NULL PRIMARY KEY);
CREATE TABLE `TRACKER` (
`tracker_id` INTEGER NOT NULL PRIMARY KEY,
`cfr_status` VARCHAR(255) NOT NULL,
`mdu` BOOLEAN, `iwb` BOOLEAN,
`obsolete` BOOLEAN, `comment` VARCHAR(255), `date_id` VARCHAR(255) NOT NULL, FOREIGN KEY (`date_id`) REFERENCES DATE(`date_id`));
Any help is appreciated.

The issue is that you are just concatenating variables into your insert statements without worrying about whether the format they are in makes any sense. For example, to insert a date literal into SQLite you should be using this:
'2018-10-24'
Here is an example of how you may use prepared statements to do a proper insert:
today_date = datetime.now().strftime("%Y-%m-%d")
c.execute("INSERT INTO DATE (date_id) VALUES (?)", ("'" + today_date + "'",))

How to commit MySql table name as variable in python?

I want to create a new mysqldb table for each unique user, but i'm getting errors:
1.'bytes' object has no attribute 'encode'
2.Can't convert 'bytes' object to str implicitly
3.You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''Text'(
id int(11) NOT NULL AUTO_INCREMENT,
UserName text NOT NULL' at line 1
c.execute("CREATE TABLE IF NOT EXISTS {table_name}".format(table_name=belekas), (belekas) + """(
`id` int(11) NOT NULL AUTO_INCREMENT,
`UserName` text NOT NULL,
`Data` date NOT NULL,
`Laikas` time NOT NULL,
`KeyStrokes` text NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8""")
con.commit()
c.execute("INSERT INTO {table_name} VALUES (id, %s, Data, Laikas, %s)".format(table_name=belekas),
(belekas, vartotojas, tekstas))
con.commit()
I tried using:
c.execute("CREATE TABLE IF NOT EXISTS" + vartotojas + """(
and this:
c.execute("CREATE TABLE IF NOT EXISTS" + repr(vartotojas.decode('utf-8')) + """(
and this:
c.execute("CREATE TABLE IF NOT EXISTS {this_table}".format(this_table=vartotojas), (vartotojas.encode("utf-8")) + """(
Can someone suggest solution for this problem?