I have a SQL statement that retrieve data from SQL Server using pyodbc package.
The problem is that the query doesn't return all the data instead it return several records.
Example:
CREATE TABLE [dbo].[t1]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[first] [nvarchar](50) NULL,
[last] [nchar](50) NULL,
[Rating] [int] NULL,
CONSTRAINT [PK_t1]
PRIMARY KEY CLUSTERED ([ID] ASC)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
Data:
SQL statement:
advanced_search_term_list =[]
sql = (
r"select ID, first, last, Rating "
r"from dbo.t3 "
r"where "
r"(first LIKE CONCAT('%', ?, '%') OR ? IS NULL) AND "
r"(last LIKE CONCAT('%', ?, '%') OR ? IS NULL) AND "
r"(Rating = ? OR ? IS NULL)"
)
param1 = advanced_search_term_list[0]
param2 = advanced_search_term_list[1]
param3 = advanced_search_term_list[2]
rows = cursor.execute(sql, [param1, param1, param2, param2, param3, param3]).fetchall()
for row in rows:
print(row.ID)
print(row.first)
print(row.last)
print(row.Rating)
If the user runs this SQL statement without any user input, the expected output is all the rows.
But instead, it returns these rows only:
based on the comments of #Larnu i edit the sql statement by adding for the Rating field parameter = 0 like the below line :
r"(Rating = ? OR ? IS NULL OR ? = 0)"
Related
Error Message
You have an error in your SQL syntax; check the manual that
corresponds to your MariaDB server version for the right syntax to use
near '%s' at line 1
MySQL Database Table
CREATE TABLE `tblorders` (
`order_id` int(11) NOT NULL,
`order_date` date NOT NULL,
`order_number` varchar(50) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
ALTER TABLE `tblorders`
ADD PRIMARY KEY (`order_id`),
ADD UNIQUE KEY `order_number` (`order_number`);
ALTER TABLE `tblorders`
MODIFY `order_id` int(11) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=4;
Code
mydb = mysql.connector.connect(host = "localhost", user = "root", password = "", database = "mydb")
mycursor = mydb.cursor()
sql = "Select order_id from tblorders where order_number=%s"
val = ("1221212")
mycursor.execute(sql, val)
Am I missing anything?
You must pass a list or a tuple as the arguments, but a tuple of a single value is just a scalar in parentheses.
Here are some workarounds to ensure that val is interpreted as a tuple or a list:
sql = "Select order_id from tblorders where order_number=%s"
val = ("1221212",)
mycursor.execute(sql, val)
sql = "Select order_id from tblorders where order_number=%s"
val = ["1221212"]
mycursor.execute(sql, val)
This is a thing about Python that I always find weird, but it makes a kind of sense.
In case you want to insert data you have to modify your SQL. Use INSERT instead of SELECT like this:
INSERT INTO tblorders (order_number) VALUES ("122121");
That statement will add new record to the table. Besides, in MariaDB you need to use ? instead of %s that works on Mysql database.
sql = "INSERT INTO tblorders (order_number) VALUES (?);"
val = "1231231"
mycursor.execute(sql, [val])
So I'm using sqlalchemy for a project I'm working on. I've got an issue where I will eventually have thousands of records that need to be saved every hour. These records may be inserted or updated. I've been using bulk_save_objects for this and it's worked great. However now I have to introduce a history to these records being saved, which means I need the IDs returned so I can link these entries to an entry in a history table. I know about using return_defaults, and that works. However, it introduces a problem that my bulk_save_objects inserts and updates one entry at a time, instead of in bulk, which removes the purpose. Is there another option, where I can bulk insert and update at the same time, but retain the IDs?
The desired result can be achieved using a technique similar to the one described in the answer here by uploading the rows to a temporary table and then performing an UPDATE followed by an INSERT that returns the inserted ID values. For SQL Server, that would be an OUTPUT clause on the INSERT statement:
main_table = "team"
# <set up test environment>
with engine.begin() as conn:
conn.execute(sa.text(f"DROP TABLE IF EXISTS [{main_table}]"))
conn.execute(
sa.text(
f"""
CREATE TABLE [dbo].[{main_table}](
[id] [int] IDENTITY(1,1) NOT NULL,
[prov] [varchar](2) NOT NULL,
[city] [varchar](50) NOT NULL,
[name] [varchar](50) NOT NULL,
[comments] [varchar](max) NULL,
CONSTRAINT [PK_team] PRIMARY KEY CLUSTERED
(
[id] ASC
)
)
"""
)
)
conn.execute(
sa.text(
f"""
CREATE UNIQUE NONCLUSTERED INDEX [UX_team_prov_city] ON [dbo].[{main_table}]
(
[prov] ASC,
[city] ASC
)
"""
)
)
conn.execute(
sa.text(
f"""
INSERT INTO [{main_table}] ([prov], [city], [name])
VALUES ('AB', 'Calgary', 'Flames')
"""
)
)
# <data for upsert>
df = pd.DataFrame(
[
("AB", "Calgary", "Flames", "hard-working, handsome lads"),
("AB", "Edmonton", "Oilers", "ruffians and scalawags"),
],
columns=["prov", "city", "name", "comments"],
)
# <perform upsert, returning IDs>
temp_table = "#so65525098"
with engine.begin() as conn:
df.to_sql(temp_table, conn, index=False, if_exists="replace")
conn.execute(
sa.text(
f"""
UPDATE main SET main.name = temp.name,
main.comments = temp.comments
FROM [{main_table}] main INNER JOIN [{temp_table}] temp
ON main.prov = temp.prov AND main.city = temp.city
"""
)
)
inserted = conn.execute(
sa.text(
f"""
INSERT INTO [{main_table}] (prov, city, name, comments)
OUTPUT INSERTED.prov, INSERTED.city, INSERTED.id
SELECT prov, city, name, comments FROM [{temp_table}] temp
WHERE NOT EXISTS (
SELECT * FROM [{main_table}] main
WHERE main.prov = temp.prov AND main.city = temp.city
)
"""
)
).fetchall()
print(inserted)
"""console output:
[('AB', 'Edmonton', 2)]
"""
# <check results>
with engine.begin() as conn:
pprint(conn.execute(sa.text(f"SELECT * FROM {main_table}")).fetchall())
"""console output:
[(1, 'AB', 'Calgary', 'Flames', 'hard-working, handsome lads'),
(2, 'AB', 'Edmonton', 'Oilers', 'ruffians and scalawags')]
"""
I'm pushing data from a data-frame into MySQL, right now it is only adding new data to the table if the data does not exists(appending). This works perfect, however I also want my code to check if the record already exists then it needs to update. So I need it to append + update. I really don't know how to start fixing this as I got stuck....someone tried this before?
This is my code:
engine = create_engine("mysql+pymysql://{user}:{pw}#localhost/{db}"
.format(user="root",
pw="*****",
db="my_db"))
my_df.to_sql('my_table', con = engine, if_exists = 'append')
You can use next solution on DB side:
First: create table for insert data from Pandas (let call it test):
CREATE TABLE `test` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(100) NOT NULL,
`capacity` INT(11) NOT NULL,
PRIMARY KEY (`id`)
);
Second: Create table for resulting data (let call it cumulative_test) exactly same structure as test:
CREATE TABLE `cumulative_test` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(100) NOT NULL,
`capacity` INT(11) NOT NULL,
PRIMARY KEY (`id`)
);
Third: set trigger on each insert into the test table will insert ore update record in the second table like:
DELIMITER $$
CREATE
/*!50017 DEFINER = 'root'#'localhost' */
TRIGGER `before_test_insert` BEFORE INSERT ON `test`
FOR EACH ROW BEGIN
DECLARE _id INT;
SELECT id INTO _id
FROM `cumulative_test` WHERE `cumulative_test`.`name` = new.name;
IF _id IS NOT NULL THEN
UPDATE cumulative_test
SET `cumulative_test`.`capacity` = `cumulative_test`.`capacity` + new.capacity;
ELSE
INSERT INTO `cumulative_test` (`name`, `capacity`)
VALUES (NEW.name, NEW.capacity);
END IF;
END;
$$
DELIMITER ;
So you will already insert values into the test table and get calculated results in the second table. The logic inside the trigger can be matched for your needs.
Similar to the approach used for PostgreSQL here, you can use INSERT … ON DUPLICATE KEY in MySQL:
with engine.begin() as conn:
# step 0.0 - create test environment
conn.execute(sa.text("DROP TABLE IF EXISTS main_table"))
conn.execute(
sa.text(
"CREATE TABLE main_table (id int primary key, txt varchar(50))"
)
)
conn.execute(
sa.text(
"INSERT INTO main_table (id, txt) VALUES (1, 'row 1 old text')"
)
)
# step 0.1 - create DataFrame to UPSERT
df = pd.DataFrame(
[(2, "new row 2 text"), (1, "row 1 new text")], columns=["id", "txt"]
)
# step 1 - create temporary table and upload DataFrame
conn.execute(
sa.text(
"CREATE TEMPORARY TABLE temp_table (id int primary key, txt varchar(50))"
)
)
df.to_sql("temp_table", conn, index=False, if_exists="append")
# step 2 - merge temp_table into main_table
conn.execute(
sa.text(
"""\
INSERT INTO main_table (id, txt)
SELECT id, txt FROM temp_table
ON DUPLICATE KEY UPDATE txt = VALUES(txt)
"""
)
)
# step 3 - confirm results
result = conn.execute(
sa.text("SELECT * FROM main_table ORDER BY id")
).fetchall()
print(result) # [(1, 'row 1 new text'), (2, 'new row 2 text')]
I have been trying to set the ObjectID as Identity column using python on SQL. The below SQL statment works on Management Studio (SSMS) and sets the ObjectID as Identity column. The code works, but when checking the table on SSMS, I don't see ObjectID as Identity column.
The following ways work on python but do not change the identity column on SSMS.
Adding conn.commit() after each execution.
Running the .sql file with python file reader.
Shown in the image, the Identity column is still empty after code execution in python.
The code was generated by SSMS, and my purpose is to SET ObjectID field Identity column. Maybe there is a better way.
Here is the code:
newTableName = "A_Test_DashAutomation"
try:
cursor.execute("""
BEGIN TRANSACTION
SET QUOTED_IDENTIFIER ON
SET ARITHABORT ON
SET NUMERIC_ROUNDABORT OFF
SET CONCAT_NULL_YIELDS_NULL ON
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
COMMIT
BEGIN TRANSACTION """)
cursor.execute("""
CREATE TABLE dbo.Tmp_""" + newTableName + """
(
SubProjectTempId bigint NULL,
CIPNumber varchar(16) NOT NULL,
Label nvarchar(50) NULL,
Date_Started datetime2(7) NULL,
Date_Completed datetime2(7) NULL,
Status nvarchar(25) NULL,
Shape geography NULL,
Type varchar(2) NOT NULL,
ProjectCode varchar(16) NULL,
ActiveFlag int NULL,
Category varchar(32) NULL,
ProjectDescription varchar(64) NULL,
UserDefined varchar(1024) NULL,
InactiveReasonDate datetime NULL,
FYTDBudget money NULL,
LTDBudget money NULL,
PeriodExpenses money NULL,
FYTDExpenses money NULL,
LTDExpenses money NULL,
LTDEncumbrances money NULL,
LTDBalance money NULL,
FiscalYear int NULL,
ToPeriod int NULL,
_LastImported datetime NOT NULL,
OBJECTID int NOT NULL IDENTITY (1, 1),
GDB_GEOMATTR_DATA varbinary(MAX) NULL
) ON [PRIMARY]
TEXTIMAGE_ON [PRIMARY]
""")
cursor.execute("ALTER TABLE dbo.Tmp_" + newTableName + " SET (LOCK_ESCALATION = TABLE)")
cursor.execute("SET IDENTITY_INSERT dbo.Tmp_" + newTableName + " ON")
cursor.execute("""
IF EXISTS(SELECT * FROM dbo.""" + newTableName + """)
EXEC('INSERT INTO dbo.Tmp_""" + newTableName + """ (SubProjectTempId, CIPNumber, Label, Date_Started, Date_Completed, Status, Shape, Type, ProjectCode, ActiveFlag, Category, ProjectDescription, UserDefined, InactiveReasonDate, FYTDBudget, LTDBudget, PeriodExpenses, FYTDExpenses, LTDExpenses, LTDEncumbrances, LTDBalance, FiscalYear, ToPeriod, _LastImported, OBJECTID, GDB_GEOMATTR_DATA)
SELECT SubProjectTempId, CIPNumber, Label, Date_Started, Date_Completed, Status, Shape, Type, ProjectCode, ActiveFlag, Category, ProjectDescription, UserDefined, InactiveReasonDate, FYTDBudget, LTDBudget, PeriodExpenses, FYTDExpenses, LTDExpenses, LTDEncumbrances, LTDBalance, FiscalYear, ToPeriod, _LastImported, OBJECTID, GDB_GEOMATTR_DATA FROM dbo.""" + newTableName + """ WITH (HOLDLOCK TABLOCKX)')
""")
cursor.execute("SET IDENTITY_INSERT dbo.Tmp_" + newTableName + " OFF")
cursor.execute("DROP TABLE dbo." + newTableName)
cursor.execute("EXECUTE sp_rename N'dbo.Tmp_" + newTableName + "', N'" + newTableName + "', 'OBJECT'")
cursor.execute("""
ALTER TABLE dbo.""" + newTableName + """ ADD CONSTRAINT
R1143_pk PRIMARY KEY CLUSTERED
(
OBJECTID
) WITH( PAD_INDEX = OFF, FILLFACTOR = 75, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
""")
cursor.execute("""
CREATE SPATIAL INDEX SIndx ON dbo.""" + newTableName + """(Shape) USING GEOGRAPHY_AUTO_GRID
WITH( CELLS_PER_OBJECT = 16, STATISTICS_NORECOMPUTE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
""")
cursor.execute("""
ALTER TABLE dbo.""" + newTableName + """ ADD CONSTRAINT
g1084_ck CHECK (([Shape].[STSrid]=(4326)))
""")
cursor.execute("COMMIT")
conn.commit()
print("Object ID is set to Identify column.")
except pymssql.DatabaseError, err:
print(str(err))
Any help would be appreciated.
Thanks
Problem solved. The STSrid changes everytime you create a new table, so taking this portion out of the SQL code helped. However, I am not sure if the spatial index remained the same.
Cheers
cursor.execute("""
ALTER TABLE dbo.""" + newTableName + """ ADD CONSTRAINT
g1084_ck CHECK (([Shape].[STSrid]=(4326))) """)
I have that query in a python program:
And i should create a multidimensional array (if it possible) or four arrays from this query for every column from the query.
Can you suggest an elegant way to solve it?
conn = #connection to the server
cursor=conn.cursor()
query = (" select id, name, phone, city from guest")
cursor.execute(query)
results = cursor.fetchall
for i in results:
print i
cursor.close()
conn.close()
Not elegant but it may assist to unravel the mysterious Python Connector Cursor Class and transfers the list of tuples (see Copperfield comment) with the data from the query, into a list (phoneList) of dictionaries (entries) with details of each entry in the database, that might be easier to work with in your python script:
# ref: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor.html
import mysql.connector
db = 'test'
table = 'phonebook'
phoneList = []
drop_table = ("DROP TABLE IF EXISTS {};").format(table)
# By default, the starting value for AUTO_INCREMENT is 1, and it will increment by 1 for each new record.
# To let the AUTO_INCREMENT sequence start with another value, use the following SQL statement:
# ALTER TABLE phonebook AUTO_INCREMENT=100;
create_table = ("CREATE TABLE {} ("
"id int NOT NULL AUTO_INCREMENT,"
"name varchar(30) NOT NULL,"
"phone varchar(30) NOT NULL,"
"city varchar(30) NOT NULL,"
"PRIMARY KEY (id))"
" ENGINE=InnoDB DEFAULT CHARSET=latin1;").format(table)
Names = {'Bill':{'phone':'55123123','city':'Melbourne'},
'Mary':{'phone':'77111123','city':'Sydney'},
'Sue':{'phone':'55888123','city':'Melbourne'},
'Harry':{'phone':'77777123','city':'Sydney'},
'Fred':{'phone':'88123444','city':'Yongala'},
'Peter':{'phone':'55999123','city':'Melbourne'}}
cnx = mysql.connector.connect(user='mysqluser', password='xxxx',host='127.0.0.1',database=db)
cursor = cnx.cursor(dictionary=True) # key to using **row format
cursor.execute(drop_table)
cursor.execute(create_table)
# populate db
for name,detail in dict.items(Names):
sql = ("INSERT INTO {} (name,phone,city) VALUES ('{}','{}','{}')".format(table,name,detail['phone'],detail['city']))
cursor.execute(sql)
sql = ("SELECT id,name,phone,city FROM {}".format(table))
cursor.execute(sql)
for row in cursor:
print("{id} {name} {phone} {city}".format(**row))
phoneList.append(row)
print phoneList[0]['name'],phoneList[0]['city']
print phoneList[3]['name'],phoneList[3]['phone']
for entries in phoneList: # list of dictionaries
print entries['name'],entries
for entries in phoneList:
for k,v in dict.items(entries):
print k,v
print "\n"
cnx.close()