I am using the following command to enter list values in a SQLi Database.
sql_comm = "INSERT INTO Revisions (Work_ID,Rev_No,Name,Date) VALUES (?,?,?,?);"
data = (wid,i,rev_name,rev_date)
crsr.execute(sql_comm,data)
sql_com2 = "INSERT INTO Revisions (Field,New_Value) VALUES (?,?)"
for elem in zip(changes,vals):
crsr.execute(sql_com2,elem)
connection.commit()
This is the resulting output.
Work_ID Rev_No Name Date Field New Value
172 1 Raul Gitta <raoul_g#ware.23biz> 2021-10-21T10:08:10.653Z null null
null null null null System.Rev 2
null null null null System.AuthorizedDate 2021-10-20T06:42:01.32Z
null null null null System.RevisedDate 2021-10-21T10:08:10.653Z
null null null null System.ChangedDate 2021-10-20T06:42:01.32Z
null null null null System.Watermark 198
Is there a way to avoid these null values?
All the null values stem from that fact that they are not inserted in the appropriate INSERT query. It might be easier to use lists for the parameters instead of tuples, since lists are mutable.
Something like:
data = [wid,i,rev_name,rev_date]
sql_com2 = "INSERT INTO Revisions (Work_ID,Rev_No,Name,Date,Field,New_Value)
VALUES (?,?,?,?,?,?)"
for elem in zip(changes,vals):
crsr.execute(sql_com2,data + list(elem))
connection.commit()
Related
Hi im trying to make a python script which will update a line each time the script is run.
im having a bit of a headscratcher to how best to tackle the part where i have to update each line (a select statement) based on the value of a dataframe.
simplified i got a string 'select null r_cnt, null t_cnt, null r_dur, null t_dur from my_table'
and i got a list containing the fields for this line [t_cnt, r_dur]
I then want the new string output to be the first string where we have removed the null in front of the the values which was present in my list but kept null in front of those not in the list.
'select null r_cnt, t_cnt, r_dur, null t_dur from my_table'
my whole code looks something like this now where im stuck at the point i mentioned above
str_to_execute = f"select * from {db}.table_desc where grp_id in (400,500,300,1200) and id not in(127,140,125)"
cursor.execute(str_to_execute)
df = as_pandas(cursor)
for index, row in df.iterrows():
# print(row['name'])
str_to_execute = f"SHOW COLUMN STATS {db}.ctrl_{row['id']}"
cursor.execute(str_to_execute)
loop = as_pandas(cursor)
for index, row in loop.iterrows():
print(row['Column'])
str_to_execute = f"select concat(cast(ctrl_id as string),cast(ctrl_date as string)) primarykey, ctrl_id, ctrl_date,null r_cnt, null t_cnt, null r_dur, null t_dur,null r_amt, null t_amt,null p_cnt, null p_dur,null p_amt, null ro_vol, null t_vol, null r_vol, null p_vol, null ro_amt, null ro_cnt, from {db}.ctrl_{row['id']}"
if #This is where im stuck
Try:
s = 'select null r_cnt, null t_cnt, null r_dur, null t_dur from my_table'
lst = ['t_cnt', 'r_dur']
checklist = ['null r_cnt', 'null t_cnt', 'null r_dur']
checkliststr = ','.join(checklist)
for itm in lst:
if itm in checkliststr:
print('null ' + itm)
s=s.replace('null ' + itm, itm)
print(s)
You could just split the functionality of your string in this case Select and from my table from the input of your statement, which is more or less if I understand your correctly a list of strings.
One possible solution would be the following:
Define or get your lists of strings:
origin_list = ["null r_cnt", "null t_cnt", "null r_dur", "null t_dur"]
goal_list = ["t_cnt", "r_dur"]
For each element of your origin_list you want to edit the element according to its existence in the goal_list, I would do it with a map and lambda:
edited_list = list(map(lambda x: edit(x, goal_list), origin_list))
Now we have to define the function and logic from your post I derived the following logic or something similar:
Now you have the adjusted string and can merge it back together with your functionality.
Although I am quite new to SQL I have already used python to build DBs, but now I am stuck.
To put it simple, I have a schema with three tables, which are related to one another via foreign keys. They were created using python, as described below (not showing the definitions of c and conn, as I am pretty sure that the error does not lie there):
import sqlalchemy
import pandas as pd
# create the runsMaster table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`runsMaster` (
`run_ID` INT NOT NULL AUTO_INCREMENT,
`run_name` VARCHAR(50) NULL,
PRIMARY KEY (`run_ID`))
ENGINE = InnoDB""")
# Create the samplesMaster table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`samplesMaster` (
`sample_ID` INT NOT NULL AUTO_INCREMENT,
`run_ID` INT NULL,
`sample_name` VARCHAR(50) NULL,
PRIMARY KEY (`sample_ID`),
INDEX `fk_table1_runsMaster1_idx` (`run_ID` ASC),
CONSTRAINT `fk_table1_runsMaster1`
FOREIGN KEY (`run_ID`)
REFERENCES `ngsRunStats_FK`.`runsMaster` (`run_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION)
ENGINE = InnoDB""")
# Create the XYStats table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`XYstats` (
`XYstats_ID` INT NOT NULL AUTO_INCREMENT,
`run_ID` INT NULL,
`sample_ID` INT NULL,
`X_TOTAL_COVERAGE` FLOAT NULL,
`X_TARGET_COUNT` FLOAT NULL,
`X_MEAN_TARGET_COVERAGE` FLOAT NULL,
`Y_TOTAL_COVERAGE` FLOAT NULL,
`Y_TARGET_COUNT` FLOAT NULL,
`Y_MEAN_TARGET_COVERAGE` FLOAT NULL,
`Ymeancov_Xmeancov` FLOAT NULL,
PRIMARY KEY (`XYstats_ID`),
INDEX `fk_XYstats_runsMaster_idx` (`run_ID` ASC),
INDEX `fk_XYstats_samplesMaster1_idx` (`sample_ID` ASC),
CONSTRAINT `fk_XYstats_runsMaster`
FOREIGN KEY (`run_ID`)
REFERENCES `ngsRunStats_FK`.`runsMaster` (`run_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION,
CONSTRAINT `fk_XYstats_samplesMaster1`
FOREIGN KEY (`sample_ID`)
REFERENCES `ngsRunStats_FK`.`samplesMaster` (`sample_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION)
ENGINE = InnoDB""")
Both the samplesMaster and the runsMaster table are working fine. They are automatically populated from other iterations that are not all that important for the understanding of this problem.
After a few operations, I want to extract some values from a pandas df (XY_df) and insert into the XYStats table. My pandas df looks like the following
0 1 2 3
0 X 121424.000000 64.0 1897.26000
1 Y 14.019900 4.0 3.50497
2 Ymeancov/Xmeancov 0.001847 NaN NaN
Below is the dictionary that can be obtained from the table with XY_df.to_dict()
{0: {0: 'X', 1: 'Y', 2: 'Ymeancov/Xmeancov'},
1: {0: 121424.0, 1: 14.0199, 2: 0.00184739},
2: {0: 64.0, 1: 4.0, 2: nan},
3: {0: 1897.26, 1: 3.5049699999999997, 2: nan}}
The code that I am using to populate the XYStats table is shown below:
c.execute(f"""INSERT INTO XYstats (run_ID, sample_ID, X_TOTAL_COVERAGE, X_TARGET_COUNT, X_MEAN_TARGET_COVERAGE, Y_TOTAL_COVERAGE, Y_TARGET_COUNT, Y_MEAN_TARGET_COVERAGE, Ymeancov_Xmeancov)
VALUES
('{runID}',
'{sampleID}',
'{XY_df.iloc[0,1]}',
'{XY_df.iloc[0,2]}',
'{XY_df.iloc[0,3]}',
'{XY_df.iloc[1,1]}',
'{XY_df.iloc[1,2]}',
'{XY_df.iloc[1,3]}',
'{XY_df.iloc[2,1]}'
""")
conn.commit()
But then I get
ProgrammingError: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 11
Which is not informative at all I reckon
I am quite sure that my error does not lie in
The tables creation. I have been using the runsMaster as well as the samplesMaster the way they are
The data type that I am trying to insert into the XYStats table> XY_df is a pandas data frame and what I am trying to insert (e.g. XY_df.iloc[0,3]) are numpy.float64 (type(XY_df.iloc[0,1]))
But other than that I am quite clueless on what's going on as the error message that I am getting is very vague.
The error is a syntax error in the query you are executing on SQL. You have an unclosed bracket after VALUES. All you need to do is add a closing bracket at the end of the query string and you're good to go:
c.execute(f"""INSERT INTO XYstats (run_ID, sample_ID, X_TOTAL_COVERAGE, X_TARGET_COUNT, X_MEAN_TARGET_COVERAGE, Y_TOTAL_COVERAGE, Y_TARGET_COUNT, Y_MEAN_TARGET_COVERAGE, Ymeancov_Xmeancov)
VALUES
('{runID}',
'{sampleID}',
'{XY_df.iloc[0,1]}',
'{XY_df.iloc[0,2]}',
'{XY_df.iloc[0,3]}',
'{XY_df.iloc[1,1]}',
'{XY_df.iloc[1,2]}',
'{XY_df.iloc[1,3]}',
'{XY_df.iloc[2,1]}')
""")
I'm trying to create an SQL database with the following fields:
connection= sqlite3.connect('Main Database')
crsr = connection.cursor()
#Creates a table for the teacher data if no table is found on the system
crsr.execute("""CREATE TABLE IF NOT EXISTS Teacher_Table(Teacher_ID INTEGER PRIMARY KEY,
TFirst_Name VARCHAR(25) NOT NULL,
TLast_Name VARCHAR (25) NOT NULL,
Gender CHAR(1) NOT NULL,
Home_Address VARCHAR (50) NOT NULL,
Contact_Number VARCHAR (14) NOT NULL);""")
connection.commit()
connection.close()
But when I input values, the gender field accepts more than one value
Database View
How can I make sure it only accepts one character for that field
How can I make sure it only accepts one character for that field
SQLite does not check the length constraints defined at type level, as is specified in the documentation on types:
(...) Note that numeric arguments in parentheses that following the type name (ex: "VARCHAR(255)") are ignored by SQLite - SQLite does not impose any length restrictions (other than the large global SQLITE_MAX_LENGTH limit) on the length of strings, BLOBs or numeric values.
So you can not enforce this at the database level. You will thus need to enforce this through your views, etc.
We can however, like #Ilja Everilä says, use a CHECK constraint:
CREATE TABLE IF NOT EXISTS Teacher_Table(
Teacher_ID INTEGER PRIMARY KEY,
TFirst_Name VARCHAR(25) NOT NULL,
TLast_Name VARCHAR (25) NOT NULL,
Gender CHAR(1) NOT NULL CHECK (length(Gender) < 2),
Home_Address VARCHAR (50) NOT NULL,
Contact_Number VARCHAR (14) NOT NULL
)
I have to check if there is any null in my database,
and I need to check 11 columns (by or) and plus and like year (ex. 2017%).
def test():
sql= "select date from a000760 where (total_assets is null or total_liabilities is null or sales_figures is null or sales_cost is null or business_profits is null or gross_margin is null or current_income is null or depreciation_expense_of_tangible_assets is null or liquid_asset is null or noncurrent_asset is null or liquid_liability is null) and (date like '2010%')"
curs.execute(sql)
#year="2010"
#curs.execute("select date from a000760 where (total_assets is null or total_liabilities is null or sales_figures is null or sales_cost is null or business_profits is null or gross_margin is null or current_income is null or depreciation_expense_of_tangible_assets is null or liquid_asset is null or noncurrent_asset is null or liquid_liability is null) and (date like %s)",year)
result = curs.fetchall()
if len(result)>0: // print shows () even if it's none. so, I use this
print "a000760"
print "2010 null exists"
It's the test version of one table.
I have to check more than 2000 tables.
It works for this def and shows error (only for one table).
But it doesn't work for total tables.
And I get this error:
Warning: (1292, "Incorrect date value: '2010%' for column 'date' at row 1")
I don't know how...
I've searched for the whole grammar,
but when I type %2017% it doesn't work.
Do not use like with dates! Dates are not strings.
You can just do:
year(date) = 2010
Or:
date >= '2010-01-01' and date < '2011-01-01'
I am trying to create a copy of a table through a python script that has all the qualities of the original except for the partitions. I want to do this multiple times in my script (through a for loop) because I want to mysqldump daily files of old data from that table, so I'm trying to use something like:
CREATE TABLE temp_utilization LIKE utilization WITHOUT PARTITIONING;
Here is the original table:
CREATE TABLE `utilization` (
`wrep_time` timestamp NULL DEFAULT NULL,
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (UNIX_TIMESTAMP(wrep_time))
(PARTITION p0 VALUES LESS THAN (1391990400) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN (1392076800) ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN (1392163200) ENGINE = InnoDB,
PARTITION p3 VALUES LESS THAN (1392249600) ENGINE = InnoDB,
PARTITION p492 VALUES LESS THAN (1434499200) ENGINE = InnoDB,
PARTITION p493 VALUES LESS THAN (1434585600) ENGINE = InnoDB,
PARTITION p494 VALUES LESS THAN (1434672000) ENGINE = InnoDB,
PARTITION p495 VALUES LESS THAN (1434758400) ENGINE = InnoDB,
PARTITION p496 VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */
I would like to create a temp table which contains a create table like this:
CREATE TABLE `temp_utilization` (
`wrep_time` timestamp NULL DEFAULT NULL,
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
mysql> alter table utilization remove partitioning;
Query OK, 0 rows affected (0.40 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> show create table utilization\G
*************************** 1. row ***************************
Table: utilization
Create Table: CREATE TABLE `utilization` (
`wrep_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
So, for your example:
CREATE TABLE temp_utilization LIKE utilization;
ALTER TABLE temp_utilization REMOVE PARTITIONING;
Then during your loop you can CREATE TABLE t1 LIKE temp_utilization or however you wish to name the tables
No, it does not appear that you can create a table like another table without partitions, if it is already partitioned, in one command as you suggested above.
The partition is part of the table definition and is stored in the metadata. You can check that by executing show create table yourtablename;
If you just want to create the table over and over again in a loop without the partitions and the data I see three (added one b/c of Cez) options.
have the table definitions hard coded in your script
create the table in the DB without the partitions. So you have one temp table already created and use that as your template to loop through.
run two separate command from your script: A create table like and then an alter table to remove the partitions in a loop.
You can choose which options best suits you for your environment.
You can reference your options when creating a table at dev.mysql.