I'm using Python to insert JSON data into a PostgreSQL table and I wanted to update a column automatically when a row is updated.
Table definition is:
CREATE TABLE public.customer_data (
id serial4 NOT NULL,
sno int4 NOT NULL,
org public.org NULL,
cust_nbr int8 NULL,
fdc_customer_number int8 NOT NULL,
gender bpchar(1) NULL DEFAULT NULL::bpchar,
mar_status public.mar_status NULL,
spous_name varchar(40) NULL DEFAULT NULL::character varying,
employer varchar(40) NULL DEFAULT NULL::character varying,
designation varchar(30) NULL DEFAULT NULL::character varying,
c_statement_flag public.c_statement_flag NULL,
c_city_code bpchar(2) NULL DEFAULT NULL::bpchar,
c_marital_status public.c_marital_status NULL,
card_vip int4 NULL,
createdon timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
updatedon timestamp NULL,
CONSTRAINT customer_data_pk PRIMARY KEY (fdc_customer_number));
createdon and updatedon columns should have the same timestamp in case of new inserted row. In case of update, only the updatedon column should be updated automatically. How can I achieve this? Or should this be done from Python?.
I use a trigger for this:
CREATE OR REPLACE FUNCTION public.ts_update()
RETURNS trigger
LANGUAGE plpgsql
AS $function$
BEGIN
NEW.updatedon := clock_timestamp();
RETURN NEW;
END;
$function$
customer_data_ts_update BEFORE UPDATE ON public.customer_data FOR EACH ROW EXECUTE FUNCTION ts_update()
This example clock_timestamp() which represents wall clock time. For other choices see Current Date/Time. The plus to the trigger approach is that the field will get set no matter what client is updating the row.
Related
I am using pyodbc to fetch total quantity of a product based on a certain criteria called "Strength". The "Strength" column also has string value for some records, so it is a varchar column.
The user enters details like brand, product type, product line, date range, minimum quantity (in this case, 12), and strength range
This is my query:
SELECT SUM(CAST([Qty] AS decimal(10, 2))) AS Qty
FROM (
SELECT
[Brand],
[ProdType],
[Lot],
CAST([Strength] AS DECIMAL(10,4)) AS [Strength],
[ProductLine],
[Size],
[Stage],
[Customer],
[PackedOn],
[Qty],
[RefreshedBy],
[RefreshedOn]
FROM SalesData
WHERE
(isnumeric([Strength]) = 1)
AND [Stage]='WIP'
AND [PackedOn]>='2018-06-03'
AND [PackedOn]<='2020-06-03'
AND [Brand]='ABC'
AND [ProductLine]='DEF'
AND [Size]='15'
AND [Qty]>='12.0'
AND [Strength]>=0.2
AND [Strength]<=0.4
AND [ProdType] Is Null
) as outputdata
This is my table:
ID Brand ProdType Lot Strength ProductLine Size Stage Province PackedOn Qty
1 ABC NULL XXXXXXX 0.16 DEF 15 WIP NULL 2018-12-07 1200
This is the create statement
CREATE TABLE [dbo].[SalesData](
[ID] [int] NOT NULL,
[Brand] [varchar](max) NOT NULL,
[ProdType] [varchar](max) NULL,
[Lot] [varchar](max) NOT NULL,
[Strength] [varchar](max) NOT NULL,
[ProductLine] [varchar](max) NOT NULL,
[Size] [varchar](max) NOT NULL,
[Stage] [varchar](max) NOT NULL,
[Province] [varchar](max) NULL,
[PackedOn] [date] NOT NULL,
[Qty] [float] NOT NULL,
[RefreshedBy] [varchar](max) NULL,
[RefreshedOn] [varchar](max) NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
My problem is that this query results in a Quantity of 1200, even though it is outside the strength range. I am using SQL Server Management Studio V 18.4. How do I solve this?
In your WHERE clause you should use.
TRY_CAST([Strength] AS DECIMAL(10,4))>=0.2 AND TRY_CAST([Strength] AS DECIMAL(10,4))<=0.4
Because sql queries start working from where clauses( and joins) then executes other parts. SELECT is the least important part and if you only use CAST in your select it will be only useful for printing data as your preferred format.
SELECT SUM(CAST([Qty] AS decimal(10, 2))) AS Qty FROM
(SELECT [Brand], [ProdType], [Lot], CAST([Strength] AS DECIMAL(10,4)) AS [Strength], [ProductLine], [Size], [Stage], [Customer], [PackedOn], [Qty], [RefreshedBy], [RefreshedOn]
FROM SalesData
WHERE (isnumeric([Strength]) = 1) AND [Stage]='WIP' AND [PackedOn]>='2018-06-03'
AND [PackedOn]<='2020-06-03' AND [Brand]='ABC' AND [ProductLine]='DEF'
AND [Size]='15' AND [Qty]>='12.0' AND TRY_CAST([Strength] AS DECIMAL(10,4))>=0.2 AND TRY_CAST([Strength] AS DECIMAL(10,4))<=0.4 AND [ProdType] Is Null) as outputdata
You need to CAST() before doing numeric comparison, otherwise SQL Server compares strings, not numbers, which leads to unexpected results: as an example, string-wise, '2' is greater than '12' (since it starts with '2', which is greater than '1') That's true of all numeric comparisons involved in the query (Size is also concerned).
I would suggest TRY_CAST(), which avoids error and returns null if conversion fails (which will effectively fail the condition, and remove the corresponding row from the query).
Also, the subquery is unnecessary.
Consider:
SELECT SUM(Qty) Qty
FROM SalesData
WHERE
Stage = 'WIP'
AND PackedOn >= '2018-06-03'
AND PackedOn <= '2020-06-03'
AND Brand = 'ABC'
AND ProductLine = 'DEF'
AND ProdType Is Null
AND Qty >= 12
AND TRY_CAST(Strength AS DECIMAL(10, 4)) >= 0.2
AND TRY_CAST(Strength AS DECIMAL(10, 4)) <= 0.4
AND TRY_CAST(Size AS INT) = 15
If you want to cast your float output to decimal, it is more accurate to this after the sum(), so:
SELECT CAST(SUM(Qty) AS DECIMAL(10, 2)) Qty
FROM ...
Although I am quite new to SQL I have already used python to build DBs, but now I am stuck.
To put it simple, I have a schema with three tables, which are related to one another via foreign keys. They were created using python, as described below (not showing the definitions of c and conn, as I am pretty sure that the error does not lie there):
import sqlalchemy
import pandas as pd
# create the runsMaster table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`runsMaster` (
`run_ID` INT NOT NULL AUTO_INCREMENT,
`run_name` VARCHAR(50) NULL,
PRIMARY KEY (`run_ID`))
ENGINE = InnoDB""")
# Create the samplesMaster table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`samplesMaster` (
`sample_ID` INT NOT NULL AUTO_INCREMENT,
`run_ID` INT NULL,
`sample_name` VARCHAR(50) NULL,
PRIMARY KEY (`sample_ID`),
INDEX `fk_table1_runsMaster1_idx` (`run_ID` ASC),
CONSTRAINT `fk_table1_runsMaster1`
FOREIGN KEY (`run_ID`)
REFERENCES `ngsRunStats_FK`.`runsMaster` (`run_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION)
ENGINE = InnoDB""")
# Create the XYStats table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`XYstats` (
`XYstats_ID` INT NOT NULL AUTO_INCREMENT,
`run_ID` INT NULL,
`sample_ID` INT NULL,
`X_TOTAL_COVERAGE` FLOAT NULL,
`X_TARGET_COUNT` FLOAT NULL,
`X_MEAN_TARGET_COVERAGE` FLOAT NULL,
`Y_TOTAL_COVERAGE` FLOAT NULL,
`Y_TARGET_COUNT` FLOAT NULL,
`Y_MEAN_TARGET_COVERAGE` FLOAT NULL,
`Ymeancov_Xmeancov` FLOAT NULL,
PRIMARY KEY (`XYstats_ID`),
INDEX `fk_XYstats_runsMaster_idx` (`run_ID` ASC),
INDEX `fk_XYstats_samplesMaster1_idx` (`sample_ID` ASC),
CONSTRAINT `fk_XYstats_runsMaster`
FOREIGN KEY (`run_ID`)
REFERENCES `ngsRunStats_FK`.`runsMaster` (`run_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION,
CONSTRAINT `fk_XYstats_samplesMaster1`
FOREIGN KEY (`sample_ID`)
REFERENCES `ngsRunStats_FK`.`samplesMaster` (`sample_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION)
ENGINE = InnoDB""")
Both the samplesMaster and the runsMaster table are working fine. They are automatically populated from other iterations that are not all that important for the understanding of this problem.
After a few operations, I want to extract some values from a pandas df (XY_df) and insert into the XYStats table. My pandas df looks like the following
0 1 2 3
0 X 121424.000000 64.0 1897.26000
1 Y 14.019900 4.0 3.50497
2 Ymeancov/Xmeancov 0.001847 NaN NaN
Below is the dictionary that can be obtained from the table with XY_df.to_dict()
{0: {0: 'X', 1: 'Y', 2: 'Ymeancov/Xmeancov'},
1: {0: 121424.0, 1: 14.0199, 2: 0.00184739},
2: {0: 64.0, 1: 4.0, 2: nan},
3: {0: 1897.26, 1: 3.5049699999999997, 2: nan}}
The code that I am using to populate the XYStats table is shown below:
c.execute(f"""INSERT INTO XYstats (run_ID, sample_ID, X_TOTAL_COVERAGE, X_TARGET_COUNT, X_MEAN_TARGET_COVERAGE, Y_TOTAL_COVERAGE, Y_TARGET_COUNT, Y_MEAN_TARGET_COVERAGE, Ymeancov_Xmeancov)
VALUES
('{runID}',
'{sampleID}',
'{XY_df.iloc[0,1]}',
'{XY_df.iloc[0,2]}',
'{XY_df.iloc[0,3]}',
'{XY_df.iloc[1,1]}',
'{XY_df.iloc[1,2]}',
'{XY_df.iloc[1,3]}',
'{XY_df.iloc[2,1]}'
""")
conn.commit()
But then I get
ProgrammingError: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 11
Which is not informative at all I reckon
I am quite sure that my error does not lie in
The tables creation. I have been using the runsMaster as well as the samplesMaster the way they are
The data type that I am trying to insert into the XYStats table> XY_df is a pandas data frame and what I am trying to insert (e.g. XY_df.iloc[0,3]) are numpy.float64 (type(XY_df.iloc[0,1]))
But other than that I am quite clueless on what's going on as the error message that I am getting is very vague.
The error is a syntax error in the query you are executing on SQL. You have an unclosed bracket after VALUES. All you need to do is add a closing bracket at the end of the query string and you're good to go:
c.execute(f"""INSERT INTO XYstats (run_ID, sample_ID, X_TOTAL_COVERAGE, X_TARGET_COUNT, X_MEAN_TARGET_COVERAGE, Y_TOTAL_COVERAGE, Y_TARGET_COUNT, Y_MEAN_TARGET_COVERAGE, Ymeancov_Xmeancov)
VALUES
('{runID}',
'{sampleID}',
'{XY_df.iloc[0,1]}',
'{XY_df.iloc[0,2]}',
'{XY_df.iloc[0,3]}',
'{XY_df.iloc[1,1]}',
'{XY_df.iloc[1,2]}',
'{XY_df.iloc[1,3]}',
'{XY_df.iloc[2,1]}')
""")
I have three main tables to keep track of products, location and the logistics between them which includes moving products to and from various locations. I have made another table balance to keep a final balance of the quantity of each product in respective locations.
Here are the schemas:
products(prod_id INTEGER PRIMARY KEY AUTOINCREMENT,
prod_name TEXT UNIQUE NOT NULL,
prod_quantity INTEGER NOT NULL,
unallocated_quantity INTEGER)
Initially, when products are added, prod_quantity and unallocated_quantity have the same values. unallocated_quantity is then subtracted from, each time a certain quantity of the respective product is allocated.
location(loc_id INTEGER PRIMARY KEY AUTOINCREMENT,
loc_name TEXT UNIQUE NOT NULL)
logistics(trans_id INTEGER PRIMARY KEY AUTOINCREMENT,
prod_id INTEGER NOT NULL,
from_loc_id INTEGER NULL,
to_loc_id INTEGER NOT NULL,
prod_quantity INTEGER NOT NULL,
trans_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY(prod_id) REFERENCES products(prod_id),
FOREIGN KEY(from_loc_id) REFERENCES location(loc_id),
FOREIGN KEY(to_loc_id) REFERENCES location(loc_id))
balance(prod_id INTEGER NOT NULL,
loc_id INTEGER NOT NULL,
quantity INTEGER NOT NULL,
FOREIGN KEY(prod_id) REFERENCES products(prod_id),
FOREIGN KEY(loc_id) REFERENCES location(loc_id))
At each entry made in logistics, I want a trigger to update the values in balance thereby keeping a summary of all the transactions (moving products between locations)
I thought of a trigger solution which checks if for each insert on the table logistics, there already exists the same prod_id, loc_id entry in the balance table, which if exists will be updated appropriately. However, I don't have the experience in SQLite to implement this idea.
I believe that your TRIGGER would be along the lines of either :-
CREATE TRIGGER IF NOT EXISTS logistics_added AFTER INSERT ON logistics
BEGIN
UPDATE balance SET quantity = ((SELECT quantity FROM balance WHERE prod_id = new.prod_id AND loc_id = new.from_loc_id) - new.prod_quantity) WHERE prod_id = new.prod_id AND loc_id = new.from_loc_id;
UPDATE balance SET quantity = ((SELECT quantity FROM balance WHERE prod_id = new.prod_id AND loc_id = new.to_loc_id) + new.prod_quantity) WHERE prod_id = new.prod_id AND loc_id = new.to_loc_id;
END;
or :-
CREATE TRIGGER IF NOT EXISTS logistics_added AFTER INSERT ON logistics
BEGIN
INSERT OR REPLACE INTO balance VALUES(new.prod_id,new.from_loc_id,(SELECT quantity FROM balance WHERE prod_id = new.prod_id AND loc_id = new.from_loc_id) - new.prod_quantity);
INSERT OR REPLACE INTO balance VALUES(new.prod_id,new.to_loc_id,(SELECT quantity FROM balance WHERE prod_id = new.prod_id AND loc_id = new.to_loc_id) + new.prod_quantity);
END;
Note that the second relies upon adding a UNIQUE constraint to the balance table by using PRIMARY KEY (prod_id,loc_id) or alternately UNIQUE (prod_id,loc_id). The UNIQUE constraint would probably be required/wanted anyway.
The subtle difference is that the second would INSERT a balance row if and appropriate one didn't exist. The latter would do nothing if the appropriate balance row didn't exist.
I'm trying to create an SQL database with the following fields:
connection= sqlite3.connect('Main Database')
crsr = connection.cursor()
#Creates a table for the teacher data if no table is found on the system
crsr.execute("""CREATE TABLE IF NOT EXISTS Teacher_Table(Teacher_ID INTEGER PRIMARY KEY,
TFirst_Name VARCHAR(25) NOT NULL,
TLast_Name VARCHAR (25) NOT NULL,
Gender CHAR(1) NOT NULL,
Home_Address VARCHAR (50) NOT NULL,
Contact_Number VARCHAR (14) NOT NULL);""")
connection.commit()
connection.close()
But when I input values, the gender field accepts more than one value
Database View
How can I make sure it only accepts one character for that field
How can I make sure it only accepts one character for that field
SQLite does not check the length constraints defined at type level, as is specified in the documentation on types:
(...) Note that numeric arguments in parentheses that following the type name (ex: "VARCHAR(255)") are ignored by SQLite - SQLite does not impose any length restrictions (other than the large global SQLITE_MAX_LENGTH limit) on the length of strings, BLOBs or numeric values.
So you can not enforce this at the database level. You will thus need to enforce this through your views, etc.
We can however, like #Ilja Everilä says, use a CHECK constraint:
CREATE TABLE IF NOT EXISTS Teacher_Table(
Teacher_ID INTEGER PRIMARY KEY,
TFirst_Name VARCHAR(25) NOT NULL,
TLast_Name VARCHAR (25) NOT NULL,
Gender CHAR(1) NOT NULL CHECK (length(Gender) < 2),
Home_Address VARCHAR (50) NOT NULL,
Contact_Number VARCHAR (14) NOT NULL
)
I am trying to create a copy of a table through a python script that has all the qualities of the original except for the partitions. I want to do this multiple times in my script (through a for loop) because I want to mysqldump daily files of old data from that table, so I'm trying to use something like:
CREATE TABLE temp_utilization LIKE utilization WITHOUT PARTITIONING;
Here is the original table:
CREATE TABLE `utilization` (
`wrep_time` timestamp NULL DEFAULT NULL,
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (UNIX_TIMESTAMP(wrep_time))
(PARTITION p0 VALUES LESS THAN (1391990400) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN (1392076800) ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN (1392163200) ENGINE = InnoDB,
PARTITION p3 VALUES LESS THAN (1392249600) ENGINE = InnoDB,
PARTITION p492 VALUES LESS THAN (1434499200) ENGINE = InnoDB,
PARTITION p493 VALUES LESS THAN (1434585600) ENGINE = InnoDB,
PARTITION p494 VALUES LESS THAN (1434672000) ENGINE = InnoDB,
PARTITION p495 VALUES LESS THAN (1434758400) ENGINE = InnoDB,
PARTITION p496 VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */
I would like to create a temp table which contains a create table like this:
CREATE TABLE `temp_utilization` (
`wrep_time` timestamp NULL DEFAULT NULL,
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
mysql> alter table utilization remove partitioning;
Query OK, 0 rows affected (0.40 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> show create table utilization\G
*************************** 1. row ***************************
Table: utilization
Create Table: CREATE TABLE `utilization` (
`wrep_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
So, for your example:
CREATE TABLE temp_utilization LIKE utilization;
ALTER TABLE temp_utilization REMOVE PARTITIONING;
Then during your loop you can CREATE TABLE t1 LIKE temp_utilization or however you wish to name the tables
No, it does not appear that you can create a table like another table without partitions, if it is already partitioned, in one command as you suggested above.
The partition is part of the table definition and is stored in the metadata. You can check that by executing show create table yourtablename;
If you just want to create the table over and over again in a loop without the partitions and the data I see three (added one b/c of Cez) options.
have the table definitions hard coded in your script
create the table in the DB without the partitions. So you have one temp table already created and use that as your template to loop through.
run two separate command from your script: A create table like and then an alter table to remove the partitions in a loop.
You can choose which options best suits you for your environment.
You can reference your options when creating a table at dev.mysql.