I'm trying to create an SQL database with the following fields:
connection= sqlite3.connect('Main Database')
crsr = connection.cursor()
#Creates a table for the teacher data if no table is found on the system
crsr.execute("""CREATE TABLE IF NOT EXISTS Teacher_Table(Teacher_ID INTEGER PRIMARY KEY,
TFirst_Name VARCHAR(25) NOT NULL,
TLast_Name VARCHAR (25) NOT NULL,
Gender CHAR(1) NOT NULL,
Home_Address VARCHAR (50) NOT NULL,
Contact_Number VARCHAR (14) NOT NULL);""")
connection.commit()
connection.close()
But when I input values, the gender field accepts more than one value
Database View
How can I make sure it only accepts one character for that field
How can I make sure it only accepts one character for that field
SQLite does not check the length constraints defined at type level, as is specified in the documentation on types:
(...) Note that numeric arguments in parentheses that following the type name (ex: "VARCHAR(255)") are ignored by SQLite - SQLite does not impose any length restrictions (other than the large global SQLITE_MAX_LENGTH limit) on the length of strings, BLOBs or numeric values.
So you can not enforce this at the database level. You will thus need to enforce this through your views, etc.
We can however, like #Ilja Everilä says, use a CHECK constraint:
CREATE TABLE IF NOT EXISTS Teacher_Table(
Teacher_ID INTEGER PRIMARY KEY,
TFirst_Name VARCHAR(25) NOT NULL,
TLast_Name VARCHAR (25) NOT NULL,
Gender CHAR(1) NOT NULL CHECK (length(Gender) < 2),
Home_Address VARCHAR (50) NOT NULL,
Contact_Number VARCHAR (14) NOT NULL
)
Related
I'm using Python to insert JSON data into a PostgreSQL table and I wanted to update a column automatically when a row is updated.
Table definition is:
CREATE TABLE public.customer_data (
id serial4 NOT NULL,
sno int4 NOT NULL,
org public.org NULL,
cust_nbr int8 NULL,
fdc_customer_number int8 NOT NULL,
gender bpchar(1) NULL DEFAULT NULL::bpchar,
mar_status public.mar_status NULL,
spous_name varchar(40) NULL DEFAULT NULL::character varying,
employer varchar(40) NULL DEFAULT NULL::character varying,
designation varchar(30) NULL DEFAULT NULL::character varying,
c_statement_flag public.c_statement_flag NULL,
c_city_code bpchar(2) NULL DEFAULT NULL::bpchar,
c_marital_status public.c_marital_status NULL,
card_vip int4 NULL,
createdon timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
updatedon timestamp NULL,
CONSTRAINT customer_data_pk PRIMARY KEY (fdc_customer_number));
createdon and updatedon columns should have the same timestamp in case of new inserted row. In case of update, only the updatedon column should be updated automatically. How can I achieve this? Or should this be done from Python?.
I use a trigger for this:
CREATE OR REPLACE FUNCTION public.ts_update()
RETURNS trigger
LANGUAGE plpgsql
AS $function$
BEGIN
NEW.updatedon := clock_timestamp();
RETURN NEW;
END;
$function$
customer_data_ts_update BEFORE UPDATE ON public.customer_data FOR EACH ROW EXECUTE FUNCTION ts_update()
This example clock_timestamp() which represents wall clock time. For other choices see Current Date/Time. The plus to the trigger approach is that the field will get set no matter what client is updating the row.
What is the best approach to export byte/graphic data from Teradata ? For a data migration project, can someone guide me on how to export data from teradata to snowflake. I am using TPT scripts and tdload approach, however, it doesn't seem to work.
Approach that I have followed :
Convert the data from byte to ASCII using FROM_BYTES() method in Teradata. However, during the ingestion process I was not able to get it bacl to original state.
Using FROM_BYTES() method with base10/base16 to get into the desired format, however, I am facing same issue with the process.
Below is Table structure :
CREATE SET TABLE DBC.AccessRights ,FALLBACK ,
NO BEFORE JOURNAL,
NO AFTER JOURNAL,
CHECKSUM = DEFAULT,
DEFAULT MERGEBLOCKRATIO,
MAP = TD_DATADICTIONARYMAP
(
UserId BYTE(4) NOT NULL,
DatabaseId BYTE(4) NOT NULL,
TVMId BYTE(6) NOT NULL,
FieldId SMALLINT FORMAT '---,--9' NOT NULL,
AccessRight CHAR(2) CHARACTER SET LATIN UPPERCASE NOT CASESPECIFIC NOT NULL,
WithGrant CHAR(1) CHARACTER SET LATIN UPPERCASE NOT CASESPECIFIC NOT NULL,
GrantorID BYTE(4) NOT NULL,
AllnessFlag CHAR(1) CHARACTER SET LATIN UPPERCASE NOT CASESPECIFIC NOT NULL,
CreateUID BYTE(4),
CreateTimeStamp TIMESTAMP(0),
LastAccessTimeStamp TIMESTAMP(0),
AccessCount INTEGER FORMAT '--,---,---,--9')
PRIMARY INDEX ( UserId )
PARTITION BY ( RANGE_N((ID2BIGINT(DatabaseId )) MOD 1073741824 BETWEEN 0 AND 1073741823 EACH 1 ),
RANGE_N(ID2BIGINT(TVMId ) BETWEEN 0 AND 4294967295. EACH 1 )ADD 2 );
Tpt Script :
USING CHARACTER SET UTF8
DEFINE JOB EXPORT_DELIMITED_FILE
DESCRIPTION 'Export rows from a Teradata table to a unformatted file' ( DEFINE SCHEMA FILE_SCHEMA (
UserId BYTE(4),
DatabaseId BYTE(4),
TVMId BYTE(6),
FieldId SMALLINT,
AccessRight CHAR(4),
WithGrant CHAR(2),
GrantorID BYTE(4),
AllnessFlag CHAR(2),
CreateUID BYTE(4),
CreateTimeStamp TIMESTAMP(0),
LastAccessTimeStamp TIMESTAMP(0),
AccessCount INTEGER
);
DEFINE OPERATOR SQL_SELECTOR
TYPE SELECTOR SCHEMA FILE_SCHEMA ATTRIBUTES
(
VARCHAR PrivateLogName = 'selector_log',
VARCHAR TdpId = <host_id>,
VARCHAR UserName = <user_name>,
VARCHAR UserPassword = <password>,,
VARCHAR SelectStmt = 'SELECT * FROM DBC.AccessRights;',
VARCHAR LobDirectoryPath = <lob_dir>
);
DEFINE OPERATOR FILE_WRITER TYPE DATACONNECTOR CONSUMER SCHEMA * ATTRIBUTES
(
VARCHAR PrivateLogName = 'dataconnector_log',
VARCHAR DirectoryPath = <dir_path>,
VARCHAR FileName = 'file.csv',
VARCHAR FORMAT= 'BINARY',
VARCHAR OpenMode = 'Write'
);
APPLY TO OPERATOR (FILE_WRITER)
SELECT * FROM OPERATOR (SQL_SELECTOR);
);
Snowflake defaults to hex string representation for input/output of binary data. If you have TPT output DELIMITED format with a schema defined or generated as all VARCHAR fields, you won't be able to use SELECT * but you can use FROM_BYTES to generate the hex string. There are some quirks that are simple enough to work around as long as you are aware, namely
If the high order bit of the BYTE value is set the returned value will be the two's complement preceded by a negative sign
leading zeros are removed from the result string
But you can prefix a "positive, non-zero" byte, convert, and then remove the extra characters from the string.
SUBSTRING(FROM_BYTES('12'xb||DatabaseId, 'base16') FROM 3)
Although I am quite new to SQL I have already used python to build DBs, but now I am stuck.
To put it simple, I have a schema with three tables, which are related to one another via foreign keys. They were created using python, as described below (not showing the definitions of c and conn, as I am pretty sure that the error does not lie there):
import sqlalchemy
import pandas as pd
# create the runsMaster table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`runsMaster` (
`run_ID` INT NOT NULL AUTO_INCREMENT,
`run_name` VARCHAR(50) NULL,
PRIMARY KEY (`run_ID`))
ENGINE = InnoDB""")
# Create the samplesMaster table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`samplesMaster` (
`sample_ID` INT NOT NULL AUTO_INCREMENT,
`run_ID` INT NULL,
`sample_name` VARCHAR(50) NULL,
PRIMARY KEY (`sample_ID`),
INDEX `fk_table1_runsMaster1_idx` (`run_ID` ASC),
CONSTRAINT `fk_table1_runsMaster1`
FOREIGN KEY (`run_ID`)
REFERENCES `ngsRunStats_FK`.`runsMaster` (`run_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION)
ENGINE = InnoDB""")
# Create the XYStats table
c.execute("""CREATE TABLE IF NOT EXISTS `ngsRunStats_FK`.`XYstats` (
`XYstats_ID` INT NOT NULL AUTO_INCREMENT,
`run_ID` INT NULL,
`sample_ID` INT NULL,
`X_TOTAL_COVERAGE` FLOAT NULL,
`X_TARGET_COUNT` FLOAT NULL,
`X_MEAN_TARGET_COVERAGE` FLOAT NULL,
`Y_TOTAL_COVERAGE` FLOAT NULL,
`Y_TARGET_COUNT` FLOAT NULL,
`Y_MEAN_TARGET_COVERAGE` FLOAT NULL,
`Ymeancov_Xmeancov` FLOAT NULL,
PRIMARY KEY (`XYstats_ID`),
INDEX `fk_XYstats_runsMaster_idx` (`run_ID` ASC),
INDEX `fk_XYstats_samplesMaster1_idx` (`sample_ID` ASC),
CONSTRAINT `fk_XYstats_runsMaster`
FOREIGN KEY (`run_ID`)
REFERENCES `ngsRunStats_FK`.`runsMaster` (`run_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION,
CONSTRAINT `fk_XYstats_samplesMaster1`
FOREIGN KEY (`sample_ID`)
REFERENCES `ngsRunStats_FK`.`samplesMaster` (`sample_ID`)
ON DELETE CASCADE
ON UPDATE NO ACTION)
ENGINE = InnoDB""")
Both the samplesMaster and the runsMaster table are working fine. They are automatically populated from other iterations that are not all that important for the understanding of this problem.
After a few operations, I want to extract some values from a pandas df (XY_df) and insert into the XYStats table. My pandas df looks like the following
0 1 2 3
0 X 121424.000000 64.0 1897.26000
1 Y 14.019900 4.0 3.50497
2 Ymeancov/Xmeancov 0.001847 NaN NaN
Below is the dictionary that can be obtained from the table with XY_df.to_dict()
{0: {0: 'X', 1: 'Y', 2: 'Ymeancov/Xmeancov'},
1: {0: 121424.0, 1: 14.0199, 2: 0.00184739},
2: {0: 64.0, 1: 4.0, 2: nan},
3: {0: 1897.26, 1: 3.5049699999999997, 2: nan}}
The code that I am using to populate the XYStats table is shown below:
c.execute(f"""INSERT INTO XYstats (run_ID, sample_ID, X_TOTAL_COVERAGE, X_TARGET_COUNT, X_MEAN_TARGET_COVERAGE, Y_TOTAL_COVERAGE, Y_TARGET_COUNT, Y_MEAN_TARGET_COVERAGE, Ymeancov_Xmeancov)
VALUES
('{runID}',
'{sampleID}',
'{XY_df.iloc[0,1]}',
'{XY_df.iloc[0,2]}',
'{XY_df.iloc[0,3]}',
'{XY_df.iloc[1,1]}',
'{XY_df.iloc[1,2]}',
'{XY_df.iloc[1,3]}',
'{XY_df.iloc[2,1]}'
""")
conn.commit()
But then I get
ProgrammingError: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 11
Which is not informative at all I reckon
I am quite sure that my error does not lie in
The tables creation. I have been using the runsMaster as well as the samplesMaster the way they are
The data type that I am trying to insert into the XYStats table> XY_df is a pandas data frame and what I am trying to insert (e.g. XY_df.iloc[0,3]) are numpy.float64 (type(XY_df.iloc[0,1]))
But other than that I am quite clueless on what's going on as the error message that I am getting is very vague.
The error is a syntax error in the query you are executing on SQL. You have an unclosed bracket after VALUES. All you need to do is add a closing bracket at the end of the query string and you're good to go:
c.execute(f"""INSERT INTO XYstats (run_ID, sample_ID, X_TOTAL_COVERAGE, X_TARGET_COUNT, X_MEAN_TARGET_COVERAGE, Y_TOTAL_COVERAGE, Y_TARGET_COUNT, Y_MEAN_TARGET_COVERAGE, Ymeancov_Xmeancov)
VALUES
('{runID}',
'{sampleID}',
'{XY_df.iloc[0,1]}',
'{XY_df.iloc[0,2]}',
'{XY_df.iloc[0,3]}',
'{XY_df.iloc[1,1]}',
'{XY_df.iloc[1,2]}',
'{XY_df.iloc[1,3]}',
'{XY_df.iloc[2,1]}')
""")
I am working with Python and SQLite. I am constantly getting this message
"near ")": syntax error".
I tried to add a semi-colon to all the queries but still, I get this error message.
tables.append("""
CREATE TABLE IF NOT EXISTS payment (
p_id integer PRIMARY KEY,
o_id integer NON NULL,
FOREIGN KEY(o_id) REFERENCES orders(o_id),
);"""
)
You have a comma before the final closing ). Simply remove it.
i.e. use :-
tables.append("""
CREATE TABLE IF NOT EXISTS payment (
p_id integer PRIMARY KEY,
o_id integer NON NULL,
FOREIGN KEY(o_id) REFERENCES orders(o_id)
);"""
)
Remove the comma in the end of the FOREIGN KEY(o_id) REFERENCES orders(o_id),
The working code will be:
tables.append("""
CREATE TABLE IF NOT EXISTS payment (
p_id integer PRIMARY KEY,
o_id integer NON NULL,
FOREIGN KEY(o_id) REFERENCES orders(o_id)
);"""
)
Try this:
tables = []
tables.append("""
CREATE TABLE IF NOT EXISTS payment p_id integer PRIMARY KEY,
o_id integer NON NULL FOREIGN KEY(o_id) REFERENCES orders(o_id),
""")
print(tables)
I am trying to create a copy of a table through a python script that has all the qualities of the original except for the partitions. I want to do this multiple times in my script (through a for loop) because I want to mysqldump daily files of old data from that table, so I'm trying to use something like:
CREATE TABLE temp_utilization LIKE utilization WITHOUT PARTITIONING;
Here is the original table:
CREATE TABLE `utilization` (
`wrep_time` timestamp NULL DEFAULT NULL,
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (UNIX_TIMESTAMP(wrep_time))
(PARTITION p0 VALUES LESS THAN (1391990400) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN (1392076800) ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN (1392163200) ENGINE = InnoDB,
PARTITION p3 VALUES LESS THAN (1392249600) ENGINE = InnoDB,
PARTITION p492 VALUES LESS THAN (1434499200) ENGINE = InnoDB,
PARTITION p493 VALUES LESS THAN (1434585600) ENGINE = InnoDB,
PARTITION p494 VALUES LESS THAN (1434672000) ENGINE = InnoDB,
PARTITION p495 VALUES LESS THAN (1434758400) ENGINE = InnoDB,
PARTITION p496 VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */
I would like to create a temp table which contains a create table like this:
CREATE TABLE `temp_utilization` (
`wrep_time` timestamp NULL DEFAULT NULL,
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
mysql> alter table utilization remove partitioning;
Query OK, 0 rows affected (0.40 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> show create table utilization\G
*************************** 1. row ***************************
Table: utilization
Create Table: CREATE TABLE `utilization` (
`wrep_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`end_time` timestamp NULL DEFAULT NULL,
`location` varchar(64) NOT NULL,
`sub_location` varchar(64) NOT NULL,
`model_id` varchar(255) DEFAULT NULL,
`offline` int(11) DEFAULT NULL,
`disabled` int(11) NOT NULL DEFAULT '0',
`total` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`location`,`sub_location`,`wrep_time`),
KEY `key_location` (`location`),
KEY `key_sub_location` (`sub_location`),
KEY `end_time` (`end_time`),
KEY `wrep_time` (`wrep_time`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
So, for your example:
CREATE TABLE temp_utilization LIKE utilization;
ALTER TABLE temp_utilization REMOVE PARTITIONING;
Then during your loop you can CREATE TABLE t1 LIKE temp_utilization or however you wish to name the tables
No, it does not appear that you can create a table like another table without partitions, if it is already partitioned, in one command as you suggested above.
The partition is part of the table definition and is stored in the metadata. You can check that by executing show create table yourtablename;
If you just want to create the table over and over again in a loop without the partitions and the data I see three (added one b/c of Cez) options.
have the table definitions hard coded in your script
create the table in the DB without the partitions. So you have one temp table already created and use that as your template to loop through.
run two separate command from your script: A create table like and then an alter table to remove the partitions in a loop.
You can choose which options best suits you for your environment.
You can reference your options when creating a table at dev.mysql.