I'm trying to move to SQLAlchemy from psycopg2. At a point in my code, I've got a list of column names in columnList to be created as text types in a table name stored in tableName. Using psycopg2, this query works as follows:
ct = 'CREATE TABLE ' + tableName + ' ( id integer PRIMARY KEY, '
for _col in columnList:
ct += '"' + _col + '" text, '
ct = ct[:-2] + ');'
cur.execute(ct)
conn.commit()
How can I achieve the same thing in SQLAlchemy, bearing in mind that I cannot hard code the column names or table names as they are drawn from a dynamic list?
Also, once this table is created, how should it and the columns be referenced later in the code when querying or inserting values?
Thanks.
Using this sample data:
data = {'table1name': ['textcol1', 'textcol2'],
'table2name': ['textcol3', 'textcol4']}
Here's one way to do it:
from sqlalchemy import Table, Column, Text, Integer, MetaData
metadata = MetaData()
for tblname, colnames in data.items():
Table(
tblname, metadata,
Column('id', Integer, primary_key=True),
*[Column(name, Text) for name in colnames]
)
The tables are automatically mapped to their name in the MetaData.tables dictionary...
print(type(metadata.tables['table1name'])) # <class 'sqlalchemy.sql.schema.Table'>
... so you can always access them through there to perform queries, as long as you have the name of the table.
This will render the create table statements:
from sqlalchemy.schema import CreateTable
for table in metadata.tables.values():
print(CreateTable(table))
Which prints:
CREATE TABLE table1name (
id INTEGER NOT NULL,
textcol1 TEXT,
textcol2 TEXT,
PRIMARY KEY (id) )
CREATE TABLE table2name (
id INTEGER NOT NULL,
textcol3 TEXT,
textcol4 TEXT,
PRIMARY KEY (id) )
Related
I want to create three tables fremdgehen.com, molligundwillig.de and reifer6.de.
But I am not able to transfer each element to CREATE TABLE. Why?
import sqlite3
con = sqlite3.connect('emails.db')
cur = con.cursor()
pages = ['fremdgehen.com', 'molligundwillig.de', 'reifer6.de']
for i in pages:
try:
sql_command = f'''
CREATE TABLE {i} (
email INTEGER,
password VARCHAR,
PRIMARY KEY (id));
cur.executescript(sql_command)
con.commit()
'''
except:
pass
con.close()
Since the table names contain a . character, you have to escape the names.
sql_command = f'''
CREATE TABLE `{i}` (
email INTEGER,
password VARCHAR,
PRIMARY KEY (id));
cur.executescript(sql_command)
con.commit()
'''
Note that having . in table and column names is inconvenient, because . is the separator between database, table, and column names. So you'll have to escape the table name in all your queries. Consider removing the . before using it as a table name to simplify this.
I have a function that I use to update tables in PostgreSQL. It works great to avoid duplicate insertions by creating a temp table and dropping it upon completion. However, I have a few tables with serial ids and I have to pass the serial id in a column. Otherwise, I get an error that the keys are missing. How can I insert values in those tables and have the serial key get assigned automatically? I would prefer to modify the function below if possible.
def export_to_sql(df, table_name):
from sqlalchemy import create_engine
engine = create_engine(f'postgresql://{user}:{password}#{host}:5432/{user}')
df.to_sql(con=engine,
name='temporary_table',
if_exists='append',
index=False,
method = 'multi')
with engine.begin() as cnx:
insert_sql = f'INSERT INTO {table_name} (SELECT * FROM temporary_table) ON CONFLICT DO NOTHING; DROP TABLE temporary_table'
cnx.execute(insert_sql)
code used to create the tables
CREATE TABLE symbols
(
symbol_id serial NOT NULL,
symbol varchar(50) NOT NULL,
CONSTRAINT PK_symbols PRIMARY KEY ( symbol_id )
);
CREATE TABLE tweet_symols(
tweet_id varchar(50) REFERENCES tweets,
symbol_id int REFERENCES symbols,
PRIMARY KEY (tweet_id, symbol_id),
UNIQUE (tweet_id, symbol_id)
);
CREATE TABLE hashtags
(
hashtag_id serial NOT NULL,
hashtag varchar(140) NOT NULL,
CONSTRAINT PK_hashtags PRIMARY KEY ( hashtag_id )
);
CREATE TABLE tweet_hashtags
(
tweet_id varchar(50) NOT NULL,
hashtag_id integer NOT NULL,
CONSTRAINT FK_344 FOREIGN KEY ( tweet_id ) REFERENCES tweets ( tweet_id )
);
CREATE INDEX fkIdx_345 ON tweet_hashtags
(
tweet_id
);
The INSERT statement does not define the target columns, so Postgresql will attempt to insert values into a column that was defined as SERIAL.
We can work around this by providing a list of target columns, omitting the serial types. To do this we use SQLAlchemy to fetch the metadata of the table that we are inserting into from the database, then make a list of target columns. SQLAlchemy doesn't tell us if a column was created using SERIAL, but we will assume that it is if it is a primary key and is set to autoincrement. Primary key columns defined with GENERATED ... AS IDENTITY will also be filtered out - this is probably desirable as they behave in the same way as SERIAL columns.
import sqlalchemy as sa
def export_to_sql(df, table_name):
engine = sa.create_engine(f'postgresql://{user}:{password}#{host}:5432/{user}')
df.to_sql(con=engine,
name='temporary_table',
if_exists='append',
index=False,
method='multi')
# Fetch table metadata from the database
table = sa.Table(table_name, sa.MetaData(), autoload_with=engine)
# Get the names of columns to be inserted,
# assuming auto-incrementing PKs are serial types
column_names = ','.join(
[f'"{c.name}"' for c in table.columns
if not (c.primary_key and c.autoincrement)]
)
with engine.begin() as cnx:
insert_sql = sa.text(
f'INSERT INTO {table_name} ({column_names}) (SELECT * FROM temporary_table) ON CONFLICT DO NOTHING; DROP TABLE temporary_table'
)
cnx.execute(insert_sql)
i want to create a table in sqlite3 with python put when i change number1 the name of table to the name of variable make the name of variable the name of table not the value in variable
db.execute("create table if not exists number1 (ID integer primary key autoincrement, item text, quantity integer,price_per_once integer, total integer)")
Thanks in advance
Concatenate a variable:
db.execute("create table if not exists " + table_name + " (ID integer primary key autoincrement, item text, quantity integer,price_per_once integer, total integer)")
Although if you're creating lots of tables with the same stucture, there's probably a better way to design your schema. Whatever the difference is between all these tables, it should probably just be a column in the table.
Have you tried something like this:
db.execute("create table " + tableNameVariable + " (...)")
I have a list of tuples like so:
>>> all_names = c.execute("""select name from fb_friends""")
>>> for name in all_names:
... print(name)
('Jody Ann Elizabeth Lill',)
('Georgia Gee Smith',)
...(282 more)...
('Josh Firth',)
('Danny Hallas',)
And I want to create a table for each individual person. First I need to replace all spaces with an underscore in order to be SQLite3 table names, so I do that like so:
>>> all_names = c.execute("""select name from fb_friends""")
>>> for name in all_names:
... friends_name = name[0].replace(" ", "_")
... print(friends_name)
Jody_Ann_Elizabeth_Lill
Georgia_Gee_Smith
...(282 more)...
Josh_Firth
Danny_Hallas
So if I understand correctly I now have a list, not a list of tuples..? Which should be simple to create all my tables from this list like so:
>>> all_names = c.execute("""select name from fb_friends""")
>>> for name in all_names:
... friends_name = name[0].replace(" ", "_")
... c.execute("""create table {0} (`id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, `work` TEXT NOT NULL, `education` TEXT, `current_city` TEXT, `phone_number` TEXT, `dob` TEXT, `gender` TEXT, `sexual_orientation` TEXT, `religion` TEXT, `relationship_status` TEXT, `about_me` TEXT )""".format(friends_name))
But all it does is create a table from the first name in the list, I would have thought the for loop would iterate over the list of names and create a table for each one, which is what I want, can people please advise me on a few things:
Is using the .replace method the best way to get the underscores in the names? If no then what is?
Why is the for loop not iterating over each name to create the tables? And how do I make it do that?
Are my methods correct at all? If not then what methods would do it better?
While your immediate issue is resolved likely due to the raw cursor not retrieving all records from database whereas cursor.fetchall() or list(cursor) imports all rows to client side (Python), a slight memory footprint but may ensure rows import in script, I would be remiss to not encourage best practices in database design.
As mentioned in comments, reconsider this approach and normalize your entire database model for one-to-many or many-to-many relationships between tables as relational databases are intended to do. This way you avoid the overhead, maintenance, and querying challenges of 280+ identically structured tables.
From your explanation, you sound like you need a distinct friends table with one-to-many links with workplaces and posts table and any others, all connected to a friendid as foreign keys. Ultimately, you would create tables only once and populate data with append queries:
-- CREATE TABLES
CREATE TABLE friends (
`friendid` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
`first_name` TEXT, last_name TEXT,
`work` TEXT, `education` TEXT,
`current_city` TEXT, `phone_number` TEXT,
`dob` TEXT, `gender` TEXT, `sexual_orientation` TEXT,
`religion` TEXT, `relationship_status` TEXT, `about_me` TEXT);
CREATE TABLE workplaces (
`workplaceid` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
`friendid` INTEGER,
`workplace` TEXT NOT NULL,
`city` TEXT, `state` TEXT);
CREATE TABLE posts (
`postid` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
`friendid` INTEGER,
`text` TEXT NOT NULL,
`postdate` TEXT, `postlocation` TEXT);
-- POPULATE TABLES (adjust to actual tables/fields)
INSERT INTO friends (first_name, last_name)
SELECT SUBSTR(`name`, 1, INSTR(`name`, " ")) AS first_name,
SUBSTR(`name`, INSTR(`name`, " ")+1) AS last_name,
FROM fb_friends;
INSERT INTO workplaces (friendid, work, city)
SELECT f.friendid, w.work, w.city
FROM fb_work w INNER JOIN friends f ON w.name = f.first_name & ' ' & f.last_name;
INSERT INTO posts (friendid, `text`, postdate)
SELECT f.friendid, p.post_text, p.post_date
FROM fb_posts p INNER JOIN friends f ON p.name = f.first_name & ' ' & f.last_name;
So after continually playing around with various methods I managed to figure out how to do this. In order to get the the code to work all that had to be done was to convert it to a list, like so:
>>> all_names = c.execute("""select name from fb_friends""")
>>> names = list(all_names) # I just added this line and changed the variable names below
>>> for name in names:
... friends_name = name[0].replace(" ", "_")
... c.execute("""create table {0} (`id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, `work` TEXT NOT NULL, `education` TEXT, `current_city` TEXT, `phone_number` TEXT, `dob` TEXT, `gender` TEXT, `sexual_orientation` TEXT, `religion` TEXT, `relationship_status` TEXT, `about_me` TEXT )""".format(name))
please try out this solution:
staff = ['a', 'b', 'c']
for name in staff:
c.execute("CREATE TABLE IF NOT EXISTS %s (name TEXT);" % (name))
May this question is silly, but i am not able to generate the regex expression to fetch table Name and PRIMARY KEY.
TABLE:
CREATE TABLE 'dhcpr_dhcprelayinterface' (
'vrId' integer default 0,
'ifName' string ,
PRIMARY KEY(ifName,vrId),
FOREIGN KEY (vrId) REFERENCES 'vr_vr'(vrId) ON DELETE CASCADE ON UPDATE CASCADE);
I am using:
begin = re.compile(r"CREATE TABLE[ \"]*([^ \"]+)[ \"]*[(]([^/;]+)[/;]",re.IGNORECASE) to fetch all table name and data.
But i would need data only with table name and PRIMARY KEY.
Expected Output:
dhcpr_dhcprelayinterface
PRIMARY KEY(ifName,vrId)
This solution takes care of some issues you seem not worried about (but which are good to worry about), e.g., SQLite allows you to write escaped ' as '', and there may be any number of spaces, even newlines, between CREATE and TABLE, and between PRIMARY, KEY, and (:
s = """\
CREATE TABLE 'dhcpr_dhcprelayinterface' (
'vrId' integer default 0,
'ifName' string ,
PRIMARY KEY(ifName,vrId),
FOREIGN KEY (vrId) REFERENCES 'vr_vr'(vrId)
ON DELETE CASCADE ON UPDATE CASCADE);
"""
pattern = """
CREATE \s+ TABLE \s+
'((?:[^']|'')*)' # allows escaped single quote
.+ # stuff between table name and primary key
(PRIMARY \s+ KEY\s? \([^)]*\))
"""
mo = re.search(pattern, s, re.IGNORECASE | re.VERBOSE | re.DOTALL)
print(mo.groups())
Output:
('dhcpr_dhcprelayinterface', 'PRIMARY KEY(ifName,vrId)')
The following was tested using python2.7:
>>> table_string = """
... CREATE TABLE 'dhcpr_dhcprelayinterface' (
... 'vrId' integer default 0,
... 'ifName' string ,
... PRIMARY KEY(ifName,vrId),
... FOREIGN KEY (vrId) REFERENCES 'vr_vr'(vrId) ON DELETE CASCADE ON UPDATE CASCAD
E);"""
>>> p = r'CREATE TABLE\s+\'([^\']+)[\s\S]+PRIMARY KEY\(([^,]+),([^\)]+)\)'
>>> re.findall(p,table_string)
[('dhcpr_dhcprelayinterface', 'ifName', 'vrId')]
The explanation can be found here.
I'm sure you can solve it with regular expressions or sqlparse, but here is a "fun" way of approaching the problem just for educational purposes - using sqlite3 in memory database - actually create the table and get the table_name from the sqlite_master internal table and primary key columns from the PRAGMA table_info:
import sqlite3
query = """
CREATE TABLE 'dhcpr_dhcprelayinterface' (
'vrId' integer default 0,
'ifName' string ,
PRIMARY KEY(ifName,vrId),
FOREIGN KEY (vrId) REFERENCES 'vr_vr'(vrId) ON DELETE CASCADE ON UPDATE CASCADE);
"""
db = sqlite3.connect(":memory:")
cursor = db.cursor()
cursor.execute(query)
db.commit()
# get table name
cursor.execute("select name from sqlite_master where type = 'table'")
table_name = cursor.fetchone()[0]
print(table_name)
# get primary key columns
cursor.execute("PRAGMA table_info(%s);" % table_name)
pk_columns = [row[1] for row in cursor.fetchall()[::-1]]
print(pk_columns)
Prints:
dhcpr_dhcprelayinterface
['ifName', 'vrId']
You can use this to just get the table name and primary key.
begin = re.compile(r"CREATE TABLE[ ']*([^ ']+)[ ']*[(][^/;]+(PRIMARY KEY.*),[^/;]+;$", re.IGNORECASE)
begin.findall(YOUR_STR)
Outputs:
In [1]: a = """CREATE TABLE 'dhcpr_dhcprelayinterface' (
...: 'vrId' integer default 0,
...: 'ifName' string ,
...: PRIMARY KEY(ifName,vrId),
...: FOREIGN KEY (vrId) REFERENCES 'vr_vr'(vrId) ON DELETE CASCADE ON
UPDATE CASCADE);"""
In [2]: begin = re.compile(r"CREATE TABLE[ ']*([^ ']+)[ ']*[(][^/;]+(PRIMARY KEY.*),[^/;]+;$", re.IGNORECASE)
In [3]: begin.findall(a)
Out[3]: [('dhcpr_dhcprelayinterface', 'PRIMARY KEY(ifName,vrId)')]