I have a table in which all the 120 fields have got type varchar(75). I have coded like this.
sql = "create table " + tableName + "("
for i in range(len(flds)):
if i == len(flds) - 1:
sql += flds[i] + " varchar(75))"
else:
sql += flds[i] + " varchar(75), "
Is it possible to get a one-liner for it?
Thanks!
First, let's use join so we don't need the commas and if. And, while we're at it, we can just loop over flds instead of range(len(flds)):
columns = []
for fld in flds:
columns.append(fld + " varchar(75)"
Of course this means we have to add the ) on at the end:
sql += ', '.join(columns) + ')'
Now we can turn that loop into a comprehension:
columns = (fld + " varchar(75)" for fld in flds)
And now, we can inline that into the join:
sql += ', '.join(fld + " varchar(75)" for fld in flds) + ')'
And now, we have two lines that can obviously be combined into one:
sql = "create table " + tableName + "(" + ', '.join(fld + " varchar(75)" for fld in flds) + ')'
But that's way over 80 characters, so probably better to write it as two lines anyway. I'd probably do it like this:
columns = ', '.join(fld + " varchar(75)" for fld in flds)
sql = "create table " + tableName + "(" + columns + ")"
And finally, let's use an f-string instead of concatenating with +, which makes things only a little shorter, but a lot more readable.
columns = ', '.join(f'{fld} varchar(75)' for fld in flds)
sql = f'create table {tableName} ({columns})'
You can use join with format:
v = "create table {} ({} varchar(75));".format(tableName, " varchar(75), ".join(flds))
Related
Instead of enumerating the items in a list in this scenario:
cursor.execute("INSERT INTO " + table_name + " (url, date, " + column_headers + "total_keywords" + ")" \
"VALUES (%s,%s,%s,%s,%s,%s)",
(response.url,
datetime.date.today(),
some_list[0],
some_list[1],
some_list[2],
some_list[3],
...
some_list[n]
)
)
I would like to pass num_keywords through more generally. Is it possible to use a list in an SQL query specifically in the VALUES part of the query?
The following code snippet does not work, but this is the idea:
cursor.execute("INSERT INTO " + table_name + " (url, date, " + column_headers + "total_keywords" + ")" \
"VALUES (%s,%s,%s)",
(response.url,
datetime.date.today(),
some_list
)
)
Is this possible? How would I do it?
Use the * operator to spread the list.
cursor.execute("INSERT INTO " + table_name + " (url, date, " + column_headers + "total_keywords" + ")" \
"VALUES (%s,%s,%s,%s,%s,%s)",
(response.url, datetime.date.today(), *list)
)
BTW, don't use list as a variable name, it will replace the built-in class with that name.
I'm inserting data into SQLite using Python
database_tasks.bookId is a class variable
def insert(self, conn,dbname, name, writer='none'):
c = conn.cursor()
if dbname == "Book_database":
database_tasks.bookId += 1
sql= "INSERT INTO Book_database VALUES(" + database_tasks.bookId + "," + name + "," + writer + ")"
else:
database_tasks.studentId += 1
sql= "INSERT INTO Student VALUES("+ database_tasks.studentId + "," + name + ")"
c.execute(sql)
conn.commit()
return None
I'm getting an error with Insert command.
sql= "INSERT INTO Book_database VALUES(" + database_tasks.bookId + "," + name + "," + writer + ")"
TypeError: can only concatenate str (not "int") to str
You should never build a query string including values to insert that way but used parameterized queries. It is an anti-pattern that has been used for decades to build SQL injection attacks. You should use:
if dbname == "Book_database":
database_tasks.bookId += 1
c.execute("INSERT INTO Book_database VALUES(?,?,?)",
(database_tasks.bookId, name, writer))
else:
...
it looks like your bookId is int. Cast it to str before combining.
sql= "INSERT INTO Book_database VALUES(" + str(database_tasks.bookId) + "," + name + "," + writer + ")"
The error is fairly self explanatory - you are trying to concatenate an integer to a string.
my assumption is that database_tasks.studentId is an int, so changing your query to
sql= "INSERT INTO Book_database VALUES(" + str(database_tasks.bookId) + "," + name + "," + writer + ")"
would fix this, however be aware that if this is public facing, it would be very easy to exploit this, I'd recommend fully validating input, or changing your approach to this problem.
I have a Pandas DataFrame with many columns out of which one column is 'value' containing an HTML web page. I am doing an Upsert query for each row of the DataFrame but I am getting the following error:
I have tried escaping the HTML with the following methods:
df.value = df.value.apply(lambda x: re.escape(x))
df.value = df.value.apply(lambda x: MySQLdb.escape_string(x))
Here is my function:
non_key_cols = df.columns.tolist()
non_key_cols.remove(primary_key)
# df.value = df.value.apply(lambda x: re.escape(x))
df.value = df.value.apply(lambda x: MySQLdb.escape_string(x))
enclose_with_quote = [True if type_name.name=='object' else False for type_name in df.dtypes]
all_cols = df.columns.tolist()
#enclose df columns in inverted commas
for i in range(len(enclose_with_quote)):
if enclose_with_quote[i]:
df[all_cols[i]] = df[all_cols[i]].apply(lambda x: '"' + x + '"')
else:
df[all_cols[i]] = df[all_cols[i]].apply(lambda x: str(x))
sql = "INSERT INTO " \
+ tablename \
+ "(" + ", ".join([col for col in df.columns]) + ")" \
+ " VALUES " \
+ ", ".join(["(" + ", ".join(list(row)) + ")" for row in df.itertuples(index=False, name=None)]) \
+ " ON CONFLICT (" + primary_key + ") DO UPDATE SET " \
+ ", ".join([col + "=EXCLUDED." + col for col in non_key_cols])
conn = _getpostgres_connection()
cur = conn.cursor()
cur.execute(sql)
cur.close()
conn.commit()
conn.close()
This is the error I get:
ProgrammingError: syntax error at or near "margin" LINE 1:
...t_of_nums_not_in_table_regex) VALUES ("<p style=\"margin: 0p...
The issue that you write string within doubles quotes. In Postgres, doubles quotes means a column/table name. You would have to use single quotes for strings.
if enclose_with_quote[i]:
df[all_cols[i]] = df[all_cols[i]].apply(lambda x: "'" + x + "'")
That being said, you will then get an error if your string contains single quotes. The safest - and simplest - way is to use a parameterized query, which will handle the quote escaping by itself. Else have a look at this post for using a custom string delimiter.
I have following query to delete the duplicates from the table in database.
WITH x AS (SELECT "region_code" dup, min(ctid)
FROM public.test2 GROUP BY "region_code"
HAVING count(*) > 1)
DELETE FROM public.test2
USING x
WHERE (region_code) = (dup) AND public.test2.ctid <> x.min
RETURNING *;
Now I want to execute this query using python. When I run this query in python, nothing happens. I am using sqlalchemy with python 3.6.
query = "WITH x AS (SELECT \"region_code\" dup, min(ctid) FROM " + schema + "." + table_name + " GROUP BY \"region_code\" HAVING count(*) > 1) DELETE FROM " + schema + "." + table_name +" USING x WHERE (region_code) = (dup) AND " + schema + "." + table_name +".ctid <> x.min RETURNING *;"
data = con.execute(query)
I am new in python, and using Python & PostgreSQL (9.03) (and psycopg2 to interface between the two) in Windows XP environment.
I am working on a huge spatial dataset road network dataset, and seperating the data per Country through ArcGIS Geoprocessing, and automatically store and them in a PostGIS (1.5) Database.
While when retrieving values from the database everything works as planned:
...
try:
conn = psycopg2.connect("host = '" + HostName + "' dbname='" + DBName + "' user='" + Username + "' password='" + Password + "'")
curs = conn.cursor()
except:
print "Unable to connect to the database"
SQLStatement = "SELECT data_partition FROM datasets WHERE map_partition='" + MapPartitions[0] + "'"
curs.execute(SQLStatement)
...
When I am trying to pass the following Union Statement to Postgres, there is no resulting table, while if I take the printed SQL Statement and run it in as an SQL Statement and run it PostgresSQL, it creates the desired resulting table:
conn = psycopg2.connect("host = '" + HostName + "' dbname='" + DBName + "' user='" + Username + "' password='" + Password + "'")
cur = conn.cursor()
SQLStatement = (
"CREATE TABLE " + Schema + "." + PartitionTableName + " AS \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net0 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net1 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net2 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net3 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net4 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net5;\n"
"\n"
"\n"
"ALTER TABLE " + Schema + "." + partName + "_Lines_" + Rel + "\n"
"DROP COLUMN gid;\n"
cur.execute(SQLStatement)
conn.commit()
cur.close()
If we print the SQL Statement, this is the resulting query:
print SQLStatement
CREATE TABLE compresseddata.FRA24_Lines_2011_03 AS
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net0 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net1 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net2 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net3 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net4 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net5;
ALTER TABLE compresseddata.FRA24_Lines_2011_03
DROP COLUMN gid;
I am using variables in the to Merge different Road Network Classes, and due to different Partitions of my dataset, I need to iterate through, them, but for some reason that I cannot still understand, there is no table being produced.
Any ideas?
Thanx in advance for the help
THe SQL you are sending are actually 3 statements, not 1.
I never tried this but I expect execute to complain about this.
Additionally there is a semicolon missing in the ALTER TABLE statement.
I would recommend to add exception handling to your code and execute each SQL statement separately so you get better error reporting on what might go wrong.
Indeed Peter, this seems to be the case.
More specifically Each SQL Statement must be passed separately through:
curs.execute(SQLStatement)
and them committed via:
conn.commit()
All the changes will then be apparent in the database.
Thanx again
As already mentioned, individually executing each statement and checking the exception can provide good insight to what is occurring.
In particular psycopg2 will raise psycopg2.ProgrammingError. If the error message is not useful, you may have better luck looking up the exception's pgcode and then investigating that.
PGCodes for 9.1:
http://www.postgresql.org/docs/9.1/static/errcodes-appendix.html ).
try:
cur.execute(SQLQUERY)
except psycopg2.ProgrammingError as e:
# Err code lookup at http://www.postgresql.org/docs/9.1/static/errcodes-appendix.html
print "psycopg2 error code %s" % e.pgcode
raise e
NOTE: A cursors execute statement CAN take multiple sql statements in a single string.
ex: cur.execute('create table ABBA (); create table BETA ();') is a perfectly legitimate statement.
For this reason, do not expect cursor.execute to perform any sanity checks on a string only input!
I'd suggest (except for special rare circumstances) to execute each statement individually.