How to convert WITH as SELECT query expression in python? - python

I have following query to delete the duplicates from the table in database.
WITH x AS (SELECT "region_code" dup, min(ctid)
FROM public.test2 GROUP BY "region_code"
HAVING count(*) > 1)
DELETE FROM public.test2
USING x
WHERE (region_code) = (dup) AND public.test2.ctid <> x.min
RETURNING *;
Now I want to execute this query using python. When I run this query in python, nothing happens. I am using sqlalchemy with python 3.6.
query = "WITH x AS (SELECT \"region_code\" dup, min(ctid) FROM " + schema + "." + table_name + " GROUP BY \"region_code\" HAVING count(*) > 1) DELETE FROM " + schema + "." + table_name +" USING x WHERE (region_code) = (dup) AND " + schema + "." + table_name +".ctid <> x.min RETURNING *;"
data = con.execute(query)

Related

Error : Insert Command error with Database as SQLITE

I'm inserting data into SQLite using Python
database_tasks.bookId is a class variable
def insert(self, conn,dbname, name, writer='none'):
c = conn.cursor()
if dbname == "Book_database":
database_tasks.bookId += 1
sql= "INSERT INTO Book_database VALUES(" + database_tasks.bookId + "," + name + "," + writer + ")"
else:
database_tasks.studentId += 1
sql= "INSERT INTO Student VALUES("+ database_tasks.studentId + "," + name + ")"
c.execute(sql)
conn.commit()
return None
I'm getting an error with Insert command.
sql= "INSERT INTO Book_database VALUES(" + database_tasks.bookId + "," + name + "," + writer + ")"
TypeError: can only concatenate str (not "int") to str
You should never build a query string including values to insert that way but used parameterized queries. It is an anti-pattern that has been used for decades to build SQL injection attacks. You should use:
if dbname == "Book_database":
database_tasks.bookId += 1
c.execute("INSERT INTO Book_database VALUES(?,?,?)",
(database_tasks.bookId, name, writer))
else:
...
it looks like your bookId is int. Cast it to str before combining.
sql= "INSERT INTO Book_database VALUES(" + str(database_tasks.bookId) + "," + name + "," + writer + ")"
The error is fairly self explanatory - you are trying to concatenate an integer to a string.
my assumption is that database_tasks.studentId is an int, so changing your query to
sql= "INSERT INTO Book_database VALUES(" + str(database_tasks.bookId) + "," + name + "," + writer + ")"
would fix this, however be aware that if this is public facing, it would be very easy to exploit this, I'd recommend fully validating input, or changing your approach to this problem.

Select Column from Table in Django Not Working

I am trying to access the two specific column from the table in Django it is not Working but When I am trying to access select * it is working
I am using postgresql
When I am trying to access select all its working this is I am trying to access for particular column
def bigdataDatabase(X):
engine = sqlalchemy.create_engine('postgresql+psycopg2://postgres:password#localhost/db_name')
con = engine.connect()
result = con.execute(
"Select Orign,Departure From 'table_name' WHERE index = '" + str(X) + "'")
This is not working
I have also tried with this
result = con.execute("Select tablename.Orign,tablename.departure From 'table_name' WHERE index = '" + str(X) + "'")
both the above code is not working
Programming Error column does not exist
But When I am executing all this it is working
result = con.execute("Select * From 'table_name' WHERE index = '" + str(X) + "'")
I have found the solution of the problem the query should be executed like this
result = con.execute('Select "Orign","Departure" From "Table_name" WHERE index = ' + str(X))

Python Generate Create Table

I have a table in which all the 120 fields have got type varchar(75). I have coded like this.
sql = "create table " + tableName + "("
for i in range(len(flds)):
if i == len(flds) - 1:
sql += flds[i] + " varchar(75))"
else:
sql += flds[i] + " varchar(75), "
Is it possible to get a one-liner for it?
Thanks!
First, let's use join so we don't need the commas and if. And, while we're at it, we can just loop over flds instead of range(len(flds)):
columns = []
for fld in flds:
columns.append(fld + " varchar(75)"
Of course this means we have to add the ) on at the end:
sql += ', '.join(columns) + ')'
Now we can turn that loop into a comprehension:
columns = (fld + " varchar(75)" for fld in flds)
And now, we can inline that into the join:
sql += ', '.join(fld + " varchar(75)" for fld in flds) + ')'
And now, we have two lines that can obviously be combined into one:
sql = "create table " + tableName + "(" + ', '.join(fld + " varchar(75)" for fld in flds) + ')'
But that's way over 80 characters, so probably better to write it as two lines anyway. I'd probably do it like this:
columns = ', '.join(fld + " varchar(75)" for fld in flds)
sql = "create table " + tableName + "(" + columns + ")"
And finally, let's use an f-string instead of concatenating with +, which makes things only a little shorter, but a lot more readable.
columns = ', '.join(f'{fld} varchar(75)' for fld in flds)
sql = f'create table {tableName} ({columns})'
You can use join with format:
v = "create table {} ({} varchar(75));".format(tableName, " varchar(75), ".join(flds))

Get MSSQL table column names using pyodbc in python

I am trying to get the mssql table column names using pyodbc, and getting an error saying
ProgrammingError: No results. Previous SQL was not a query.
Here is my code:
class get_Fields:
def GET(self,r):
web.header('Access-Control-Allow-Origin', '*')
web.header('Access-Control-Allow-Credentials', 'true')
fields = []
datasetname = web.input().datasetName
tablename = web.input().tableName
cnxn = pyodbc.connect(connection_string)
cursor = cnxn.cursor()
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,* FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = " + "'"+ tablename + "'"
cursor.execute(query)
DF = DataFrame(cursor.fetchall())
columns = [column[0] for column in cursor.description]
return json.dumps(columns)
how to solve this?
You can avoid this by using some of pyodbc's built in methods. For example, instead of:
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,* FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = " + "'"+ tablename + "'"
cursor.execute(query)
DF = DataFrame(cursor.fetchall())
Try:
column_data = cursor.columns(table=tablename, catalog=datasetname, schema='dbo').fetchall()
print(column_data)
That will return the column names (and other column metadata). I believe the column name is the fourth element per row. This also relieves the very valid concerns about SQL injection. You can then figure out how to build your DataFrame from the resulting data.
Good luck!
Your line
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,*...
Will produce something like
USE[databasename]SELECT ...
In SSMS this would work, but I'd suggest to look on proper spacing and to separate the USE-statement with a semicolon:
query = "USE " + "[" +datasetname+ "]; " + "SELECT COLUMN_NAME,*...
Set the database context using the Database attribute when building the connection string
Use parameters any time you are passing user input (especially from HTTP requests!) to a WHERE clause.
These changes eliminate the need for dynamic SQL, which can be insecure and difficult to maintain.

Python PostgreSQL Statement Problem psycopg2 cursor.execute(Table Union)

I am new in python, and using Python & PostgreSQL (9.03) (and psycopg2 to interface between the two) in Windows XP environment.
I am working on a huge spatial dataset road network dataset, and seperating the data per Country through ArcGIS Geoprocessing, and automatically store and them in a PostGIS (1.5) Database.
While when retrieving values from the database everything works as planned:
...
try:
conn = psycopg2.connect("host = '" + HostName + "' dbname='" + DBName + "' user='" + Username + "' password='" + Password + "'")
curs = conn.cursor()
except:
print "Unable to connect to the database"
SQLStatement = "SELECT data_partition FROM datasets WHERE map_partition='" + MapPartitions[0] + "'"
curs.execute(SQLStatement)
...
When I am trying to pass the following Union Statement to Postgres, there is no resulting table, while if I take the printed SQL Statement and run it in as an SQL Statement and run it PostgresSQL, it creates the desired resulting table:
conn = psycopg2.connect("host = '" + HostName + "' dbname='" + DBName + "' user='" + Username + "' password='" + Password + "'")
cur = conn.cursor()
SQLStatement = (
"CREATE TABLE " + Schema + "." + PartitionTableName + " AS \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net0 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net1 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net2 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net3 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net4 UNION \n"
"SELECT * FROM " + Schema + "." + partName + "_Lines_" + Rel + "_Net5;\n"
"\n"
"\n"
"ALTER TABLE " + Schema + "." + partName + "_Lines_" + Rel + "\n"
"DROP COLUMN gid;\n"
cur.execute(SQLStatement)
conn.commit()
cur.close()
If we print the SQL Statement, this is the resulting query:
print SQLStatement
CREATE TABLE compresseddata.FRA24_Lines_2011_03 AS
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net0 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net1 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net2 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net3 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net4 UNION
SELECT * FROM compresseddata.FRA24_Lines_2011_03_Net5;
ALTER TABLE compresseddata.FRA24_Lines_2011_03
DROP COLUMN gid;
I am using variables in the to Merge different Road Network Classes, and due to different Partitions of my dataset, I need to iterate through, them, but for some reason that I cannot still understand, there is no table being produced.
Any ideas?
Thanx in advance for the help
THe SQL you are sending are actually 3 statements, not 1.
I never tried this but I expect execute to complain about this.
Additionally there is a semicolon missing in the ALTER TABLE statement.
I would recommend to add exception handling to your code and execute each SQL statement separately so you get better error reporting on what might go wrong.
Indeed Peter, this seems to be the case.
More specifically Each SQL Statement must be passed separately through:
curs.execute(SQLStatement)
and them committed via:
conn.commit()
All the changes will then be apparent in the database.
Thanx again
As already mentioned, individually executing each statement and checking the exception can provide good insight to what is occurring.
In particular psycopg2 will raise psycopg2.ProgrammingError. If the error message is not useful, you may have better luck looking up the exception's pgcode and then investigating that.
PGCodes for 9.1:
http://www.postgresql.org/docs/9.1/static/errcodes-appendix.html ).
try:
cur.execute(SQLQUERY)
except psycopg2.ProgrammingError as e:
# Err code lookup at http://www.postgresql.org/docs/9.1/static/errcodes-appendix.html
print "psycopg2 error code %s" % e.pgcode
raise e
NOTE: A cursors execute statement CAN take multiple sql statements in a single string.
ex: cur.execute('create table ABBA (); create table BETA ();') is a perfectly legitimate statement.
For this reason, do not expect cursor.execute to perform any sanity checks on a string only input!
I'd suggest (except for special rare circumstances) to execute each statement individually.

Categories