psycopg2 dictionary dump to json - python

Sorry if this is a noob question, but I am trying to dump a psycopg2 dictionary directly into a json string. I do get a return value in the browser, but it isn't formatted like most of the other json examples I see. The idea being to dump the result of a select statement into a json string and unbundle it on the other end to add into a database on the client side. The code is below and a sample of the return. Is there a better way to do this operation with json and psycopg2?
# initializing variables
location_connection = location_cursor = 0
sql_string = coordinate_return = data = ""
# opening connection and setting cursor
location_connection = psycopg2.connect("dbname='' user='' password=''")
location_cursor = location_connection.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
# setting sql string and executing query
sql_string = "select * from " + tablename + " where left(alphacoordinate," + str(len(coordinate)) + ") = '" + coordinate + "' order by alphacoordinate;"
location_cursor.execute(sql_string)
data = json.dumps(location_cursor.fetchall())
# closing database connection
location_connection.close()
# returning coordinate string
return data
sample return
"[{\"alphacoordinate\": \"nmaan-001-01\", \"xcoordinate\":
3072951151886, \"planetarydiameter\": 288499, \"planetarymass\":
2.020936938e+27, \"planetarydescription\": \"PCCGQAAA\", \"planetarydescriptionsecondary\": 0, \"moons\": 1"\"}]"

You could create the JSON string directly in Postgres using row_to_json:
# setting sql string and executing query
sql_string = "select row_to_json(" + tablename + ") from " + tablename + " where left(alphacoordinate," + str(len(coordinate)) + ") = '" + coordinate + "' order by alphacoordinate;"
location_cursor.execute(sql_string)
data = location_cursor.fetchall()

Related

I am using Flask and MySQL database but the update query is giving error [duplicate]

I have multiple tables that are updated after a value is changed in a grid. These tables don't always have the same keys or columns so I cannot explicitly name the columns or formats. The only thing that is ever the same, is the column where the keys reside. I know the way I am currently doing this is not correct and leaves me open to injection attacks.
I also ran into an issue where some of the values contain keys that throw an error in the SQL statement. For example, updating WHERE email = t'est#email.com.
I am not really sure of the proper way to write these statements. I did some research and see multiple methods for different purposes but am not sure which is proper. I am looking to do this as dynamically as possible. Can anyone point me in the right direction?
To connect:
import mysql.connector as sql
import MySQLdb
#Connect
self.db_name = 'database'
self.server = 'server'
self.user_id = 'user'
self.pw = 'password'
try:
self.db_con = MySQLdb.connect(user=self.user_id,password=self.pw,database=self.db_name,
host=self.server,charset='utf8',autocommit=True)
self.cursor = self.db_con.cursor()
except:
print("Error connecting")
SQL Statements:
key_id = str("'") + self.GetCellValue(event.GetRow(),1) + str("'")
target_col = self.GetColLabelValue(event.GetCol())
key_col = self.GetColLabelValue(1)
nVal = str("'") + self.GetCellValue(event.GetRow(),event.GetCol()) + str("'")
#SQL STATEMENTS
sql_update = "UPDATE " + tbl + " SET " + target_col + " = " + nVal + " WHERE " + key_col + " = " + key_id + ""
#INSERT
sql_update = ("INSERT INTO " + str(self.tbl) + "(" + self.key_col + ")" + "VALUES (" + str("'") + str(val) + str("'") + ")")
#DELETE
sql_update = "DELETE FROM " + tbl + " WHERE " + self.key_col + " = " + self.key_id + ""
#SELECT
sql_query = "SELECT * FROM " + self.tbl
#Excecute
try:
self.cursor.execute(sql_update)
except:
print('Error')
self.db_con.rollback()
Databases have different notations for "quoting" identifiers (table and column names etc) and values (data).
MySQL uses backticks to quote identifiers. For values, it's best to use the parameter substitution mechanism provided by the connector package: it's more likely to handle tricky cases like embedded quotes correctly, and will reduce the risk of SQL injection.
Here's an example for inserts; the same techniques can be used for the other types of query.
key_id = str("'") + self.GetCellValue(event.GetRow(),1) + str("'")
target_col = self.GetColLabelValue(event.GetCol())
key_col = self.GetColLabelValue(1)
nVal = str("'") + self.GetCellValue(event.GetRow(),event.GetCol()) + str("'")
#INSERT (using f-strings for brevity)
sql_update = (f"INSERT INTO `{self.tbl}` (`{self.key_col}`) VALUES (%s)")
# Pass the statement and values to cursor.execute.
# The values are assumed to be a sequence, so a single value should be
# placed in a tuple or list.
self.cursor.execute(sql_update, (nVal,))
If you have more than one column / value pair you could do something like this:
cols = ['A', 'B', 'C']
vals = ['a', 'b', 'c']
col_names = ','.join([f'`{c}`' for c in cols])
values_placeholder = ','.join(['%s'] * len(cols))
sql_update = (f"INSERT INTO `{self.tbl}` (col_names) VALUES ({values_placeholder})")
self.cursor.execute(sql_update, vals)
Values are not only data for insertion, but also data that we are using for comparison, for example in WHERE clauses. So an update statement with a filter might be created like this:
sql_update = (f"UPDATE `{tbl}` SET (`{target_col}`) = (%s) WHERE (`{key_col}`) = %s")
self.cursor.execute(sql_update, (nVal, key_id))
However sometimes the target of a SET or WHERE clause may be a column, for example we want to do an update based on other values in the row. For example, this statement will set target_col to the value of other_col for all rows where key_col is equal to other_key_col:
sql_update = (f"UPDATE `{tbl}` SET (`{target_col}`) = `{other_col}` WHERE (`{key_col}`) = `{other_key_col}`")
self.cursor.execute(sql_update)

Select Column from Table in Django Not Working

I am trying to access the two specific column from the table in Django it is not Working but When I am trying to access select * it is working
I am using postgresql
When I am trying to access select all its working this is I am trying to access for particular column
def bigdataDatabase(X):
engine = sqlalchemy.create_engine('postgresql+psycopg2://postgres:password#localhost/db_name')
con = engine.connect()
result = con.execute(
"Select Orign,Departure From 'table_name' WHERE index = '" + str(X) + "'")
This is not working
I have also tried with this
result = con.execute("Select tablename.Orign,tablename.departure From 'table_name' WHERE index = '" + str(X) + "'")
both the above code is not working
Programming Error column does not exist
But When I am executing all this it is working
result = con.execute("Select * From 'table_name' WHERE index = '" + str(X) + "'")
I have found the solution of the problem the query should be executed like this
result = con.execute('Select "Orign","Departure" From "Table_name" WHERE index = ' + str(X))

How to execute mysql query with LIKE statement in Django

I am trying to search in my mysql database with LIKE statement :
indexstr = request.GET['index']
indexstr = '%' + indexstr + '%'
offset = int(request.GET['offset'])
for row_data in advertisement.objects.raw(
'select * from requests_advertisement WHERE short_description LIKE ' + indexstr + ' LIMIT 10 OFFSET ' + str(offset*5)):
But it has this error:
Error Image
It seems that it cannot work with % character. When I remove % it works correctly.
There's no reason to use a raw query here. You need the __icontains lookup:
advertisement.objects.filter(short_description__icontains=request.GET["index"])

sqlite3.OperationalError: near "........": syntax error

windows 7
python 2.7
Django 1.11
I have used Django to develop a website. In the backend I have the sqlite database which have 2 tables. One table accepts the form user submitted, and the other is for comparison.
Once a form A is submitted by the user, it will be save under table catalog_fw, and the catalog_fw.ODM and catalog_fw.project_name will be compared with the ones in the table catalog_fw_instance. If one line have the exact same content for catalog_fw.ODM and catalog_fw.project, catalog_fw_instance.level will be combined with A to pass to the an .exe to generate a txtx file.
However, error occurs in this line: c.execute("catalog_fw_instance.level,......
`
when I run this python file:
sqlite3.OperationalError: near "catalog_fw_instance": syntax error
The code to get sqlite data, compare and pass to the .exe is here:
def when_call_exe():
with sqlite3.connect('db.sqlite3') as con:
c = con.cursor()
#c.execute("catalog_fw_instance.level, SELECT catalog_fw.ODM_name, catalog_fw.project_name, catalog_fw.UAP, catalog_fw.NAP, catalog_fw.LAP, catalog_fw.num_address FROM catalog_fw INNER JOIN catalog_fw_instance ON catalog_fw.ODM_name=catalog_fw_instance.ODM_name AND catalog_fw.project_name=catalog_fw_instance.project_name")
sql = ("SELECT catalog_fw.ODM_name, catalog_fw.project_name, catalog_fw.UAP, catalog_fw.NAP, catalog_fw.LAP, " +
"catalog_fw.num_address, catalog_fw_instance.level " +
"FROM catalog_fw catalog_fw" +
"INNER JOIN catalog_fw_instance catalog_fw_instanc" +
" ON catalog_fw.ODM_name = catalog_fwi.ODM_name AND catalog_fw.project_name = catalog_fw_instance.project_name")
c.execute(sql)
print '1:', c.fetchone()
parameter = c.fetchone()
print '2', parameter
#pass to exe
args = ['.//exe//Test.exe', parameter[0], parameter[1]+parameter[2], parameter[3], parameter[4], parameter[5], parameter[6]]
output = my_check_output(args)
if 'SUCCESS' in output:
filename = output[28:-1]
else:
filename = output[8:-1]
downloadlink = os.path.join('/exe', '%s' % filename)
#save link to sqlite db
c.execute('''UPDATE catalog_fw SET download = %s WHERE
ODM_Name=parameter[1] AND project_Name=parameter[2] ''' % downloadlink)
here shows the 2 tables in the sqlite database
table 1
table 2
As far as I know, when calling cursor#execute() in Python, we should be passing a single string containing the query to be run. It looks like you are passing one of the select parameters, followed by a query, all together as a single string. Consider the following version:
c = con.cursor()
sql = ("SELECT cf.ODM_name, cf.project_name, cf.UAP, cf.NAP, cf.LAP, " +
"cf.num_address, cfi.level " +
"FROM catalog_fw cf " +
"INNER JOIN catalog_fw_instance cfi " +
" ON cf.ODM_name = cfi.ODM_name AND cf.project_name = cfi.project_name")
c.execute(sql)
print(c.fetchone())
parameter = c.fetchone()

Get MSSQL table column names using pyodbc in python

I am trying to get the mssql table column names using pyodbc, and getting an error saying
ProgrammingError: No results. Previous SQL was not a query.
Here is my code:
class get_Fields:
def GET(self,r):
web.header('Access-Control-Allow-Origin', '*')
web.header('Access-Control-Allow-Credentials', 'true')
fields = []
datasetname = web.input().datasetName
tablename = web.input().tableName
cnxn = pyodbc.connect(connection_string)
cursor = cnxn.cursor()
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,* FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = " + "'"+ tablename + "'"
cursor.execute(query)
DF = DataFrame(cursor.fetchall())
columns = [column[0] for column in cursor.description]
return json.dumps(columns)
how to solve this?
You can avoid this by using some of pyodbc's built in methods. For example, instead of:
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,* FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = " + "'"+ tablename + "'"
cursor.execute(query)
DF = DataFrame(cursor.fetchall())
Try:
column_data = cursor.columns(table=tablename, catalog=datasetname, schema='dbo').fetchall()
print(column_data)
That will return the column names (and other column metadata). I believe the column name is the fourth element per row. This also relieves the very valid concerns about SQL injection. You can then figure out how to build your DataFrame from the resulting data.
Good luck!
Your line
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,*...
Will produce something like
USE[databasename]SELECT ...
In SSMS this would work, but I'd suggest to look on proper spacing and to separate the USE-statement with a semicolon:
query = "USE " + "[" +datasetname+ "]; " + "SELECT COLUMN_NAME,*...
Set the database context using the Database attribute when building the connection string
Use parameters any time you are passing user input (especially from HTTP requests!) to a WHERE clause.
These changes eliminate the need for dynamic SQL, which can be insecure and difficult to maintain.

Categories