Context
I've written a python script designed to run on a server. The script accepts a user input to make a search. I.E the user types in an arbitrary string and the database returns all usernames with a similar string.
Description of problem
I'm very uncertain about the security of the input. The program uses a stored procedure and a parameterised procedure, but despite this, if a user types in nothing i.e a blank string or if they enter something like % then the script returns every single username in the database.
Code
import json
import mysql.connector
class json_read():
def __init__(self,name):
self.name = name
def json_caller(self):
with open(self.name) as f:
f = json.load(f)[0]
return f
f = json_read("database_connect.json")
config = f.json_caller()
def mysql_connect(func):
def wrapper(*args, **kwargs):
try:
cnx = mysql.connector.connect(**config)
cursor = cnx.cursor()
cursor.execute(*args, **kwargs)
result = cursor.fetchall()
print("\nConnection is stable # " + func.__name__)
print(result)
except:
print("\nConnection failed # " + func.__name__)
return wrapper
class query_dbh():
f2 = json_read("queries.json")
def __init__(self, index):
self.index = self.f2.json_caller()[index]
#mysql_connect
def query(*args, **kwargs):
pass
search_query = query_dbh("Search_uid").index
search_param = [raw_input("Search: ") + "%"]
query(search_query,(search_param))
Queries are kept in a JSON file and loaded by the script
[
{
"Select_names" : "SELECT first,last FROM user",
"Select_id" : "SELECT id FROM user",
"Order_names" : "SELECT first, last FROM user ORDER BY first ASC",
"Search_uid" : "SELECT uid FROM user WHERE uid LIKE %s"
}
]
Where Search_uid is the query being used.
Related
I created my first tkinter+sqlite app using lubuntu and it works fine but when I ran it on windows I kept getting a database not found error.
this is my code:
class App():
...
class Data():
def __init__(self, username=None, password=None, inst=None):
self.serverlist = []
self.username = username
self.password = password
self.inst = inst
self.populate_serverlist()
self.populate_attributes()
print(self.username + self.password + self.inst)
def populate_serverlist(self):
...
def populate_attributes(self):
...
def add_new_profile(self, username, password, inst):
...
def get_profile(self):
...
#staticmethod
def run_query(sql, data=None, receive=False):
conn = sqlite3.connect("profile.db")
cursor = conn.cursor()
if data:
cursor.execute(sql, data)
else:
cursor.execute(sql)
if receive:
return cursor.fetchall()
else:
conn.commit()
conn.close()
u/staticmethod
def first_timeDB():
create_table = "CREATE TABLE profile (username text, password text, inst text)"
Data.run_query(create_table)
if __name__ == '__main__':
app = App()
if not os.path.isfile("profile.db"):
app.data.first_timeDB()
app.mainloop()
I tried replacing "profile.db" for a full path ('C:\User\Doc\profile.db') but still would not find it.
Then I also tried this tip but also didn't work, this is the modified code:
class Data():
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
db_path = os.path.join(BASE_DIR, "profile.db")
...
def run_query(sql, data=None, receive=False):
conn = sqlite3.connect(db_path)
I get an "NameError: name 'db_path' is not defined" error (even though it creates the profile.db file as per my main code). So anyone knows what I'm doing wrong???
Found the answer. Turns out it has nothing to do with the path. The method 'populate_attributes()' was trying to get values from the table before it was created, as this method was being called by the class Data's init().
The DB path should be double slash \\ or / when you run it on windows.
When I testing the function of update_or_create in multi threading condition, I found the result is not what I wanted,they created more than one record in MySQL. As the code show, update_or_create used select .. for update to lock rows in MySQL, then it should be only one record in MySQL. I used SQLAlchemy and row sql has proved that.
So, is the Django codes wrong?
with Django code:
def get_or_create_ins():
p, created = OxalicAcid.objects.update_or_create(defaults={"formula": "20", "degree": "80"}, name="smart")
def run():
for i in range(10):
t = threading.Thread(target=get_or_create_ins, args=())
t.start()
if __name__ == "__main__":
# more than one record will be created
run()
with SQLAlchemy code:
#contextmanager
def transaction_atomic():
session = Session()
try:
yield session
session.commit()
except Exception as e:
session.rollback()
raise e
def get_result_with_update(session, name):
sql = text("""
select * from acid_oxalicacid where name = :name for update
""")
params = dict(name=name)
cursor = session.execute(sql, params)
result = cursor.fetchall()
return result
def get_result(session, name):
sql = text("""
select * from acid_oxalicacid where name = :name
""")
params = dict(name=name)
cursor = session.execute(sql, params)
result = cursor.fetchall()
return result
def create_data(session, name, degree, formula):
sql = text("""
insert into acid_oxalicacid (name, degree, formula) values (:name, :degree, :formula)
""")
params = dict(
name=name,
degree=degree,
formula=formula
)
session.execute(sql, params)
def get_or_create():
name = "smart"
degree = "50"
formula = "100"
with transaction_atomic() as session:
res = get_result_with_update(session, name)
if not res:
create_data(session, name, degree, formula)
res = get_result(session, name)
return res
if __name__ == "__main__":
# Only one record be created, that's correct
for i in range(10):
t = threading.Thread(target=get_or_create, args=())
t.start()
Because Django use the 'read committed' transaction isolation level, so it will be multiply records, if change it to 'repeatable read', it's will be only one record in database.
I created a CRUD endpoint wit Flask but when I try to GET data, I receive a 404 error. I tried to access this endpoint with 'http://127.0.0.1:5002/albums/beck//' and 'http://127.0.0.1:5002/albums/beck' but still get a 404. Since I supplied 'beck' as the artist name I thought the get method would run fine. I think I added the resource incorrectly.
class Artistdetails(Resource):
def get(self, artist_name):
conn = db_connect.connect()
# Protect against SQL injection
restricted_char = "!=<>*0&|/\\"
for char in restricted_char:
artist_name = artist_name.replace(char, "")
query_db = conn.execute("SELECT DISTINCT album FROM album WHERE artist='{0}'".format(artist_name.title()))
result = jsonify({'artistAlbumList': [i[0] for i in query_db.cursor.fetchall()]})
return result
def put(self, artist_name, album_name, album_name_new):
conn = db_connect.connect()
# Protect against SQL injection
restricted_char = "!=<>*0&|/\\"
for char in restricted_char:
artist_name = artist_name.replace(char, "")
query_db = conn.execute("UPDATE album SET album='{0}' WHERE artist='{1}' AND"
" album='{2}'".format(artist_name.title(), album_name.title(), album_name_new.title()))
result = jsonify({'putAlbumId': [i[0] for i in query_db.cursor.fetchall()]})
return result, 201
def post(self, artist_name, album_name):
conn = db_connect.connect()
# Protect against SQL injection
restricted_char = "!=<>*0&|/\\"
for char in restricted_char:
artist_name = artist_name.replace(char, "")
query_db = conn.execute("INSERT INTO album (album, artist) VALUES"
" ({0},{1})".format(artist_name.title(), album_name.title()))
result = jsonify({'postAlbumId': [i[0] for i in query_db.cursor.fetchall()]})
return result, 201
def delete(self, artist_name, album_name):
conn = db_connect.connect()
# Protect against SQL injection
restricted_char = "!=<>*0&|/\\"
for char in restricted_char:
artist_id = artist_name.replace(char, "")
album_id = album_name.replace(char, "")
query_db = conn.execute("DELETE FROM album WHERE"
" artist_id='{0}' AND album_id='{1}'".format(artist_name, album_name)
)
result = jsonify({'deleteAlbumId': [i[0] for i in query_db.cursor.fetchall()]})
return result, 204
Create API routes
api.add_resource(Api, '/')
api.add_resource(Albums, '/albums')
api.add_resource(Artistdetails, '/albums/<string:artist_name>/<string:album_name>/<string:album_name_new>')
api.add_resource(Genreyear, '/albums/yr')
api.add_resource(Genrenum, '/albums/genre')
api.add_resource(Artists, '/artists')
This line:
api.add_resource(Artistdetails,
'/albums/<string:artist_name>/<string:album_name>/<string:album_name_new>')
It adds a path to the Flask router that makes it expect /albums/<artist_name>/<album_name>/<album_name_new>, whereas you're trying to request /albums/<artist_name>, which doesn't match anything.
A quick fix for you would be:
api.add_resource(Artistdetails, '/albums/<string:artist_name>')
However, you might instead want to support query string parameters for your search interface so that requests look more like this:
/albums?artist=<string>&album_name=<string>
To do that, the documentation for Flask-RESTful reqparse would be useful.
I'm learning python since last few weeks. For better learning, I decided to work on some project. So here is my Class for MySQL connection and demo example as well. Can you please tell me. What other improvement can be possible for following code?
Structure?
What else I can do to optimize code?
And Please forgive. If I'm doing some silly mistakes in code. (I'm learning)
#!/usr/bin/python
import pymysql
# select (table, parameter)
# insert (table, data)
# update (table, id, data)
# delete (table, id)
class MySQL:
def __init__(self):
self.sort_by = ""
self.order = ""
# initiate database connection.
self.connection = pymysql.connect(host='localhost',
user='root',
password='',
db='sherlock',
charset='utf8mb4')
self.cursor = self.connection.cursor(pymysql.cursors.DictCursor)
# this function is for selecting any feild on any table.(feilds veriable is optinal)
def select(self, table, *feilds):
flds = "" #differnt name for feilds veriable.
if not feilds:
flds = '*'
else:
for f in feilds:
if not flds:
flds = f
else:
flds += ",`%s`" % f
sql = "SELECT %s FROM `%s` " % (flds, table)
if self.sort_by:
sql = sql +"order by "+ str(self.sort_by) +" "+ str(self.order)
print sql
self.cursor.execute(sql)
result = self.cursor.fetchall()
return result
# This function is for data sorting for Mysql; but optinal.
# example : SELECT * FROM `users` order by id asc
def order_by(self, sort_by="", order="", *args, **kwargs):
self.sort_by = sort_by
self.order = order
# this function is for closing Mysql connection
def close(self):
self.connection.close()
########### END OF MySQL CLASS #############
sql = MySQL()
# sql.order_by function should be called before the sql.select() function.
sql.order_by("email")
# this will select all the feilds from `users` table.
# you can specify whichever feilds you want to return. like : sql.select("users", "id, email")
result = sql.select("users", "password")
for email in result:
print email["password"]
sql.close()
I have a database class in python which I use to query the database.
class Database():
def __init__(self, user, password, host, port, service_name, mode, *args):
#mode should be 0 if not cx_Oracle.SYSDBA
self.user = user
self.password = password
self.host = host
self.port = port
self.user = user
self.service_name = service_name
self.logger = logging.getLogger(__name__)
self.mode = 0
self.connection = None
self.connect_string = self.user + '/' + self.password + '#' + dsn
try:
self.connection = cx_Oracle.connect(self.connect_string, mode=self.mode, threaded=True)
self.connection.stmtcachesize = 1000
self.connection.client_identifier = 'my_app_scheduler'
self.cursor = self.connection.cursor()
self.cursor.arraysize = 10000
self.idVar = self.cursor.var(cx_Oracle.NUMBER)
except cx_Oracle.DatabaseError, exc:
error, = exc
self.logger.exception('Exception occured while trying to create database object : %s', error.message)
raise exc
def query(self, q):
try:
self.cursor.execute(q)
return self.cursor.fetchall(), self.cursor.rowcount
except cx_Oracle.DatabaseError, exc:
raise exc
And this is the code to manipulate the fetched data and convert it.
output, rowcount = db_run_query.query(sql_text)
#self.logger.debug('output : %s, type : %s', output, type(output))
end_time=time.time()
time_taken=end_time - start_time
self.logger.debug('Rowcount : %s, time_taken : %s', rowcount, time_taken)
column_name = [d[0] for d in db_run_query.cursor.description]
result = [dict(zip(column_name, row)) for row in output]
#Convert everything to string : Eg: datetime
try:
for each_dict in result:
for key in each_dict:
if isinstance(each_dict[key], cx_Oracle.LOB):
self.logger.debug('%s', each_dict[key].size())
each_dict[key]=each_dict[key].read()
#self.logger.debug('%s %s %s %s %s %s %s', key, each_dict, type(key), type(each_dict[key]), type(each_dict), temp_each_dict, type(temp_each_dict))
else:
each_dict[key]=str(each_dict[key])
except Exception as e:
self.logger.debug(e)
So without self.cursor.arraysize = 10000
and for a query like select clob_value from table it was able to fetch the data and logged Rowcount : 4901, time_taken : 0.196296930313 but was giving me an error like
LOB variable no longer valid after subsequent fetch
but when I mention the arraysize parameter the error goes away. ( Is arraysize only for lob columns coz it works fine for select other_column from table where rownum<20000 <- other_column in varchar)
Why does that happen?
Turns out CLOBs and fetchall don't place nice together:
Internally, Oracle uses LOB locators which are allocated based on the
cursor array size. Thus, it is important that the data in the LOB
object be manipulated before another internal fetch takes place. The
safest way to do this is to use the cursor as an iterator. In
particular, do not use the fetchall() method.
Avoiding cursor.fetchall() and using it like an iterator (e.g. for row in cursor: ...) and i was able to get around this problem.