I'm working on a project with standard Django+SQLite bundle.
And I've come to situation where Django querySet API and even django 'raw()' are not enough for me to retrieve appropriate information from the database.
So I execute custom SQL directly to retrieve data. Also I need to define 'lower' function through 'create_function' interface cause SQLite can't perform case-insensitive sorting on unicode field (https://www.sqlite.org/faq.html#q18).
My question is where should I put this 'create_function' statement? Is it normal to put it directly into a django view, so it would be executed every time the view is loaded, or I should put it somewere else, where it would be executed only once?
from django.db import connection
def lower(text):
if text:
return text.lower()
else:
return text
def my_view(request):
...
with connection.cursor() as cursor:
if (connection.vendor == 'sqlite'):
connection.connection.create_function('lower', 1, lower)
cursor.execute('SELECT * FROM <my complicated select using lower>;')
data = dictfetchall(cursor)
...
Related
I have the following function:
def create_tables(tables: list) -> None:
template = jinja2.Template(
open("/opt/airflow/etl_scripts/postgres/staging_orientdb_create_tables.sql", "r").read()
)
pg_hook = PostgresHook(POSTGRES_CONN_ID) # this is airflow module for Postgres
conn = pg_hook.get_conn()
for table in tables:
sql = template.render(TABLE_NAME=table)
with conn.cursor() as cur:
cur.execute(sql)
conn.commit()
Is there a solution to check the content of the "execute" or "sql" internal variable?
My test looks like this, but because I return nothing, I can't check:
def test_create_tables(mocker):
pg_mock = MagicMock(name="pg")
mocker.patch.object(
PostgresHook,
"get_conn",
return_value=pg_mock
)
mock_file_content = "CREATE TABLE IF NOT EXISTS {{table}}"
mocker.patch("builtins.open", mock_open(read_data=mock_file_content))
create_tables(tables=["mock_table_1", "mock_table_2"])
Thanks,
You cannot access internal variables of a function from the outside.
I strongly suggest refactoring your create_tables function then, if you already see that you want to test a specific part of its algorithm.
You could create a function like get_sql_from_table for example. Then test that separately and mock it out in your test_create_tables.
Although, since all it does is call the template rendering function form an external library, I am not sure, if that is even something you should be testing. You should assume/know that they test their functions themselves.
Same goes for the Postgres functions.
But the general advice stands: If you want to verify that a specific part of your code does what you expect it to do, factor it out into its own unit/function and test that separately.
I'm currently performing a raw query in my database because i use the MySQL function instr. I would like to translate is into a django Func class.I've spend several days reading the docs, Django custom for complex Func (sql function) and Annotate SQL Function to Django ORM Queryset `FUNC` / `AGGREGATE` but i still fail to write succesfully my custom Func.
This is my database
from django.db import models
class Car(models.Model):
brand = models.CharField("brand name", max_length=50)
#then add the __str__ function
Then I populate my database for this test
Car.objects.create(brand="mercedes")
Car.objects.create(brand="bmw")
Car.objects.create(brand="audi")
I want to check if something in my table is in my user input. This is how i perform my SQL query currently
query = Car.objects.raw("SELECT * FROM myAppName_car WHERE instr(%s, brand)>0", ["my audi A3"])
# this return an sql query with one element in this example
I'm trying to tranform it in something that would look like this
from django.db.models import Func
class Instr(Func):
function = 'INSTR'
query = Car.objects.filter(brand=Instr('brand'))
EDIT
Thank to the response, the correct answer is
from django.db.models import Value
from django.db.models.functions import StrIndex
query = Car.objects.annotate(pos=StrIndex(Value("my audi A3"), "brand")).filter(pos__gt=0)
Your custom database function is totally correct, but you're using it in the wrong way.
When you look at your usage in the raw SQL function, you can clearly see that you need 3 parameters for your filtering to work correctly: a string, a column name and a threshold (in your case it is always zero)
If you want to use this function in the same way in your query, you should do it like this:
query = Car.objects.annotate(
position_in_brand=Instr("my audi A3")
).filter(position_in_brand__gt=0)
For this to work properly, you also need to add output_field = IntegerField() to your function class, so Django knows that the result will always be an integer.
But... You've missed 2 things and your function is not needed at all, because it already exist in Django as StrIndex.
And the 2nd thing is: you can achieve the same outcome by using already existing django lookups, like contains or icontains:
query = Car.objects.filter(brand__contains="my audi A3")
You can find out more about lookups in Django docs
While reading someone's code in Python/Flask, I came across the line:
results = Page.query.filter(Page.url.contains(url))
I've searched it but can't get a satisfying answer. What do the functions query.filter and url.contains do exactly and what values they return under different conditions like if there are no matches or multiple matches or table doesn't exist. Is page the name of the table or just the name of the class?
Edit: Function in which the line is used
#pages.route('/<url>/', methods=('GET', 'POST'))
def url_view(url):
from app import get_locale
page = DataGetter.get_page_by_url('/' + url, get_locale())
return render_template('gentelella/guest/page.html', page=page)
def get_page_by_url(url, selected_language=False):
if selected_language:
results = Page.query.filter_by(language=selected_language).filter(Page.url.contains(url))
else:
results = Page.query.filter(Page.url.contains(url))
if results:
return results.first()
return results
I think you need to do a bit more research on your own, but to get you started this looks like a Flask-SQLAlchemy project which is a Flask around the SQLAlchemy library. The documentation you need to read is mostly in SQLAlchemy and Flask-SQLAlchemy.
Page appears to be a ORM mapping of a database table. The filter/filter_by methods encapsulates the SQL SELECT functionality. The contains method encapsulates the SQL LIKE %<OTHER>% functionality.
I have a web app that allows users to enter a search query which will then retrieve models that match this search criteria. Here are my methods:
#staticmethod
def searchBody(query):
'''
Return all entries whose body text contains the query.
'''
return Entry.objects.get(text__icontains=query)
#staticmethod
def searchTitle(query):
'''
Return all entries whose title text contains the query.
'''
return Entry.objects.get(title__icontains=query)
#staticmethod
def searchAuthor(query):
'''
Return all entries whose author text contains the query.
'''
return Entry.objects.get(author.icontains=query)
My question is simply: is this secure as it stands? In other words, does incontains perform the necessary string escaping operations so a person can't inject SQL or Python code into the query to launch an attack?
Yes, the Django ORM protects you against SQL injection.
Of course you can never be entirely sure that there is no security vulnerability in an application. Nevertheless, the ORM is the component responsible for protecting you against SQL injection, so you should assume it's safe and keep your django install up to date!
On an unrelated note, there is a typo in Entry.objects.get(author.icontains=query).
Also, using .get is going to throw a lot of errors here (whenever the object doesn't exist, or more than one exist). It doesn't do what your docstring says either.
You probably want to be using .filter instead.
I am using SQLAlchemy + Pyramid to operate on my database. However, there are some optional tables which are not always expected to be present in the DB. So while querying them I try to catch such cases with the NoSuchTableError
try:
x = session.query(ABC.name.label('sig_name'),func.count('*').label('count_')).join(DEF).join(MNO).filter(MNO.relevance >= relevance_threshold).group_by(DEF.signature).order_by(desc('count_')).all()[:val]
except NoSuchTableError:
x = [-1,]
But on executing this statement, I get a ProgrammingError
ProgrammingError: (ProgrammingError) (1146, "Table 'db.mno' doesn't exist")
Why does SQLAlchemy raise the more general ProgrammingError instead of the more specific NoSuchTableError? And if this is indeed expected behaviour, how do I ensure the app displays correct information depending on whether tables are present/absent?
EDIT1
Since this is part of my webapp, the model of DB is in models.py (under my pyramid webapp). I do have a setting in my .ini file that asks user to select whether additional tables are available or not. But not trusting the user, I want to be able to check for myself (in the views) whether table exists or not. The contentious table is something like (in models.py)
class MNO(Base):
__tablename__="mno"
id=Column(Integer,primary_key=True,autoincrement=True)
sid=Column(Integer)
cid=Column(mysql.MSInteger(unsigned=True))
affectability=Column(Integer)
cvss_base=Column(Float)
relevance=Column(Float)
__table_args__=(ForeignKeyConstraint(['sid','cid',],['def.sid','def.cid',]),UniqueConstraint('sid','cid'),)
How and Where should the check be made so that a variable can be set (preferably during app setup) which tells me whether the tables are present or not?
Note: In this case I would have to try if...else rather than 'ask for forgiveness'
According to the sqlalchemy docs, a NoSuchTableError is only thrown when "SQLAlchemy [is] asked to load a table's definition from the database, but the table doesn't exist." You could try loading a table's definition, catching the error there, and doing your query otherwise.
If you want to do things via "asking for forgiveness":
try:
table = Table(table_name, MetaData(engine))
except NoSuchTableError:
pass
Alternatively, you could just check whether the table exists:
Edit:
Better yet, why don't you use the has_table method:
if engine.dialect.has_table(connection, table_name):
#do your crazy query
Why don't you use Inspector to grab the table names first?
Maybe something like this:
from sqlalchemy import create_engine
from sqlalchemy.engine import reflection
#whatever code you already have
engine = create_engine('...')
insp = reflection.Inspector.from_engine(engine)
table_name = 'foo'
table_names = insp.get_table_names()
if table_name in table_names:
x = session.query(ABC.name.label('sig_name'),func.count('*').label('count_')).join(DEF).join(MNO).filter(MNO.relevance >= relevance_threshold).group_by(DEF.signature).order_by(desc('count_')).all()[:val]