We Keep Coding

Is there a way to check if a path is absolute in jinja2?

In the pydata-sphinx-theme we need to check if a path is absolute or not before adding it to the template. Currently we use the following:
{% set image_light = image_light if image_light.startswith("http") else pathto('_static/' + image_light, 1) %}
It's working but fails to capture local files and many other absolute configurations. Is there a more elegant way to perform this check ?

I would consider implementing this logic in Python proper, and bundle it as a custom template function. This way it'd be much easier to implement, debug and test.

thanks #klas Š.for the guidances.
for anyone coming here I did add:
from urllib.parse import urlparse
# The registration function
def setup_is_absolute(app, pagename, templatename, context, doctree):
def is_absolute(link):
return bool(urlparse(link).netloc) or link.startswith("/")
context['is_absolute'] = is_absolute
# Your extension's setup function
def setup(app):
app.connect("html-page-context", setup_is_absolute)
and in my template:
{{ is_absolute(logo) }}

Airflow Variable Usage in DAG file

I have a short DAG where I need to get the variable stored in Airflow (Airflow -> Admin -> variables). But, when we use as a template I'm getting below error.
Sample code and error shown below:
from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator
def display_variable():
my_var = Variable.get("my_var")
print('variable' + my_var)
return my_var
def display_variable1():
my_var = {{ var.value.my_var }}
print('variable' + my_var)
return my_var
dag = DAG(dag_id="variable_dag", start_date=airflow.utils.dates.days_ago(14),
schedule_interval='#daily')
task = PythonOperator(task_id='display_variable', python_callable=display_variable, dag=dag)
task1 = PythonOperator(task_id='display_variable1', python_callable=display_variable1, dag=dag)
task >> task1
Here the usage to get the value of a variable using:
Variable.ger("my_var") --> is working
But, I'm getting an error using the other way:
{{ var.value.my_var }}
Error:
File "/home/airflow_home/dags/variable_dag.py", line 12, in display_variable1
my_var = {{ var.value.my_var }}
NameError: name 'var' is not defined

Both display_variable functions run Python code, so Variable.get() works as intended. The {{ ... }} syntax is used for templated strings. Some arguments of most Airflow operators support templated strings, which can be given as "{{ expression to be evaluated at runtime }}". Look up Jinja templating for more information. Before a task is executed, templated strings are evaluated. For example:
BashOperator(
task_id="print_now",
bash_command="echo It is currently {{ macros.datetime.now() }}",
)
Airflow evaluates the bash_command just before executing it, and as a result the bash_command will hold e.g. "Today is Wednesday".
However, running {{ ... }} as if it were Python code would actually try to create a nested set:
{{ variable }}
^^
|└── inner set
|
outer set
Since sets are not hashable in Python, this will never evaluate, even if the statement inside is valid.
Additional resources:
https://www.astronomer.io/guides/templating
The template_fields attribute on each operator defines which attributes are template-able, see docs for your operator to see the value of template_fields: https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/operators/python/index.html#airflow.operators.python.PythonOperator.template_fields

How to use dag_run.conf for typed arguments

I have a DAG that create a Google Dataproc cluster and submit a job to it.
I would like to be able to customize the cluster (number of workers) and the job (arguments passed to it) through the dag_run.conf parameter.
Cluster creation
For the cluster creation, I wrote a logic with something like:
DataprocCreateClusterOperator(...
cluster_config = {...
num_workers = "{% if 'cluster' is in dag_run.conf and 'secondary_worker_config' is in dag_run.conf['cluster'] and 'num_instances' is in dag_run.conf['cluster']['secondary_worker_config'] %}{{ dag_run.conf['cluster']['secondary_worker_config']['num_instances'] }}{% else %}16{% endif %}"
}
)
That is to say, if cluster.secondary_worker_config.num_instances is available in dag_run.conf, use it, else fallback on default value 16.
However, when rendered, this is expanded as a Python string, like "16", leading to failure because the num_workers parameter must be an int or a long.
I cannot parse it to int during operator declaration:
num_workers = int("{% ... %}")
because this will try to interpret the whole jinja script as an integer (and not the resulting value).
Using the | int jinja filter neither solve the problem.
Job submission
I have a similar problem for job submission.
Operator expect a job dict argument, with field spark.args to provide arguments to the spark job. This field must be an iterable, and is expected to be a list of strings, e.g: ["--arg=foo", "bar"].
I want to be able to add some arguments by providing them through dag_run.conf:
{
args = ["--new_arg=baz", "bar2"]
}
But adding these arguments to the initial list doesn't seem to be possible. You either get a single argument for all additional args: ["--arg=foo", "bar", "--new_arg=baz bar2"], or a single string with all arguments.
In any case, the resulting job submission is not working as expected...
Is there an existing way to workaround this problem?
If not, is there a way to add a "casting step" after "template rendering" one, either in the provider operators or directly in the BaseOperator abstract class?
Edit
I think that the solution proposed by Josh Fell is the way to go. However, for those that don't want to upgrade Airflow, I tried to implement the solution proposed by Jarek.
import unittest
import datetime
from typing import Any
from airflow import DAG
from airflow.models import BaseOperator, TaskInstance
# Define an operator which check its argument type at runtime (during "execute")
class TypedOperator(BaseOperator):
def __init__(self, int_param: int, **kwargs):
super(TypedOperator, self).__init__(**kwargs)
self.int_param = int_param
def execute(self, context: Any):
assert(type(self.int_param) is int)
# Extend the "typed" operator with an operator handling templating
class TemplatedOperator(TypedOperator):
template_fields = ['templated_param']
def __init__(self,
templated_param: str = "{% if 'value' is in dag_run.conf %}{{ dag_run.conf['value'] }}{% else %}16{% endif %}",
**kwargs):
super(TemplatedOperator, self).__init__(int_param=int(templated_param), **kwargs)
# Run a test, instantiating a task and executing it
class JinjaTest(unittest.TestCase):
def test_templating(self):
print("Start test")
dag = DAG("jinja_test_dag", default_args=dict(
start_date=datetime.date.today().isoformat()
))
print("Task intanciation (regularly done by scheduler)")
task = TemplatedOperator(task_id="my_task", dag=dag)
print("Done")
print("Task execution (only done when DAG triggered)")
context = TaskInstance(task=task, execution_date=datetime.datetime.now()).get_template_context()
task.execute(context)
print("Done")
self.assertTrue(True)
Which give the output:
Start test
Task intanciation (regularly done by scheduler)
Ran 1 test in 0.006s
FAILED (errors=1)
Error
Traceback (most recent call last):
File "/home/alexis/AdYouLike/Repositories/data-airflow-dags/tests/data_airflow_dags/utils/tasks/test_jinja.py", line 38, in test_templating
task = TemplatedOperator(task_id="my_task", dag=dag)
File "/home/alexis/AdYouLike/Repositories/data-airflow-dags/.venv/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
obj: BaseOperator = type.__call__(cls, *args, **kwargs)
File "/home/alexis/AdYouLike/Repositories/data-airflow-dags/tests/data_airflow_dags/utils/tasks/test_jinja.py", line 26, in __init__
super(TemplatedOperator, self).__init__(int_param=int(templated_param), **kwargs)
ValueError: invalid literal for int() with base 10: "{% if 'value' is in dag_run.conf %}{{ dag_run.conf['value'] }}{% else %}16{% endif %}"
As you can see, this fails at the task instanciation step, because in the TemplatedOperator.__init__ we try to cast to int the JINJA template (and not the rendered value).
Maybe I missed a point in this solution, but it seems to be unusable as is.

Unfortunately all Jinja templates are rendered as strings so the solution proposed by #JarekPotiuk is your best bet.
However, for anyone using Airflow 2.1+ or if you'd like to upgrade, there is a new parameter that can be set at the DAG level: render_template_as_native_obj
When enabling this parameter, the output from Jinja templating will be returned as native Python types (e.g. list, tuple, int, etc.). Learn more here: https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects

THe easiest way is to define your custom operator deriving from DataprocCreateClusterOperator . It's super easy and you can even do it within the dag file:
Conceptually something like that:
class MyDataprocCreateClusterOperator(DataprocCreateClusterOperator):
template_fields = DataprocCreateClusterOperator.template_fields + ['my_param']
def __init__(my_param='{{ ... }}', .....):
super(int_param=int(my_param), ....)

Display all jinja object attributes

Is there a way to display the name/content/functions of all attributes of a given object in a jinja template. This would make it easier to debug a template that is not acting as expected.
I am building a website using the hyde framework and this would come in quite handy since I am still learning the intricacies of both jinja and hyde.
Originally, I had thought it would work to use the attr filter, but this seems to require a name value. I would like to to not have to specify the name in order to get all available attributes for the object.
Some google searching showed django syntax looks like the following, but I am not familiar with django so this may only apply to database items. Long story short, I would like a method that works kind of like this for any object named obj
{% for field, value in obj.get_fields %}
{{ field }} : {{ value }} </br>
{% endfor %}
final solution:
#jayven was right, I could create my own jinja2 filter. Unfortunately, using the stable version of hyde (0.8.4), this is not a trivial act of having a filter in the pythonpath and setting a simple yaml value in the site.yaml file (There is a pull-request for that). That being said, I was able to figure it out! So the following is my final solution which ends up being very helpful for debugging any unkown attributes.
It's easy enough to create site-specific hyde extensions just create a local python package with the following directory tree
hyde_ext
__init__.py
custom_filters.py
Now create the extension:
from hyde.plugin import Plugin
from jinja2 import environmentfilter, Environment
debug_attr_fmt = '''name: %s
type: %r
value: %r'''
#environmentfilter
def debug_attr(env, value, verbose=False):
'''
A jinja2 filter that creates a <pre> block
that lists all the attributes of a given object
inlcuding the value of those attributes and type.
This filter takes an optional variable "verbose",
which prints underscore attributes if set to True.
Verbose printing is off by default.
'''
begin = "<pre class='debug'>\n"
end = "\n</pre>"
result = ["{% filter escape %}"]
for attr_name in dir(value):
if not verbose and attr_name[0] == "_":
continue
a = getattr(value, attr_name)
result.append(debug_attr_fmt % (attr_name, type(a), a))
result.append("{% endfilter %} ")
tmpl = Environment().from_string("\n\n".join(result))
return begin + tmpl.render() + end
#return "\n\n".join(result)
# list of custom-filters for jinja2
filters = {
'debug_attr' : debug_attr
}
class CustomFilterPlugin(Plugin):
'''
The curstom-filter plugin allows any
filters added to the "filters" dictionary
to be added to hyde
'''
def __init__(self, site):
super(CustomFilterPlugin, self).__init__(site)
def template_loaded(self,template):
super(CustomFilterPlugin, self).template_loaded(template)
self.template.env.filters.update(filters)
To let hyde know about the extension add hyde_ext.custom_filters.CustomFilterPlugin to the "plugins" list of the site.yaml file.
Lastly, test it out on a file, you can add this to some random page {{resource|debug_attr}} or the following to get even the underscore-attributes {{resource|debug_attr(verbose=True)}}
Of course, I should add, that it seems like this might become much easier in the future whenever hyde 1.0 is released. Especially since there is already a pull request waiting to implement a simpler solution. This was a great way to learn a little more about how to use jinja and hyde though!

I think you can implement a filter yourself, for example:
from jinja2 import *
def show_all_attrs(value):
res = []
for k in dir(value):
res.append('%r %r\n' % (k, getattr(value, k)))
return '\n'.join(res)
env = Environment()
env.filters['show_all_attrs'] = show_all_attrs
# using the filter
tmpl = env.from_string('''{{v|show_all_attrs}}''')
class Myobj(object):
a = 1
b = 2
print tmpl.render(v=Myobj())
Also see the doc for details: http://jinja.pocoo.org/docs/api/#custom-filters

Math on Django Templates

Here's another question about Django.
I have this code:
views.py
cursor = connections['cdr'].cursor()
calls = cursor.execute("SELECT * FROM cdr where calldate > '%s'" %(start_date))
result = [SQLRow(cursor, r) for r in cursor.fetchall()]
return render_to_response("cdr_user.html",
{'calls':result }, context_instance=RequestContext(request))
I use a MySQL query like that because the database is not part of a django project.
My cdr table has a field called duration, I need to divide that by 60 and multiply the result by a float number like 0.16.
Is there a way to multiply this values using the template tags? If not, is there a good way to do it in my views?
My template is like this:
{% for call in calls %}
<tr class="{% cycle 'odd' 'even' %}"><h3>
<td valign="middle" align="center"><h3>{{ call.calldate }}</h3></td>
<td valign="middle" align="center"><h3>{{ call.disposition }}</h3></td>
<td valign="middle" align="center"><h3>{{ call.dst }}</h3></td>
<td valign="middle" align="center"><h3>{{ call.billsec }}</h3></td>
<td valign="middle" align="center">{{ (call.billsec/60)*0.16 }}</td></h3>
</tr>
{% endfor %}
The last is where I need to show the value, I know the "(call.billsec/60)*0.16" is impossible to be done there. I wrote it just to represent what I need to show.

You can do it on three different layers:
Database level. SQL is a powerful language capable of mathematics. You could write your equation in the select part of your query. In your case, that should be along the lines SELECT (duration/60*0.16) FROM cdr;. Examples can be found here and on Google. Note that in this case, stress (algorithm complexity) is put on the MySQL server process and not the Python process.
View level. In your example, just before your return, you could loop over every element of your result variable to modify its value. You can follow the example that was given by Lie Ryan for this level.
Template level. This is done by a custom filter. You can write your custom filter as written in the documentation and pipe your template variable through this filter in order to get the desired value.
Something along these lines would represent a custom filter applicable on your template(s):
#register.filter
def customFilter(value):
return value / 60.0 * 0.16
You would then use it this way in your template, after {% load %}-ing the custom filter (read the documentation for more implementation information):
{{ billsec|customFilter }}

If the math operations are not too complex I normally use custom template tags. Add operation is already available as a template tag and I use the below snippet in my project for multiplication, subtraction and division respectively. Put this code inside a .py file inside your app/templatetags location and also add a __init__.py in there.
from django import template
#Django template custom math filters
#Ref : https://code.djangoproject.com/ticket/361
register = template.Library()
def mult(value, arg):
"Multiplies the arg and the value"
return int(value) * int(arg)
def sub(value, arg):
"Subtracts the arg from the value"
return int(value) - int(arg)
def div(value, arg):
"Divides the value by the arg"
return int(value) / int(arg)
register.filter('mult', mult)
register.filter('sub', sub)
register.filter('div', div)

EDIT: the following answer is totally wrong, since I thought OP was using sqlite. MySQL has its own way of wrapping things into dictionary, see far below.
You can subclass sqlite3.Row and write your own "computed field":
class MyRow(sqlite3.Row):
def comp_billsec(self):
return (self['billsec'] / 60) * 0.16
cursor = ...
cursor.row_factory = MyRow
for r in cursor.execute('...'):
print r['billsec'], r.comp_billsec()
note that our comp_billsec is accessed using method call syntax, while sqlite3.Row factory provides access through dictionary syntax. This discrepancy would disappear in django template since inside django template, dictionary access and zero-argument function call have the same syntax, so you can do {{ call.billsec }} and {{ call.comp_billsec }}.
EDIT: In MySQL, you can insert computed values in the view along the lines of:
cursor = connections['cdr'].cursor(cursorclass=MySQLdb.cursors.DictCursor)
calls = cursor.execute("...")
result = [r + dict(comp_billsec=r['billsec'] / 60 * 0.16) for r in cursor.fetchall()]
return render_to_response("cdr_user.html",
{'calls':result }, context_instance=RequestContext(request))
Additionaly, you should use parameterized query (note the comma instead of %):
cursor.execute("SELECT * FROM cdr where calldate > '%s'", (start_date,))
Your previous code is subject to SQL injection security issue since you're interpolating start_date into the SQL query directly. If start_date contains ' OR 1 OR ', for example, your query will be interpolated as SELECT * FROM cdr where calldate > '' OR 1 OR '' which will select all rows in the table; it could be even worse.