We Keep Coding

Python creating dynamic YAML file using jinja2 for nested structures

I need to create a YAML file that is based on simple YAML that needs to be updated based on properties supplied by developers and contain the following YAML structure:
- switch: null
title: switch
case:
- condition: (parameter1==='Parameter1_value1')
execute:
- switch:
case:
- condition: $(parameter2) == "Parameter2_value1"
execute:
- invoke:
parameter3:
parameter3_value1: null
- condition: $(parameter2) == "Parameter2_value2"
execute:
- invoke:
parameter3:
parameter3_value2: null
- condition: (parameter1==='Parameter1_value2')
execute:
- switch:
case:
- condition: $(parameter2) == "Parameter2_value1"
execute:
- invoke:
parameter3:
parameter3_value1: null
- condition: $(parameter2) == "Parameter2_value2"
execute:
- invoke:
parameter3:
parameter3_value2: null
Both parameter1 and parameter2 can have multiple values, so I need to populate the structure dynamically, according to the values that I receive.
I tried to do the following:
Import the following
import ruamel.yaml
from jinja2 import Template
Load the basic file -
yaml = ruamel.yaml.YAML()
data_src = yaml.load(file_name)
In parallel receive the values from another JSON file, and once I have the data, I created using Jinja the following:
parameter2_data_tmpl = Template(""" - condition: $(parameter2) == "{{ parameter2_value }}"
execute:
- invoke:
parameter3: {{ parameter3_value }}
""");
parameter2_data = parameter2_data_tmpl.render(parameter2_value = parameter2_value, parameter3_value = parameter3_value)
This works like a charm, and when I print it - it looks great. but then I tried to add the new YAML piece to the structure I have by first add it to relevant array (Using the append method), and then assign the array to the relevant element in the original YAML structure.
But when I add it to the array, it added it in different format:
case: [' - condition: parameter2 == \"\
parameter2_value\"\\n execute:\\n -\
parameter3: parameter3_value\\n \
It's like jinja2 created it correctly, but not as YAML syntax.
Why doesn't this work? Is there an alternative to creating these code dynamically?

The result of the render is a string:
from jinja2 import Template
parameter2_data_tmpl = Template(""" - condition: $(parameter2) == "{{ parameter2_value }}"
execute:
- invoke:
parameter3: {{ parameter3_value }}
""");
parameter2_value = 'parameter2_value'
parameter3_value = 'parameter3_value'
parameter2_data = parameter2_data_tmpl.render(parameter2_value = parameter2_value, parameter3_value = parameter3_value)
print(type(parameter2_data))
This gives:
<class 'str'>
When you add that string somewhere in the datastructure that you loaded using ruamel.yaml you
get a quoted scalar.
What you want to do is add a data structure, but you cannot get from
the template, because the output of the render is not valid YAML.
You can directly create the data structure without rendering, if you don't want
to do that, you should change the template, so that it generates valid YAML input, load that
and then insert that data structure in the data_src at the appropriate place
(which is unclear from your incomplete program):
import sys
import ruamel.yaml
from jinja2 import Template
parameter2_data_tmpl = Template("""\
- condition: $(parameter2) == "{{ parameter2_value }}"
execute:
- invoke:
parameter3: {{ parameter3_value }}
""");
parameter2_value = 'parameter2_value'
parameter3_value = 'parameter3_value'
parameter2_data = parameter2_data_tmpl.render(parameter2_value = parameter2_value, parameter3_value = parameter3_value)
yaml = ruamel.yaml.YAML()
yaml.indent(sequence=4, offset=2)
new_data = yaml.load(parameter2_data)
data = yaml.load("""\
case: some value to overwrite
""")
data['case'] = new_data
yaml.dump(data, sys.stdout)
which gives:
case:
- condition: $(parameter2) == "parameter2_value"
execute:
- invoke:
parameter3: parameter3_value
Please note that there also exists a plugin ruamel.yaml.jinja2
that allows you to load
unrendered Jinja2 templates that generate YAML output and modify values.

Render variables from a file

There is a file api.yml containing a config for ansible:
config/application.properties:
server.port: 6081
system.default.lang: rus
api.pdd.url: "http://{{ stage['PDD'] }}"
api.policy.alias: "integration"
api.order.url: "http://{{ stage['Order'] }}
api.foo.url: "http://{{ stage['FOO'] }}
There is a stage.yml containing the key and the stage values:
default_node:
Order: '172.16.100.40:8811'
PDD: '172.16.100.41:8090'
FOO: '10.100.0.11:3165
In fact, these files are larger and the 'stage' variables are also many.
My task is to parse api.yml and turn it into properties-config. The problem is that I can not pull up the values ​​{{stage ['value']}} I'm trying to do it this way:
stream = yaml.load(open('api.yml'))
result={}
result.update(stream['config/application.properties'])
context= yaml.load(open('stage.yml'))
stage={}
stage.update(context['default_node'])
text = '{% for items in result | dictsort(true)%} {{ items[0] }} = {{
items[1] }} {%endfor%}'
template = Template(text)
properti = (template.render(result=result, stage=stage))
At the output I get this:
server.port = 6081
system.default.lang = rus
api.pdd.url = http://{{ stage['PDD'] }}
api.policy.alias = integration
api.order.url = http://{{ stage['Order'] }}
api.foo.url = http://{{ stage['FOO'] }}
And you need to get this:
server.port = 6081
system.default.lang = rus
api.pdd.url = 172.16.100.41:8090
api.policy.alias = "integration"
api.order.url = 172.16.100.40:8811
api.foo.url = 10.100.0.11:3165
Can I do it with jinja or ansible lib?
Sorry for my bad english

Following this approach, you would need to treat api.yml as a template itself and render it. Otherwise, jinja2 will treat it as a simple value of the property. Something like this would do:
import yaml
from jinja2 import Environment, Template
import json
stream = yaml.load(open('api.yml'))
result={}
result.update(stream['config/application.properties'])
context= yaml.load(open('stage.yml'))
stage={}
stage.update(context['default_node'])
text = """{% for items in result | dictsort(true)%} {{ items[0] }} = {{ items[1] }} {%endfor%}"""
#Then render the results dic as well
resultsTemplate = Template(json.dumps(result))
resultsRendered = json.loads( resultsTemplate.render(stage=stage) )
template = Template(text)
properti = (template.render(result=resultsRendered, stage=stage))
After this you will see the wanted values in the properti var:
' api.foo.url = http://10.100.0.11:3165 api.order.url = http://172.16.100.40:8811 api.pdd.url = http://172.16.100.41:8090 api.policy.alias = integration server.port = 6081 system.default.lang = rus'
It would be nice though if jinja2 was able to render recursively. Maybe spending some time working out with the globals and shared modes of the Environment this can be achieved.
Hope this helps.

Django template syntax for nested dictionary item using variable names

On a Django template page, I'm trying to access the value inside a nested dictionary.
books =
{
1: { 1: 'Alice', 2: 'Bob', 3: 'Marta' },
2: { 1: 'Alice', 3: 'Marta' },
3: { 1: 'Alice', 2: 'Bob' },
}
Somewhere on my page, I have these two variables
info.id = 1
detail.id = 2
What I want to do is print (if it exists) the item books[1][2], or in other words books[info.id][detail.id]. I ran into trouble because I couldn't access this nested variable. This got solved here. However, the solution proposed was to access nested dictionary items using the dot notation. But the problem is that this doesn't seem to work when using variables. Using that logic, I would do:
{{ books.info.id.detail.id }}
But this doesn't yield any result. How should I approach the situation when using variables to access the items in a dictionary? Do note that the actual item may or may not exist, which is why I run into trouble using books[info.id][detail.id]

You can't do this in the template directly. You'll need to use a custom template tag. This would work:
#register.simple_tag
def nested_get(dct, key1, key2):
return dct.get(key1, {}).get(key2)
Now you can use it in the template:
{% load my_tags_library %}
{% nested_get books item.id detail.id %}

How Turning List into Class Object for Flask Templating

How to I turn a list of lists into a class that I can call for each object like foo.bar.spam?
list of lists:
information =[['BlueLake1','MO','North','98812'], ['BlueLake2','TX','West','65343'], ['BlueLake3','NY','sales','87645'],['RedLake1','NY','sales','58923'],['RedLake2','NY','sales','12644'],['RedLake3','KY','sales','32642']]
This would be to create variables for a very large html table using jinja2 templating in Flask.
I would want to be able to to do something like this:
{% for x in information %}
<tr>
<td>{{x.name}}</td>
<td>Via: {{x.location}} | Loop: {{x.region}}</td>
<td>{{x.idcode}}</td>
</tr>
{% endfor %}
There will be other uses then just this one template with this information, hence why I want it to be a callable class to use in other places.

Using collections.namedtuple:
>>> from collections import namedtuple
>>> Info = namedtuple('Info', ['name', 'location', 'region', 'idcode'])
>>>
>>> information =[
... ['BlueLake1','MO','North','98812'],
... ['BlueLake2','TX','West','65343'],
... ['BlueLake3','NY','sales','87645'],
... ['RedLake1','NY','sales','58923'],
... ['RedLake2','NY','sales','12644'],
... ['RedLake3','KY','sales','32642']
... ]
>>> [Info(*x) for x in information]
[Info(name='BlueLake1', location='MO', region='North', idcode='98812'),
Info(name='BlueLake2', location='TX', region='West', idcode='65343'),
Info(name='BlueLake3', location='NY', region='sales', idcode='87645'),
Info(name='RedLake1', location='NY', region='sales', idcode='58923'),
Info(name='RedLake2', location='NY', region='sales', idcode='12644'),
Info(name='RedLake3', location='KY', region='sales', idcode='32642')]

Probably the most common way is to put each of the records into a dict
info = []
for r in information:
record = dict(name=r[0], location=r[1], region=r[2], idcode=r[3])
info.append(record)
Jinja2 then allows you to use x.name etc to access the properties exactly as you do in your example.
{% for x in info %}
<tr>
<td>{{x.name}}</td>
<td>Via: {{x.location}} | Loop: {{x.region}}</td>
<td>{{x.idcode}}</td>
</tr>
{% endfor %}
NOTE this way of indexing into the data (x.name) is a jinja2 specific shortcut (though it's stolen from django templates, which probably stole it from something else).
Within python itself you'd have to do:
for x in info:
print(x['name'])
# x.name will throw an error since name isn't an 'attribute' within x
# x['name'] works because 'name' is a 'key' that we added to the dict

lambda as argument to jinja2 filter?

I'd like to have a custom filter in jinja2 like this:
{{ my_list|my_real_map_filter(lambda i: i.something.else)|some_other_filter }}
But when I implement it, I get this error:
TemplateSyntaxError: expected token ',', got 'i'
It appears jinja2's syntax does not allow for lambdas as arguments? Is there some nice workaround? For now, I'm creating the lambda in python then passing it to the template as a variable, but I'd rather be able to just create it in the template.

No, you cannot pass general Python expression to filter in Jinja2 template
The confusion comes from jinja2 templates being similar to Python syntax in many aspects, but you shall take it as code with completely independent syntax.
Jinja2 has strict rules, what can be expected at which part of the template and it generally does not allow python code as is, it expect exact types of expressions, which are quite limited.
This is in line with the concept, that presentation and model shall be separated, so template shall not allow too much logic. Anyway, comparing to many other templating options, Jinja2 is quite permissible and allows quite a lot of logic in templates.

I have a workaround, I'm sorting a dict object:
registers = dict(
CMD = dict(
address = 0x00020,
name = 'command register'),
SR = dict(
address = 0x00010,
name = 'status register'),
)
I wanted to loop over the register dict, but sort by address. So I needed a way to sort by the 'address' field. To do this, I created a custom filter and pass the lambda expression as a string, then I use Python's builtin eval() to create the real lambda:
def my_dictsort(value, by='key', reverse = False):
if by == 'key':
sort_by = lambda x: x[0].lower() # assumes key is a str
elif by == 'value':
sort_by = lambda x: x[1]
else:
sort_by = eval(by) # assumes lambda string, you should error check
return sorted(value, key = sort_by, reverse = reverse)
With this function, you can inject it into the jinja2 environment like so:
env = jinja2.Environment(...)
env.filters['my_dictsort'] = my_dictsort
env.globals['lookup'] = lookup # queries a database, returns dict
And then call it from your template:
{% for key, value in lookup('registers') | my_dict_sort("lambda x:x[1]['address']") %}
{{"""\
static const unsigned int ADDR_{key} = 0x0{address:04X}; // {name}
""" | format(key = key, address = value['address'], name = value['name'])
}}
{% endfor %}
Output:
static const unsigned int ADDR_SR = 0x00010; // status register
static const unsigned int ADDR_CMD = 0x00020; // command register
So you can pass a lambda as a string, but you'll have to add a custom filter to do it.

i've had to handle the same issue recently, I've had to create a list of dict in my Ansible template, and this is not included in the base filters.
Here's my workaroud :
def generate_list_from_list(list, target, var_name="item"):
"""
:param list: the input data
:param target: the applied transformation on each item of the list
:param var_name: the name of the parameter in the lambda, to be able to change it if needed
:return: A list containing for each item the target format
"""
# I didn't put the error handling to keep it short
# Here I evaluate the lambda template, inserting the name of the parameter and the output format
f = eval("lambda {}: {}".format(var_name, target))
return [f(item) for item in list]
# ---- Ansible filters ----
class FilterModule(object):
def filters(self):
return {
'generate_list_from_list': generate_list_from_list
}
I'm then able to use it this way :
(my_input_list is a listof string, but it would work with a list of anything)
# I have to put quotes between the <target> parameter because it evaluates as a string
# The variable is not available at the time of the templating so it would fail otherwise
my_variable: "{{ my_input_list | generate_list_from_list(\"{ 'host': item, 'port': 8000 }\") }}"
Or if I want to rename the lambda parameter :
my_variable: "{{ my_input_list | generate_list_from_list(\"{ 'host': variable, 'port': 8000 }\", var_name="variable") }}"
This outputs :
when called directly in Python:
[{'host': 'item1', 'port': 8000}, {'host': 'item2', 'port': 8000}]
when called in a template (a yaml file in my example, but as it returns a list you can do whatever you want once the list is transformed) :
my_variable:
- host: item1
port: 8000
- host: item2
port: 8000
Hope this helps someone