We Keep Coding

Easier way to use csv file with jinja

stackoverflow.csv:
name,age,country
Dutchie, 10, Netherlands
Germie, 20, Germany
Swisie, 30, Switzerland
stackoverflow.j2:
Name: {{ name }}
Age: {{ age }}
Country: {{ country }}
#####
Python script:
#! /usr/bin/env python
import csv
from jinja2 import Template
import time
source_file = "stackoverflow.csv"
template_file = "stackoverflow.j2"
# String that will hold final full text
full_text = ""
# Open up the Jinja template file (as text) and then create a Jinja Template Object
with open(template_file) as f:
template = Template(f.read(), keep_trailing_newline=True)
# Open up the CSV file containing the data
with open(source_file) as f:
# Use DictReader to access data from CSV
reader = csv.DictReader(f)
# For each row in the CSV, generate a configuration using the jinja template
for row in reader:
text = template.render(
name=row["name"],
age=row["age"],
country=row["country"]
)
# Append this text to the full text
full_text += text
output_file = f"{template_file.split('.')[0]}_{source_file.split('.')[0]}.txt"
# Save the final configuration to a file
with open(output_file, "w") as f:
f.write(full_text)
output:
Name: Dutchie
Age: 10
Country: Netherlands
#####
Name: Germie
Age: 20
Country: Germany
#####
Name: Swisie
Age: 30
Country: Switzerland
#####
See the script and input file above. Everything is working at the moment, but I would like to optimize the script that when I add a new column in the CSV file, I **don'**t need to add the script.
Example: when I add to the CSV file the column "address", I would need the update the template.render with the following:
text = template.render(
name=row["name"],
age=row["age"],
country=row["country"],
address=row["address"]
)
Is there a way to do this more efficient? I once had a code example to do this, but I cannot find it anymore :(.

You can unpack the dict into key and value variables in a for loop with items().
{% for key, value in row.items() %}
{{ key }}: {{ value }}
{% endfor %}
You can also pass the list of rows to the template and use another for loop so that you only have to render the template once.

Add comment inline with new item, or before

I'm using ruamel.yaml to insert some values into yaml files that i alredy have.
I am able to insert new items into the yaml file, but i can not insert comments.
This is what i'm trying to achieve
Resources:
Statement:
- Sid: Sid2
Resource:
# 1 - Account 1
- item1
- item # New Comment
This is the python code i'm using, but i'm not able to insert the item2 and the comments.
import sys
from ruamel.yaml import YAML
yaml_doc = """\
Resources:
Statement:
- Sid: Sid2
Resource:
# 1 - Account 1
- item1
"""
yaml = YAML()
data = yaml.load(yaml_doc)
data['Resources']['Statement'][0]['Resource'].append("item2")
data['Resources']['Statement'][0]['Resource'].yaml_add_eol_comment('New Comment', 'item2', column=0)
yaml.dump(data, sys.stdout)
Can I get some help on how to add the comment in line with the newly added item?
Thanks

Replacing strings in YAML using Python

I have the following YAML:
instance:
name: test
flavor: x-large
image: centos7
tasks:
centos-7-prepare:
priority: 1
details::
ha: 0
args:
template: &startup
name: startup-centos-7
version: 1.2
timeout: 1800
centos-7-crawl:
priority: 5
details::
ha: 1
args:
template: *startup
timeout: 0
The first task defines template name and version, which is then used by other tasks. Template definition should not change, however others especially task name will.
What would be the best way to change template name and version in Python?
I have the following regex for matching (using re.DOTALL):
template:.*name: (.*?)version: (.*?)\s
However did not figure out re.sub usage so far. Or is there any more convenient way of doing this?

For this kind of round-tripping (load-modify-dump) of YAML you should be using ruamel.yaml (disclaimer: I am the author of that package).
If your input is in input.yaml, you can then relatively easily find
the name and version under key template and update them:
import sys
import ruamel.yaml
def find_template(d):
if isinstance(d, list):
for elem in d:
x = find_template(elem)
if x is not None:
return x
elif isinstance(d, dict):
for k in d:
v = d[k]
if k == 'template':
if 'name' in v and 'version' in v:
return v
x = find_template(v)
if x is not None:
return x
return None
yaml = ruamel.yaml.YAML()
# yaml.indent(mapping=4, sequence=4, offset=2)
yaml.preserve_quotes = True
with open('input.yaml') as ifp:
data = yaml.load(ifp)
template = find_template(data)
template['name'] = 'startup-centos-8'
template['version'] = '1.3'
yaml.dump(data, sys.stdout)
which gives:
instance:
name: test
flavor: x-large
image: centos7
tasks:
centos-7-prepare:
priority: 1
'details:':
ha: 0
args:
template: &startup
name: startup-centos-8
version: '1.3'
timeout: 1800
centos-7-crawl:
priority: 5
'details:':
ha: 1
args:
template: *startup
timeout: 0
Please note that the (superfluous) quotes that I inserted in the input, as well as the comment and the name of the alias are preserved.

I would parse the yaml file into a dictionary, and the edit the field and write the dictionary back out to yaml.
See this question for discussion on parsing yaml in python How can I parse a YAML file in Python but I think you would end up with something like this.
from ruamel.yaml import YAML
from io import StringIO
yaml=YAML(typ='safe')
yaml.default_flow_style = False
#Parse from string
myConfig = yaml.load(doc)
#Example replacement code
for task in myConfig["tasks"]:
if myConfig["tasks"][task]["details"]["args"]["template"]["name"] == "&startup":
myConfig["tasks"][task]["details"]["args"]["template"]["name"] = "new value"
#Convert back to string
buf = StringIO()
yaml.dump(myConfig, buf)
updatedYml = buf.getvalue()

Render variables from a file

There is a file api.yml containing a config for ansible:
config/application.properties:
server.port: 6081
system.default.lang: rus
api.pdd.url: "http://{{ stage['PDD'] }}"
api.policy.alias: "integration"
api.order.url: "http://{{ stage['Order'] }}
api.foo.url: "http://{{ stage['FOO'] }}
There is a stage.yml containing the key and the stage values:
default_node:
Order: '172.16.100.40:8811'
PDD: '172.16.100.41:8090'
FOO: '10.100.0.11:3165
In fact, these files are larger and the 'stage' variables are also many.
My task is to parse api.yml and turn it into properties-config. The problem is that I can not pull up the values ​​{{stage ['value']}} I'm trying to do it this way:
stream = yaml.load(open('api.yml'))
result={}
result.update(stream['config/application.properties'])
context= yaml.load(open('stage.yml'))
stage={}
stage.update(context['default_node'])
text = '{% for items in result | dictsort(true)%} {{ items[0] }} = {{
items[1] }} {%endfor%}'
template = Template(text)
properti = (template.render(result=result, stage=stage))
At the output I get this:
server.port = 6081
system.default.lang = rus
api.pdd.url = http://{{ stage['PDD'] }}
api.policy.alias = integration
api.order.url = http://{{ stage['Order'] }}
api.foo.url = http://{{ stage['FOO'] }}
And you need to get this:
server.port = 6081
system.default.lang = rus
api.pdd.url = 172.16.100.41:8090
api.policy.alias = "integration"
api.order.url = 172.16.100.40:8811
api.foo.url = 10.100.0.11:3165
Can I do it with jinja or ansible lib?
Sorry for my bad english

Following this approach, you would need to treat api.yml as a template itself and render it. Otherwise, jinja2 will treat it as a simple value of the property. Something like this would do:
import yaml
from jinja2 import Environment, Template
import json
stream = yaml.load(open('api.yml'))
result={}
result.update(stream['config/application.properties'])
context= yaml.load(open('stage.yml'))
stage={}
stage.update(context['default_node'])
text = """{% for items in result | dictsort(true)%} {{ items[0] }} = {{ items[1] }} {%endfor%}"""
#Then render the results dic as well
resultsTemplate = Template(json.dumps(result))
resultsRendered = json.loads( resultsTemplate.render(stage=stage) )
template = Template(text)
properti = (template.render(result=resultsRendered, stage=stage))
After this you will see the wanted values in the properti var:
' api.foo.url = http://10.100.0.11:3165 api.order.url = http://172.16.100.40:8811 api.pdd.url = http://172.16.100.41:8090 api.policy.alias = integration server.port = 6081 system.default.lang = rus'
It would be nice though if jinja2 was able to render recursively. Maybe spending some time working out with the globals and shared modes of the Environment this can be achieved.
Hope this helps.

lambda as argument to jinja2 filter?

I'd like to have a custom filter in jinja2 like this:
{{ my_list|my_real_map_filter(lambda i: i.something.else)|some_other_filter }}
But when I implement it, I get this error:
TemplateSyntaxError: expected token ',', got 'i'
It appears jinja2's syntax does not allow for lambdas as arguments? Is there some nice workaround? For now, I'm creating the lambda in python then passing it to the template as a variable, but I'd rather be able to just create it in the template.

No, you cannot pass general Python expression to filter in Jinja2 template
The confusion comes from jinja2 templates being similar to Python syntax in many aspects, but you shall take it as code with completely independent syntax.
Jinja2 has strict rules, what can be expected at which part of the template and it generally does not allow python code as is, it expect exact types of expressions, which are quite limited.
This is in line with the concept, that presentation and model shall be separated, so template shall not allow too much logic. Anyway, comparing to many other templating options, Jinja2 is quite permissible and allows quite a lot of logic in templates.

I have a workaround, I'm sorting a dict object:
registers = dict(
CMD = dict(
address = 0x00020,
name = 'command register'),
SR = dict(
address = 0x00010,
name = 'status register'),
)
I wanted to loop over the register dict, but sort by address. So I needed a way to sort by the 'address' field. To do this, I created a custom filter and pass the lambda expression as a string, then I use Python's builtin eval() to create the real lambda:
def my_dictsort(value, by='key', reverse = False):
if by == 'key':
sort_by = lambda x: x[0].lower() # assumes key is a str
elif by == 'value':
sort_by = lambda x: x[1]
else:
sort_by = eval(by) # assumes lambda string, you should error check
return sorted(value, key = sort_by, reverse = reverse)
With this function, you can inject it into the jinja2 environment like so:
env = jinja2.Environment(...)
env.filters['my_dictsort'] = my_dictsort
env.globals['lookup'] = lookup # queries a database, returns dict
And then call it from your template:
{% for key, value in lookup('registers') | my_dict_sort("lambda x:x[1]['address']") %}
{{"""\
static const unsigned int ADDR_{key} = 0x0{address:04X}; // {name}
""" | format(key = key, address = value['address'], name = value['name'])
}}
{% endfor %}
Output:
static const unsigned int ADDR_SR = 0x00010; // status register
static const unsigned int ADDR_CMD = 0x00020; // command register
So you can pass a lambda as a string, but you'll have to add a custom filter to do it.

i've had to handle the same issue recently, I've had to create a list of dict in my Ansible template, and this is not included in the base filters.
Here's my workaroud :
def generate_list_from_list(list, target, var_name="item"):
"""
:param list: the input data
:param target: the applied transformation on each item of the list
:param var_name: the name of the parameter in the lambda, to be able to change it if needed
:return: A list containing for each item the target format
"""
# I didn't put the error handling to keep it short
# Here I evaluate the lambda template, inserting the name of the parameter and the output format
f = eval("lambda {}: {}".format(var_name, target))
return [f(item) for item in list]
# ---- Ansible filters ----
class FilterModule(object):
def filters(self):
return {
'generate_list_from_list': generate_list_from_list
}
I'm then able to use it this way :
(my_input_list is a listof string, but it would work with a list of anything)
# I have to put quotes between the <target> parameter because it evaluates as a string
# The variable is not available at the time of the templating so it would fail otherwise
my_variable: "{{ my_input_list | generate_list_from_list(\"{ 'host': item, 'port': 8000 }\") }}"
Or if I want to rename the lambda parameter :
my_variable: "{{ my_input_list | generate_list_from_list(\"{ 'host': variable, 'port': 8000 }\", var_name="variable") }}"
This outputs :
when called directly in Python:
[{'host': 'item1', 'port': 8000}, {'host': 'item2', 'port': 8000}]
when called in a template (a yaml file in my example, but as it returns a list you can do whatever you want once the list is transformed) :
my_variable:
- host: item1
port: 8000
- host: item2
port: 8000
Hope this helps someone