We Keep Coding

Extending Jinja's {% trans %} to use JavaScript variables

I'd like to extend the behaviour of trans by rendering variables not as as values from the context, but instead as html (without using the context). My aim is to be able to populate those variables on the client through JavaScript.
Jinja as it seems doesn't allow for a great deal of customisation of this kind or I'm just unable to find the right hooks.
Here's what I'd like to achieve:
{% etrans name=username %}
My name is {{ name }}
{% endetrans %}
This should render to:
My name is <span id='#username'></span>
Of course, I could just use the normal {% trans %} directive and pass my html code to template.render(html_code_params), but that would require to have them defined in the template and the rendering code which I'd like to avoid.
Here's what I got so far (not much) which allows for a new etrans tag and the ability to use whatever goodies InternationalizationExtension has to offer.
from jinja2.ext import InternationalizationExtension
from jinja2.runtime import concat
class JavaScriptVariableExtension(InternationalizationExtension):
tagname = 'etrans'
tags = set([tagname])
def _parse_block(self, parser, allow_pluralize):
"""Parse until the next block tag with a given name.
Copy from InternationalizationExtension, as this uses hardcoded
`name:endtrans` instead of relying on tag name
"""
referenced = []
buf = []
while 1:
if parser.stream.current.type == 'data':
buf.append(parser.stream.current.value.replace('%', '%%'))
next(parser.stream)
elif parser.stream.current.type == 'variable_begin':
next(parser.stream)
name = parser.stream.expect('name').value
referenced.append(name)
buf.append('%%(%s)s' % name)
parser.stream.expect('variable_end')
elif parser.stream.current.type == 'block_begin':
next(parser.stream)
# can't use hardcoded "endtrans"
# if parser.stream.current.test('name:endtrans'):
if parser.stream.current.test('name:end%s' % self.tagname):
break
elif parser.stream.current.test('name:pluralize'):
if allow_pluralize:
break
parser.fail('a translatable section can have only one '
'pluralize section')
parser.fail('control structures in translatable sections are '
'not allowed')
elif parser.stream.eos:
parser.fail('unclosed translation block')
else:
assert False, 'internal parser error'
return referenced, concat(buf)
i18n_extended = JavaScriptVariableExtension
I don't mind overloading more methods (although the reason for above one should perhaps fixed upstream).
Stepping through the code is quite an interesting adventure. However, I hit a snag and am interested if anyone can give some advice.
The problem I see is that during the compilation, the function context.resolve() gets baked into the compiled code. jinja2.jinja2.compiler.CodeGenerator doesn't really allow any different handling here (correct me if I'm wrong). Ideally, I would define another node (for the variable) and this node would handle the way it's dealt with during compilation, but I don't see how this is possible. I might be too focussed on this as a solution, so perhaps someone can provide alternatives.

As suggested by #Garrett's comment, a much easier solution is to pass in a function to the template renderer that interpolates the variables. In my case, my target client-side framework is Angular, but this also works for any JS variables that you want to use within a {% trans %} environment. Here are the building blocks:
def text_to_javascript(string):
# modify as needed...
return "<span>{{ %s }}</span>" % string
def render():
tmpl = jinja_env.get_template(template_filename)
return tmpl.render({'js': text_to_javascript})
And this how I make use of it in the template file:
{% trans username=js('user.name') %}
My name is {{ username }}
{% endtrans %}
In the Angular controller, the variable user is bound to the $scope like so:
$scope.user = {'name': 'Bugs Bunny'}

Parsing/extracting from OrderedDict using Jinja in Salt

I am running into a rather weird issue while parsing results of a salt command. The command I am running is
{% set hostname = salt['publish.publish']('roles:*{}*'.format(role), 'grains.item', 'fqdn', 'grain') %}
And output looks below:
OrderedDict([('1.server.com', OrderedDict([('fqdn', '1.server.com')])), ('0.server.com', OrderedDict([('fqdn', '0.server.com')]))])
Now my understanding is when I do items() on above result with a line below, it should work
{% for hostname, fqdn in salt['publish.publish']('roles:*{}*'.format(role), 'grains.item', 'fqdn', 'grain').items() %}
But the moment I use items() in above line I start running into an error:
failed: Jinja variable 'None' has no attribute 'items'
I tried a couple of other ways (Doing items().items() or storing result in a variable and then running for loop over) to get the list out of OrderedDict but none of ways seem to help.

Either I don't know Python enough or there is something weird going on. Simply adding a check has made the above work. So working block looks like (Partial code of course):
{% set hostname = salt['publish.publish']('roles:*{}*'.format(role), 'grains.item', 'fqdn', 'grain') %}
{% if hostname is not none %}
{% for host, site in hostname.items() %}
My understanding is if check was only meant for checking just in case hostname is empty. But looks like even if there is data - an if check is needed. Still curious to know the mystery!

jinja2.exceptions.UndefinedError with Python dict

I'm trying to render a jinja2 template from a python dict created from a CSV file, but jinja2 is taking exception to the dict and I don't understand why.
This is the template:
{% for key, value in hostname.iteritems() %}
interface {{ key }}
description {{ value }}
{% endfor %}
and this is the python code:
import csv
from pprint import pprint
import os
import jinja2
CSVDATA_FILENAME = 'port_descriptions.csv'
TEMPLATE_FILENAME = 'cisco.j2'
hostnames = []
env = jinja2.Environment(
loader=jinja2.FileSystemLoader(os.getcwd()),
trim_blocks=True, lstrip_blocks=True)
template = env.get_template(TEMPLATE_FILENAME)
for row in csv.DictReader(open(CSVDATA_FILENAME)):
if row['hostname'] not in hostnames:
hostnames.append(row['hostname'])
for hostname in hostnames:
x = hostname
hostname = {}
for row in csv.DictReader(open(CSVDATA_FILENAME)):
if x == row['hostname']:
hostname[row['port']] = row['des']
pprint(hostname)
print template.render(hostname)
At the penultimate line (pprint(hostname)) the code will print the hostname dictionaries as desired so I know they are there, but jinja2 won't render them. Am I missing something obvious here, in the template maybe?
As far as I can see my code follows this - http://keepingitclassless.net/2014/03/network-config-templates-jinja2/ pretty closely, but I must be overlooking something?

Looking at this example in the docs, you probably need to pass your dictionary as a named keyword argument, like this: print template.render(hostname=hostname)
Note that only the part on the left of the equals sign is relevant to the template; you could just do template.render(hostname={'testy': 'testy'}) and the same template would keep working.

SaltStack: Use regexps in pillar file

I would like to do some basic pillar value settings for all boxes so that I can use them later in a unified way. Our minions are usually named in this format:
<project>-<env>.<role>-<sequence>.<domain>
Example pillar/base/top.sls:
base:
'*':
- basics
'I#project:mycoolproject and I#role:nginx':
- etc.
Example pillar/base/basics/init.sls:
{% if '-live.' in grains['id'] %}
env: production
{% elif '-qa.' in grains['id'] %}
env: qa
{% elif '-staging.' in grains['id'] %}
env: staging
{% else %}
env:
{% endif %}
{% set role = re.match("(?:live|qa|staging)\.([a-z_\-]+)\-', grains['id']).group(1) -%}
role: {{ role }}
The env part obviously works but I can't get the regex working. As far as I understood there is no way to import python module (i.e. import re) in jinja template. Any suggestions how to get regex functionality available in the pillar file if possible at all?

The simple answer is, "no". There is not a way to inject regex functionality directly into the jinja environment (I'm sure there's a way to extend jinja, but anyway..)
The way I addressed this was with an external module function, id_info.explode() and an external pillar.
Enable external modules on the master:
external_modules: /srv/extmod
External modules do not require any sort of special infrastructure--they are just regular python modules (not packages, mind you--the loader doesn't currently know how to properly side-load a package yet)
Put your python+regex logic there. Return a dictionary, assembled to your your liking.
Your external module would go in /srv/extmod/modules. You can call call this function from your pillar.sls
{% id_info = __salt__[id_info.explode()] -%}
{% subcomponent = id_info['subcomponent'] -%}
{% project = id_info['project'] -%}
etc...
A couple things to know:
The salt-master has to be restarted when an external module is added or modified. There isn't a way that I know of to incite the equivalent of a saltutil.refresh_modules() call on the salt-master, so there ya go.
The external_modules directive is not just for execution modules. In this scenario, you would also create /srv/extmod/{pillar,runners,outputers,etc}.
These modules are only available on the master

How do I pass parameters to a salt state file?

I want to create a group and user using salt state files, but I do not know the group, gid, user, uid, sshkey until I need to execute the salt state file which I would like to pass in as parameters.
I have read about Pillar to create the variable. How do I create pillars before execution?
/srv/salt/group.sls:
{{ name }}:
group.present:
- gid: {{ gid }}
- system: True
Command line:
salt 'SaltStack-01' state.sls group name=awesome gid=123456

If you really want to pass in the data on the command like you can also do it like this:
{{ pillar['name'] }}:
group.present:
- gid: {{ pillar['gid'] }}
- system: True
Then on the command line you can pass in the data like this:
salt 'SaltStack-01' state.sls group pillar='{"name": "awesome", "gid": "123456"}'

You use Pillars to create "dictionaries" that you can reference into State files. I'm not sure if I'm understanding you correctly, but here's an example of what you can do:
mkdir /srv/pillar/
Create /srv/pillar/groups.sls and paste something like this into it:
groups:
first: 1234
second: 5678
These are names and GIDs of the groups you want to create.
Create /srv/pillar/top.sls so you can apply this pillar to your minions. This is very similar to a salt top file, so you can either apply it to all minions ('*') or just the one ('SaltStack-01'):
base:
'hc01*':
- groups
To test that that has worked, you can run salt '*' pillar.items and you should find the groups pillar somewhere in the output.
Now, your /srv/salt/group.sls file should look like this:
{% for group,gid in pillar.get('groups',{}).items() %}
{{ group }}:
group.present:
- gid: {{ gid }}
{% endfor %}
This is a for loop: for every group and gid in the pillar groups, do the rest. So basically, you can look at it as if the state file is running twice:
first:
group.present:
- gid: 1234
And then:
second:
group.present:
- gid: 5678
This was incorporated from this guide.

if you do not want use Pillar
you can do as:
# /srv/salt/params.yaml
name: awesome
gid: 123456
and then:
# /srv/salt/groups.sls
{% import_yaml "params.yaml" as params %}
{{ params['name'] }}:
group.present:
- gid: {{ parmas['gid'] }}
- system: True
more details:doc

Another nice way to pass (incase you don't want to use pillars Nor create a file as other answers shows) - you can pass a local environment variable to salt and read it from within the sls file, like this:
Command:
MYVAR=world salt 'SaltStack-01' state.sls somesalt # Note the env variable passed at the beginning
sls file:
# /srv/salt/somesalt.sls
foo:
cmd.run:
- name: |
echo "hello {{ salt['environ.get']('MYVAR') }}"
Will print to stdout:
hello world
Another good thing to know is that the env variable also gets passed on to any included salt states as well.