I would like to be able to read in python a YAML jinja configuration file like using the PyYAML library but I'm receiving errors:
{% set name = "abawaca" %}
{% set version = "1.00" %}
package:
name: {{ name }}
version: {{ version }}
source:
fn: {{ name }}-{{ version }}.tar.gz
url: https://github.com/CK7/abawaca/archive/v{{ version }}.tar.gz
sha256: 57465bb291c3a9af93605ffb11d704324079036205e5ac279601c9e98c467529
build:
number: 0
requirements:
build:
- gcc # [not osx]
- llvm # [osx]
Your input is not valid YAML, as you can easily check, e.g. here
You should first expand the {% %} constructs, and then process the YAML, or you should make your file into valid YAML.
This is a partly consequence of choosing jinja2 for which the macro sequences {% ... %} start with a character ({) that has special meaning in YAML.
If you need to change the YAML, and write it out again, you can define your own delimiters and choose them so that don't have special meaning in YAML.
The {% %} you should put in a YAML comment block as at the top-level you have a mapping and should only have key-value pairs. One way to achieve that is by redefining the start as #% %# (you don't necessarily have to change the end, but I prefer the symmetry).
Then after updating, run the correct YAML through a small script that processes the file and replaces the delimiters to those that jinja2 understands, or tweak the environment, to change the actual definitions used by jinja2.
corrected data.yaml:
#% set name = "abawaca" %#
#% set version = "1.00" %#
package:
name: <{ name }>
version: 42
source:
fn: <{ name }>-<{ version }>.tar.gz
url: https://github.com/CK7/abawaca/archive/v<{ version }>.tar.gz
sha256: 57465bb291c3a9af93605ffb11d704324079036205e5ac279601c9e98c467529
build:
number: 0
requirements:
build:
- gcc # [not osx]
- llvm # [osx]
This can be processed by:
import jinja2
from ruamel import yaml
yaml_file = 'data.yaml'
tmp_file = 'tmp.yaml'
data = yaml.round_trip_load(open(yaml_file))
data['package']['version'] = '<{ version }>'
with open(tmp_file, 'w') as fp:
yaml.round_trip_dump(data, fp)
environment = jinja2.Environment(
loader=jinja2.FileSystemLoader(searchpath='.'),
trim_blocks=True,
block_start_string='#%', block_end_string='%#',
variable_start_string='<{', variable_end_string='}>')
print(environment.get_template(tmp_file).render())
to give:
package:
name: abawaca
version: 1.00
source:
fn: abawaca-1.00.tar.gz
url: https://github.com/CK7/abawaca/archive/v1.00.tar.gz
sha256: 57465bb291c3a9af93605ffb11d704324079036205e5ac279601c9e98c467529
build:
number: 0
requirements:
build:
- gcc # [not osx]
- llvm # [osx]
Please note that you have to use `ruamel.yaml (disclaimer: I am the author of that package), you cannot do this as easily with PyYAML as it throws away the comments on reading the YAML file. Since all of the jinja2 within comments occurs at the beginning of the file you can work around this with this particular example, but in general that will not be the case.
Related
1. Summary
I can’t automatically generate the correct title from filenames of articles/pages.
For example, I can’t automatically generate metadata key title Kira Goddess from article Kira-Goddess.md
2. Argumentation
DRY, automation. I don’t want to manually write the title every time for each article and page if I can do it automatically.
An exception — files with words, that contain hyphens — “well-known”, “English-speaking”. In this case, I must explicitly specify title in the metadata of my articles. But words with hyphens are rare in filenames of my articles.
3. MCVE
3.1. Data
You can see it in my KiraTitleFromFilename branch of my repository for Pelican debugging.
pelicanconf.py:
"""MCVE."""
AUTHOR = 'Sasha Chernykh'
SITENAME = 'SashaPelicanDebugging'
SITEURL = '.'
PATH = 'content'
TIMEZONE = 'Europe/Moscow'
DEFAULT_LANG = 'en'
# [INFO] Use article name when preserve the slug:
# https://docs.getpelican.com/en/stable/settings.html#url-settings
SLUGIFY_SOURCE = 'basename'
# [INFO] Preserve case of article filename
SLUGIFY_PRESERVE_CASE = True
# [INFO] Get title from article filename:
# https://docs.getpelican.com/en/stable/settings.html#metadata
# https://github.com/getpelican/pelican/issues/2107
# https://github.com/getpelican/pelican/commit/2e82a53cdf3f1f9d66557850cc2811479d5bb645
FILENAME_METADATA = '(?P<title>.*)'
Kira-Goddess.md:
Date: 2020-09-24 18:57:33
Kira Goddess!
Another Pelican files generated by pelican-quickstart.
Simplified part of base.html:
<title>{{ article.title }}</title>
3.2. Steps to reproduce
See .travis.yml:
Run Pelican build:
pelican content -s pelicanconf.py --fatal warnings --verbose
Finding content of <title> tag:
grep -E "<title>.*</title>" output/Kira-Goddess.html
3.3. Behavior
3.3.1. Current
See Travis build:
<title>Kira-Goddess</title>
3.3.2. Desired
It would be nice, if:
<title>{{ article.title }}</title>
will transform to:
<title>Kira Goddess</title>
4. Not helped
In the description of EXTRA_PATH_METADATA variable I read, that Pelican used Python group name notation syntax (?P<name>…). I couldn’t find, how I can make substitutions in Python <class 'str'> (print(type(FILENAME_METADATA)) → <class 'str'>). I tried variants as:
import re
KIRA_DEFAULT_FILENAME_REGEX = '(?P<title>.*)'
FILENAME_METADATA = re.sub(KIRA_DEFAULT_FILENAME_REGEX, "-", " ")
or
KIRA_DEFAULT_FILENAME_REGEX = '(?P<title>.*)'
FILENAME_METADATA = KIRA_DEFAULT_FILENAME_REGEX.replace("-", "")
It doesn’t work.
5. Don’t offer
5.1. Use Jinja2 filters in templates
5.1.1. Suggestion
Use replace() filter in your template files like this:
<title>{{ article.title|replace('-', " ") }}</title>
5.1.2. Why is it not good
Pelican plugins (e.g. Pelican Open Graph) still will use article.title. Unwanted data as Kira-Goddess, not Kira Goddess still will pass to plugins.
5.2. Use spaces in your Markdown
5.2.1. Suggestion
For example, name your file Kira Goddess.md, not Kira-Goddess.md.
5.2.2. Why is it not good
Whitespaces in filenames is a bad practice — 1, 2, 3, 4, 5.
It seems like pelican doesn't provide a way to implement what you want.
FILENAME_METADATA regex can only select a name from a filename, but you can't substitute - with whitespaces there.
So I think the current best way for you is to specify title tag manually in each file.
I'm trying to pick up a value from a list randomly in jinja2, this way:
{{ ['#EDD7C9', '#EDD7C9', '#EEE'] | random }}
It selects an element randomly, but it reuses it every next time. If I change something in my template, then it picks up a different number again.
But weirdly, if I use range, it works just fine:
{{ range(1,100) |random }}
Any ideas of how I can pick a random item from a fixed list without having to create my own filters?
Just to add a more complete example:
{% for i in range(5) %}
{{ ['#EDD7C9', '#EDD7C9', '#EEE'] | random }}
{{ range(1,100) | random }} <br/>
{% endfor %}
outputs:
#EEE 8
#EEE 61
#EEE 58
#EEE 64
#EEE 76
Any help is appreciated!
thanks :)
This appears to be a bug introduced in Jinja 2.9. I can reproduce it with any of the 2.9 releases (2.9 - 2.9.6). Jinja 2.8 and Jinja 2.8.1 work fine, and random values are mixed in the rendering:
$ cat test.py
from jinja2 import Template
template = Template('''
{%- for i in range(10) -%}
{{ ['foo', 'bar', 'baz', 'eggs', 'ham', 'spam'] | random }}{{ ' ' }}
{%- endfor %}
''')
print(template.render())
$ bin/python test.py
eggs eggs eggs eggs eggs eggs eggs eggs eggs eggs
$ bin/pip -U Jinja2==2.8.1
Collecting jinja2==2.8.1
Using cached Jinja2-2.8.1-py2.py3-none-any.whl
Requirement already up-to-date: MarkupSafe in ./lib/python3.6/site-packages (from jinja2==2.8.1)
Installing collected packages: jinja2
Found existing installation: Jinja2 2.9
Uninstalling Jinja2-2.9:
Successfully uninstalled Jinja2-2.9
Successfully installed jinja2-2.8.1
$ bin/python test.py
spam ham eggs spam spam eggs eggs baz bar foo
Note that this only applies to a list literal embedded in the template with random directly after. If you pass in the list as a variable, then you do get a random() result each time; that variable can also be set in the template with {% set ... %}. It appears the template compiler is optimising one step too far somewhere.
So for now the work-arounds are:
Use {% set seq = ['#EDD7C9', '#EDD7C9', '#EEE'] %} and {{ seq | random }}
Pass in a list as a variable into the template and apply random to that
or to downgrade to 2.8.1.
I've filed issue #739 with the project to track this. The maintainers have excellent temporal skills and have fixed this a day ahead of time. A future version will include this fix (be it 2.9.7 or 2.10).
I would like to do some basic pillar value settings for all boxes so that I can use them later in a unified way. Our minions are usually named in this format:
<project>-<env>.<role>-<sequence>.<domain>
Example pillar/base/top.sls:
base:
'*':
- basics
'I#project:mycoolproject and I#role:nginx':
- etc.
Example pillar/base/basics/init.sls:
{% if '-live.' in grains['id'] %}
env: production
{% elif '-qa.' in grains['id'] %}
env: qa
{% elif '-staging.' in grains['id'] %}
env: staging
{% else %}
env:
{% endif %}
{% set role = re.match("(?:live|qa|staging)\.([a-z_\-]+)\-', grains['id']).group(1) -%}
role: {{ role }}
The env part obviously works but I can't get the regex working. As far as I understood there is no way to import python module (i.e. import re) in jinja template. Any suggestions how to get regex functionality available in the pillar file if possible at all?
The simple answer is, "no". There is not a way to inject regex functionality directly into the jinja environment (I'm sure there's a way to extend jinja, but anyway..)
The way I addressed this was with an external module function, id_info.explode() and an external pillar.
Enable external modules on the master:
external_modules: /srv/extmod
External modules do not require any sort of special infrastructure--they are just regular python modules (not packages, mind you--the loader doesn't currently know how to properly side-load a package yet)
Put your python+regex logic there. Return a dictionary, assembled to your your liking.
Your external module would go in /srv/extmod/modules. You can call call this function from your pillar.sls
{% id_info = __salt__[id_info.explode()] -%}
{% subcomponent = id_info['subcomponent'] -%}
{% project = id_info['project'] -%}
etc...
A couple things to know:
The salt-master has to be restarted when an external module is added or modified. There isn't a way that I know of to incite the equivalent of a saltutil.refresh_modules() call on the salt-master, so there ya go.
The external_modules directive is not just for execution modules. In this scenario, you would also create /srv/extmod/{pillar,runners,outputers,etc}.
These modules are only available on the master
I want to create a group and user using salt state files, but I do not know the group, gid, user, uid, sshkey until I need to execute the salt state file which I would like to pass in as parameters.
I have read about Pillar to create the variable. How do I create pillars before execution?
/srv/salt/group.sls:
{{ name }}:
group.present:
- gid: {{ gid }}
- system: True
Command line:
salt 'SaltStack-01' state.sls group name=awesome gid=123456
If you really want to pass in the data on the command like you can also do it like this:
{{ pillar['name'] }}:
group.present:
- gid: {{ pillar['gid'] }}
- system: True
Then on the command line you can pass in the data like this:
salt 'SaltStack-01' state.sls group pillar='{"name": "awesome", "gid": "123456"}'
You use Pillars to create "dictionaries" that you can reference into State files. I'm not sure if I'm understanding you correctly, but here's an example of what you can do:
mkdir /srv/pillar/
Create /srv/pillar/groups.sls and paste something like this into it:
groups:
first: 1234
second: 5678
These are names and GIDs of the groups you want to create.
Create /srv/pillar/top.sls so you can apply this pillar to your minions. This is very similar to a salt top file, so you can either apply it to all minions ('*') or just the one ('SaltStack-01'):
base:
'hc01*':
- groups
To test that that has worked, you can run salt '*' pillar.items and you should find the groups pillar somewhere in the output.
Now, your /srv/salt/group.sls file should look like this:
{% for group,gid in pillar.get('groups',{}).items() %}
{{ group }}:
group.present:
- gid: {{ gid }}
{% endfor %}
This is a for loop: for every group and gid in the pillar groups, do the rest. So basically, you can look at it as if the state file is running twice:
first:
group.present:
- gid: 1234
And then:
second:
group.present:
- gid: 5678
This was incorporated from this guide.
if you do not want use Pillar
you can do as:
# /srv/salt/params.yaml
name: awesome
gid: 123456
and then:
# /srv/salt/groups.sls
{% import_yaml "params.yaml" as params %}
{{ params['name'] }}:
group.present:
- gid: {{ parmas['gid'] }}
- system: True
more details:doc
Another nice way to pass (incase you don't want to use pillars Nor create a file as other answers shows) - you can pass a local environment variable to salt and read it from within the sls file, like this:
Command:
MYVAR=world salt 'SaltStack-01' state.sls somesalt # Note the env variable passed at the beginning
sls file:
# /srv/salt/somesalt.sls
foo:
cmd.run:
- name: |
echo "hello {{ salt['environ.get']('MYVAR') }}"
Will print to stdout:
hello world
Another good thing to know is that the env variable also gets passed on to any included salt states as well.
How does one install a package from a tar ball using salt stack? For example, I want to install Apache Tomcat from a remote tar ball - how can I achieve this? I tried using file.managed, but it's showing mismatch in source_hash - how can I predict checksum dynamically using the source url?
{% set tomcat = pillar.get('tomcat', {}) -%}
{% set version = tomcat.get('version', 'x.x.x') -%}
{% set tomcat_package = source + '/tomact-' + version + '.tar.gz' -%}
file.managed:
- name: {{ tomcat_package }}
- source: http://tomcat.org/download/tomcat-{{ version }}.tar.gz
- source_hash: {{ checksum }}
file.managed only download that file for you. To extract it, take a look at the archive.extracted:
tomcat:
archive:
- extracted
- name: /opt/
- source: http://mirrors.digipower.vn/apache/tomcat/tomcat-7/v7.0.47/bin/apache-tomcat-7.0.47.tar.gz
- source_hash: md5=efbae77efad579b655ae175754cad3df
- archive_format: tar
- options: z
- if_missing: /opt/apache-tomcat-7.0.47