We Keep Coding

Jinja2 {% include file %} outside of search path doesn't work

This is an elementary issue which is probably related to Jinja2 PrefixLoader or ChoiceLoader.
On Python 3.6 we load with this command
jinja2.FileSystemLoader( searchpath= "\\template_folder\\")
On Windows 7, our file structure is as follows.
- folder_bbb
* subfile.txt
- template_folder
* template_file
- folder_aaa
* subfile.txt
From the template_file we are successful with this command
{% include "folder_aaa/subfile.txt" %}
Now we wish to move the file one level up, and write
{% include "../folder_bbb/subfile.txt" %}
but that doesn't work, complaining file not found.
What is the correct way to write? Thanks.

You may specify all paths in the the loader
jinja2.FileSystemLoader(["c:\\template_folder\\", "c:\\folder_bbb\\"])
and refer the including block without a specific path
{% include "subfile.txt" %}
The path will be searched in order so that as you say, moving the file one level up, the file will be found. (You need the template_folder path for the template itself.)

Parsing/extracting from OrderedDict using Jinja in Salt

I am running into a rather weird issue while parsing results of a salt command. The command I am running is
{% set hostname = salt['publish.publish']('roles:*{}*'.format(role), 'grains.item', 'fqdn', 'grain') %}
And output looks below:
OrderedDict([('1.server.com', OrderedDict([('fqdn', '1.server.com')])), ('0.server.com', OrderedDict([('fqdn', '0.server.com')]))])
Now my understanding is when I do items() on above result with a line below, it should work
{% for hostname, fqdn in salt['publish.publish']('roles:*{}*'.format(role), 'grains.item', 'fqdn', 'grain').items() %}
But the moment I use items() in above line I start running into an error:
failed: Jinja variable 'None' has no attribute 'items'
I tried a couple of other ways (Doing items().items() or storing result in a variable and then running for loop over) to get the list out of OrderedDict but none of ways seem to help.

Either I don't know Python enough or there is something weird going on. Simply adding a check has made the above work. So working block looks like (Partial code of course):
{% set hostname = salt['publish.publish']('roles:*{}*'.format(role), 'grains.item', 'fqdn', 'grain') %}
{% if hostname is not none %}
{% for host, site in hostname.items() %}
My understanding is if check was only meant for checking just in case hostname is empty. But looks like even if there is data - an if check is needed. Still curious to know the mystery!

SaltStack: Use regexps in pillar file

I would like to do some basic pillar value settings for all boxes so that I can use them later in a unified way. Our minions are usually named in this format:
<project>-<env>.<role>-<sequence>.<domain>
Example pillar/base/top.sls:
base:
'*':
- basics
'I#project:mycoolproject and I#role:nginx':
- etc.
Example pillar/base/basics/init.sls:
{% if '-live.' in grains['id'] %}
env: production
{% elif '-qa.' in grains['id'] %}
env: qa
{% elif '-staging.' in grains['id'] %}
env: staging
{% else %}
env:
{% endif %}
{% set role = re.match("(?:live|qa|staging)\.([a-z_\-]+)\-', grains['id']).group(1) -%}
role: {{ role }}
The env part obviously works but I can't get the regex working. As far as I understood there is no way to import python module (i.e. import re) in jinja template. Any suggestions how to get regex functionality available in the pillar file if possible at all?

The simple answer is, "no". There is not a way to inject regex functionality directly into the jinja environment (I'm sure there's a way to extend jinja, but anyway..)
The way I addressed this was with an external module function, id_info.explode() and an external pillar.
Enable external modules on the master:
external_modules: /srv/extmod
External modules do not require any sort of special infrastructure--they are just regular python modules (not packages, mind you--the loader doesn't currently know how to properly side-load a package yet)
Put your python+regex logic there. Return a dictionary, assembled to your your liking.
Your external module would go in /srv/extmod/modules. You can call call this function from your pillar.sls
{% id_info = __salt__[id_info.explode()] -%}
{% subcomponent = id_info['subcomponent'] -%}
{% project = id_info['project'] -%}
etc...
A couple things to know:
The salt-master has to be restarted when an external module is added or modified. There isn't a way that I know of to incite the equivalent of a saltutil.refresh_modules() call on the salt-master, so there ya go.
The external_modules directive is not just for execution modules. In this scenario, you would also create /srv/extmod/{pillar,runners,outputers,etc}.
These modules are only available on the master

Unescape html entities from JSON in Django templates

I'm getting the response from server that is escaped:
'item':'<b> Some Data </b>'
I pass such data to template useing item= json.loads(response)
By default django templates (in Google App Engine) escapes it further,
so its double escaped in results.
I can use safe to remove one level of escaping like:
{{item|safe}}
How do i turn entities to their corresponding signs?

You can do this:
{% autoescape off %}
{{ your_text_var }}
{% endautoescape %}

Warning - THIS IS NOT A RECOMMENDED SOLUTION. You should be using autoescaping instead (check Rafael's answer).
Following should do the job.
response.replace('&', '&').replace('<', '<').replace('>', '>')
Update -
After suggestion by Jan Schär, you should rather use the following :
response.replace('<', '<').replace('>', '>').replace('&', '&')
Because, if response is &gt;, it would result in > instead of the correct >. You should resolve & in the last.

RSS Feed aggregator using Google App Engine - Python

I am trying to build a GAE app that processes an RSS feed and stores all the data from the feed into Google Datastore. I use Minidom to extract content from the RSS feed. I also tried using Feedparser and BeautifulSoup but they did not work for me.
My app currently parses the feed and saves it in the Google datastore in about 25 seconds on my local machine. I uploaded the app and I when I tried to use it, I got the "DeadLine Exceeded Error".
I would like to know if there are any possible ways to speed up this process? The feed I use will eventually grow to have more than a 100 items over time.

It shouldn't take anywhere near that long. Here is how you might use the Universal Feed Parser.
# easy_install feedparser
And an example of using it:
import feedparser
feed = 'http://stackoverflow.com/feeds/tag?tagnames=python&sort=newest'
d = feedparser.parse(feed)
for entry in d['entries']:
print entry.title
The documentation shows you how to pull other things out of a feed. If there is a specific issue you have, please post the details.

I found a way to work around this issue, though I am not sure if this is the optimal solution.
Instead of Minidom I have used cElementTree to parse the RSS feed. I process each "item" tag and its children in a seperate task and add these tasks to the task queue.
This has helped me avoid the DeadlineExceededError. I get the "This resource uses a lot of CPU resources" warning though.
Any idea on how to avoid the warning?
A_iyer

I have a GAE RSS reader demo / prototype working using Feedparser - http://deliciourss.appspot.com/. Here's some code -
Fetch your feed.
data = urlfetch.fetch(feedUrl)
Parse with Feedparser
parsedData = feedparser.parse(data.content)
Change some features of the feed
# set main section to description if empty
for ix in range(len(parsedData.entries)):
bItem = 0
if hasattr(parsedData.entries[ix],'content'):
for item in parsedData.entries[ix].content:
if item.value:
bItem = 1
break
if bItem == 0:
parsedData.entries[ix].content[0].value = parsedData.entries[ix].summary
else:
parsedData.entries[ix].content = [{'value':parsedData.entries[ix].summary}]
Template if you are using Django/webapp
<?xml version="1.0" encoding="utf-8"?>
<channel>
<title>{{parsedData.channel.title}}</title>
<url>{{feedUrl}}</url>
<id>{{parsedData.channel.id}}</id>
<updated>{{parsedData.channel.updated}}</updated>
{% for entry in parsedData.entries %}
<item>
<id>{{entry.id}}</id>
<title>{{entry.title}}</title>
<link>
{% for link in entry.links %}
{% ifequal link.rel "alternate" %}
{{link.href|escape}}
{% endifequal %}
{% endfor %}
</link>
<author>{{entry.author_detail.name}}</author>
<pubDate>{{entry.published}}</pubDate>
<description>{{entry.summary|escape}}</description>
{% for item in entry.content %}
{% if item.value %}
<content>{{item.value|escape}}</content>
{% endif %}
{% endfor %}
</item>{% endfor %}
</channel>