I am using python yaml module to load the content from a yaml file, modified some properties, and re-writing back to a file using "yaml.dump()", however, it is changing the double quotes with 2 single quotes which is finally getting failed when it is run by ansible. Is there any way to preserve quotes that was exactly the same way when reading it
Sample section of the code:
def update_status(playbook, masterfile):
with open(masterfile, 'w') as file:
#yaml.preserve_quotes = True
playbook1 = get_default_loader()
print(playbook1)
print("===========================================================\n")
print(playbook)
yaml.dump(playbook, file, default_flow_style=False, allow_unicode=True)
file.close()
playbook=reload_queue("test.yml")
update_status(playbook,'play.yml')
Content of test.yml where i am reading from :
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"
Content of play.yml written by python yaml module :
- import_playbook: "some.yaml"
vars:
param: '{{ hostvars[''text_node''][''test_server''] | default(''all'') }}'
This problem is not seen when using "ruamel.yaml". but we have a restriction to use this library. Can this be done using only yaml module?
Your code is incomplete, but you are probably mixing up ruamel.yaml deprecated API with the new one. You should
never use yaml.dump(....., default_flow_style=False, allow_unicode=True).
ruamel.yaml properly keeps the quotes as is, when you set .preserve_quotes:
import sys
import ruamel.yaml
yaml_str = """\
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"
"""
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(yaml_str)
yaml.dump(data, sys.stdout)
which gives:
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"
Related
I have this YAML file:
id: "bundle-1"
version: "1"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
I want to modify the file by increasing the version number by 1.
I tried this code:
import sys
from ruamel.yaml import YAML
import yaml
file_name = 'bundle.yaml'
yaml.preserve_quotes = True
with open(file_name) as yml_file:
data = yaml.safe_load(yml_file)
value = int(data['version'])
print(type(value))
value += 1
str(value)
print(type(value))
data['version'] = str(value)
data = str(data)
print(value)
with open(file_name, 'w') as yaml_file:
yaml_file.write( yaml.dump(data, sys.stdout))
But I get this output, without double quotes and ordered differently:
id: bundle-1
apiVersion: 1
description: Desc
jcasc:
- jenkins.yaml
plugins:
- plugins.yaml
version: 1
But I get this output, without double quotes and ordered differently:
Since this is an object, the order of the keys don't matter. Also, yaml doesn't require quotes around strings. It looks like the library you are using omits them.
This is not a problem since the YAML is valid.
A more important problem I see is that the version is not incremented. You will have to debug your code to figure out why. I don't see anything obviously wrong with what you are doing.
On a side note, this line looks strange to me:
yaml_file.write( yaml.dump(data, sys.stdout))
I don't know what yaml library you are using, but I doubt that yaml.dump() returns anything. You probably need to do this instead:
yaml.dump(data, yaml_file)
```
You should refer to the documentation for this library to learn the correct usage of the `dump()` function.
What you need to do delete the line import yaml, instantiate a YAML() instance, and set the preserve_quotes attribute
on it. And then use the 'load() and 'dump() methods.
There is never, ever use for safe_load() and safe_dump().
import sys
import ruamel.yaml
yaml_str = """\
id: "bundle-1"
version: "1"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
"""
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=4, sequence=4, offset=2)
yaml.preserve_quotes = True
data = yaml.load(yaml_str)
data['version'] = ruamel.yaml.scalarstring.DoubleQuotedScalarString(int(data['version']) + 1)
yaml.dump(data, sys.stdout)
which gives:
id: "bundle-1"
version: "2"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
Please note that I do not import yaml and neither should you.
This code:
with open(file_name, 'w') as yaml_file:
yaml_file.write( yaml.dump(data, sys.stdout))
will write None to the file you open, as that is what yaml.dump() returns. Instead do:
from pathlib import Path
path = Path('bundle.yaml')
yaml.dump(data, path) # with yaml being a YAML instance
i want create yml file using python dictionary how to make dictionary format that i can get below format yml file
responses:
utter_greet:
- text: Hey! How are you?
buttons:
- title: "good"
payload: "/greet"
- title: "bad"
payload: "/health"
You can use this package to convert to dict
https://github.com/Infinidat/munch
pip3 install munch
convert to dict
import yaml
from munch import Munch
mydict = yaml.safe_load("""
responses:
utter_greet:
- text: Hey! How are you?
buttons:
- title: "good"
payload: "/greet"
- title: "bad"
payload: "/health"
""")
print(mydict)
convert dict to yaml
with open('output.yml', 'w') as yaml_file:
yaml.dump(mydict, yaml_file, default_flow_style=False)
How to get this yml template using python:
It has to be generating a UUID for a file and the file generated should have this yml template:
import uuid
print(uuid.uuid1())
u = str(uuid.uuid1())
u
open(u+".yml", "a+")
YML template format:
- id: 7049e3ec-b822-4fdf-a4ac-18190f9b66d1
name: Powerkatz (Staged)
description: Use Invoke-Mimikatz
tactic: credential-access
technique:
attack_id: T1003.001
name: "OS Credential Dumping: LSASS Memory"
privilege: Elevated
platforms:
windows:
psh:
command: |
Import-Module .\invoke-mimi.ps1;
Invoke-Mimikatz -DumpCreds
parsers:
plugins.stockpile.app.parsers.katz:
- source: domain.user.name
edge: has_password
target: domain.user.password
- source: domain.user.name
edge: has_hash
target: domain.user.ntlm
- source: domain.user.name
edge: has_hash
target: domain.user.sha1
payloads:
- invoke-mimi.ps1
I have a task. I need to write python code to generate a yaml file for kubernetes. So far I have been using pyyaml and it works fine. Here is my generated yaml file:
apiVersion: v1
kind: ConfigMap
data:
info:
name: hostname.com
aio-max-nr: 262144
cpu:
cpuLogicalCores: 4
memory:
memTotal: 33567170560
net.core.somaxconn: 1024
...
However, when I try to create this configMap the error is that info expects a string() but not a map. So I explored a bit and it seem the easiest way to resolve this is to add a pipe after info like this:
apiVersion: v1
kind: ConfigMap
data:
info: | # this will translate everything in data into a string but still keep the format in yaml file for readability
name: hostname.com
aio-max-nr: 262144
cpu:
cpuLogicalCores: 4
memory:
memTotal: 33567170560
net.core.somaxconn: 1024
...
This way, my configmap is created successfully. My struggling is I dont know how to add that pipe bar from python code. Here I manually added it, but I want to automate this whole process.
part of the python code I wrote is, pretend data is a dict():
content = dict()
content["apiVersion"] = "v1"
content["kind"] = "ConfigMap"
data = {...}
info = {"info": data}
content["data"] = info
# Get all contents ready. Now write into a yaml file
fileName = "out.yaml"
with open(fileName, 'w') as outfile:
yaml.dump(content, outfile, default_flow_style=False)
I searched online and found a lot of cases, but none of them fits my needs. Thanks in advance.
The pipe makes the contained values a string. That string is not processed by YAML, even if it contains data with YAML syntax. Consequently, you will need to give a string as value.
Since the string contains data in YAML syntax, you can create the string by processing the contained data with YAML in a previous step. To make PyYAML dump the scalar in literal block style (i.e. with |), you need a custom representer:
import yaml, sys
from yaml.resolver import BaseResolver
class AsLiteral(str):
pass
def represent_literal(dumper, data):
return dumper.represent_scalar(BaseResolver.DEFAULT_SCALAR_TAG,
data, style="|")
yaml.add_representer(AsLiteral, represent_literal)
info = {
"name": "hostname.com",
"aio-max-nr": 262144,
"cpu": {
"cpuLogicalCores": 4
}
}
info_str = AsLiteral(yaml.dump(info))
data = {
"apiVersion": "v1",
"kind": "ConfigMap",
"data": {
"info": info_str
}
}
yaml.dump(data, sys.stdout)
By putting the rendered YAML data into the type AsLiteral, the registered custom representer will be called which will set the desired style to |.
I have the following YAML file:
heat_template_version: 2015-10-15
parameters:
image:
type: string
label: Image name or ID
default: CirrOS
private_network_id:
type: string
label: Private network name or ID
floating_ip:
type: string
I want to add key-> default to private_network_id and floating_ip (if default doesn't exist) and to the default key I want to add the value (which I get from user)
How can I achieve this in python?
The resulting YAML should look like:
heat_template_version: 2015-10-15
parameters:
image:
type: string
label: Image name or ID
default: CirrOS
private_network_id:
type: string
label: Private network name or ID
default: <private_network_id>
floating_ip:
type: string
default: <floating_ip>
For this kind of round-tripping you should do use ruamel.yaml (disclaimer: I am the author of the package).
Assuming your input is in a file input.yaml and the following program:
from ruamel.yaml import YAML
from pathlib import Path
yaml = YAML()
path = Path('input.yaml')
data = yaml.load(path)
parameters = data['parameters']
# replace assigned values with user input
parameters['private_network_id']['default'] = '<private_network_id>'
parameters['floating_ip']['default'] = '<floating_ip>'
yaml.dump(data, path)
After that your file will exact match the output you requested.
Please note that comments in the YAML file, as well as the key ordering are automatically preserved (not guaranteed by the YAML specification).
If you are still using Python2 (which has no pathlib in the standard library) use from ruamel.std.pathlib import Path or rewrite the .load() and .dump() lines with appropriately opened, old style, file objects. E.g.
with open('input.yaml', 'w') as fp:
yaml.dump(data, fp)
I have a YAML file (all.yaml) that looks like:
...
var1: val1
var2: val2
var3: {{var1}}-{{var2}}.txt
...
If I load it in Python like this:
import yaml
f = open('all.yaml')
dataMap = yaml.safe_load(f)
f.close()
print(dataMap["var3"])
the output is {{var1}}-{{var2}}.txt and not val1-val2.txt.
Is it possible to replace the nested vars with the value?
I tried to load it with:
import jinja2
templateLoader = jinja2.FileSystemLoader( searchpath="/path/to/dir" )
templateEnv = jinja2.Environment( loader=templateLoader )
TEMPLATE_FILE = "all.yaml"
template = templateEnv.get_template( TEMPLATE_FILE )
The exception is no longer thrown, now I am stuck and have to research how to proceed.
First define an Undefined class and load yaml to get known values. Then load it again and render with known values.
#!/usr/bin/env python
import yaml
from jinja2 import Template, Undefined
str1 = '''var1: val1
var2: val2
var3: {{var1}}-{{var2}}.txt
'''
class NullUndefined(Undefined):
def __getattr__(self, key):
return ''
t = Template(str1, undefined=NullUndefined)
c = yaml.safe_load(t.render())
print t.render(c)
Run it:
$ ./test.py
var1: val1
var2: val2
var3: val1-val2.txt
Here is one possible solution:
Parse your YAML document with the yaml module
Iterate over the keys in your YAML document, treating each value as a Jinja2 template to which you pass in the keys of the YAML document as parameters.
For example:
import yaml
from jinja2 import Template
with open('sample.yml') as fd:
data = yaml.load(fd)
for k, v in data.items():
t = Template(v)
data[k] = t.render(**data)
print yaml.safe_dump(data, default_flow_style=False)
This will work fine with your particular example, but wouldn't do anything useful for, say, nested data structures (in fact, it would probably just blow up).
There is no replacement/substitution of scalar parts within the YAML specification.
Anything you want to do on that level has to be done in your application. For me, and for YAML, {{var1}} is just a nested mapping. {{var1}} is short for {{var1: null}: null}. After that the - is not allowed.
There are however multiple problems with your post:
You are using PyYAML which only supports the old (2005) YAML 1.1. Therefore you cannot you cannot have multiple documents (i.e. ended with ...) without using an explicit document start (---) like you can in YAML 1.2
Even if you correct the first line to read --- instead of ... your file will not load as a dict {{var1}} cannot be followed by a scalar - (from -{{var2}}.txt)
And if you would just use {{var1}} in your file, PyYAML cannot load this as it loads YAML mappings as Python dict and Python doesn't allow mutable keys for a dict. Just like you get an TypeError in Python when you try to do: {dict(var1=None): None}
So you should at least change your input file all.yaml to:
---
var1: val1
var2: val2
var3: '{{var1}}-{{var2}}.txt'
...
to get this to load in YAML.
You'll have to load this file two times:
once by PyYAML to get the values that you can use to render template
once as template by jinja2
After you render the template you load that (string) once more in PyYAML and you have the value that you want.
Given the corrected all.yaml as specified above in the current directory and this program:
import yaml
import jinja2
YAML_FILE = 'all.yaml'
with open(YAML_FILE) as fp:
dataMap = yaml.safe_load(fp)
env = jinja2.Environment(loader=jinja2.FileSystemLoader(searchpath='.'))
template = env.get_template(YAML_FILE)
data = yaml.safe_load(template.render(**dataMap))
print(data["var3"])
will print what you wanted:
val1-val2.txt
I do not believe you can use:
yaml.load
or
yaml.safe_load
on a file containing jinja2 variables as values. The {{variable}} will attempt to be interpreted as a dict by yaml.