We Keep Coding

How can I update a value in a YAML file?

I have this YAML file:
id: "bundle-1"
version: "1"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
I want to modify the file by increasing the version number by 1.
I tried this code:
import sys
from ruamel.yaml import YAML
import yaml
file_name = 'bundle.yaml'
yaml.preserve_quotes = True
with open(file_name) as yml_file:
data = yaml.safe_load(yml_file)
value = int(data['version'])
print(type(value))
value += 1
str(value)
print(type(value))
data['version'] = str(value)
data = str(data)
print(value)
with open(file_name, 'w') as yaml_file:
yaml_file.write( yaml.dump(data, sys.stdout))
But I get this output, without double quotes and ordered differently:
id: bundle-1
apiVersion: 1
description: Desc
jcasc:
- jenkins.yaml
plugins:
- plugins.yaml
version: 1

But I get this output, without double quotes and ordered differently:
Since this is an object, the order of the keys don't matter. Also, yaml doesn't require quotes around strings. It looks like the library you are using omits them.
This is not a problem since the YAML is valid.
A more important problem I see is that the version is not incremented. You will have to debug your code to figure out why. I don't see anything obviously wrong with what you are doing.
On a side note, this line looks strange to me:
yaml_file.write( yaml.dump(data, sys.stdout))
I don't know what yaml library you are using, but I doubt that yaml.dump() returns anything. You probably need to do this instead:
yaml.dump(data, yaml_file)
```
You should refer to the documentation for this library to learn the correct usage of the `dump()` function.

What you need to do delete the line import yaml, instantiate a YAML() instance, and set the preserve_quotes attribute
on it. And then use the 'load() and 'dump() methods.
There is never, ever use for safe_load() and safe_dump().
import sys
import ruamel.yaml
yaml_str = """\
id: "bundle-1"
version: "1"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
"""
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=4, sequence=4, offset=2)
yaml.preserve_quotes = True
data = yaml.load(yaml_str)
data['version'] = ruamel.yaml.scalarstring.DoubleQuotedScalarString(int(data['version']) + 1)
yaml.dump(data, sys.stdout)
which gives:
id: "bundle-1"
version: "2"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
Please note that I do not import yaml and neither should you.
This code:
with open(file_name, 'w') as yaml_file:
yaml_file.write( yaml.dump(data, sys.stdout))
will write None to the file you open, as that is what yaml.dump() returns. Instead do:
from pathlib import Path
path = Path('bundle.yaml')
yaml.dump(data, path) # with yaml being a YAML instance

Python YAML dump is changing the double quotes with 2 single quote

I am using python yaml module to load the content from a yaml file, modified some properties, and re-writing back to a file using "yaml.dump()", however, it is changing the double quotes with 2 single quotes which is finally getting failed when it is run by ansible. Is there any way to preserve quotes that was exactly the same way when reading it
Sample section of the code:
def update_status(playbook, masterfile):
with open(masterfile, 'w') as file:
#yaml.preserve_quotes = True
playbook1 = get_default_loader()
print(playbook1)
print("===========================================================\n")
print(playbook)
yaml.dump(playbook, file, default_flow_style=False, allow_unicode=True)
file.close()
playbook=reload_queue("test.yml")
update_status(playbook,'play.yml')
Content of test.yml where i am reading from :
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"
Content of play.yml written by python yaml module :
- import_playbook: "some.yaml"
vars:
param: '{{ hostvars[''text_node''][''test_server''] | default(''all'') }}'
This problem is not seen when using "ruamel.yaml". but we have a restriction to use this library. Can this be done using only yaml module?

Your code is incomplete, but you are probably mixing up ruamel.yaml deprecated API with the new one. You should
never use yaml.dump(....., default_flow_style=False, allow_unicode=True).
ruamel.yaml properly keeps the quotes as is, when you set .preserve_quotes:
import sys
import ruamel.yaml
yaml_str = """\
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"
"""
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(yaml_str)
yaml.dump(data, sys.stdout)
which gives:
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"

How to write yml file using python pyml package?

i want create yml file using python dictionary how to make dictionary format that i can get below format yml file
responses:
utter_greet:
- text: Hey! How are you?
buttons:
- title: "good"
payload: "/greet"
- title: "bad"
payload: "/health"

You can use this package to convert to dict
https://github.com/Infinidat/munch
pip3 install munch
convert to dict
import yaml
from munch import Munch
mydict = yaml.safe_load("""
responses:
utter_greet:
- text: Hey! How are you?
buttons:
- title: "good"
payload: "/greet"
- title: "bad"
payload: "/health"
""")
print(mydict)
convert dict to yaml
with open('output.yml', 'w') as yaml_file:
yaml.dump(mydict, yaml_file, default_flow_style=False)

How to get this yml template using python:
It has to be generating a UUID for a file and the file generated should have this yml template:
import uuid
print(uuid.uuid1())
u = str(uuid.uuid1())
u
open(u+".yml", "a+")
YML template format:
- id: 7049e3ec-b822-4fdf-a4ac-18190f9b66d1
name: Powerkatz (Staged)
description: Use Invoke-Mimikatz
tactic: credential-access
technique:
attack_id: T1003.001
name: "OS Credential Dumping: LSASS Memory"
privilege: Elevated
platforms:
windows:
psh:
command: |
Import-Module .\invoke-mimi.ps1;
Invoke-Mimikatz -DumpCreds
parsers:
plugins.stockpile.app.parsers.katz:
- source: domain.user.name
edge: has_password
target: domain.user.password
- source: domain.user.name
edge: has_hash
target: domain.user.ntlm
- source: domain.user.name
edge: has_hash
target: domain.user.sha1
payloads:
- invoke-mimi.ps1

How do I add a pipe the vertical bar (|) into a yaml file from Python

I have a task. I need to write python code to generate a yaml file for kubernetes. So far I have been using pyyaml and it works fine. Here is my generated yaml file:
apiVersion: v1
kind: ConfigMap
data:
info:
name: hostname.com
aio-max-nr: 262144
cpu:
cpuLogicalCores: 4
memory:
memTotal: 33567170560
net.core.somaxconn: 1024
...
However, when I try to create this configMap the error is that info expects a string() but not a map. So I explored a bit and it seem the easiest way to resolve this is to add a pipe after info like this:
apiVersion: v1
kind: ConfigMap
data:
info: | # this will translate everything in data into a string but still keep the format in yaml file for readability
name: hostname.com
aio-max-nr: 262144
cpu:
cpuLogicalCores: 4
memory:
memTotal: 33567170560
net.core.somaxconn: 1024
...
This way, my configmap is created successfully. My struggling is I dont know how to add that pipe bar from python code. Here I manually added it, but I want to automate this whole process.
part of the python code I wrote is, pretend data is a dict():
content = dict()
content["apiVersion"] = "v1"
content["kind"] = "ConfigMap"
data = {...}
info = {"info": data}
content["data"] = info
# Get all contents ready. Now write into a yaml file
fileName = "out.yaml"
with open(fileName, 'w') as outfile:
yaml.dump(content, outfile, default_flow_style=False)
I searched online and found a lot of cases, but none of them fits my needs. Thanks in advance.

The pipe makes the contained values a string. That string is not processed by YAML, even if it contains data with YAML syntax. Consequently, you will need to give a string as value.
Since the string contains data in YAML syntax, you can create the string by processing the contained data with YAML in a previous step. To make PyYAML dump the scalar in literal block style (i.e. with |), you need a custom representer:
import yaml, sys
from yaml.resolver import BaseResolver
class AsLiteral(str):
pass
def represent_literal(dumper, data):
return dumper.represent_scalar(BaseResolver.DEFAULT_SCALAR_TAG,
data, style="|")
yaml.add_representer(AsLiteral, represent_literal)
info = {
"name": "hostname.com",
"aio-max-nr": 262144,
"cpu": {
"cpuLogicalCores": 4
}
}
info_str = AsLiteral(yaml.dump(info))
data = {
"apiVersion": "v1",
"kind": "ConfigMap",
"data": {
"info": info_str
}
}
yaml.dump(data, sys.stdout)
By putting the rendered YAML data into the type AsLiteral, the registered custom representer will be called which will set the desired style to |.

Using ruamel.yaml to update yaml block in yaml file which contains multiple yamls

I've got a yaml file which contains several yaml blocks. The file can be seen here.
Basically, I am trying to update the value for one of the image keys (ctrl+f blockfreight/go-bftx:) in the last yaml block of the file. However, I want to preserve everything about the file (including comments) except for the one value that I am updating.
I've got the following code:
"""
Retrieves a truncated version of the latest git commit sha and updates
the go-bftx container image tag in app.yaml
"""
import sys
import ruamel.yaml
from subprocess import check_output
yaml_path = 'app.yaml'
for yaml_block in ruamel.yaml.round_trip_load_all(stream=open(yaml_path)):
pass
# parse the most recent git commit sha from command line
docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()
# update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
yaml_block['spec']['template']['spec']['containers'][1]['image'] = docker_image
ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)
This successfully edits the value, keeps comments intact, but only sends the final yaml block to stdout.
Is there any way for me to edit only the sixth (and final) yaml block in app.yaml, and keep the rest of the file intact (comments included)?
I tried replacing pass in the above code with an if statement that sent the first five yamls to stdout, and then edited the value in the sixth value, sending that to stdout as well. My thinking was to use bash to send all of the stdout to a file (e.g. python app_modifier.py > app1.yaml), but this only sent the output from the sixth yaml.
That code looked like this:
for i, yaml_block in enumerate(ruamel.yaml.round_trip_load_all(stream=open(yaml_path))):
if i != 5:
ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)
else:
# parse the most recent git commit sha from command line
docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 --pretty=format:%h'.split()).decode()
# update go-bftx image with most recent git-commit-sha tag in the StatefulSet blocks
yaml_block['spec']['template']['spec']['containers'][1]['image'] = docker_image
ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)
Any assistance would be greatly appreciated! Thank you!

Your file consists of multiple YAML documents, this is what you read in using round_trip_load_all, which gives you a generator.
If you write that back using round_trip_dump(), you are never going to get the --- YAML document separators that are in the original file.
You can probably use a generator and round_trip_dump_all since you know in which document to make changes, but it is probably easier to make a list of what round_trip_load_all generates and work on that. I would also use ruamel.yaml's new API to do so:
import sys
import ruamel.yaml
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = list(yaml.load_all(open('app.yaml')))
# parse the most recent git commit sha from command line
# docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()
docker_image = 'blockfreight/go-bftx:ci-cd-' + 'check_output_output'
# update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
data[-1]['spec']['template']['spec']['containers'][1]['image'] = docker_image
with open('out.yaml', 'w') as ofp:
yaml.dump_all(data, ofp)
The above mostly works. The out.yaml diff's with the change you want to make, and some whitespace standardisations (indents, between [ ") which are of course one off.
There are two major issues with this approach:
The first YAML document in the file has no content, so its comments cannot be hung from any mapping/sequence and are not preserved
The second YAML document has a final comment ( # // Initializes BFTX Service to interact with endpoints ) that gets dropped. That is most likely a bug in ruamel.yaml that shows up in multi-document files only and that I have to investigate
To solve the first issue, you can only read that document as normal lines without ruamel.yaml. When you do so, you might as well do all but the rest, as this solves (i.e. circumvents) the second issue as well:
import sys
import ruamel.yaml
with open('out.yaml', 'w') as ofp:
lines = ''
with open('app.yaml') as ifp:
for line in ifp:
lines += line
if line == '---\n':
ofp.write(lines)
lines = ''
# process lines from the last document
# print(lines)
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(lines)
# parse the most recent git commit sha from command line
# docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()
docker_image = 'blockfreight/go-bftx:ci-cd-' + 'check_output_output'
# update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
data['spec']['template']['spec']['containers'][1]['image'] = docker_image
yaml.dump(data, ofp)
This should do what you want unless you care about the leading trailing space in a flow style sequence, or preserve the three places with inconsistent indentation. The output of diff -u app.yaml out.yaml:
--- app.yaml 2018-06-23 14:41:02.256290577 +0200
+++ out.yaml 2018-06-23 14:58:09.933991459 +0200
## -143,7 +143,7 ##
spec:
selector:
matchLabels:
- app: bftx
+ app: bftx
serviceName: blockfreight
replicas: 1
template:
## -151,7 +151,7 ##
labels:
app: bftx
spec:
- containers:
+ containers:
- name: tm
imagePullPolicy: IfNotPresent
image: tendermint/tendermint:0.20.0
## -199,7 +199,7 ##
tendermint node --moniker="`hostname`" --p2p.seeds="aeabbf6b891435013f2a800fa9e22a1451ca90fd#bftx0.blockfreight.net:8888,6e9515c2cfed19464e6ce11ba2297ecdb411103b#bftx1.blockfreight.net:8888,b8b988370783bd0e58bf926d621a47160af2bdae#bftx2.blockfreight.net:8888,8c091f4e3dc4ac27db1efd38beee012d99967fd8#bftx3.blockfreight.net:8888" --proxy_app="tcp://localhost:46658" --consensus.create_empty_blocks=false
- name: app
imagePullPolicy: Always
- image: blockfreight/go-bftx:rc1
+ image: blockfreight/go-bftx:ci-cd-check_output_output
ports:
- containerPort: 12345
- containerPort: 46658
## -247,7 +247,7 ##
- mountPath: /etc/nginx/conf.d/pub_key.conf
name: tmconfigdir
subPath: pub_key_nginx.conf
- volumes:
+ volumes:
- name: tmconfigdir
configMap:
name: bftx-config
## -262,7 +262,7 ##
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
- accessModes: [ "ReadWriteOnce" ]
+ accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
## -271,7 +271,7 ##
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
- accessModes: [ "ReadWriteOnce" ]
+ accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 12Mi