i want create yml file using python dictionary how to make dictionary format that i can get below format yml file
responses:
utter_greet:
- text: Hey! How are you?
buttons:
- title: "good"
payload: "/greet"
- title: "bad"
payload: "/health"
You can use this package to convert to dict
https://github.com/Infinidat/munch
pip3 install munch
convert to dict
import yaml
from munch import Munch
mydict = yaml.safe_load("""
responses:
utter_greet:
- text: Hey! How are you?
buttons:
- title: "good"
payload: "/greet"
- title: "bad"
payload: "/health"
""")
print(mydict)
convert dict to yaml
with open('output.yml', 'w') as yaml_file:
yaml.dump(mydict, yaml_file, default_flow_style=False)
How to get this yml template using python:
It has to be generating a UUID for a file and the file generated should have this yml template:
import uuid
print(uuid.uuid1())
u = str(uuid.uuid1())
u
open(u+".yml", "a+")
YML template format:
- id: 7049e3ec-b822-4fdf-a4ac-18190f9b66d1
name: Powerkatz (Staged)
description: Use Invoke-Mimikatz
tactic: credential-access
technique:
attack_id: T1003.001
name: "OS Credential Dumping: LSASS Memory"
privilege: Elevated
platforms:
windows:
psh:
command: |
Import-Module .\invoke-mimi.ps1;
Invoke-Mimikatz -DumpCreds
parsers:
plugins.stockpile.app.parsers.katz:
- source: domain.user.name
edge: has_password
target: domain.user.password
- source: domain.user.name
edge: has_hash
target: domain.user.ntlm
- source: domain.user.name
edge: has_hash
target: domain.user.sha1
payloads:
- invoke-mimi.ps1
Related
I have this YAML file:
id: "bundle-1"
version: "1"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
I want to modify the file by increasing the version number by 1.
I tried this code:
import sys
from ruamel.yaml import YAML
import yaml
file_name = 'bundle.yaml'
yaml.preserve_quotes = True
with open(file_name) as yml_file:
data = yaml.safe_load(yml_file)
value = int(data['version'])
print(type(value))
value += 1
str(value)
print(type(value))
data['version'] = str(value)
data = str(data)
print(value)
with open(file_name, 'w') as yaml_file:
yaml_file.write( yaml.dump(data, sys.stdout))
But I get this output, without double quotes and ordered differently:
id: bundle-1
apiVersion: 1
description: Desc
jcasc:
- jenkins.yaml
plugins:
- plugins.yaml
version: 1
But I get this output, without double quotes and ordered differently:
Since this is an object, the order of the keys don't matter. Also, yaml doesn't require quotes around strings. It looks like the library you are using omits them.
This is not a problem since the YAML is valid.
A more important problem I see is that the version is not incremented. You will have to debug your code to figure out why. I don't see anything obviously wrong with what you are doing.
On a side note, this line looks strange to me:
yaml_file.write( yaml.dump(data, sys.stdout))
I don't know what yaml library you are using, but I doubt that yaml.dump() returns anything. You probably need to do this instead:
yaml.dump(data, yaml_file)
```
You should refer to the documentation for this library to learn the correct usage of the `dump()` function.
What you need to do delete the line import yaml, instantiate a YAML() instance, and set the preserve_quotes attribute
on it. And then use the 'load() and 'dump() methods.
There is never, ever use for safe_load() and safe_dump().
import sys
import ruamel.yaml
yaml_str = """\
id: "bundle-1"
version: "1"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
"""
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=4, sequence=4, offset=2)
yaml.preserve_quotes = True
data = yaml.load(yaml_str)
data['version'] = ruamel.yaml.scalarstring.DoubleQuotedScalarString(int(data['version']) + 1)
yaml.dump(data, sys.stdout)
which gives:
id: "bundle-1"
version: "2"
apiVersion: "1"
description: "Desc"
jcasc:
- "jenkins.yaml"
plugins:
- "plugins.yaml"
Please note that I do not import yaml and neither should you.
This code:
with open(file_name, 'w') as yaml_file:
yaml_file.write( yaml.dump(data, sys.stdout))
will write None to the file you open, as that is what yaml.dump() returns. Instead do:
from pathlib import Path
path = Path('bundle.yaml')
yaml.dump(data, path) # with yaml being a YAML instance
I am using python yaml module to load the content from a yaml file, modified some properties, and re-writing back to a file using "yaml.dump()", however, it is changing the double quotes with 2 single quotes which is finally getting failed when it is run by ansible. Is there any way to preserve quotes that was exactly the same way when reading it
Sample section of the code:
def update_status(playbook, masterfile):
with open(masterfile, 'w') as file:
#yaml.preserve_quotes = True
playbook1 = get_default_loader()
print(playbook1)
print("===========================================================\n")
print(playbook)
yaml.dump(playbook, file, default_flow_style=False, allow_unicode=True)
file.close()
playbook=reload_queue("test.yml")
update_status(playbook,'play.yml')
Content of test.yml where i am reading from :
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"
Content of play.yml written by python yaml module :
- import_playbook: "some.yaml"
vars:
param: '{{ hostvars[''text_node''][''test_server''] | default(''all'') }}'
This problem is not seen when using "ruamel.yaml". but we have a restriction to use this library. Can this be done using only yaml module?
Your code is incomplete, but you are probably mixing up ruamel.yaml deprecated API with the new one. You should
never use yaml.dump(....., default_flow_style=False, allow_unicode=True).
ruamel.yaml properly keeps the quotes as is, when you set .preserve_quotes:
import sys
import ruamel.yaml
yaml_str = """\
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"
"""
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(yaml_str)
yaml.dump(data, sys.stdout)
which gives:
- import_playbook: "some.yaml"
vars:
param: "{{ hostvars['text_node']['test_server'] | default(''all'') }}"
I'm trying to edit a yaml file but when I write the new file the whole structure of the file is messed up.
Line breaks are wrong, some indentations are wrong as well and it even deletes some of my parameters for some reason. Here's an example:
Before
AWSTemplateFormatVersion: "2010-09-09"
Metadata:
Generator: "user"
Description: "CloudFormation blah blah"
Parameters:
VPCEndpointServiceServiceName:
Description: VPCEndpointServiceServiceName
Type: String
Mappings:
PrivateLink:
EndPoint:
EndPointName: test
EndPointVpcId: vpc-123
SecurityGroupIds: sg-123
SubnetId1: subnet-123
SubnetId2: subnet-123
Resources:
EC2VPCEndpoint:
Type: "AWS::EC2::VPCEndpoint"
Properties:
VpcEndpointType: "Interface"
VpcId: !FindInMap [PrivateLink, EndPoint, EndPointVpcId]
ServiceName: !Ref VPCEndpointServiceServiceName
SubnetIds:
- !FindInMap [PrivateLink, EndPoint, SubnetId1]
- !FindInMap [PrivateLink, EndPoint, SubnetId2]
PrivateDnsEnabled: false
SecurityGroupIds:
- !FindInMap [PrivateLink, EndPoint, SecurityGroupIds]
After
AWSTemplateFormatVersion: '2010-09-09'
Metadata:
Generator: user
Description: CloudFormation blah blah
Parameters:
VPCEndpointServiceServiceName:
Description: VPCEndpointServiceServiceName
Type: String
Mappings:
PrivateLink:
EndPoint:
EndPointName: test
EndPointVpcId: vpc-123
SecurityGroupIds: sg-123
SubnetId1: subnet-123
SubnetId2: subnet-123
Resources:
EC2VPCEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
VpcEndpointType: Interface
VpcId:
- PrivateLink
- EndPoint
- EndPointVpcId
ServiceName: VPCEndpointServiceServiceName
SubnetIds:
- - PrivateLink
- EndPoint
- SubnetId1
- - PrivateLink
- EndPoint
- SubnetId2
PrivateDnsEnabled: 'false'
SecurityGroupIds:
- - PrivateLink
- EndPoint
- SecurityGroupIds
What's really weird is that it also removes parameters like !FindInMap and !Ref.
Here's my function:
def editEndpointTemplate(endpoint_tempplate_path):
#read yaml file
with open(endpoint_tempplate_path) as file:
data = yaml.load(file, Loader=yaml.BaseLoader)
data['Mappings']['PrivateLink']['EndPoint']['EndPointName'] = "New Name"
#write yaml file and overwrite old one.
with open(endpoint_tempplate_path, 'w') as file:
yaml.dump(data, file, sort_keys=False)
I want to preserve the file exactly as it is because all I do is update a few keys.
I also had issues with the file being sorted alphabetically, but a simple sort_keys=False fixed that. I figured any other of PyYaml's attributes would solve this but couldn't figure it out.
Any help would be appreciated.
Found a workaround. Instead of using PyYaml or cfn-tools, I used ruamel.yaml.
import sys
from ruamel.yaml import YAML
def editEndpointTemplate(endpoint_template_path):
yaml = YAML()
yaml.indent(mapping=3)
#Load yaml file
with open(endpoint_template_path) as fp:
data = yaml.load(fp)
#Change data
data['Mappings']['PrivateLink']['EndPoint']['EndPointName'] = service_name
#Dump it back to the same file
# yaml.dump(data, sys.stdout)
with open(endpoint_template_path, 'w') as fp:
yaml.dump(data, fp)
The new YAML file is preserved exactly like my template, even after editing.
I've got a yaml file which contains several yaml blocks. The file can be seen here.
Basically, I am trying to update the value for one of the image keys (ctrl+f blockfreight/go-bftx:) in the last yaml block of the file. However, I want to preserve everything about the file (including comments) except for the one value that I am updating.
I've got the following code:
"""
Retrieves a truncated version of the latest git commit sha and updates
the go-bftx container image tag in app.yaml
"""
import sys
import ruamel.yaml
from subprocess import check_output
yaml_path = 'app.yaml'
for yaml_block in ruamel.yaml.round_trip_load_all(stream=open(yaml_path)):
pass
# parse the most recent git commit sha from command line
docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()
# update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
yaml_block['spec']['template']['spec']['containers'][1]['image'] = docker_image
ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)
This successfully edits the value, keeps comments intact, but only sends the final yaml block to stdout.
Is there any way for me to edit only the sixth (and final) yaml block in app.yaml, and keep the rest of the file intact (comments included)?
I tried replacing pass in the above code with an if statement that sent the first five yamls to stdout, and then edited the value in the sixth value, sending that to stdout as well. My thinking was to use bash to send all of the stdout to a file (e.g. python app_modifier.py > app1.yaml), but this only sent the output from the sixth yaml.
That code looked like this:
for i, yaml_block in enumerate(ruamel.yaml.round_trip_load_all(stream=open(yaml_path))):
if i != 5:
ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)
else:
# parse the most recent git commit sha from command line
docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 --pretty=format:%h'.split()).decode()
# update go-bftx image with most recent git-commit-sha tag in the StatefulSet blocks
yaml_block['spec']['template']['spec']['containers'][1]['image'] = docker_image
ruamel.yaml.round_trip_dump(yaml_block, sys.stdout)
Any assistance would be greatly appreciated! Thank you!
Your file consists of multiple YAML documents, this is what you read in using round_trip_load_all, which gives you a generator.
If you write that back using round_trip_dump(), you are never going to get the --- YAML document separators that are in the original file.
You can probably use a generator and round_trip_dump_all since you know in which document to make changes, but it is probably easier to make a list of what round_trip_load_all generates and work on that. I would also use ruamel.yaml's new API to do so:
import sys
import ruamel.yaml
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = list(yaml.load_all(open('app.yaml')))
# parse the most recent git commit sha from command line
# docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()
docker_image = 'blockfreight/go-bftx:ci-cd-' + 'check_output_output'
# update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
data[-1]['spec']['template']['spec']['containers'][1]['image'] = docker_image
with open('out.yaml', 'w') as ofp:
yaml.dump_all(data, ofp)
The above mostly works. The out.yaml diff's with the change you want to make, and some whitespace standardisations (indents, between [ ") which are of course one off.
There are two major issues with this approach:
The first YAML document in the file has no content, so its comments cannot be hung from any mapping/sequence and are not preserved
The second YAML document has a final comment ( # // Initializes BFTX Service to interact with endpoints ) that gets dropped. That is most likely a bug in ruamel.yaml that shows up in multi-document files only and that I have to investigate
To solve the first issue, you can only read that document as normal lines without ruamel.yaml. When you do so, you might as well do all but the rest, as this solves (i.e. circumvents) the second issue as well:
import sys
import ruamel.yaml
with open('out.yaml', 'w') as ofp:
lines = ''
with open('app.yaml') as ifp:
for line in ifp:
lines += line
if line == '---\n':
ofp.write(lines)
lines = ''
# process lines from the last document
# print(lines)
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(lines)
# parse the most recent git commit sha from command line
# docker_image = 'blockfreight/go-bftx:ci-cd-' + check_output('git log -1 -- pretty=format:%h'.split()).decode()
docker_image = 'blockfreight/go-bftx:ci-cd-' + 'check_output_output'
# update go-bftx image with most recent git-commit-sha tag in the StatefulSet block
data['spec']['template']['spec']['containers'][1]['image'] = docker_image
yaml.dump(data, ofp)
This should do what you want unless you care about the leading trailing space in a flow style sequence, or preserve the three places with inconsistent indentation. The output of diff -u app.yaml out.yaml:
--- app.yaml 2018-06-23 14:41:02.256290577 +0200
+++ out.yaml 2018-06-23 14:58:09.933991459 +0200
## -143,7 +143,7 ##
spec:
selector:
matchLabels:
- app: bftx
+ app: bftx
serviceName: blockfreight
replicas: 1
template:
## -151,7 +151,7 ##
labels:
app: bftx
spec:
- containers:
+ containers:
- name: tm
imagePullPolicy: IfNotPresent
image: tendermint/tendermint:0.20.0
## -199,7 +199,7 ##
tendermint node --moniker="`hostname`" --p2p.seeds="aeabbf6b891435013f2a800fa9e22a1451ca90fd#bftx0.blockfreight.net:8888,6e9515c2cfed19464e6ce11ba2297ecdb411103b#bftx1.blockfreight.net:8888,b8b988370783bd0e58bf926d621a47160af2bdae#bftx2.blockfreight.net:8888,8c091f4e3dc4ac27db1efd38beee012d99967fd8#bftx3.blockfreight.net:8888" --proxy_app="tcp://localhost:46658" --consensus.create_empty_blocks=false
- name: app
imagePullPolicy: Always
- image: blockfreight/go-bftx:rc1
+ image: blockfreight/go-bftx:ci-cd-check_output_output
ports:
- containerPort: 12345
- containerPort: 46658
## -247,7 +247,7 ##
- mountPath: /etc/nginx/conf.d/pub_key.conf
name: tmconfigdir
subPath: pub_key_nginx.conf
- volumes:
+ volumes:
- name: tmconfigdir
configMap:
name: bftx-config
## -262,7 +262,7 ##
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
- accessModes: [ "ReadWriteOnce" ]
+ accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
## -271,7 +271,7 ##
annotations:
volume.alpha.kubernetes.io/storage-class: anything
spec:
- accessModes: [ "ReadWriteOnce" ]
+ accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 12Mi
I have the following YAML file:
heat_template_version: 2015-10-15
parameters:
image:
type: string
label: Image name or ID
default: CirrOS
private_network_id:
type: string
label: Private network name or ID
floating_ip:
type: string
I want to add key-> default to private_network_id and floating_ip (if default doesn't exist) and to the default key I want to add the value (which I get from user)
How can I achieve this in python?
The resulting YAML should look like:
heat_template_version: 2015-10-15
parameters:
image:
type: string
label: Image name or ID
default: CirrOS
private_network_id:
type: string
label: Private network name or ID
default: <private_network_id>
floating_ip:
type: string
default: <floating_ip>
For this kind of round-tripping you should do use ruamel.yaml (disclaimer: I am the author of the package).
Assuming your input is in a file input.yaml and the following program:
from ruamel.yaml import YAML
from pathlib import Path
yaml = YAML()
path = Path('input.yaml')
data = yaml.load(path)
parameters = data['parameters']
# replace assigned values with user input
parameters['private_network_id']['default'] = '<private_network_id>'
parameters['floating_ip']['default'] = '<floating_ip>'
yaml.dump(data, path)
After that your file will exact match the output you requested.
Please note that comments in the YAML file, as well as the key ordering are automatically preserved (not guaranteed by the YAML specification).
If you are still using Python2 (which has no pathlib in the standard library) use from ruamel.std.pathlib import Path or rewrite the .load() and .dump() lines with appropriately opened, old style, file objects. E.g.
with open('input.yaml', 'w') as fp:
yaml.dump(data, fp)