yaml params edit with python3 - python

I have a list of yaml files which I would like to edit in the same way. Put the following block under spec.template.imagePullSecret and spec.template.container.imagePullPolicy in each file:
imagePullSecrets:
- dockerpullsecret
container:
imagePullPolicy: IfNotPresent
This is an example of one file:
apiVersion: argoproj.io/v1alpha1
kind: Sensor
meta data:
name: sensor-finanda-ci
namespace: argo-events
spec:
template:
serviceAccountName: argo-events-sa
dependencies:
- name: eventsource-iv
eventSourceName: eventsource-iv
eventName: iv
This is my desired output:
apiVersion: argoproj.io/v1alpha1
kind: Sensor
meta data:
name: sensor-finanda-ci
namespace: argo-events
spec:
template:
serviceAccountName: argo-events-sa
imagePullSecrets:
- dockerpullsecret
container:
imagePullPolicy: IfNotPresent
dependencies:
- name: eventsource-iv
eventSourceName: eventsource-iv
eventName: iv


If your heart is set on a Python solution, something like this might do:
import os
import yaml
for yaml_filename in [filename for filename in os.listdir('.') if filename.endswith('.yaml')]:
with open(yaml_filename, 'r') as yaml_file:
yaml_obj = yaml.safe_load(yaml_file)
if 'spec' not in yaml_obj:
yaml_obj['spec'] = {"template": {}}
if 'template' not in yaml_obj['spec']:
yaml_obj['spec']['template'] = {"imagePullSecrets": [], "container": {}}
if 'container' not in yaml_obj['spec']['template']:
yaml_obj['spec']['template']['container'] = {}
yaml_obj['spec']['template']['imagePullSecrets'] = ["dockerpullsecret"]
yaml_obj['spec']['template']['container']['imagePullPolicy'] = 'IfNotPresent'
with open(yaml_filename, 'w') as yaml_file:
yaml.dump(yaml_obj, yaml_file)
If you're open to command-line utils, this is more concise:
Loop over the files and use yq to edit them in-place.
for file in *.yaml; do
yq -i e '.spec.template.imagePullSecrets = ["dockerpullsecret"] | .spec.template.container.imagePullPolicy = "IfNotPresent"' "$file"
done

Related

python iterate yaml and filter result

I have this yaml file
data:
- name: acme_aws1
source: aws
path: acme/acme_aws1.zip
- name: acme_gke1
source: gke
path: acme/acme_gke1.zip
- name: acme_oci
source: oci
path: acme/acme_oci1.zip
- name: acme_aws2
source: aws
path: acme/acme_aws2.zip
- name: acme_gke2
source: gke
path: acme/acme_gke2.zip
- name: acme_oci2
source: oci
path: acme/acme_oci2.zip
i want to filter out the data containing "source=gke" and for loop assign the value of path to variable., can any one please share how-to when using python with pyyaml as import module.

This code would do what you need, it just reads, and uses filter standard function to return an iterable with the elements passing a condition. Then such elements are put into a new list
import yaml
# for files you can use
# with open("data.yaml", "r") as file:
# yaml_data = yaml.safe_load(file)
yaml_data = yaml.safe_load("""
data:
- name: acme_aws1
source: aws
path: acme/acme_aws1.zip
- name: acme_gke1
source: gke
path: acme/acme_gke1.zip
- name: acme_oci
source: oci
path: acme/acme_oci1.zip
- name: acme_aws2
source: aws
path: acme/acme_aws2.zip
- name: acme_gke2
source: gke
path: acme/acme_gke2.zip
- name: acme_oci2
source: oci
path: acme/acme_oci2.zip
""")
data = yaml_data['data']
filtered = list(filter(lambda x: x.get('source') == 'gke', data))
print(filtered)
It prints
[{'name': 'acme_gke1', 'source': 'gke', 'path': 'acme/acme_gke1.zip'}, {'name': 'acme_gke2', 'source': 'gke', 'path': 'acme/acme_gke2.zip'}]

import yaml
# Read the file.
content = yaml.safe_load('your_file.yaml')
# Get rid of 'gke' elements.
not_gke_sources = [block for block in content if block.source != 'gke']
# Iterate over to access all 'path's.
for block in not_gke_sources:
path = block.path
# Some actions.

python creating string from yaml vars

i have the following yaml file:
rule_groups:
- name: "APPSTREAM"
allowed-domains:
- ".github.com"
- ".google.com"
source: "10.143.80.0/24"
- name: "TEST"
allowed-domains:
- ".microsoft.com"
- ".amazonaws.com"
source: "10.143.70.0/24"
i am calling inside a python script :
#!/usr/bin/env python3
import yaml
with open('settings.yaml', 'rb') as f:
config = yaml.safe_load(f)
core_config = config['rule_groups']
for workload in core_config:
print(workload)
{'name': 'APPSTREAM', 'allowed-domains': ['.github.com', '.google.com'], 'source': '10.143.80.0/24'}
{'name': 'TEST', 'allowed-domains': ['.microsoft.com', '.amazonaws.com'], 'source': '10.143.70.0/24'}
i am trying to create a dynamic string for every name , allowed-domains and sid and write the outpot into a file , as following:
$APPSTREAM ,.google.com, sid:1
$APPSTREAM ,.github.com, sid:2
$TEST, .microsoft.com, sid:3
$TEST, .amazonaws.com,sid:4
any help will be appreciated

solution
#!/usr/bin/env python3
import yaml
with open('settings.yaml', 'rb') as f:
config = yaml.safe_load(f)
core_config = config['rule_groups']
i = 0
for workload in core_config:
for domain in (workload['allowed-domains']):
i = i+1
print(f"${workload['name']},'{domain}',sid:{i}")

change a specific value in yaml file with lot of indentation using python

kind: Deployment
apiVersion: apps/v1
metadata:
name: websitemanager
namespace: white
selfLink: /apis/apps/v1/namespaces/white/deployments/websitemanager
generation: 89
labels:
app: websitemanager
app.kubernetes.io/instance: websitemanager
backup: kube-noah
annotations:
deployment.kubernetes.io/revision: '75'
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"deployment.kubernetes.io/revision":"75"},"labels":{"app":"websitemanager","app.kubernetes.io/instance":"websitemanager","backup":"kube-noah"},"name":"websitemanager","namespace":"white","selfLink":"/apis/apps/v1/namespaces/white/deployments/websitemanager"},"spec":{"progressDeadlineSeconds":600,"replicas":4,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app":"websitemanager"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"annotations":{"co.elastic.logs.json-logging/json.add_error_key":"true","co.elastic.logs.json-logging/json.keys_under_root":"true","co.elastic.logs.json-logging/json.message_key":"message_key","co.elastic.logs/enabled":"true"},"creationTimestamp":null,"labels":{"app":"websitemanager"}},"spec":{"containers":[{"env":[{"name":"APP","value":"websitemanager"},{"name":"PORT","value":"80"},{"name":"ALLOW_SKIPS","valueFrom":{"secretKeyRef":{"key":"ALLOW_SKIPS","name":"env-vars"}}},{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"env-vars"}}},{"name":"AWS_REGION","valueFrom":{"secretKeyRef":{"key":"AWS_REGION","name":"env-vars"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"env-vars"}}},{"name":"BO_RENDERER_URL","valueFrom":{"secretKeyRef":{"key":"BO_RENDERER_URL","name":"env-vars"}}},{"name":"BO_S3_URL","valueFrom":{"secretKeyRef":{"key":"BO_S3_URL","name":"env-vars"}}},{"name":"CHAT_V3","valueFrom":{"secretKeyRef":{"key":"CHAT_V3","name":"env-vars"}}},{"name":"CMS_RENDERER_URL","valueFrom":{"secretKeyRef":{"key":"CMS_RENDERER_URL","name":"env-vars"}}},{"name":"CRON_EXPRESSION","valueFrom":{"secretKeyRef":{"key":"CRON_EXPRESSION","name":"env-vars"}}},{"name":"CRON_GROUP_SCHEDULE_HOURS","valueFrom":{"secretKeyRef":{"key":"CRON_GROUP_SCHEDULE_HOURS","name":"env-vars"}}},{"name":"CRON_SCHEDULE_HOURS","valueFrom":{"secretKeyRef":{"key":"CRON_SCHEDULE_HOURS","name":"env-vars"}}},{"name":"DYNAMIC_PAGE_URL","valueFrom":{"secretKeyRef":{"key":"DYNAMIC_PAGE_URL","name":"env-vars"}}},{"name":"ENVIRONMENT_BUCKET_ENDPOINT","valueFrom":{"secretKeyRef":{"key":"ENVIRONMENT_BUCKET_ENDPOINT","name":"env-vars"}}},{"name":"ENVIRONMENT_BUCKET_NAME","valueFrom":{"secretKeyRef":{"key":"ENVIRONMENT_BUCKET_NAME","name":"env-vars"}}},{"name":"FIREBASE_KEY","valueFrom":{"secretKeyRef":{"key":"FIREBASE_KEY","name":"env-vars"}}},{"name":"FO_RENDERER_URL","valueFrom":{"secretKeyRef":{"key":"FO_RENDERER_URL","name":"env-vars"}}},{"name":"FO_S3_URL","valueFrom":{"secretKeyRef":{"key":"FO_S3_URL","name":"env-vars"}}},{"name":"GEFEN_ENV","valueFrom":{"secretKeyRef":{"key":"GEFEN_ENV","name":"env-vars"}}},{"name":"LEADS_SQS_DL_QUEUE","valueFrom":{"secretKeyRef":{"key":"LEADS_SQS_DL_QUEUE","name":"env-vars"}}},{"name":"LEADS_SQS_QUEUE","valueFrom":{"secretKeyRef":{"key":"LEADS_SQS_QUEUE","name":"env-vars"}}},{"name":"LOGGLY_INPUT_TOKEN","valueFrom":{"secretKeyRef":{"key":"LOGGLY_INPUT_TOKEN","name":"env-vars"}}},{"name":"LOGGLY_SUBDOMAIN","valueFrom":{"secretKeyRef":{"key":"LOGGLY_SUBDOMAIN","name":"env-vars"}}},{"name":"LOKALISE_API_KEY","valueFrom":{"secretKeyRef":{"key":"LOKALISE_API_KEY","name":"env-vars"}}},{"name":"LOKALISE_PROJECT_ID","valueFrom":{"secretKeyRef":{"key":"LOKALISE_PROJECT_ID","name":"env-vars"}}},{"name":"LP_RENDERER_URL","valueFrom":{"secretKeyRef":{"key":"LP_RENDERER_URL","name":"env-vars"}}},{"name":"MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"env-vars"}}},{"name":"MONGO_URLS","valueFrom":{"secretKeyRef":{"key":"MONGO_URLS","name":"env-vars"}}},{"name":"NEW_RELIC_LICENSE_KEY","valueFrom":{"secretKeyRef":{"key":"NEW_RELIC_LICENSE_KEY","name":"env-vars"}}},{"name":"NEXTGEN_S3_URL","valueFrom":{"secretKeyRef":{"key":"NEXTGEN_S3_URL","name":"env-vars"}}},{"name":"NODE_ENV","valueFrom":{"secretKeyRef":{"key":"NODE_ENV","name":"env-vars"}}},{"name":"OPERATION_BUCKET_ENV_PART","valueFrom":{"secretKeyRef":{"key":"OPERATION_BUCKET_ENV_PART","name":"env-vars"}}},{"name":"PG_URL","valueFrom":{"secretKeyRef":{"key":"PG_URL","name":"env-vars"}}},{"name":"QUICKSIGHT_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"QUICKSIGHT_ACCESS_KEY_ID","name":"env-vars"}}},{"name":"QUICKSIGHT_ACCOUNT_ID","valueFrom":{"secretKeyRef":{"key":"QUICKSIGHT_ACCOUNT_ID","name":"env-vars"}}},{"name":"QUICKSIGHT_SECRET_KEY_ID","valueFrom":{"secretKeyRef":{"key":"QUICKSIGHT_SECRET_KEY_ID","name":"env-vars"}}},{"name":"REDIS_URL","valueFrom":{"secretKeyRef":{"key":"REDIS_URL","name":"env-vars"}}},{"name":"ROLLBAR_API_TOKEN","valueFrom":{"secretKeyRef":{"key":"ROLLBAR_API_TOKEN","name":"env-vars"}}},{"name":"RS_URL","valueFrom":{"secretKeyRef":{"key":"RS_URL","name":"env-vars"}}},{"name":"SES_PASS","valueFrom":{"secretKeyRef":{"key":"SES_PASS","name":"env-vars"}}},{"name":"SES_USER","valueFrom":{"secretKeyRef":{"key":"SES_USER","name":"env-vars"}}},{"name":"STELLAR_REDIS_URL","valueFrom":{"secretKeyRef":{"key":"STELLAR_REDIS_URL","name":"env-vars"}}},{"name":"USE_HTTP_SERVER","valueFrom":{"secretKeyRef":{"key":"USE_HTTP_SERVER","name":"env-vars"}}},{"name":"USE_LEADS_V3","valueFrom":{"secretKeyRef":{"key":"USE_LEADS_V3","name":"env-vars"}}},{"name":"androidBuildName","valueFrom":{"secretKeyRef":{"key":"androidBuildName","name":"env-vars"}}},{"name":"iosAppStoreId","valueFrom":{"secretKeyRef":{"key":"iosAppStoreId","name":"env-vars"}}},{"name":"iosBuildName","valueFrom":{"secretKeyRef":{"key":"iosBuildName","name":"env-vars"}}}],"image":"gefenonline/websitemanager:develop-111","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":80,"scheme":"HTTP"},"initialDelaySeconds":5,"periodSeconds":3,"successThreshold":1,"timeoutSeconds":1},"name":"websitemanager","resources":{"limits":{"memory":"700Mi"},"requests":{"cpu":"150m","memory":"400Mi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","imagePullSecrets":[{"name":"docker-registry"}],"restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}}},"status":null}
spec:
replicas: 4
selector:
matchLabels:
app: websitemanager
template:
metadata:
creationTimestamp: null
labels:
app: websitemanager
annotations:
co.elastic.logs.json-logging/json.add_error_key: 'true'
co.elastic.logs.json-logging/json.keys_under_root: 'true'
co.elastic.logs.json-logging/json.message_key: message_key
co.elastic.logs/enabled: 'true'
spec:
containers:
- name: websitemanager
image: 'gefenonline/websitemanager:develop-111'
resources:
limits:
memory: 700Mi
requests:
cpu: 150m
memory: 400Mi
livenessProbe:
httpGet:
path: /health
port: 80
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 3
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
imagePullSecrets:
- name: docker-registry
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
status:
observedGeneration: 89
replicas: 4
updatedReplicas: 4
readyReplicas: 4
availableReplicas: 4
conditions:
- type: Progressing
status: 'True'
lastUpdateTime: '2021-08-03T06:34:01Z'
lastTransitionTime: '2021-08-03T06:33:40Z'
reason: NewReplicaSetAvailable
message: ReplicaSet "websitemanager-7cdbcf6488" has successfully progressed.
- type: Available
status: 'True'
lastUpdateTime: '2021-08-04T13:57:07Z'
lastTransitionTime: '2021-08-04T13:57:07Z'
reason: MinimumReplicasAvailable
message: Deployment has minimum availability.
I have a yaml file and i want to make change in the image key in the file which is under spec-containers by a python script.so far i have got this script but not able to access the key-value-pair and i am quite new to this so can anybody help me on this.i am not able to change the key-value pair in the deployments.yaml file
import yaml
with open('deployment.yaml', 'w') as f:
content = yaml.load(f)
# for k,v in content.items():
# print(k['selector'])
print(content['spec']['template']['spec']['containers'])
#doc['spec']['spec']['template']['spec']['containers']['image'] = 'letsencrypt-prod'
# for k,v in (content['spec']['template']['spec']['containers']):
# print(v)
yaml.dump(content, f)
# with open("deployment.yaml", "w") as f:
# yaml.dump(content, f)

You were just missing that the level at key "containers" is a list, so the zeroth index must be used to get to the image key:
import yaml
with open('deployment.yaml', 'r') as fin:
content = yaml.load(fin, Loader=yaml.FullLoader)
content['spec']['template']['spec']['containers'][0]['image'] = "letsencrypt-prod"
with open('deployment2.yaml', 'w') as fout:
yaml.dump(content, fout)

Editing YAML file by Python and ruamel.yaml module

I need to edit this yaml file changing cpu and memory variable. I'm a begginer python developer.
I'm using python 3, and ruamel.yaml with sys module.
The yaml to change:
- apiVersion: v1
kind: ResourceQuota
metadata:
annotations:
openshift.io/quota-tier: Medium
creationTimestamp:
labels:
quota-tier: Medium
name: burst-quota
namespace: testing
resourceVersion: ""
selfLink: /api/v1/namespaces/testing/resourcequotas/burst-quota
uid:
spec:
hard:
cpu: "8"
memory: 16Gi
status:
hard:
cpu: "8"
memory: 16Gi
used:
cpu: 20m
memory: 256Mi
Here is my code:
import sys
import ruamel.yaml
yaml = ruamel.yaml.YAML()
with open('quota-edit.yaml') as fp:
data = yaml.load(fp)
for elem in data:
if elem['kind'] == 'ResourceQuota':
elem['spec'] = None
elem['hard'] = None
elem['cpu'] = 123
break
yaml.dump(data, sys.stdout)
The output of my code is:
- apiVersion: v1
kind: ResourceQuota
metadata:
annotations:
openshift.io/quota-tier: Medium
creationTimestamp:
labels:
quota-tier: Medium
name: burst-quota
namespace: sre
resourceVersion: ""
selfLink: /api/v1/namespaces/testing/resourcequotas/burst-quota
uid:
spec:
hard:
cpu: '8'
memory: 16Gi
status:
hard:
cpu: '8'
memory: 16Gi
used:
cpu: 20m
memory: 256Mi
cpu: 123
What am I doing wrong?
Thanks,
Regards!

YAML is a hierarchical structure. When you read it, that hierarchy is preserved. Your code accesses a 2nd level element (kind), and that works as you expect. But the elements you want to change are on deeper nesting levels. Your code tries to modify them at the wrong level. Don't modify child elements of elem, but go all the way down the hierarchy:
import sys
import ruamel.yaml
yaml = ruamel.yaml.YAML()
with open('quota-edit.yaml') as fp:
data = yaml.load(fp)
for elem in data:
if elem['kind'] == 'ResourceQuota':
elem['spec']['hard']['cpu'] = 123
break
yaml.dump(data, sys.stdout)
This replaces the CPU item in spec.hard.cpu. Adjust for the other cpu entries as needed.

how to store register value as dict in ansible?

I'm trying to convert a string
From "{ip: 10.213.151.76, mask: 255.255.252.0},{ip: 10.213.151.799, mask: 255.255.252.0}"
to [{ip: 10.213.151.76, mask: 255.255.252.0}, {ip: 10.213.151.76, mask: 255.255.252.0}].
Playbook code
- hosts: localhost
vars:
- vari: "[{ip: 10.213.151.76, mask: 255.255.252.0},{ip: 10.213.151.799, mask: 255.255.252.0}]"
- foo: []
tasks:
- set_fact: testing={{vari[1:-1] | regex_findall('\{(.*?)\}')}}
# - set_fact: testing={{vari[1:-1]}}
- debug: var=testing
- name: run my script!
command: python ../test.py "{{item}}"
delegate_to: 127.0.0.1
register: hash
with_items: "{{testing}}"
- debug: var=hash
- set_fact: foo={{foo + item.stdout_lines}}
with_items: "{{hash.results}}"
- debug: var=foo
pyhon script which converts a string to dictionary.
#!/usr/bin/python
import sys
import json
ip_str = str(sys.argv[1])
print dict(s.split(': ') for s in (ip_str.split(', ')))
Current foo variable value coming like this.
ok: [localhost] => {
"foo": [
"{'ip': '10.213.151.76', 'mask': '255.255.252.0'}",
"{'ip': '10.213.151.799', 'mask': '255.255.252.0'}"
]
}
Basically, i want this value in the list of hash format: [{ip: 10.213.151.76, mask: 255.255.252.0}, {ip: 10.213.151.76, mask: 255.255.252.0}].
Python script return value as a dictionary but register store it as a string. ansible not able to convert it as a dictonary.
Any help? Thanks in Advance.

For the record, if it's this complex, I think you should consider redesigning. However...
You can write a variables file in one play, and read it in the next.
- name: dynamic vars file write test
hosts: localhost
gather_facts: no
tasks:
- name: write it
shell: ' echo "foo: bar">tst.yml '
- name: dynamic vars file read test
hosts: localhost
gather_facts: no
vars_files:
- tst.yml
tasks:
- name: read it
debug:
msg: "{{foo}}"
Which outputs:
PLAY [dynamic vars file write test]
********************************************
TASK [write it]
********************************************
changed: [localhost]
PLAY [dynamic vars file read test]
********************************************
TASK [read it]
********************************************
ok: [localhost] => {
"changed": false,
"msg": "bar"
}
This makes it a heck of a lot easier to format and parse, at least for me.

Categories