We Keep Coding

GitHub secrets created using API REST but returned empty

I created GitHub secrets using the GitHub REST API as it's given in the folowing documentation: https://docs.github.com/en/rest/actions/secrets
The code that I used to crypt and then create my list of GitHub environnment secrets is te following :
from github import Github
import requests
access_token = base["TOKEN"]
user = gitLogin["user"]
api_url = f"https://api.github.com/users/"+user+"/repos"
response = requests.get(api_url, auth=(user, access_token))
print(response)
dev_secrets_names = [...]
dev_secrets_list = [...]
dev_encrypted_secrets_list = []
def encrypt(public_key: str, secret_value: str) -> str:
"""Encrypt a Unicode string using the public key."""
public_key = public.PublicKey(public_key.encode("utf-8"), encoding.Base64Encoder())
sealed_box = public.SealedBox(public_key)
encrypted = sealed_box.encrypt(secret_value.encode("utf-8"))
return b64encode(encrypted).decode("utf-8")
for i in range(len(dev_secrets_list)):
encrypted_secret = encrypt(public_key, dev_secrets_list[i])
dev_encrypted_secrets_list.append(encrypted_secret)
for i in range(len(dev_secrets_list)):
url = f"https://api.github.com/repositories/"+"${{ steps.repos.outputs.repos_id }}"+"/environments/"+environnments["dev"]+"/secrets/"+dev_secrets_names[i]
print(url)
body = {"encrypted_value": f"{dev_encrypted_secrets_list[i]}", "key_id": "${{ steps.keys.outputs.key_id }}"}
response = requests.put(url, json=body, auth=(user, access_token))
The code executes correctly and when I go to check the secrets are well created in GitHub. Only, when I try to retrieve the secrets in a task, they are not read as if they were empty.
The folowing code is where I'm trying to use the secrets :
on:
workflow_call:
inputs:
Organization:
required: true
type: string
Repository:
required: true
type: string
devEnv:
required: true
type: string
uatEnv:
required: true
type: string
prodEnv:
required: true
type: string
devBranch:
required: true
type: string
uatBranch:
required: true
type: string
prodBranch:
required: true
type: string
releaseBranch:
required: true
type: string
rootFolder:
required: true
type: string
secrets:
DEV_SF_ACCOUNT:
required: true
DEV_SF_USERNAME:
required: true
DEV_SNOWFLAKE_PASSWORD:
required: true
DEV_SF_ROLE:
required: true
DEV_SF_WAREHOUSE:
required: true
deploy-snowflake-changes-dev:
name: deploy schamas changes to dev
needs: ShitTest
if: needs.ShitTest.outputs.output == 'true'
environment:
name: ${{inputs.devEnv}}
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout#v2
- name: Use Python 3.8.x
uses: actions/setup-python#v2.2.1
with:
python-version: 3.8.x
- name: Run schemachange
shell: pwsh
run: |
python --version
echo "Step 1: Installing schemachange"
pip install schemachange
echo "Step 3: Running schemachange"
schemachange deploy -f ./${{inputs.rootFolder}} -a ${{secrets.DEV_SF_ACCOUNT}} -u ${{secrets.DEV_SF_USERNAME}} -r ${{secrets.DEV_SF_ROLE}} -w ${{secrets.DEV_SF_WAREHOUSE}} -d DEV_${{secrets.SF_DATABASE}} -c DEV_${{secrets.SF_DATABASE}}.${{secrets.SF_SCHEMA}}.${{secrets.SF_HISTORY_TABLE}} --vars $varsString --create-change-history-table -v
env:
SNOWFLAKE_PASSWORD: ${{ secrets.DEV_SNOWFLAKE_PASSWORD }}
I voluntarily removed some part of the code to keep it simple.
Can you help to understand why secrets are returned as empty ?
note: when I update the secrets manually, everything works.
The following code is the one from witch I call the previous workflow :
jobs:
snowflake-devops:
uses: ./.github/workflows/snowflake-devops.yml
with:
Organization: $($parametersFileContent.organization)
Repository: $($parametersFileContent.repository)
devEnv: $($parametersFileContent.environnments.dev)
uatEnv: $($parametersFileContent.environnments.uat)
prodEnv: $($parametersFileContent.environnments.prod)
devBranch: ${{ env.dev }}
uatBranch: ${{ env.uat }}
prodBranch: ${{ env.prod }}
releaseBranch: ${{ env.release }}
rootFolder: $($parametersFileContent.rootFolder)
secrets:
TOKEN: $TOKEN
SF_DATABASE: $SF_DATABASE
SF_SCHEMA: $SF_SCHEMA
SF_HISTORY_TABLE: $SF_HISTORY_TABLE
DEV_SF_ACCOUNT: $DEV_SF_ACCOUNT
DEV_SF_USERNAME: $DEV_SF_USERNAME
DEV_SF_ROLE: $DEV_SF_ROLE
DEV_SF_WAREHOUSE: $DEV_SF_WAREHOUSE
DEV_SNOWFLAKE_PASSWORD: $DEV_SNOWFLAKE_PASSWORD

I don't think this has anything to do with secret creation, based on your caller workflow (the workflow at the bottom of the post). It appears you are not actually referencing the secrets correctly to pass them into the reusable workflow (snowflake-devops.yml).
The proper syntax would be:
jobs:
snowflake-devops:
uses: ./.github/workflows/snowflake-devops.yml
with:
foo: bar
secrets:
TOKEN: ${{ secrets.TOKEN }}
foo: ${{ secrets.bar }}
But, to save yourself more headaches, try using secrets: inherit. ref.
e.g.
jobs:
snowflake-devops:
uses: ./.github/workflows/snowflake-devops.yml
with:
Organization: $($parametersFileContent.organization)
Repository: $($parametersFileContent.repository)
devEnv: $($parametersFileContent.environnments.dev)
uatEnv: $($parametersFileContent.environnments.uat)
prodEnv: $($parametersFileContent.environnments.prod)
devBranch: ${{ env.dev }}
uatBranch: ${{ env.uat }}
prodBranch: ${{ env.prod }}
releaseBranch: ${{ env.release }}
rootFolder: $($parametersFileContent.rootFolder)
secrets: inherit

How to use python output as input for next step in argo workflow?

Based on the last line of the output of a python script (I cannot adapt the output-format) I want to trigger multiple new steps in argo-wf. How can I ignore all output lines except the last one in below example? I cannot adapt thy python-code so I guess I have to include an additional step to exclude all lines except last one.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: loops-param-result-
spec:
entrypoint: loop-param-result-example
templates:
- name: loop-param-result-example
steps:
- - name: generate
template: gen-number-list
- - name: sleep
template: sleep-n-sec
arguments:
parameters:
- name: seconds
value: "{{item}}"
withParam: "{{steps.generate.outputs.result}}"
- name: gen-number-list
script:
image: python:alpine3.6
command: [python]
source: |
import json
import sys
print("abc")
print("def")
print("1, 2, 3")
- name: sleep-n-sec
inputs:
parameters:
- name: seconds
container:
image: alpine:latest
command: [sh, -c]
args: ["echo sleeping for {{inputs.parameters.seconds}} seconds; sleep {{inputs.parameters.seconds}}; echo done"]
As I am not able to adapt the python-script I cannot use the standard-solution as shown below:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: loops-param-result-
spec:
entrypoint: loop-param-result-example
templates:
- name: loop-param-result-example
steps:
- - name: generate
template: gen-number-list
- - name: sleep
template: sleep-n-sec
arguments:
parameters:
- name: seconds
value: "{{item}}"
withParam: "{{steps.generate.outputs.result}}"
- name: gen-number-list
script:
image: python:alpine3.6
command: [python]
source: |
import json
import sys
json.dump([i for i in range(20, 31)], sys.stdout)
- name: sleep-n-sec
inputs:
parameters:
- name: seconds
container:
image: alpine:latest
command: [sh, -c]
args: ["echo sleeping for {{inputs.parameters.seconds}} seconds; sleep {{inputs.parameters.seconds}}; echo done"]

I never got the
command: [ python ]
source |
import json
to work, however, this works for me:
command: [ python, -c ]
args: &script
- |
import json

Write multi-line secret to windows runner in GitHub workflow

Summary
What specific syntax must be changed in the code below in order for the multi-line contents of the $MY_SECRETS environment variable to be 1.) successfully written into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file on a Windows runner in the GitHub workflow whose code is given below, and 2.) read by the simple Python 3 main.py program given below?
PROBLEM DEFINITION:
The echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml command is only printing the string literal MY_SECRETS into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file instead of printing the multi-line contents of the MY_SECRETS variable.
We confirmed that this same echo command does successfully print the same multi-line secret in an ubuntu-latest runner, and we manually validated the correct contents of the secrets.LIST_OF_SECRETS environment variable. ... This problem seems entirely isolated to either the windows command syntax, or perhaps to the windows configuration of the GitHub windows-latest runner, either of which should be fixable by changing the workflow code below.
EXPECTED RESULT:
The multi-line secret should be printed into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file and read by main.py.
The resulting printout of the contents of the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file should look like:
***
***
***
***
LOGS THAT DEMONSTRATE THE FAILURE:
The result of running main.py in the GitHub Actions log is:
ccc item is: $MY_SECRETS
As you can see, the string literal $MY_SECRETS is being wrongly printed out instead of the 4 *** secret lines.
REPO FILE STRUCTURE:
Reproducing this error requires only 2 files in a repo file structure as follows:
.github/
workflows/
test.yml
main.py
WORKFLOW CODE:
The minimal code for the workflow to reproduce this problem is as follows:
name: write-secrets-to-file
on:
push:
branches:
- dev
jobs:
write-the-secrets-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout#v3
- shell: python
name: Configure agent
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
import subprocess
import pathlib
pathlib.Path("C:\\Users\\runneradmin\\somedir\\").mkdir(parents=True, exist_ok=True)
print('About to: echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')
output = subprocess.getoutput('echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')
print(output)
os.chdir('D:\\a\\myRepoName\\')
mycmd = "python myRepoName\\main.py"
p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
# returns None while subprocess is running
retcode = p.poll()
line = p.stdout.readline()
print(line)
if retcode is not None:
break
MINIMAL APP CODE:
Then the minimal main.py program that demonstrates what was actually written into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file is:
with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml') as file:
for item in file:
print('ccc item is: ', str(item))
if "var1" in item:
print("Found var1")
STRUCTURE OF MULTI-LINE SECRET:
The structure of the multi-line secret contained in the secrets.LIST_OF_SECRETS environment variable is:
var1:value1
var2:value2
var3:value3
var4:value4
These 4 lines should be what gets printed out when main.py is run by the workflow, though the print for each line should look like *** because each line is a secret.

The problem is - as it is so often - the quirks of Python with byte arrays and strings and en- and de-coding them in the right places...
Here is what I used:
test.yml:
name: write-secrets-to-file
on:
push:
branches:
- dev
jobs:
write-the-secrets-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout#v3
- shell: python
name: Configure agent
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
import subprocess
import pathlib
import os
# using os.path.expanduser() instead of hard-coding the user's home directory
pathlib.Path(os.path.expanduser("~/somedir")).mkdir(parents=True, exist_ok=True)
secrets = os.getenv("MY_SECRETS")
with open(os.path.expanduser("~/somedir/mykeys.yaml"),"w",encoding="UTF-8") as file:
file.write(secrets)
mycmd = ["python","./main.py"]
p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
# returns None while subprocess is running
retcode = p.poll()
line = p.stdout.readline()
# If len(line)==0 we are at EOF and do not need to print this line.
# An empty line from main.py would be '\n' with len('\n')==1!
if len(line)>0:
# We decode the byte array to a string and strip the
# new-line characters \r and \n from the end of the line,
# which were read from stdout of main.py
print(line.decode('UTF-8').rstrip('\r\n'))
if retcode is not None:
break
main.py:
import os
# using os.path.expanduser instead of hard-coding user home directory
with open(os.path.expanduser('~/somedir/mykeys.yaml'),encoding='UTF-8') as file:
for item in file:
# strip the new-line characters \r and \n from the end of the line
item=item.rstrip('\r\n')
print('ccc item is: ', str(item))
if "var1" in item:
print("Found var1")
secrets.LIST_OF_SECRETS:
var1: secret1
var2: secret2
var3: secret3
var4: secret4
And my output in the log was
ccc item is: ***
Found var1
ccc item is: ***
ccc item is: ***
ccc item is: ***

Edit: updated with fixed main.py and how to run it.
You can write the key file directly with Python:
- shell: python
name: Configure agent
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
import os
import pathlib
pathlib.Path('C:\\Users\\runneradmin\\somedir\\').mkdir(parents=True, exist_ok=True)
with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml', 'w') as key_file:
key_file.write(os.environ['MY_SECRETS'])
- uses: actions/checkout#v3
- name: Run main
run: python main.py
To avoid newline characters in your output, you need a main.py that removes the newlines (here with .strip().splitlines()):
main.py
with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml') as file:
for item in file.read().strip().splitlines():
print('ccc item is: ', str(item))
if "var1" in item:
print("Found var1")
Here's the input:
LIST_OF_SECRETS = '
key:value
key2:value
key3:value
'
And the output:
ccc item is: ***
Found var1
ccc item is: ***
ccc item is: ***
ccc item is: ***
Here is my complete workflow:
name: write-secrets-to-file
on:
push:
branches:
- master
jobs:
write-the-secrets-windows:
runs-on: windows-latest
steps:
- shell: python
name: Configure agent
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
import os
import pathlib
pathlib.Path('C:\\Users\\runneradmin\\somedir\\').mkdir(parents=True, exist_ok=True)
with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml', 'w') as key_file:
key_file.write(os.environ['MY_SECRETS'])
- uses: actions/checkout#v3
- name: Run main
run: python main.py
Also, a simpler version using only Windows shell (Powershell):
- name: Create key file
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
mkdir C:\\Users\\runneradmin\\somedir
echo "$env:MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml
- uses: actions/checkout#v3
- name: Run main
run: python main.py

I tried the following code and it worked fine :
LIST_OF_SECRETS
key1:val1
key2:val2
Github action (test.yml)
name: write-secrets-to-file
on:
push:
branches:
- main
jobs:
write-the-secrets-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout#v3
- shell: python
name: Configure agentt
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
import base64, subprocess, sys
import os
secrets = os.environ["MY_SECRETS"]
def powershell(cmd, input=None):
cmd64 = base64.encodebytes(cmd.encode('utf-16-le')).decode('ascii').strip()
stdin = None if input is None else subprocess.PIPE
process = subprocess.Popen(["powershell.exe", "-NonInteractive", "-EncodedCommand", cmd64], stdin=stdin, stdout=subprocess.PIPE)
if input is not None:
input = input.encode(sys.stdout.encoding)
output, stderr = process.communicate(input)
output = output.decode(sys.stdout.encoding).replace('\r\n', '\n')
return output
command = r"""$secrets = #'
{}
'#
$secrets | Out-File -FilePath .\mykeys.yaml""".format(secrets)
command1 = r"""Get-Content -Path .\mykeys.yaml"""
powershell(command)
print(powershell(command1))
Output
***
***
As you also mention in the question, Github will obfuscate any printed value containing the secrets with ***
EDIT : Updated the code to work with multiple line secrets. This answer was highly influenced by this one

You need to use yaml library:
import yaml
data = {'MY_SECRETS':'''
var1:value1
var2:value2
var3:value3
var4:value4
'''}#add your secret
with open('file.yaml', 'w') as outfile: # Your file
yaml.dump(data, outfile, default_flow_style=False)
This is result:
I used this.

How to run ansible from a python script

I am trying to run ansible playbooks within a python script used as a part of a flask server.
However, when trying to construct the playbook with python, it doesn't really work out the way its intended.
The original playbook should be:
---
- hosts: dc
gather_facts: no
tasks:
- name: create user omtest
win_domain_user:
name: omtest
firstname: om
surname: test
password: sjsd
state: present
attributes:
telephoneNumber: 123456
path: ou=Hilv,ou="Users",dc=ic,dc=com
password_expired: yes
groups:
- "_AV_NL"
and the python code for it:
play_ovirt = dict(
name="win-domain",
hosts=host_list,
gather_facts='no',
tasks=[
dict(action=dict(module='win_domain_user', args='dict(name='omtest', firstname='om', 'surname='test', password="sjsd', state='present', attributes='dict(telephoneNumber='123456')', path='ou=Hilv,ou="Users",dc=ic,dc=com', groups='['_AV_NL']', password_expired="yes"), register='shell_out')
]
)
I know there is something wrong with the python code but can't identify what is it.
Also this is my first time using ansible as part of a python code.

How to log in molecule?

I am using Molecule to test an Ansible role. I've written unit tests in python, and have been unable to print out variables or log anything to stdout. Here is my code
import os
import testinfra.utils.ansible_runner
import logging
LOG = logging.getLogger("toto")
testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')
def test_something(host):
print 'Hello World!'
LOG.warn('Hello World!')
assert True

Enable the s flag for testinfra inside molecule.yml:
---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
platforms:
- name: instance
image: centos:7
provisioner:
name: ansible
lint:
name: ansible-lint
scenario:
name: default
verifier:
name: testinfra
lint:
name: flake8
options:
s: true