We Keep Coding

Python boto3 how to parse role_arn from AWS_CONFIG_FILE?

I have an AWS config file that my boto3 session has access to, via the AWS_CONFIG_FILE environment variable.
The config file looks like this: (multi-account environment)
[profile profile1]
credential_source Environment
region=us-east-whatever
role_arn=arn:aws:iam:<ACCOUNT NUMBER 1>:role/all-profiles-same-role-name
[profile profile2]
credential_source Environment
region=us-east-whatever
role_arn=arn:aws:iam:<ACCOUNT NUMBER 2>:role/all-profiles-same-role-name
[profile profileN]
credential_source Environment
region=us-east-whatever
role_arn=arn:aws:iam:<ACCOUNT NUMBER N>:role/all-profiles-same-role-name
In my Python code, I am trying to setup RefreshableCredentails (boto3 method) using somethign like this: (excluding full code because I think the issue is mainly about parsing the aws_config_file):
def __get_session_credentials(self):
# hardcode one role_arn for now but need to variablize
session_ttl=3000
aws_role_arn="arn:aws:iam::<ACCOUNT NUM>:role/all-profiles-same-role-name
...
Can I somehow parse the "role_arn" from the config file on a per profile basis to make that function more extensible? How would I do that?

You can use the configparser module from the standard library:
import configparser
from pathlib import Path
def main():
path_to_config = Path(Path.home(), ".aws", "config")
parser = configparser.ConfigParser()
parser.read(path_to_config)
for profile in parser.sections():
if "role_arn" in parser[profile]:
print(
"Found profile", profile, "with role_arn", parser[profile]["role_arn"]
)
if __name__ == "__main__":
main()
I'm not going to share the output here, though ;-)

I think that your problem could be solved if you configure a new environment variable AWS_PROFILE
According to boto3 docs[1]:
AWS_PROFILE
The default profile to use, if any. If no value is specified, Boto3 attempts to search the shared credentials file and the config file for the default profile.
And aws-cli (just for reference) docs[2]:
AWS_PROFILE
Specifies the name of the AWS CLI profile with the credentials and options to use. This can be the name of a profile stored in a credentials or config file, or the value default to use the default profile.
If defined, this environment variable overrides the behavior of using the profile named [default] in the configuration file. You can override this environment variable by using the --profile command line parameter.
So, just set this environment variable to the profileN

Name error when trying to import API key from .env

I am trying to store my API Keys in a .env file
I created the file as a File containing settings for editor file type. Stored my APIKeys
TWILIO_ACCOUNT_SID=***
TWILIO_AUTH_TOKEN=***
TWIML_APPLICATION_SID=***
TWILIO_API_KEY=***
TWILIO_API_SECRET=***
Installed decouple, imported and used config to retrieve my API tokens in my settings.py file
from decouple import config
...
TWILIO_ACCOUNT_SID = config(TWILIO_ACCOUNT_SID)
TWILIO_AUTH_TOKEN = config(TWILIO_AUTH_TOKEN)
TWIML_APPLICATION_SID = config(TWIML_APPLICATION_SID)
TWILIO_API_KEY = config(TWILIO_API_KEY)
TWILIO_API_SECRET = config(TWILIO_API_SECRET)
I am however getting the error message:
TWILIO_ACCOUNT_SID = config(TWILIO_ACCOUNT_SID)
NameError: name 'TWILIO_ACCOUNT_SID' is not defined

You don't need to use the decouple library to read your environment variables.
Firstly download the .env plug-in supporter for PyCharm (if that's what you're using)
https://www.codestudyblog.com/cs2112pyc/1224021812.html
This will allow you to set and get variables from your file. Make sure your configuration has the correct .env file set.
my .env file has the variable set to:
TWILIO_ACCOUNT_SID=SUPER SECRET KEY
Then all you need to is:
import os
twilio_key = os.environ.get('TWILIO_ACCOUNT_SID')
print(twilio_key)
>>>SUPER SECRET KEY
Process finished with exit code 0

django-environ - Managing LDAP DN in .env

I have a django web app, with a middleware that intercepts incoming requests, extracts user details (added to header by upstream middleware) in the request header, and checks if user is authorized to access the page if user is member of a distribution group.
I'm using django-environ to manage my environment variables so i can modify the list of DL Groups which can access my page without changing the code.
# in Middleware, only included important part of code
from django.conf import settings
MINIMAL_MEMBERSHIP = settings.AUTHORIZED_MEMBERSHIP_REQUIREMENT # This should pass in as a list
server_object = Server(LDAP_SERVER)
conn = Connection(server_object, LDAP_USER, LDAP_PASS, client_strategy=SAFE_SYNC, auto_bind=True)
status, result, response, _ = conn.search(
search_base=requester_dn,
search_filter = '(objectClass=User)',
attributes = ['memberOf']
)
authorized = False
requester_membership_list = response[0]['raw_attributes']['memberOf']
for membership in requester_membership_list:
ad_group_name = membership.decode('utf-8')
if ad_group_name in MINIMAL_MEMBERSHIP:
authorized = True
break
# In settings.py
AUTHORIZED_MEMBERSHIP_REQUIREMENT = env.list('AUTHORIZED_MEMBERSHIP_REQUIREMENT')
# In .env
AUTHORIZED_MEMBERSHIP_REQUIREMENT="CN=Virtualisation team,OU=Distribution Group,OU=Exchange,OU=APPS,DC=xxx,DC=xxx,DC=xxx,DC=com",
According to django-environ, you can read .env as a list like
# .env
LIST_ENV=one,two,three,four
# settings.py
LIST_ENV=env.list(LIST_ENV)
print(LIST_ENV) # outputs ['one', 'two', 'three', 'four']
But understandably ldap DN format will break this as a full DN is already delimited by commas, so:
# .env
DN_LIST="CN=1,OU=1,OU=1,OU=1,DC=xxx,DC=xxx,DC=xxx,DC=com","CN=2,OU=2,OU=2,OU=2,DC=xxx,DC=xxx,DC=xxx,DC=com"
# settings.py
DN_LIST=env.list(DN_LIST)
# Actual Behavior
print(DN_LIST)
# DN_LIST = ['CN=1', 'OU=1', 'OU=1', ...]
# len(DN_LIST) will output 16
I would like achieve this:
# Behavior i want
print(DN_LIST)
# DN_LIST = ["CN=1,OU=1,OU=1,OU=1,DC=xxx,DC=xxx,DC=xxx,DC=com","CN=2,OU=2,OU=2,OU=2,DC=xxx,DC=xxx,DC=xxx,DC=com"]
# len(DN_LIST) will output 2
Is there anyway to do this, or maybe any alternatives to manage the list from .env without modifying the code if i need to add/remove new groups?
Thank you

Sometimes taking a night off provides a simple solution to an otherwise simple problem.. heh.
Decided to take the DN List in .env as a string and have the DNs delimited by something (e.g. ;) and manually split/process it into a list.
# .env
AUTHORIZED_MEMBERSHIP_REQUIREMENT=CN=1,OU=1,OU=1,OU=1,DC=xxx,DC=xxx,DC=xxx,DC=com;CN=2,OU=2,OU=2,OU=2,DC=xxx,DC=xxx,DC=xxx,DC=com
# .settings.py
AUTHORIZED_MEMBERSHIP_REQUIREMENT = env.str('AUTHORIZED_MEMBERSHIP_REQUIREMENT').split(';')

This is about how you read the shell environment variable (.env) into python
DN_LIST="CN=1,OU=1,OU=1,OU=1,DC=xxx" ## instead this
DN_LIST_NEW='["CN=1,OU=1,OU=1,OU=1,DC=xxx","CN=2,OU=2,OU=2,OU=2,DC=xxx"]' ## please you this
Off-topic: define these DN in .env means if we want to change the DN values, we have to change in .env and **restart/deploy** the application , which is not good. You should think about put these DN values into a table in database

How to place a created file in an existed GITLAB repository through python

I have created a text file using file operations in python. I want the file to be pushed to my existed GITLAB repository.
I have tried the below code where i get the created file in my local folders.
file_path = 'E:\My material\output.txt'
k= 'Fail/Pass'
with open (file_path, 'w+') as text:
text.write('Test case :' +k)
text.close()
What is the process or steps or any modifications in file_path to move the created text file to the GITLAB repository through python code.

Using python gitlab module :
We can push the file to the gitlab,but you need to follow the below steps:
Step 1) Clone the repository to your local
Step 2) Add the file to the clonned repository
Step 3) Push the code to the gitlab
code:
import gitlab
import base64
from gitlab import Gitlab
import shutil
callbacks = pygit2.RemoteCallbacks(pygit2.UserPass("Your_private_token", 'x-oauth-basic'))
repoClone = pygit2.clone_repository("https://gitlab.com/group/"+api_name+".git", local_clone_path,checkout_branch=target_branch,callbacks=callbacks) # clonning the repo to local
shutil.copy(src,dst_clone_repo_path) #copy your file to the cloned repo
repoClone.remotes.set_url("origin", "https://gitlab.com/group/"+api_name+".git")
index = repoClone.index
index.add_all()
index.write()
tree = index.write_tree()
oid = repoClone.create_commit('refs/heads/'+target_branch, author, commiter, "init commit",tree,[repoClone.head.peel().hex])
remote = repoClone.remotes["origin"]
credentials = pygit2.UserPass("your_private_token", 'x-oauth-basic') # passing credentials
remote.credentials = credentials
remote.push(['refs/heads/'+target_branch],callbacks=callbacks) # push the code to the gitlab repo

Do you mean executing Shell Commands with Python? Suppose this newly created file and this python script are both under the specific local git repository which connected with the remote repository you want to commit. our plan is packing all the bash command in os.system.
import os
os.system('git add E:\My material\output.txt; git commit -m "anything"; git push -u origin master')
update
import os
os.system('cd /local/repo; mv E:\My material\output.txt .; git add output.txt; git commit -m "anything"; git push -u origin master')

A bit late to the party but:
Using the gitlab python library you can do this:
def commit_file(project_id: int, file_path: str, gitlab_url: str, private_token: str, branch: str = "main") -> bool:
"""Commit a file to the repository
Parameters
----------
project_id: int
the project id to commit into. E.g. 1582
file_path: str
the file path to commit. NOTE: expecting a path relative to the
repo path. This will also be used as the path to commit into.
If you want to use absolute local path you will also have to
pass a parameter for the file relative repo path
gitlab_url: str
The gitlab url. E.g. https://gitlab.example.com
private_token: str
Private access token. See doc for more details
branch: str
The branch you are working in. See note below for more about this
"""
gl = gitlab.Gitlab(gitlab_url, private_token=private_token)
try:
# get the project by the project id
project = gl.projects.get(project_id)
# read the file contents
with open(file_path, 'r') as fin:
content = fin.read()
file_data = {'file_path': file_path,
'branch': branch,
'content': content,
'author_email': "your#email.com", # don't think this is required
'commit_message': 'Created a file'}
resp = project.files.create(file_data)
# do something with resp
except gitlab.exceptions.GitlabGetError as get_error:
# project does not exists
print(f"could not find no project with id {project_id}: {get_error}")
return False
except gitlab.exceptions.GitlabCreateError as create_error:
# project does not exists
print(f"could not create file: {create_error}")
return False
return True
Example is based on gitlab project files documentation python-gitlab package docs
Note that if you want to create the file in a new branch, you will have to also provide a start_branch attribute:
file_data = {'file_path': file_path,
'branch': your_new_branch,
'start_branch': base_branch,
'content': content,
'author_email': "your#email.com",
'commit_message': 'Created a file in new branch'}
Also, if you don't care for using the python-gitlab package, you can use the rest api directly (using request or something like that. The relevant documentation is here

How to use pyfig module in python?

This is the part of the mailer.py script:
config = pyfig.Pyfig(config_file)
svnlook = config.general.svnlook #svnlook path
sendmail = config.general.sendmail #sendmail path
From = config.general.from_email #from email address
To = config.general.to_email #to email address
what does this config variable contain? Is there a way to get the value for config variable without pyfig?

In this case config = a pyfig.Pyfig object initialised with the contents of the file named by the content of the string config_file.
To find out what that object does and contains you can either look at the documentation and/or the source code, both here, or you can print out, after the initialisation, e.g.:
config = pyfig.Pyfig(config_file)
print "Config Contains:\n\t", '\n\t'.join(dir(config))
if hasattr(config, "keys"):
print "Config Keys:\n\t", '\n\t'.join(config.keys())
or if you are using Python 3,
config = pyfig.Pyfig(config_file)
print("Config Contains:\n\t", '\n\t'.join(dir(config)))
if hasattr(config, "keys"):
print("Config Keys:\n\t", '\n\t'.join(config.keys()))
To get the same data without pyfig you would need to read and parse at the content of the file referenced by config_file within your own code.
N.B.: Note that pyfig seems to be more or less abandoned - no updates in over 5 years, web site no longer exists, etc., so I would strongly recommend converting the code to use a json configuration file instead.