We Keep Coding

trying to access environmental variables using os library python [duplicate]

How can I get the value of an environment variable in Python?

Environment variables are accessed through os.environ:
import os
print(os.environ['HOME'])
To see a list of all environment variables:
print(os.environ)
If a key is not present, attempting to access it will raise a KeyError. To avoid this:
# Returns `None` if the key doesn't exist
print(os.environ.get('KEY_THAT_MIGHT_EXIST'))
# Returns `default_value` if the key doesn't exist
print(os.environ.get('KEY_THAT_MIGHT_EXIST', default_value))
# Returns `default_value` if the key doesn't exist
print(os.getenv('KEY_THAT_MIGHT_EXIST', default_value))

To check if the key exists (returns True or False)
'HOME' in os.environ
You can also use get() when printing the key; useful if you want to use a default.
print(os.environ.get('HOME', '/home/username/'))
where /home/username/ is the default

Here's how to check if $FOO is set:
try:
os.environ["FOO"]
except KeyError:
print "Please set the environment variable FOO"
sys.exit(1)

Actually it can be done this way:
import os
for item, value in os.environ.items():
print('{}: {}'.format(item, value))
Or simply:
for i, j in os.environ.items():
print(i, j)
For viewing the value in the parameter:
print(os.environ['HOME'])
Or:
print(os.environ.get('HOME'))
To set the value:
os.environ['HOME'] = '/new/value'

You can access the environment variables using
import os
print os.environ
Try to see the content of the PYTHONPATH or PYTHONHOME environment variables. Maybe this will be helpful for your second question.

As for the environment variables:
import os
print os.environ["HOME"]

Import the os module:
import os
To get an environment variable:
os.environ.get('Env_var')
To set an environment variable:
# Set environment variables
os.environ['Env_var'] = 'Some Value'

import os
for a in os.environ:
print('Var: ', a, 'Value: ', os.getenv(a))
print("all done")
That will print all of the environment variables along with their values.

If you are planning to use the code in a production web application code, using any web framework like Django and Flask, use projects like envparse. Using it, you can read the value as your defined type.
from envparse import env
# will read WHITE_LIST=hello,world,hi to white_list = ["hello", "world", "hi"]
white_list = env.list("WHITE_LIST", default=[])
# Perfect for reading boolean
DEBUG = env.bool("DEBUG", default=False)
NOTE: kennethreitz's autoenv is a recommended tool for making project-specific environment variables. For those who are using autoenv, please note to keep the .env file private (inaccessible to public).

There are also a number of great libraries. Envs, for example, will allow you to parse objects out of your environment variables, which is rad. For example:
from envs import env
env('SECRET_KEY') # 'your_secret_key_here'
env('SERVER_NAMES',var_type='list') #['your', 'list', 'here']

You can also try this:
First, install python-decouple
pip install python-decouple
Import it in your file
from decouple import config
Then get the environment variable
SECRET_KEY=config('SECRET_KEY')
Read more about the Python library here.

Edited - October 2021
Following #Peter's comment, here's how you can test it:
main.py
#!/usr/bin/env python
from os import environ
# Initialize variables
num_of_vars = 50
for i in range(1, num_of_vars):
environ[f"_BENCHMARK_{i}"] = f"BENCHMARK VALUE {i}"
def stopwatch(repeat=1, autorun=True):
"""
Source: https://stackoverflow.com/a/68660080/5285732
stopwatch decorator to calculate the total time of a function
"""
import timeit
import functools
def outer_func(func):
#functools.wraps(func)
def time_func(*args, **kwargs):
t1 = timeit.default_timer()
for _ in range(repeat):
r = func(*args, **kwargs)
t2 = timeit.default_timer()
print(f"Function={func.__name__}, Time={t2 - t1}")
return r
if autorun:
try:
time_func()
except TypeError:
raise Exception(f"{time_func.__name__}: autorun only works with no parameters, you may want to use #stopwatch(autorun=False)") from None
return time_func
if callable(repeat):
func = repeat
repeat = 1
return outer_func(func)
return outer_func
#stopwatch(repeat=10000)
def using_environ():
for item in environ:
pass
#stopwatch
def using_dict(repeat=10000):
env_vars_dict = dict(environ)
for item in env_vars_dict:
pass
python "main.py"
# Output
Function=using_environ, Time=0.216224731
Function=using_dict, Time=0.00014206099999999888
If this is true ... It's 1500x faster to use a dict() instead of accessing environ directly.
A performance-driven approach - calling environ is expensive, so it's better to call it once and save it to a dictionary. Full example:
from os import environ
# Slower
print(environ["USER"], environ["NAME"])
# Faster
env_dict = dict(environ)
print(env_dict["USER"], env_dict["NAME"])
P.S- if you worry about exposing private environment variables, then sanitize env_dict after the assignment.

For Django, see Django-environ.
$ pip install django-environ
import environ
env = environ.Env(
# set casting, default value
DEBUG=(bool, False)
)
# reading .env file
environ.Env.read_env()
# False if not in os.environ
DEBUG = env('DEBUG')
# Raises Django's ImproperlyConfigured exception if SECRET_KEY not in os.environ
SECRET_KEY = env('SECRET_KEY')

You should first import os using
import os
and then actually print the environment variable value
print(os.environ['yourvariable'])
of course, replace yourvariable as the variable you want to access.

The tricky part of using nested for-loops in one-liners is that you have to use list comprehension. So in order to print all your environment variables, without having to import a foreign library, you can use:
python -c "import os;L=[f'{k}={v}' for k,v in os.environ.items()]; print('\n'.join(L))"

You can use python-dotenv module to access environment variables
Install the module using:
pip install python-dotenv
Then import the module into your Python file
import os
from dotenv import load_dotenv
# Load the environment variables
load_dotenv()
# Access the environment variable
print(os.getenv("BASE_URL"))

os.environ.get() does not return the Environment Value in windows?

I already set SLACK_TOKEN environment Variable. But "SLACK_TOKEN=os.environ.get('SLACK_TOKEN')" is returning "None".
The type of SLACK_TOKEN is NoneType. I think os.environ.get not fetching value of environment variable. so rest of the code is not executing.
import os
from slackclient import SlackClient
SLACK_TOKEN= os.environ.get('SLACK_TOKEN') #returning None
print(SLACK_TOKEN) # None
print(type(SLACK_TOKEN)) # NoneType class
slack_client = SlackClient(SLACK_TOKEN)
print(slack_client.api_call("api.test")) #{'ok': True}
print(slack_client.api_call("auth.test")) #{'ok': False, 'error': 'not_authed'}
def list_channels():
channels_call = slack_client.api_call("channels.list")
if channels_call['ok']:
return channels_call['channels']
return None
def channel_info(channel_id):
channel_info = slack_client.api_call("channels.info", channel=channel_id)
if channel_info:
return channel_info['channel']
return None
if __name__ == '__main__':
channels = list_channels()
if channels:
print("Channels: ")
for c in channels:
print(c['name'] + " (" + c['id'] + ")")
detailed_info = channel_info(c['id'])
if detailed_info:
print(detailed_info['latest']['text'])
else:
print("Unable to authenticate.") #Unable to authenticate

I faced similar issue.I fixed it by removing quotes from the values.
Example:
I created a local.env file wherein I stored my secret key values :
*local.env:*
export SLACK_TOKEN=xxxxxyyyyyyyzzzzzzz
*settings.py:*
SLACK_TOKEN = os.environ.get('SLACK_TOKEN')
In your python terminal or console,run the command : *source local.env*
****Involve local.env in gitignore.Make sure you dont push it to git as you have to safeguard your information.
This is applicable only to the local server.Hope this helps.Happy coding :)

In my case, I write wrong content in env file:
SLACK_TOKEN=xxxxxyyyyyyyzzzzzzz
I forgot export befor it, the correct should be:
export SLACK_TOKEN=xxxxxyyyyyyyzzzzzzz

You can use a config file to get the env vars without using export,
in the env file store varibale normally
.env:
DATABASE_URL=postgresql+asyncpg://postgres:dina#localhost/mysenseai
Then create a config file that will be used to store the env variable like so
config.py:
from pydantic import BaseSettings
class Settings(BaseSettings):
database_url: str
class Config:
env_file = '.env'
settings = Settings()
than you can use it that way
from config import settings
url = settings.database_url

If you declared the variable SLACK_TOKEN in the windows command prompt you will be able to access it in the same instance of that command prompt not anywhere including Powershell and the git bash. Be careful with that
whenever you want to run that python script, consider running it in the same command prompt where you declared those variables
you can always check if the variable exists in the cmd by running echo %SLACK_TOKEN% if it does not exists the cmd will return %SLACK_TOKEN%

How to access an AWS Lambda environment variable from Python

Using the new environment variable support in AWS Lambda, I've added an env var via the webui for my function.
How do I access this from Python? I tried:
import os
MY_ENV_VAR = os.environ['MY_ENV_VAR']
but my function stopped working (if I hard code the relevant value for MY_ENV_VAR it works fine).

AWS Lambda environment variables can be defined using the AWS Console, CLI, or SDKs. This is how you would define an AWS Lambda that uses an LD_LIBRARY_PATH environment variable using AWS CLI:
aws lambda create-function \
--region us-east-1
--function-name myTestFunction
--zip-file fileb://path/package.zip
--role role-arn
--environment Variables={LD_LIBRARY_PATH=/usr/bin/test/lib64}
--handler index.handler
--runtime nodejs4.3
--profile default
Once created, environment variables can be read using the support your language provides for accessing the environment, e.g. using process.env for Node.js. When using Python, you would need to import the os library, like in the following example:
...
import os
...
print("environment variable: " + os.environ['variable'])
Resource Link:
AWS Lambda Now Supports Environment Variables
Assuming you have created the .env file along-side your settings module.
.
├── .env
└── settings.py
Add the following code to your settings.py
# settings.py
from os.path import join, dirname
from dotenv import load_dotenv
dotenv_path = join(dirname(__file__), '.env')
load_dotenv(dotenv_path)
Alternatively, you can use find_dotenv() method that will try to find a .env file by (a) guessing where to start using file or the working directory -- allowing this to work in non-file contexts such as IPython notebooks and the REPL, and then (b) walking up the directory tree looking for the specified file -- called .env by default.
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv())
Now, you can access the variables either from system environment variable or loaded from .env file.
Resource Link:
https://github.com/theskumar/python-dotenv
gepoggio answered in this post: https://github.com/serverless/serverless/issues/577#issuecomment-192781002
A workaround is to use python-dotenv:
https://github.com/theskumar/python-dotenv
import os
import dotenv
dotenv.load_dotenv(os.path.join(here, "../.env"))
dotenv.load_dotenv(os.path.join(here, "../../.env"))
It tries to load it twice because when ran locally it's in
project/.env and when running un Lambda the .env is located in
project/component/.env

Both
import os
os.getenv('MY_ENV_VAR')
And
os.environ['MY_ENV_VAR']
are feasible solutions, just make sure in the lambda GUI that the ENV variables are actually there.

I used this code; it includes both cases, setting the variable from the handler and setting it from outside the handler.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Trying new lambda stuff"""
import os
import configparser
class BqEnv(object):
"""Env and self variables settings"""
def __init__(self, service_account, configfile=None):
config = self.parseconfig(configfile)
self.env = config
self.service_account = service_account
#staticmethod
def parseconfig(configfile):
"""Connection and conf parser"""
config = configparser.ConfigParser()
config.read(configfile)
env = config.get('BigQuery', 'env')
return env
def variable_tests(self):
"""Trying conf as a lambda variable"""
my_env_var = os.environ['MY_ENV_VAR']
print my_env_var
print self.env
return True
def lambda_handler(event, context):
"""Trying env variables."""
print event
configfile = os.environ['CONFIG_FILE']
print configfile
print type(str(configfile))
bqm = BqEnv('some-json.json', configfile)
bqm.variable_tests()
return True
I tried this with a demo config file that has this:
[BigQuery]
env = prod
And the setting on lambda was the following:
Hope this can help!

os.environ["variable_name"]
In the configuration section of AWS lambda, make sure you declare the variable with the same name that you're trying to access here. For this example, it should be variable_name

Python - bulk promote variables to parent scope

In python 2.7, I want to run:
$ ./script.py initparms.py
This is a trick to supply a parameter file to script.py, since initparms.py contains several python variables e.g.
Ldir = '/home/marzipan/jelly'
LMaps = True
# etc.
script.py contains:
X = __import__(sys.argv[1])
Ldir = X.Ldir
LMaps = X.Lmaps
# etc.
I want to do a bulk promotion of the variables in X so they are available to script.py, without spelling out each one in the code by hand.
Things like
import __import__(sys.argv[1])
or
from sys.argv[1] import *
don't work. Almost there perhaps... Any ideas? Thanks!

here's a one-liner:
globals().update(__import__(sys.argv[1]).__dict__)

You can use execfile:
execfile(sys.argv[1])
Of course, the usual warnings with exec or eval apply (Your script has no way of knowing whether it is running trusted or untrusted code).
My suggestion would be to not do what you're doing and instead use configparser and handling the configuration though there.

You could do something like this:
import os
import imp
import sys
try:
module_name = sys.argv[1]
module_info = imp.find_module(module_name, [os.path.abspath(os.path.dirname(__file__))] + sys.path)
module_properties = imp.load_module(module_name, *module_info)
except ImportError:
pass
else:
try:
attrlist = module_properties.__all__
except AttributeError:
attrlist = dir(module_properties)
for attr in attrlist:
if attr.startswith('__'):
continue
globals()[attr] = getattr(module_properties, attr)
Little complicated, but gets the job done.

Is there a standard way to list names of Python modules in a package?

Is there a straightforward way to list the names of all modules in a package, without using __all__?
For example, given this package:
/testpkg
/testpkg/__init__.py
/testpkg/modulea.py
/testpkg/moduleb.py
I'm wondering if there is a standard or built-in way to do something like this:
>>> package_contents("testpkg")
['modulea', 'moduleb']
The manual approach would be to iterate through the module search paths in order to find the package's directory. One could then list all the files in that directory, filter out the uniquely-named py/pyc/pyo files, strip the extensions, and return that list. But this seems like a fair amount of work for something the module import mechanism is already doing internally. Is that functionality exposed anywhere?

Using python2.3 and above, you could also use the pkgutil module:
>>> import pkgutil
>>> [name for _, name, _ in pkgutil.iter_modules(['testpkg'])]
['modulea', 'moduleb']
EDIT: Note that the parameter for pkgutil.iter_modules is not a list of modules, but a list of paths, so you might want to do something like this:
>>> import os.path, pkgutil
>>> import testpkg
>>> pkgpath = os.path.dirname(testpkg.__file__)
>>> print([name for _, name, _ in pkgutil.iter_modules([pkgpath])])

import module
help(module)

Maybe this will do what you're looking for?
import imp
import os
MODULE_EXTENSIONS = ('.py', '.pyc', '.pyo')
def package_contents(package_name):
file, pathname, description = imp.find_module(package_name)
if file:
raise ImportError('Not a package: %r', package_name)
# Use a set because some may be both source and compiled.
return set([os.path.splitext(module)[0]
for module in os.listdir(pathname)
if module.endswith(MODULE_EXTENSIONS)])

Don't know if I'm overlooking something, or if the answers are just out-dated but;
As stated by user815423426 this only works for live objects and the listed modules are only modules that were imported before.
Listing modules in a package seems really easy using inspect:
>>> import inspect, testpkg
>>> inspect.getmembers(testpkg, inspect.ismodule)
['modulea', 'moduleb']

This is a recursive version that works with python 3.6 and above:
import importlib.util
from pathlib import Path
import os
MODULE_EXTENSIONS = '.py'
def package_contents(package_name):
spec = importlib.util.find_spec(package_name)
if spec is None:
return set()
pathname = Path(spec.origin).parent
ret = set()
with os.scandir(pathname) as entries:
for entry in entries:
if entry.name.startswith('__'):
continue
current = '.'.join((package_name, entry.name.partition('.')[0]))
if entry.is_file():
if entry.name.endswith(MODULE_EXTENSIONS):
ret.add(current)
elif entry.is_dir():
ret.add(current)
ret |= package_contents(current)
return ret

There is a __loader__ variable inside each package instance. So, if you import the package, you can find the "module resources" inside the package:
import testpkg # change this by your package name
for mod in testpkg.__loader__.get_resource_reader().contents():
print(mod)
You can of course improve the loop to find the "module" name:
import testpkg
from pathlib import Path
for mod in testpkg.__loader__.get_resource_reader().contents():
# You can filter the name like
# Path(l).suffix not in (".py", ".pyc")
print(Path(mod).stem)
Inside the package, you can find your modules by directly using __loader__ of course.

This should list the modules:
help("modules")

If you would like to view an inforamtion about your package outside of the python code (from a command prompt) you can use pydoc for it.
# get a full list of packages that you have installed on you machine
$ python -m pydoc modules
# get information about a specific package
$ python -m pydoc <your package>
You will have the same result as pydoc but inside of interpreter using help
>>> import <my package>
>>> help(<my package>)

Based on cdleary's example, here's a recursive version listing path for all submodules:
import imp, os
def iter_submodules(package):
file, pathname, description = imp.find_module(package)
for dirpath, _, filenames in os.walk(pathname):
for filename in filenames:
if os.path.splitext(filename)[1] == ".py":
yield os.path.join(dirpath, filename)

The other answers here will run the code in the package as they inspect it. If you don't want that, you can grep the files like this answer
def _get_class_names(file_name: str) -> List[str]:
"""Get the python class name defined in a file without running code
file_name: the name of the file to search for class definitions in
return: all the classes defined in that python file, empty list if no matches"""
defined_class_names = []
# search the file for class definitions
with open(file_name, "r") as file:
for line in file:
# regular expression for class defined in the file
# searches for text that starts with "class" and ends with ( or :,
# whichever comes first
match = re.search("^class(.+?)(\(|:)", line) # noqa
if match:
# add the cleaned match to the list if there is one
defined_class_name = match.group(1).strip()
defined_class_names.append(defined_class_name)
return defined_class_names

To complete #Metal3d answer, yes you can do testpkg.__loader__.get_resource_reader().contents() to list the "module resources" but it will work only if you imported your package in the "normal" way and your loader is _frozen_importlib_external.SourceFileLoader object.
But if you imported your library with zipimport (ex: to load your package in memory), your loader will be a zipimporter object, and its get_resource_reader function is different from importlib; it will require a "fullname" argument.
To make it work in these two loaders, just specify your package name in argument to get_resource_reader :
# An example with CrackMapExec tool
import importlib
import cme.protocols as cme_protocols
class ProtocolLoader:
def get_protocols(self):
protocols = {}
protocols_names = [x for x in cme_protocols.__loader__.get_resource_reader("cme.protocols").contents()]
for prot_name in protocols_names:
prot = importlib.import_module(f"cme.protocols.{prot_name}")
protocols[prot_name] = prot
return protocols

def package_contents(package_name):
package = __import__(package_name)
return [module_name for module_name in dir(package) if not module_name.startswith("__")]