We Keep Coding

trying to access environmental variables using os library python [duplicate]

How can I get the value of an environment variable in Python?

Environment variables are accessed through os.environ:
import os
print(os.environ['HOME'])
To see a list of all environment variables:
print(os.environ)
If a key is not present, attempting to access it will raise a KeyError. To avoid this:
# Returns `None` if the key doesn't exist
print(os.environ.get('KEY_THAT_MIGHT_EXIST'))
# Returns `default_value` if the key doesn't exist
print(os.environ.get('KEY_THAT_MIGHT_EXIST', default_value))
# Returns `default_value` if the key doesn't exist
print(os.getenv('KEY_THAT_MIGHT_EXIST', default_value))

To check if the key exists (returns True or False)
'HOME' in os.environ
You can also use get() when printing the key; useful if you want to use a default.
print(os.environ.get('HOME', '/home/username/'))
where /home/username/ is the default

Here's how to check if $FOO is set:
try:
os.environ["FOO"]
except KeyError:
print "Please set the environment variable FOO"
sys.exit(1)

Actually it can be done this way:
import os
for item, value in os.environ.items():
print('{}: {}'.format(item, value))
Or simply:
for i, j in os.environ.items():
print(i, j)
For viewing the value in the parameter:
print(os.environ['HOME'])
Or:
print(os.environ.get('HOME'))
To set the value:
os.environ['HOME'] = '/new/value'

You can access the environment variables using
import os
print os.environ
Try to see the content of the PYTHONPATH or PYTHONHOME environment variables. Maybe this will be helpful for your second question.

As for the environment variables:
import os
print os.environ["HOME"]

Import the os module:
import os
To get an environment variable:
os.environ.get('Env_var')
To set an environment variable:
# Set environment variables
os.environ['Env_var'] = 'Some Value'

import os
for a in os.environ:
print('Var: ', a, 'Value: ', os.getenv(a))
print("all done")
That will print all of the environment variables along with their values.

If you are planning to use the code in a production web application code, using any web framework like Django and Flask, use projects like envparse. Using it, you can read the value as your defined type.
from envparse import env
# will read WHITE_LIST=hello,world,hi to white_list = ["hello", "world", "hi"]
white_list = env.list("WHITE_LIST", default=[])
# Perfect for reading boolean
DEBUG = env.bool("DEBUG", default=False)
NOTE: kennethreitz's autoenv is a recommended tool for making project-specific environment variables. For those who are using autoenv, please note to keep the .env file private (inaccessible to public).

There are also a number of great libraries. Envs, for example, will allow you to parse objects out of your environment variables, which is rad. For example:
from envs import env
env('SECRET_KEY') # 'your_secret_key_here'
env('SERVER_NAMES',var_type='list') #['your', 'list', 'here']

You can also try this:
First, install python-decouple
pip install python-decouple
Import it in your file
from decouple import config
Then get the environment variable
SECRET_KEY=config('SECRET_KEY')
Read more about the Python library here.

Edited - October 2021
Following #Peter's comment, here's how you can test it:
main.py
#!/usr/bin/env python
from os import environ
# Initialize variables
num_of_vars = 50
for i in range(1, num_of_vars):
environ[f"_BENCHMARK_{i}"] = f"BENCHMARK VALUE {i}"
def stopwatch(repeat=1, autorun=True):
"""
Source: https://stackoverflow.com/a/68660080/5285732
stopwatch decorator to calculate the total time of a function
"""
import timeit
import functools
def outer_func(func):
#functools.wraps(func)
def time_func(*args, **kwargs):
t1 = timeit.default_timer()
for _ in range(repeat):
r = func(*args, **kwargs)
t2 = timeit.default_timer()
print(f"Function={func.__name__}, Time={t2 - t1}")
return r
if autorun:
try:
time_func()
except TypeError:
raise Exception(f"{time_func.__name__}: autorun only works with no parameters, you may want to use #stopwatch(autorun=False)") from None
return time_func
if callable(repeat):
func = repeat
repeat = 1
return outer_func(func)
return outer_func
#stopwatch(repeat=10000)
def using_environ():
for item in environ:
pass
#stopwatch
def using_dict(repeat=10000):
env_vars_dict = dict(environ)
for item in env_vars_dict:
pass
python "main.py"
# Output
Function=using_environ, Time=0.216224731
Function=using_dict, Time=0.00014206099999999888
If this is true ... It's 1500x faster to use a dict() instead of accessing environ directly.
A performance-driven approach - calling environ is expensive, so it's better to call it once and save it to a dictionary. Full example:
from os import environ
# Slower
print(environ["USER"], environ["NAME"])
# Faster
env_dict = dict(environ)
print(env_dict["USER"], env_dict["NAME"])
P.S- if you worry about exposing private environment variables, then sanitize env_dict after the assignment.

For Django, see Django-environ.
$ pip install django-environ
import environ
env = environ.Env(
# set casting, default value
DEBUG=(bool, False)
)
# reading .env file
environ.Env.read_env()
# False if not in os.environ
DEBUG = env('DEBUG')
# Raises Django's ImproperlyConfigured exception if SECRET_KEY not in os.environ
SECRET_KEY = env('SECRET_KEY')

You should first import os using
import os
and then actually print the environment variable value
print(os.environ['yourvariable'])
of course, replace yourvariable as the variable you want to access.

The tricky part of using nested for-loops in one-liners is that you have to use list comprehension. So in order to print all your environment variables, without having to import a foreign library, you can use:
python -c "import os;L=[f'{k}={v}' for k,v in os.environ.items()]; print('\n'.join(L))"

You can use python-dotenv module to access environment variables
Install the module using:
pip install python-dotenv
Then import the module into your Python file
import os
from dotenv import load_dotenv
# Load the environment variables
load_dotenv()
# Access the environment variable
print(os.getenv("BASE_URL"))

Custom Module Import Issue

I can't seem to import my own custom NYT module. My project structure is as follows and I'm on a mac:
articulation/
articulation/
__init__.py # empty
lib/
nyt.py
__init__.py # empty
tests/
test_nyt.py
__init__.py # empty
When I try running python articulation/tests/test_nyt.py from that first parent directory, I get
File "articulation/tests/test_nyt.py", line 5, in <module>
from articulation.lib.nyt import NYT
ImportError: No module named articulation.lib.nyt
I also tried
(venv) Ericas-MacBook-Pro:articulation edohring$ Python -m articulation/tests/test_nyt.py
/Users/edohring/Desktop/articulation/venv/bin/Python: Import by filename is not supported.
test_nyt.py
import sys
sys.path.insert(0, '../../')
import unittest
#from mock import patch
# TODO: store example as fixture and complete test
from articulation.lib.nyt import NYT
class TestNYT(unittest.TestCase):
#patch('articulation.lib.nyt.NYT.fetch')
def test_nyt(self):
print "hi"
#assert issubclass(NYT, Article)
# self.assertTrue(sour_surprise.title == '')"""
nyt.py
from __future__ import division
import regex as re
import string
import urllib2
from collections import Counter
from bs4 import BeautifulSoup
from cookielib import CookieJar
PARSER_TYPE = 'html.parser'
class NYT:
def __init__(self, title, url):
self.url = url
self.title = title
self.words = get_words(url)
def get_words(url):
cj = CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
p = opener.open(url)
soup = BeautifulSoup(p.read(), PARSER_TYPE)
# title = soup.html.head.title.string
letters = soup.find_all('p', class_='story-body-text story-content')
if len(letters)==0:
letters = soup.find_all('p', class_='paragraph--story')
if len(letters)==0:
letters = soup.find_all('p', class_='story-body-text', )
words = Counter()
for element in letters:
a = element.get_text().split()
for c in a:
c = ''.join(ch for ch in c if c.isalpha())
c = c.lower()
if len(c) > 0:
words[c] += 1
return words
def test_nyt():
china_apple_stores = NYT('title_test', 'http://www.nytimes.com/2016/12/29/technology/iphone-china-apple-stores.html?_r=0')
assert(len(china_apple_stores.words) > 0)
# print china_apple_stores.words
fri_brief = NYT('Russia, Syria, 2017: Your Friday Briefing', 'http://www.nytimes.com/2016/12/30/briefing/us-briefing-russia-syria-2017.html')
assert(fri_brief.title == 'Russia, Syria, 2017: Your Friday Briefing')
assert(fri_brief.url == 'http://www.nytimes.com/2016/12/30/briefing/us-briefing-russia-syria-2017.html')
assert(len(fri_brief.words) > 0)
vet = NYT('title_test', 'http://lens.blogs.nytimes.com/2017/01/03/a-love-story-and-twins-for-a-combat-veteran-amputee/')
assert(len(vet.words)>0)
print "All NYT Tests Passed"
#test_nyt()
I've tried the following and none seem to work - does anyone know how to fix this?
- Adding an init.py file to the top directory -> Doesn't help
- Entering Memory Python couldn't find this - maybe because I'm using Python 2. If this is the issue I can post more what I tried.
- Adding sys.path at the top from suggestion below

Doing this:
import sys
sys.path.insert(0, '../../')
is usually a bad idea. Sometimes it's useful for when you're testing something, or you have a single-use program that you just need to work for a short time and then you're going to throw away, but in general it's a bad habit to get into because it might stop working once you move directories around or once you give the code to someone else. I would advise you not to let yourself get in the habit of doing that.
The most likely reason to get the kind of error you're seeing is that the directory /Users/edohring/Desktop/articulation does not appear in sys.path. The first thing to do is see what actually is in sys.path, and one good way to do that is to temporarily put these lines at the top of test_nyt.py:
import os.path, sys
for p in sys.path:
print(p)
if not os.path.isabs(p):
print(' (absolute: {})'.format(os.path.abspath(p)))
sys.exit()
Then run
python articulation/tests/test_nyt.py
and look at the output. You will get a line for each directory path that Python looks in to find its modules, and if any of those paths are relative, it will also print out the corresponding absolute path so that there is no confusion. I suspect you will find that /Users/edohring/Desktop/articulation does not appear anywhere in this list.
If that turns out to be the case, the most straightforward (but least future-proof) way to fix it is to run
export PYTHONPATH=".:$PYTHONPATH"
in the shell (not in Python!) before you use Python itself to do anything using your module. Directories named in the PYTHONPATH environment variable will be added to sys.path when Python starts up. This is only a temporary fix, unless you put it in a file like $HOME/.bashrc which will get read by the shell every time you open up a Terminal window. You can read about this and better ways to add the proper directory to sys.path in this question.
Perhaps a better way to run your script is to use the shell command
python -m articulation.tests.test_nyt
This needs to be run in the directory /Users/edohring/Desktop/articulation, or at least that directory needs to appear in sys.path in order for the command to work. But using the -m switch in this way causes Python to handle how it sets up sys.path a little differently, and it may work for you. You can read more about how sys.path is populated in this answer.

process.terminate() doesn't work for a chrome / firefox subprocess if a browser window is already present

I am launching a sub-process using the following command:
p=subprocess.Popen(["google-chrome","--new-window","http://www.hckrnews.com"])
I need to kill this process after sometime, so I'm using:
time.sleep(t)
p.terminate()
This works only if no instance of the browser is already open. If a browser window is already present, a new window opens but does not terminate after specified time.
I also tried the method given in this question, but that also doesn't work in case of already present window.

This is not a python problem, and your code doesn't contain errors. It's a problem with the browsers. When you launch the firefox or chrome's executable with --new-window a new window is opened in the existing instance of the browser.
In other words the process you just started connects to the already existing process of firefox/chrome and instructs that process to open a new window and then terminates. So when you call terminate() nothing really happens because the process you started already ended.
You can check this with few lines of code:
>>> import subprocess
>>> p = subprocess.Popen(['firefox', '-new-window'])
>>> p.wait() # uh-oh. this should be blocking!
0
>>> p.terminate()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 1551, in terminate
self.send_signal(signal.SIGTERM)
File "/usr/lib/python2.7/subprocess.py", line 1546, in send_signal
os.kill(self.pid, sig)
OSError: [Errno 3] No such process
You should tell the browser to open a new instance instead of a new window. Firefox has a -new-instance option, even though, when trying to use it, the only thing I get is that a pop-up appears saying that I cannot open two instances of the browser. Whether opening multiple instances of a browser in a certain OS session is allowed depends on the browser and may not be possible.
On Chrome I believe you can open a new session telling Chrome to use a different directory to store its data (see here for example). No idea whether this is possible
in firefox.

As explained in Bakuriu's answer, --new-window creates a new window, but under an existing instance of firefox, if there is one. If there is no existing instance, a new one is created.
It is possible to use -new-instance to tell Firefox to start a new instance for a different firefox user profile. The profile must already exist and this is limited to one instance per profile. A new profile can be created interactively with firefox -P -new-instance, and a new instance can then be started with firefox -P <profile_name> -new-instance. Mozilla's profile documentation is here.
It should be possible to programmatically create the profile - after all it is just a directory and an entry in the ~/.mozilla/profiles.ini file. Of course, this is for Firefox only, chrome might be completely different (or impossible?). An example:
import tempfile
import subprocess
import shutil
import time
import ConfigParser
MOZILLA_PROFILES_INI = '.mozilla/firefox/profiles.ini'
PROFILE_SECTION = 'Profile1'
URL = 'http://www.hckrnews.com'
profile_dir = tempfile.mkdtemp()
# quick and dirty add new profile to profiles.ini, or update if already present.
config = ConfigParser.SafeConfigParser()
config.optionxform = str # preserve case of option names
config.read(MOZILLA_PROFILES_INI)
try:
config.add_section(PROFILE_SECTION)
except ConfigParser.DuplicateSectionError, exc:
pass
config.set(PROFILE_SECTION, 'Name', 'temp_profile')
config.set(PROFILE_SECTION, 'IsRelative', '0')
config.set(PROFILE_SECTION, 'Path', profile_dir)
# write out config with hack to remove added spaces around '=' - more care needed here as it just overwrites the global mozilla config!
class NoSpaceWriter(file):
def write(self, s):
super(NoSpaceWriter, self).write(s.replace(' = ', '='))
with NoSpaceWriter(MOZILLA_PROFILES_INI, 'w') as profiles_ini:
config.write(profiles_ini)
p = subprocess.Popen(['firefox', '-P', 'temp_profile', '-new-instance', URL])
time.sleep(10)
p.terminate()
shutil.rmtree(profile_dir, True)

retrieving current URL from FireFox with python

I want to know what is the current url of active tab in running firefox instance from python module. Does FireFox have any API for this and does python know to work with it?

The most convenient way maybe insatll a firefox extension to open up a tcp service, then you can exchange info with firefox.
mozrepl can set up a telnet service, you can call js-like command to get info.
With telnetscript (http: //code.activestate.com/recipes/152043/), you can write:
import telnetscript
script = """rve
w content.location.href;
ru repl>
w repl.quit()
cl
"""
conn = telnetscript.telnetscript( '127.0.0.1', {}, 4242 )
ret = conn.RunScript( script.split( '\n' )).split( '\n' )
print ret[-2][6:]

If on windows you can use win32com
import win32clipboard
import win32com.client
shell = win32com.client.Dispatch("WScript.Shell")
shell.AppActivate('Some Application Title')
Then use shell.SendKeys to do a ctrl+l and a ctrl+c
Then read the string in the clipboard.
It's horkey though it will work, alternatly you can use something like AutoIt an compile the code to an exe that you can work with.
Hope this helps.

How to get/set logical directory path in python

In python is it possible to get or set a logical directory (as opposed to an absolute one).
For example if I have:
/real/path/to/dir
and I have
/linked/path/to/dir
linked to the same directory.
using os.getcwd and os.chdir will always use the absolute path
>>> import os
>>> os.chdir('/linked/path/to/dir')
>>> print os.getcwd()
/real/path/to/dir
The only way I have found to get around this at all is to launch 'pwd' in another process and read the output. However, this only works until you call os.chdir for the first time.

The underlying operational system / shell reports real paths to python.
So, there really is no way around it, since os.getcwd() is a wrapped call to C Library getcwd() function.
There are some workarounds in the spirit of the one that you already know which is launching pwd.
Another one would involve using os.environ['PWD']. If that environmnent variable is set you can make some getcwd function that respects it.
The solution below combines both:
import os
from subprocess import Popen, PIPE
class CwdKeeper(object):
def __init__(self):
self._cwd = os.environ.get("PWD")
if self._cwd is None: # no environment. fall back to calling pwd on shell
self._cwd = Popen('pwd', stdout=PIPE).communicate()[0].strip()
self._os_getcwd = os.getcwd
self._os_chdir = os.chdir
def chdir(self, path):
if not self._cwd:
return self._os_chdir(path)
p = os.path.normpath(os.path.join(self._cwd, path))
result = self._os_chdir(p)
self._cwd = p
os.environ["PWD"] = p
return result
def getcwd(self):
if not self._cwd:
return self._os_getcwd()
return self._cwd
cwd = CwdKeeper()
print cwd.getcwd()
# use only cwd.chdir and cwd.getcwd from now on.
# monkeypatch os if you want:
os.chdir = cwd.chdir
os.getcwd = cwd.getcwd
# now you can use os.chdir and os.getcwd as normal.

This also does the trick for me:
import os
os.popen('pwd').read().strip('\n')
Here is a demonstration in python shell:
>>> import os
>>> os.popen('pwd').read()
'/home/projteam/staging/site/proj\n'
>>> os.popen('pwd').read().strip('\n')
'/home/projteam/staging/site/proj'
>>> # Also works if PWD env var is set
>>> os.getenv('PWD')
'/home/projteam/staging/site/proj'
>>> # This gets actual path, not symlinked path
>>> import subprocess
>>> p = subprocess.Popen('pwd', stdout=subprocess.PIPE)
>>> p.communicate()[0] # returns non-symlink path
'/home/projteam/staging/deploys/20150114-141114/site/proj\n'
Getting the environment variable PWD didn't always work for me so I use the popen method. Cheers!