We Keep Coding

How does the Python subprocess module find executables to run? Can I change the location?

On Linux systems, subprocess.run(["some_command"]) appears to scrape /usr/bin for some_command by default and thus, subprocess accepts every command that also has a respective binary in this directory.
I have a use case now, where I have binaries in multiple directories and I want to execute some of them with the subprocess module. Obviously, I can do something like:
subprocess.run(["/absolute/path/to/first_binary"])
subprocess.run(["/another/absolute/path/to/another/second_binary"])
But I was wondering if there exists anything like:
subprocess.set_bin_dirs(["/usr/bin/", "/absolute/path/to/", "/another/absolute/path/to/another/"])
subprocess.run(["first_binary"])
subprocess.run(["second_binary"])

This has nothing to do with Python and the subprocess module. This is a core idea about how Unix-like(and other) operating systems work, when you do some_command, that command is looked up on your PATH. So you must add that directories of these binaries to the PATH, otherwise, you need to use the full path when you try to execute a command, e.g. /usr/bin/some_command
To do this in Python with the subprocess module, you could do something like:
import os
env_copy = os.environ.copy()
current_path = current_env['PATH']
env_copy["PATH"] = os.pathsep.join([
"/usr/bin/",
"/absolute/path/to/",
"/another/absolute/path/to/another/",
current_path
])
subprocess.run(["some_command"], env=env_copy)
Note, paths are looked up in order, so the order in which you append it matters if there are possible name-collisions. Note, you don't actually have to add the current PATH, but you might. All of this is up to you.

Is there a graceful way to use os.path.join() when the right-hand side may be /-prefixed?

In the code below, context._arguments['ConfigFile'] returns a string like '/path/file.py' (which I can't change) but due to the way os.path.join() works, I need to remove at bare minimum the first /.
Note: In my use case __file__ will always be in the appropriate position away from the config file.
I also considered giving it context._arguments['ConfigFile'][1:] but I think it's less robust.
config_file = os.path.join(
os.path.dirname(os.path.abspath(__file__)),
*context._arguments['ConfigFile'].split(os.path.sep))
I expected there to be something a little more graceful, but maybe handling paths just never is. I am using Python 2.7 but for completeness I'm open to hearing Python 3 answers.

If you use Python 3, you can benefit from the pathlib package:
from pathlib import Path
file_path = '/path/file.py'
config_file = Path(__file__).parent / file_path.lstrip('/')
print(config_file)
# /Users/darius/repos/stackoverflow/questions/path/file.py
If you use Python 2, you can install pathlib2 (pip install pathlib2) which is a backport of the standard pathlib package. To match the module names you can rename the import with import pathlib2 as pathlib.

(This is a response to a comment, really, but needs formatting.)
>>> os.path.join('/a', '/b/c')
'/b/c'
>>> os.path.join('/a', './/b/c')
'/a/.//b/c'
Use os.path.normpath to clean up:
>>> os.path.normpath(os.path.join('/a', './/b/c'))
'/a/b/c'
The other way to view this is that, at least on Unix systems, os.path.join starts with its first argument. Then, for each additional argument, it either concatenates or replaces using the return-value-so-far and the extra path component:
def unix_style_join(*args):
"low quality version, for illustration"
ret = args[0]
for extra in args[1:]:
if extra.startswith('/'):
ret = extra
else:
ret = ret + '/' + extra
return ret
Since your problem is that context._arguments['ConfigFile'] starts with /, we merely need a variant of context._arguments['ConfigFile'] that means the same thing but does not start with / ... and ./<whatever> means the same as <whatever> except that ./<whatever> starts with ., even if <whatever> starts with /.
The reason I didn't suggest this as the whole answer is that I have no idea how this all works on Windows.

Check if file system is case-insensitive in Python

Is there a simple way to check in Python if a file system is case insensitive? I'm thinking in particular of file systems like HFS+ (OSX) and NTFS (Windows), where you can access the same file as foo, Foo or FOO, even though the file case is preserved.

import os
import tempfile
# By default mkstemp() creates a file with
# a name that begins with 'tmp' (lowercase)
tmphandle, tmppath = tempfile.mkstemp()
if os.path.exists(tmppath.upper()):
# Case insensitive.
else:
# Case sensitive.

The answer provided by Amber will leave temporary file debris unless closing and deleting are handled explicitly. To avoid this I use:
import os
import tempfile
def is_fs_case_sensitive():
#
# Force case with the prefix
#
with tempfile.NamedTemporaryFile(prefix='TmP') as tmp_file:
return(not os.path.exists(tmp_file.name.lower()))
Though my usage cases generally test this more than once, so I stash the result to avoid having to touch the filesystem more than once.
def is_fs_case_sensitive():
if not hasattr(is_fs_case_sensitive, 'case_sensitive'):
with tempfile.NamedTemporaryFile(prefix='TmP') as tmp_file:
setattr(is_fs_case_sensitive,
'case_sensitive',
not os.path.exists(tmp_file.name.lower()))
return(is_fs_case_sensitive.case_sensitive)
Which is marginally slower if only called once, and significantly faster in every other case.

Good point on the different file systems, etc., Eric Smith. But why not use tempfile.NamedTemporaryFile with the dir parameter and avoid doing all that context manager lifting yourself?
def is_fs_case_sensitive(path):
#
# Force case with the prefix
#
with tempfile.NamedTemporaryFile(prefix='TmP',dir=path, delete=True) as tmp_file:
return(not os.path.exists(tmp_file.name.lower()))
I should also mention that your solution does not guarantee that you are actually testing for case sensitivity. Unless you check the default prefix (using tempfile.gettempprefix()) to make sure it contains a lower-case character. So including the prefix here is not really optional.
Your solution cleans up the temp file. I agree that it seemed obvious, but one never knows, do one?

Variation on #Shrikant's answer, applicable within a module (i.e. not in the REPL), even if your user doesn't have a home:
import os.path
is_fs_case_insensitive = os.path.exists(__file__.upper()) and os.path.exists(__file__.lower())
print(f"{is_fs_case_insensitive=}")
output (macOS):
is_fs_case_insensitive=True 👈
And the Linux side of things:
(ssha)vagrant ~$python3.8 test.py
is_fs_case_insensitive=False 👈
(ssha)vagrant ~$lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal
FWIW, I checked pathlib, os, os.path's contents via:
[k for k in vars(pathlib).keys() if "case" in k.lower()]
and nothing looks like it, though it does have a pathlib.supports_symlinks but nothing about case-sensitivity.
And the following will work in the REPL as well:
is_fs_case_insensitive = os.path.exists(os.path.__file__.upper()) and os.path.exists(os.path.__file__.lower())

Starting with Amber's answer, I came up with this code. I'm not sure it is totally robust, but it attempts to address some issues in the original (that I'll mention below).
import os
import sys
import tempfile
import contextlib
def is_case_sensitive(path):
with temp(path) as tmppath:
head, tail = os.path.split(tmppath)
testpath = os.path.join(head, tail.upper())
return not os.path.exists(testpath)
#contextlib.contextmanager
def temp(path):
tmphandle, tmppath = tempfile.mkstemp(dir=path)
os.close(tmphandle)
try:
yield tmppath
finally:
os.unlink(tmppath)
if __name__ == '__main__':
path = os.path.abspath(sys.argv[1])
print(path)
print('Case sensitive: ' + str(is_case_sensitive(path)))
Without specifying the dir parameter in mkstemp, the question of case sensitivity is vague. You're testing case sensitivity of wherever the temporary directory happens to be, but you may want to know about a specific path.
If you convert the full path returned from mkstemp to upper-case, you could potentially miss a transition somewhere in the path. For example, I have a USB flash drive on Linux mounted using vfat at /media/FLASH. Testing the existence of anything under /MEDIA/FLASH will always fail because /media is on a (case-sensitive) ext4 partition, but the flash drive itself is case-insensitive. Mounted network shares could be another situation like this.
Finally, and maybe it goes without saying in Amber's answer, you'll want to clean up the temp file created by mkstemp.

I think there's a much simpler (and probably faster) solution to this. The following seemed to be working for where I tested:
import os.path
home = os.path.expanduser('~')
is_fs_case_insensitive = os.path.exists(home.upper()) and os.path.exists(home.lower())

import os
if os.path.normcase('A') == os.path.normcase('a'):
# case insensitive
else:
# case sensitive

I think we can do this in one line with pathlib on Python 3.5+ without creating temporary files:
from pathlib import Path
def is_case_insensitive(path) -> bool:
return Path(str(Path.home()).upper()).exists()
Or for the inverse:
def is_case_sensitive(path) -> bool:
return not Path(str(Path.home()).upper()).exists()

I believe this to be the simplest solution to the question:
from fnmatch import fnmatch
os_is_case_insensitive = fnmatch('A','a')
From: https://docs.python.org/3.4/library/fnmatch.html
If the operating system is case-insensitive, then both parameters will
be normalized to all lower- or upper-case before the comparison is
performed.

os.path.basename works with URLs, why?

>>> os.path.basename('http://example.com/file.txt')
'file.txt'
.. and I thought os.path.* work only on local paths and not URLs? Note that the above example was run on Windows too .. with similar result.

In practice many functions of os.path are just string manipulation functions (which just happen to be especially handy for path manipulation) -- and since that's innocuous and occasionally handy, while formally speaking "incorrect", I doubt this will change anytime soon -- for more details, use the following simple one-liner at a shell/command prompt:
$ python -c"import sys; import StringIO; x = StringIO.StringIO(); sys.stdout = x; import this; sys.stdout = sys.__stdout__; print x.getvalue().splitlines()[10][9:]"
Or, for Python 3:
$ python -c"import sys; import io; x = io.StringIO(); sys.stdout = x; import this; sys.stdout = sys.__stdout__; print(x.getvalue().splitlines()[10][9:])"

On windows, look at the source code: C:\Python25\Lib\ntpath.py
def basename(p):
"""Returns the final component of a pathname"""
return split(p)[1]
os.path.split (in the same file) just split "\" (and sth. else)

Beware of URLs with parameters, anchors or anything that isn't a "plain" URL:
>>> import os.path
>>> os.path.basename("protocol://fully.qualifie.host/path/to/file.txt")
'file.txt'
>>> os.path.basename("protocol://fully.qualifie.host/path/to/file.txt?param1&param1#anchor")
'file.txt?param1&param1#anchor'

Use the source Luke:
def basename(p):
"""Returns the final component of a pathname"""
i = p.rfind('/') + 1
return p[i:]
Edit (response to clarification):
It works for URLs by accident, that's it. Because of that, exploiting its behaviour could be considered code smell by some.
Trying to "fix" it (check if passed path is not url) is also surprisingly difficult
www.google.com/test.php
me#other.place.com/12
./src/bin/doc/goto.c
are at the same time correct pathnames and URLs (relative), so is the http:/hello.txt (one /, and only on linux, and it's kinda stupid :)). You could "fix" it for absolute urls but relative ones will still work. Handling one special case in differently is a big no no in the python world.
To sum it up: import this

Forward slash is also an acceptable path delimiter in Windows.
It is merely that the command line does not accept paths that begin with a / because that character is reserved for args switches.

Why? Because it's useful for parsing URLs as well as local file paths. Why not?

How do I store desktop application data in a cross platform way for python?

I have a python desktop application that needs to store user data. On Windows, this is usually in %USERPROFILE%\Application Data\AppName\, on OSX it's usually ~/Library/Application Support/AppName/, and on other *nixes it's usually ~/.appname/.
There exists a function in the standard library, os.path.expanduser that will get me a user's home directory, but I know that on Windows, at least, "Application Data" is localized into the user's language. That might be true for OSX as well.
What is the correct way to get this location?
UPDATE:
Some further research indicates that the correct way to get this on OSX is by using the function NSSearchPathDirectory, but that's Cocoa, so it means calling the PyObjC bridge...

Well, I hate to have been the one to answer my own question, but no one else seems to know. I'm leaving the answer for posterity.
APPNAME = "MyApp"
import sys
from os import path, environ
if sys.platform == 'darwin':
from AppKit import NSSearchPathForDirectoriesInDomains
# http://developer.apple.com/DOCUMENTATION/Cocoa/Reference/Foundation/Miscellaneous/Foundation_Functions/Reference/reference.html#//apple_ref/c/func/NSSearchPathForDirectoriesInDomains
# NSApplicationSupportDirectory = 14
# NSUserDomainMask = 1
# True for expanding the tilde into a fully qualified path
appdata = path.join(NSSearchPathForDirectoriesInDomains(14, 1, True)[0], APPNAME)
elif sys.platform == 'win32':
appdata = path.join(environ['APPDATA'], APPNAME)
else:
appdata = path.expanduser(path.join("~", "." + APPNAME))

There's a small module available that does exactly that:
https://pypi.org/project/appdirs/

You can try to use QSettings from Qt. You can obtain the path to your MyCompany/MyApp.ini file this way:
from PySide.QtCore import QSettings, QCoreApplication
QSettings.setDefaultFormat(QSettings.IniFormat)
QCoreApplication.setOrganizationName("MyCompany")
QCoreApplication.setApplicationName("MyApp")
settings = QSettings()
print(settings.fileName())
Alternatively, without changing any global state:
QSettings(
QSettings.IniFormat, QSettings.UserScope,
"MyCompany", "MyApp"
).fileName()
On Win7 you get something like:
C:\Users\MyUser\AppData\Roaming\MyCompany\MyApp.ini
On Linux (may vary):
/home/myuser/.config/MyCompany/MyApp.ini
I don't know the possible results for OSX (but I'd like to).
QSettings functionallity seem to be nice until you want to use registerFormat, which is not available in PySide, so there is no easy way to use YAML or JSON writers for settings.

Well, for Windows APPDATA (environmental variable) points to a user's "Application Data" folder. Not sure about OSX, though.
The correct way, in my opinion, is to do it on a per-platform basis.