We Keep Coding

How to get pathlib’s unix path handling goodies without the system specific ones, eg Path.resolve() changes /tmp to /private/tmp

Hit an interesting case with pathlib where I am using it to represent paths on a android device, not the machine the python is actively running on. Is it still possible to make use of pathlib’s sugary syntax and ability to resolve unix-wide truisms such as relative paths “../../“ without resolving device specific symlink things like “/tmp/path” -> “/private/tmp/path”?
For the most part, pathlib makes handling paths on the device super easy, but i run into problems when wanting to resolve a path on the device without using the host python machine’s symlink resolving mechanisms.
I love using pathlib instead of strings. Is it possible to do path manipulations and actions with the Path object but then send off the final file command to the device ssh’d into?
The only piece from resolve i’d like is the ability to turn (Path(“/tmp/analysis/ptool”) / “../../“).resolve take care of the .. but not change /tmp to /private/tmp, which is not a symlink on the device i’m about to use this path command with.
The example is a little trivial “just do it with strings”, but the organization and automation i want to apply this on is significantly cleaner and easier to read as Path objects instead of strings. If possible, it would be great to make it work.
In [1]: import pathlib
In [2]: from pathlib import Path
In [3]: Path('/tmp/hello').resolve()
Out[3]: PosixPath('/private/tmp/hello')
MacOS, Python3.7.3

/private/tmp is where /tmp contents are actually stored on Mac OS. On this platform, /tmp is a symlink. You can see this from the operating system, without using Python at all:
$ ls -l /tmp
lrwxr-xr-x 1 root wheel 11 Oct 22 2018 /tmp -> private/tmp
$ (cd /tmp && pwd -P)
/private/tmp
pathlib.Path.resolve() replaces symlinks with the absolute paths they point to, so it's doing exactly what you're asking for.
See Why is /tmp a symlink to /private/tmp? on our sister site Ask Different.

using a whole Path in this case seems wrong as it exposes methods like stat which will try and operate on your local system, rather than the remote device. using a PurePosixPath looks better as it doesn't expose as much, and it's easy to add in a resolve method in a derived class:
from pathlib import PurePosixPath
class ResolvingPosixPath(PurePosixPath):
def resolve(self):
if not self.is_absolute():
raise ValueError('only able to resolve absolute paths')
sep = self._flavour.sep
path = ''
for name in str(self).split(sep):
if not name or name == '.':
# current dir
continue
if name == '..':
# parent dir
path, _, _ = path.rpartition(sep)
continue
path = path + sep + name
return path or sep
I've ripped the relevant code from pathlib._PosixFlavour.resolve, and
you're obviously free to use shorter class names if that's convenient!
this can be used as you suggest you want:
hello = ResolvingPosixPath('/tmp') / 'hello'
print(hello.resolve())
giving me '/tmp/hello'

current working directory mismatch in python

I have imported a existing filesystem folder as the new project folder in eclipse. I have a script which get the current working directory path of the code. I need to change directory location to acccess files in other directory related to it. But It is giving different value when executed from eclipse and from the command line. Location is same in both place. Please help me resolve this issue. Operating system is windows here
import os
print os.getcwd()
os.chdir(os.path.dirname(os.getcwd()))
print os.getcwd()
One result is this
C:\Automation\trunk\Base\TestScripts
C:\Automation\trunk\Base
Other result is this
C:\Automation\trunk\UsefulScripts
C:\Automation\trunk
Second result is the one I expect, and that is where the code is located exactly.

watch out you cannot rely on that. Do the following :
basedir = os.environ.get('PROJECT_LOC', None)
if not basedir:
basedir = sys.path[0] # We are on commandline. sys.path OK
Then use basedir to find your files
Update
you have to specify this variable in the runtime of the interpreter
window->preferences->PyDev->Interpreters->Python Interpreter TAB (environment) There you can specify PROJECT_LOC referring to project_loc by selecting NEW with name PROJECT_LOC and variable (the other button) and selecting project_loc.
For some reasons those variables are not visible in python.
You can check this now with
def read_all_variables():
for key in os.environ.keys():
print ("%30s %s" % (key,os.environ[key]))
PROJECT_LOC should be there now

I have used the sys package instead of os. It works as expected.
import os,sys
currentpath = sys.path[0]
print currentpath
I can able to run it from eclipse and also from command line to get the correct path. Thanks for the help .

Setting values persisting over python sessions

I'd like to define a constant in my script like that path to my Dropbox folder. Most my scripts will try to load some data of Dropbox which is shared among my PCs, but I find that between Mac and Ubuntu the prefix is different (/Users/<user>/Dropbox versus /home/<user>/Dropbox).
Is there a way to save this kind of information in some variable that will be loaded in each session such that I could have a global variable like DROPBOX (what would be a good convention, __DROPBOX__?) as path prefix to a file name, e.g. fname = DROPBOX + "myfile.txt".
Kind of reminds of me defining this in one's .Rprofile which holds settings in R.
Or is there a better way to handle this?

You could use the built in environment variables to get the path to the user home directory:
import os
print os.environ['HOME']
Which would solve your problem is a way that is more likely to remain stable if run on a new machine.

How about this:
os.path.expanduser('~/Dropbox')
or you can just try different alternatives:
dirs_to_try = ('/Users/Guido/Dropbox', '/home/Guido/Dropbox')
for path in dirs_to_try:
if os.path.isdir(path):
break
finally:
print 'cannot find Dropbox directory'
path = None

How do I store desktop application data in a cross platform way for python?

I have a python desktop application that needs to store user data. On Windows, this is usually in %USERPROFILE%\Application Data\AppName\, on OSX it's usually ~/Library/Application Support/AppName/, and on other *nixes it's usually ~/.appname/.
There exists a function in the standard library, os.path.expanduser that will get me a user's home directory, but I know that on Windows, at least, "Application Data" is localized into the user's language. That might be true for OSX as well.
What is the correct way to get this location?
UPDATE:
Some further research indicates that the correct way to get this on OSX is by using the function NSSearchPathDirectory, but that's Cocoa, so it means calling the PyObjC bridge...

Well, I hate to have been the one to answer my own question, but no one else seems to know. I'm leaving the answer for posterity.
APPNAME = "MyApp"
import sys
from os import path, environ
if sys.platform == 'darwin':
from AppKit import NSSearchPathForDirectoriesInDomains
# http://developer.apple.com/DOCUMENTATION/Cocoa/Reference/Foundation/Miscellaneous/Foundation_Functions/Reference/reference.html#//apple_ref/c/func/NSSearchPathForDirectoriesInDomains
# NSApplicationSupportDirectory = 14
# NSUserDomainMask = 1
# True for expanding the tilde into a fully qualified path
appdata = path.join(NSSearchPathForDirectoriesInDomains(14, 1, True)[0], APPNAME)
elif sys.platform == 'win32':
appdata = path.join(environ['APPDATA'], APPNAME)
else:
appdata = path.expanduser(path.join("~", "." + APPNAME))

There's a small module available that does exactly that:
https://pypi.org/project/appdirs/

You can try to use QSettings from Qt. You can obtain the path to your MyCompany/MyApp.ini file this way:
from PySide.QtCore import QSettings, QCoreApplication
QSettings.setDefaultFormat(QSettings.IniFormat)
QCoreApplication.setOrganizationName("MyCompany")
QCoreApplication.setApplicationName("MyApp")
settings = QSettings()
print(settings.fileName())
Alternatively, without changing any global state:
QSettings(
QSettings.IniFormat, QSettings.UserScope,
"MyCompany", "MyApp"
).fileName()
On Win7 you get something like:
C:\Users\MyUser\AppData\Roaming\MyCompany\MyApp.ini
On Linux (may vary):
/home/myuser/.config/MyCompany/MyApp.ini
I don't know the possible results for OSX (but I'd like to).
QSettings functionallity seem to be nice until you want to use registerFormat, which is not available in PySide, so there is no easy way to use YAML or JSON writers for settings.

Well, for Windows APPDATA (environmental variable) points to a user's "Application Data" folder. Not sure about OSX, though.
The correct way, in my opinion, is to do it on a per-platform basis.

Accounting for a changing path

In relation to another question, how do you account for paths that may change? For example, if a program is calling a file in the same directory as the program, you can simply use the path ".\foo.py" in *nix. However, apparently Windows likes to have the path hard-coded, e.g. "C:\Python_project\foo.py".
What happens if the path changes? For example, the file may not be on the C: drive but on a thumb drive or external drive that can change the drive letter. The file may still be in the same directory as the program but it won't match the drive letter in the code.
I want the program to be cross-platform, but I expect I may have to use os.name or something to determine which path code block to use.

Simple answer: You work out the absolute path based on the environment.
What you really need is a few pointers. There are various bits of runtime and environment information that you can glean from various places in the standard library (and they certainly help me when I want to deploy an application on windows).
So, first some general things:
os.path - standard library module with lots of cross-platform path manipulation. Your best friend. "Follow the os.path" I once read in a book.
__file__ - The location of the current module.
sys.executable - The location of the running Python.
Now you can fairly much glean anything you want from these three sources. The functions from os.path will help you get around the tree:
os.path.join('path1', 'path2') - join path segments in a cross-platform way
os.path.expanduser('a_path') - find the path a_path in the user's home directory
os.path.abspath('a_path') - convert a relative path to an absolute path
os.path.dirname('a_path') - get the directory that a path is in
many many more...
So combining this, for example:
# script1.py
# Get the path to the script2.py in the same directory
import os
this_script_path = os.path.abspath(__file__)
this_dir_path = os.path.dirname(this_script_path)
script2_path = os.path.join(this_dir_path, 'script2.py')
print script2_path
And running it:
ali#work:~/tmp$ python script1.py
/home/ali/tmp/script2.py
Now for your specific case, it seems you are slightly confused between the concept of a "working directory" and the "directory that a script is in". These can be the same, but they can also be different. For example the "working directory" can be changed, and so functions that use it might be able to find what they are looking for sometimes but not others. subprocess.Popen is an example of this.
If you always pass paths absolutely, you will never get into working directory issues.

If your file is always in the same directory as your program then:
def _isInProductionMode():
""" returns True when running the exe,
False when running from a script, ie development mode.
"""
return (hasattr(sys, "frozen") or # new py2exe
hasattr(sys, "importers") # old py2exe
or imp.is_frozen("__main__")) #tools/freeze
def _getAppDir():
""" returns the directory name of the script or the directory
name of the exe
"""
if _isInProductionMode():
return os.path.dirname(sys.executable)
return os.path.dirname(__file__)
should work. Also, I've used py2exe for my own application, and haven't tested it with other exe conversion apps.

What -- specifically -- do you mean by "calling a file...foo.py"?
Import? If so, the path is totally outside of your program. Set the PYTHONPATH environment variable with . or c:\ or whatever at the shell level. You can, for example, write 2-line shell scripts to set an environment variable and run Python.
Windows
SET PYTHONPATH=C:\path\to\library
python myapp.py
Linux
export PYTHONPATH=./relative/path
python myapp.py
Execfile? Consider using import.
Read and Eval? Consider using import.
If the PYTHONPATH is too complicated, then put your module in the Python lib/site-packages directory, where it's put onto the PYTHONPATH by default for you.

I figured out by using os.getcwd(). I also learned about using os.path.join to automatically determine the correct path format based on the OS. Here's the code:
def openNewRecord(self, event): # wxGlade: CharSheet.<event_handler>
"""Create a new, blank record sheet."""
path = os.getcwd()
subprocess.Popen(os.path.join(path, "TW2K_char_rec_sheet.py"), shell=True).stdout
It appears to be working. Thanks for the ideas.