What is the best way to do cross-platform handling of hidden files?
(preferably in Python, but other solutions still appreciated)
Simply checking for a leading '.' works for *nix/Mac, and file attributes work on Windows. However, this seems a little simplistic, and also doesn't account for alternative methods of hiding things (.hidden files, etc.). Is there a standard way to deal with this?
Here's a script that runs on Python 2.5+ and should do what you're looking for:
import ctypes
import os
def is_hidden(filepath):
name = os.path.basename(os.path.abspath(filepath))
return name.startswith('.') or has_hidden_attribute(filepath)
def has_hidden_attribute(filepath):
try:
attrs = ctypes.windll.kernel32.GetFileAttributesW(unicode(filepath))
assert attrs != -1
result = bool(attrs & 2)
except (AttributeError, AssertionError):
result = False
return result
I added something similar to has_hidden_attribute to jaraco.windows. If you have jaraco.windows >= 2.3:
from jaraco.windows import filesystem
def has_hidden_attribute(filepath):
return filesystem.GetFileAttributes(filepath).hidden
As Ben has pointed out, on Python 3.5, you can use the stdlib:
import os, stat
def has_hidden_attribute(filepath):
return bool(os.stat(filepath).st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)
Though you may still want to use jaraco.windows for the more Pythonic API.
Jason R. Coombs's answer is sufficient for Windows. And most POSIX GUI file managers/open dialogs/etc. probably follow the same "dot-prefix-means-hidden" convention as ls. But not Mac OS X.
There are at least four ways a file or directory can be hidden in Finder, file open panels, etc.:
Dot prefix.
HFS+ invisible attribute.
Finder Info hidden flag.
Matches a special blacklist built into CoreFoundation (which is different on each OS version—e.g., ~/Library is hidden in 10.7+, but not in 10.6).
Trying to write your own code to handle all of that is not going to be easy. And you'll have to keep it up-to-date, as I'm willing to bet the blacklist will change with most OS versions, Finder Info will eventually go from deprecated to completely unsupported, extended attributes may be supported more broadly than HFS+, …
But if you can require pyobjc (which is already included with recent Apple-supplied Python, and can be installed via pip otherwise), you can just call Apple's code:
import Foundation
def is_hidden(path):
url = Foundation.NSURL.fileURLWithPath_(path)
return url.getResourceValue_forKey_error_(None, Foundation.NSURLIsHiddenKey, None)[0]
def listdir_skipping_hidden(path):
url = Foundation.NSURL.fileURLWithPath_(path)
fm = Foundation.NSFileManager.defaultManager()
urls = fm.contentsOfDirectoryAtURL_includingPropertiesForKeys_options_error_(
url, [], Foundation.NSDirectoryEnumerationSkipsHiddenFiles, None)[0]
return [u.path() for u in urls]
This should work on any Python that pyobjc supports, on OS X 10.6+. If you want 10.5 or earlier, directory enumeration flags didn't exist yet, so the only option is something like filtering something like contentsOfDirectoryAtPath_error_ (or just os.listdir) on is_hidden.
If you have to get by without pyobjc, you can drop down to the CoreFoundation equivalents, and use ctypes. The key functions are CFURLCopyResourcePropertyForKey for is_hidden and CFURLEnumeratorCreateForDirectoryURL for listing a directory.
See http://pastebin.com/aCUwTumB for an implementation.
I've tested with:
OS X 10.6, 32-bit python.org 3.3.0
OS X 10.8, 32-bit Apple 2.7.2
OS X 10.8, 64-bit Apple 2.7.2
OS X 10.8, 64-bit python.org 3.3.0
It works as appropriate on each (e.g., it skips ~/Library on 10.8, but shows it on 10.6).
It should work on any OS X 10.6+ and any Python 2.6+. If you need OS X 10.5, you need to use the old APIs (or os.listdir) and filter on is_hidden. If you need Python 2.5, change the bytes checks to str checks (which of course breaks 3.x) and the with to an ugly try/finally or manual releasing.
If anyone plans on putting this code into a library, I would strongly suggest checking for pyobjc first (import Foundation and, if you don't get an ImportError you win), and only using the ctypes code if it's not available.
One last note:
Some people looking for this answer are trying to reinvent a wheel they don't need to.
Often, when people are doing something like this, they're building a GUI and want to, e.g., show a file browsers with an option to hide or show hidden files. Many of the popular cross-platform GUI frameworks (Qt, wx, etc.) have this support built in. (Also, many of them are open source, so you can read their code to see how they do it.)
That may not answer your question—e.g., they may just be passing a "filter hidden files" flag to the platform's native file-browser dialog, but you're trying to build a console-mode file-browser and can't do that. But if it does, just use it.
We actually address this in a project we write. What we do is have a number of different "hidden file checkers" that are registered with a main checker. We pass each file through these to see if it should be hidden or not.
These checkers are not only for different OS's etc, but we plug into version control "ignored" files, and optional user overrides by glob or regular expression.
It mostly amounts to what you have done, but in a pluggable, flexible and extensible way.
See source code here: https://bitbucket.org/aafshar/pida-main/src/tip/pida/services/filemanager/filemanager.py
Incorporating my previous answer as well as that from #abarnert, I've released jaraco.path 1.1 with cross-platform support for hidden file detection. With that package installed, to detect the hidden state of any file, simply invoke is_hidden:
from jaraco import path
path.is_hidden(file)
"Is there a standard way to deal with this?" Yes. Use a standard (i.e., POSIX-compliant) OS.
Since Windows is non-standard -- well -- there's no applicable standard. Wouldn't it be great if there was? I feel your pain.
Anything you try to do that's cross-platform like that will have Win32 oddities.
Your solution is -- for the present state of affairs -- excellent. At some point in the future, Microsoft may elect to write a POSIX-compliant OS. Until then, you're coping well with the situation.
Related
I'm using pyinstaller to distribute my code as executable within my team as most of them are not coding/scripting people and do not have Python Interpreter installed.
For some advanced usage of my tool, I want to make it possible for the user to implement a small custom function to adjust functionality slightly (for the few experienced people). Hence I want to let them input a python file which defines a function with a fixed name and a string as return.
Is that possible?
I mean the py-file could be drag/dropped for example, and I'd tell them that their user-defined function needs to have a certain name, e.g. "analyze()" - is it now possible to import that from the drag/dropped pythonfile within my PyInstaller Script and use it as this?
I know, it certainly will not be safe/secure and they could do evil things, delete files and so one... But that are things which we don#t care at this point, please no discussions about it. Thanks!
To answer my own question: yes it does actually work to import a module/function from a given path/pythonfile at runtime (that I knew already) even in PyInstaller (that was new for me).
I used this for my Py2.7 program:
f = r'C:\path\to\userdefined\filewithfunction.py'
if os.path.exists(f):
import imp
userdefined = imp.load_source('', f) # Only Python 2.x, for 3.x see: https://stackoverflow.com/a/67692/701049
print userdefined # just a debugging print
userdefined.imported() # here you should use try/catch; or check whether the function with the desired name really exists in the object "userdefined". This is only a small demo as example how to import, so didnt do it here.
filewithfunction.py:
--------------------
def imported():
print 'yes it worked :-)'
As written in the comments of the example code, you'll need a slightly different approach in Python 3.x. See this link: https://stackoverflow.com/a/67692/701049
Via
import win32com.client
wordapp = win32com.client.gencache.EnsureDispatch('Word.Application')
I can get a Word Application object documented e.g. here. However, ipython's autocompletion is not aware of that API, is there any way to add that?
Quick solution
Perhaps the simplest way to achieve code completion in IPython (tested with 6.2.1, see the answer below for a snippet that works with 7.1) and Jupyter is to run the following snippet:
from IPython.utils.generics import complete_object
import win32com.client
#complete_object.when_type(win32com.client.DispatchBaseClass)
def complete_dispatch_base_class(obj, prev_completions):
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
return list(ole_props) + prev_completions
except AttributeError:
pass
Short story long
With some more details being outlined in this guide, win32com ships with a script, makepy.py for generating Python types corresponding to the type library of a given COM object.
In the case of Word 2016, we would proceed as follows:
C:\Users\username\AppData\Local\Continuum\Anaconda3\pkgs\pywin32-221-py36h9c10281_0\Lib\site-packages\win32com\client>python makepy.py -i "Microsoft Word 16.0 Object Library"
Microsoft Word 16.0 Object Library
{00020905-0000-0000-C000-000000000046}, lcid=0, major=8, minor=7
>>> # Use these commands in Python code to auto generate .py support
>>> from win32com.client import gencache
>>> gencache.EnsureModule('{00020905-0000-0000-C000-000000000046}', 0, 8, 7)
The location of makepy.py will of course depend on your Python distribution. The script combrowse.py, available in the same directory, can be used to find the names of available type libraries.
With that in place, win32com.client will automatically make use of the generated types, rather than the raw IPyDispatch, and at this point, auto-completion is available in e.g. IPython or Jupyter, given that the COM object of interest actually publishes its available properties and methods (which is not a requirement).
Now, in your case, by invoking EnsureDispatch instead of Dispatch, the makepy part of the process is performed automatically, so you really should be able to obtain code completion in IPython for the published methods:
Note, though, that while this does give code completion for methods, the same will not be true for properties. It is possible to inspect those using the _prop_map_get_ attribute. For example, wordapp.Selection.Range.Font._prop_map_get_ gives all properties available on fonts.
If using IPython is not a strong requirement, note also that the PythonWin shell (located around \pkgs\pywin32\Lib\site-packages\pythonwin\Pythonwin.exe) has built-in code completion support for both properties and methods.
This, by itself, suggests that the same is achievable in IPython.
Concretely, the logic for auto-completion, which in turn relies on _prop_map_get_, can be found in scintilla.view.CScintillaView._AutoComplete. On the other hand, code completion in IPython 6.2.1 is handled by core.completer.IPCompleter. The API for adding custom code completers is provided by IPython.utils.generics.complete_object, as illustrated in the first solution above. One gotcha is that with complete_object being based on simplegeneric, only one completer may be provided for any given type. Luckily, all types generated by makepy will inherit from win32com.client.DispatchBaseClass.
If this turns out to ever be an issue, one can also circumvent complete_object entirely and simply manually patch IPython by adding the following five lines to core.completer.Completion.attr_matches:
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
words += list(ole_props)
except AttributeError:
pass
Conversely, IPython bases its code-completion on __dir__, so one could also patch gencache, which is where the code generation ultimately happens, to include something to like
def __dir__(self):
return list(set(self._prop_map_get_).union(set(self._prop_map_put_)))
to each generated DispatchBaseClass.
fuglede's answer is great, just want to update it for the newest versions of IPython (7.1+).
Since IPython.utils.generics has changes from using simplegeneric to using functools, the #complete_object.when_type method should be changed to #complete_object.register. So his initial code should changed to:
from IPython.utils.generics import complete_object
import win32com.client
#complete_object.register(win32com.client.DispatchBaseClass)
def complete_dispatch_base_class(obj, prev_completions):
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
return list(ole_props) + prev_completions
except AttributeError:
pass
At the beginning of the script, I use platform.system and platform.release to determine which OS and version the script is running on (so it knows it's data is in Application Support on Mac, home on unix-like and non-mac unix, appdata on windows <= XP, and appdata/roaming on windows >= Vista). I'd like to test my series of ifs, elifs, and elses what determine the os and release, but I only have access to Mac 10.6.7, some unknown release of Linux, and Windows 7. Is there a way to feed platform fake system and release information so I can be sure XP, Solaris, etc, would handle the script properly without having an installation?
Maybe something like
>>> platform.system = lambda: "whatever"
>>> platform.system()
'whatever'
you could create your initialization functions to take those variables as parameters so it is easy to spoof them in testing
You probably want to explore mocking platform for your testing. Alternatively, you could directly monkey patch platform, or even mess with sys.modules directly to override the default platform module, but mock is already designed to be self contained and also has the benefit of pretty clearly showing in your code what is and is not test instrumentation, so you don't accidentally get test functionality released in your production code.
For me, directly patching platform.system with pytest-mock did not work. A simple trick though is to abstract the retrieval of this information in a utility function of your own (which you can successfully patch with pytest-mock this time).
Hence, in your implementation module/class:
def _get_system() -> str:
return platform.system().lower()
def _is_windows() -> bool:
return _get_system() == 'windows'
And in your test, here using pytest + pytest-mock:
def test_win_no_pad_code():
with patch('module_name._get_system', MagicMock(return_value='windows')):
assert module_name._is_windows()
This is one way of doing using module, but you can do the equivalent using a class instead.
Given a string, I'd like to be able to send a set of keystrokes to type that string and I'd like to be able to do it in python on OSX (in python because it's part of a larger project already written in python, on OSX because I am trying to port it to OSX).
I'm able to do this now using pyobj like so (this is somewhat simplified):
from Quartz import *
CHAR_TO_SEQUENCE = {
'a':[(0, True), (0, False)]
}
def send_string(string):
for c in string:
sequence = CHAR_TO_SEQUENCE[c]
for keycode, key_down in sequence:
CGEventPost(kCGSessionEventTap, CGEventCreateKeyboardEvent(None, keycode, key_down))
and I've fleshed out CHAR_TO_SEQUENCE to include most of what I can type on my keyboard, which took a while and was tedious.
The problems with this are:
- It only works while the keyboard has the ANSII standard layout. If someone uses a french keyboard, for example, it will type the wrong things.
- It requires this ridiculous table.
I found this general solution for OSX but couldn't figure out how to apply it to python:
How to convert ASCII character to CGKeyCode?
The API mentioned there doesn't seem to be available via pyobj (or maybe I just couldn't figure out the import path).
I've seen some suggestions to set CGEventKeyboardSetUnicodeString to the desired string and not worry about the keycode. But I wasn't able to figure out how to call CGEventKeyboardSetUnicodeString from python (I can't get the arguments right) and it's not clear that this will work because the documentation says that applications can choose to ignore this in favor of the keycode.
Is there a way to do this in python on OSX?
It looks like the Carbon modules don't wrap the TIS* functions, and neither does anything else.
You could extend PyObjC, but it's much simpler to just build a trivial extension module that wraps the two functions you actually need.
Since it was faster to just do it than to explain how to do it, you can get it from https://github.com/abarnert/pykeycode and just do the usual "python setup.py build_ext --inplace" or "sudo python setup.py install", then look at test.py to see how to use it.
I have a python desktop application that needs to store user data. On Windows, this is usually in %USERPROFILE%\Application Data\AppName\, on OSX it's usually ~/Library/Application Support/AppName/, and on other *nixes it's usually ~/.appname/.
There exists a function in the standard library, os.path.expanduser that will get me a user's home directory, but I know that on Windows, at least, "Application Data" is localized into the user's language. That might be true for OSX as well.
What is the correct way to get this location?
UPDATE:
Some further research indicates that the correct way to get this on OSX is by using the function NSSearchPathDirectory, but that's Cocoa, so it means calling the PyObjC bridge...
Well, I hate to have been the one to answer my own question, but no one else seems to know. I'm leaving the answer for posterity.
APPNAME = "MyApp"
import sys
from os import path, environ
if sys.platform == 'darwin':
from AppKit import NSSearchPathForDirectoriesInDomains
# http://developer.apple.com/DOCUMENTATION/Cocoa/Reference/Foundation/Miscellaneous/Foundation_Functions/Reference/reference.html#//apple_ref/c/func/NSSearchPathForDirectoriesInDomains
# NSApplicationSupportDirectory = 14
# NSUserDomainMask = 1
# True for expanding the tilde into a fully qualified path
appdata = path.join(NSSearchPathForDirectoriesInDomains(14, 1, True)[0], APPNAME)
elif sys.platform == 'win32':
appdata = path.join(environ['APPDATA'], APPNAME)
else:
appdata = path.expanduser(path.join("~", "." + APPNAME))
There's a small module available that does exactly that:
https://pypi.org/project/appdirs/
You can try to use QSettings from Qt. You can obtain the path to your MyCompany/MyApp.ini file this way:
from PySide.QtCore import QSettings, QCoreApplication
QSettings.setDefaultFormat(QSettings.IniFormat)
QCoreApplication.setOrganizationName("MyCompany")
QCoreApplication.setApplicationName("MyApp")
settings = QSettings()
print(settings.fileName())
Alternatively, without changing any global state:
QSettings(
QSettings.IniFormat, QSettings.UserScope,
"MyCompany", "MyApp"
).fileName()
On Win7 you get something like:
C:\Users\MyUser\AppData\Roaming\MyCompany\MyApp.ini
On Linux (may vary):
/home/myuser/.config/MyCompany/MyApp.ini
I don't know the possible results for OSX (but I'd like to).
QSettings functionallity seem to be nice until you want to use registerFormat, which is not available in PySide, so there is no easy way to use YAML or JSON writers for settings.
Well, for Windows APPDATA (environmental variable) points to a user's "Application Data" folder. Not sure about OSX, though.
The correct way, in my opinion, is to do it on a per-platform basis.