I want to write a simple unit test with os.path.basename
Example
def test(path):
return os.path.basename(os.path.normpath(title))
...
result = test('/path/foo')
assert result == 'foo'
result = test(r'c:\Users\Foo\Documents\foo')
assert result == 'foo'
Problem
Running on linux the second test (windows path) is failing.
I guess the first test will fail on windows.
This actually makes pretty much sense since there are different os.path modules
Python Documentation
Since different operating systems have different path name conventions, there are several versions of this module in the standard library.
Question
Is there a way to import a specific version of os.path?
I already tried to set sys.platform to win32
Of course I could check the current platform and just run one of both tests - but I was wondering if there is a way to run both tests.
You should always use / as the path separator. Python will translate it to the correct separator for your platform.
\ is used to start escape sequences. Using / will avoid having to use raw strings to disable \ escape sequences (as in the second string in your post).
Using \ in raw strings may also cause os.path methods to behave oddly (and will only work on Windows platforms).
Related
I'm looking for a way to test, in my python script, if said script is running from Ansible so I can also run it through shell (for running unit tests etc). Calling AnsibleModule without calling from an ansible playbook will just endlessly wait for a response that will never come.
I'm expecting that there isn't a simple test and that I have to restructure in some way, but I'm open to any options.
def main():
# must test if running via ansible before next line
module = AnsibleModule(
argument_spec=dict(
server=dict(required=True, type='str'),
[...]
)
[... do things ...]
)
if __name__ == "__main__":
if running_via_ansible:
main()
else:
run_tests()
I believe there are a couple of answers, with various levels of trickery involved
since your module is written in python, ansible will use the AnsiballZ framework to run it, which means its sys.argv[0] will start with AnsiballZ_; it will also likely be written to $HOME/.ansible/tmp on the target machine, so one could sniff for .ansible/tmp showing up in argv[0] also
if the file contains the string WANT_JSON in it, then ansible will invoke it with the module's JSON payload as the first argument instead of feeding it JSON on sys.stdin (thus far the filename has been colocated with the AnsiballZ_ script, but I don't know that such a thing is guaranteed)
Similar, although apparently far more python specific: if it contains a triple-quoted sentinel """<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>""" (or the ''' flavor works, too) then that magic string is replaced by the serialized JSON that, again, would have been otherwise provided via stdin
While this may not apply, or be helpful, I actually would expect that any local testing environment would have more "fingerprints" than trying to detect the opposite, and has the pleasing side-effect of "failing open" in that the module will assume it is running in production mode unless it can prove testing mode, which should make for less weird false positives. Then again, I guess the reasonable default depends on how problematic it would be for the module to attempt to carry out its payload when not really in use
At the beginning of the script, I use platform.system and platform.release to determine which OS and version the script is running on (so it knows it's data is in Application Support on Mac, home on unix-like and non-mac unix, appdata on windows <= XP, and appdata/roaming on windows >= Vista). I'd like to test my series of ifs, elifs, and elses what determine the os and release, but I only have access to Mac 10.6.7, some unknown release of Linux, and Windows 7. Is there a way to feed platform fake system and release information so I can be sure XP, Solaris, etc, would handle the script properly without having an installation?
Maybe something like
>>> platform.system = lambda: "whatever"
>>> platform.system()
'whatever'
you could create your initialization functions to take those variables as parameters so it is easy to spoof them in testing
You probably want to explore mocking platform for your testing. Alternatively, you could directly monkey patch platform, or even mess with sys.modules directly to override the default platform module, but mock is already designed to be self contained and also has the benefit of pretty clearly showing in your code what is and is not test instrumentation, so you don't accidentally get test functionality released in your production code.
For me, directly patching platform.system with pytest-mock did not work. A simple trick though is to abstract the retrieval of this information in a utility function of your own (which you can successfully patch with pytest-mock this time).
Hence, in your implementation module/class:
def _get_system() -> str:
return platform.system().lower()
def _is_windows() -> bool:
return _get_system() == 'windows'
And in your test, here using pytest + pytest-mock:
def test_win_no_pad_code():
with patch('module_name._get_system', MagicMock(return_value='windows')):
assert module_name._is_windows()
This is one way of doing using module, but you can do the equivalent using a class instead.
In order to detect if console, correctly sys.stderr or sys.stdout, I was doing the following test:
if hasattr(sys.stderr, "isatty") and sys.stderr.isatty():
if platform.system()=='Windows':
# win code (ANSI not supported but there are alternatives)
else:
# use ANSI escapes
else:
# no colors, usually this is when you redirect the output to a file
Now the problem became more complex while running this Python code via an IDE (like PyCharm). Recently PyCharm added support for ANSI, but the first test fails: it has the isatty attribute but it is set to False.
I want to modify the logic so it will properly detect if the output supports ANSI coloring. One requirement is that under no circumstance I should output something out when the output is redirected to a file (for console it would be acceptable).
Update
Added more complex ANSI test script at https://gist.github.com/1316877
Django users can use django.core.management.color.supports_color function.
if supports_color():
...
The code they use is:
def supports_color():
"""
Returns True if the running system's terminal supports color, and False
otherwise.
"""
plat = sys.platform
supported_platform = plat != 'Pocket PC' and (plat != 'win32' or
'ANSICON' in os.environ)
# isatty is not always implemented, #6223.
is_a_tty = hasattr(sys.stdout, 'isatty') and sys.stdout.isatty()
return supported_platform and is_a_tty
See https://github.com/django/django/blob/master/django/core/management/color.py
I can tell you how others have solved this problem, but it's not pretty. If you look at ncurses as an example (which needs to be able to run on all kinds of different terminals), you'll see that they use a terminal capabilities database to store every kind of terminal and its capabilities. The point being, even they were never able to automatically "detect" these things.
I don't know if there's a cross-platform termcap, but it's probably worth your time to look for it. Even if it's out there though, it may not have your terminal listed and you may have to manually add it.
>>> os.path.basename('http://example.com/file.txt')
'file.txt'
.. and I thought os.path.* work only on local paths and not URLs? Note that the above example was run on Windows too .. with similar result.
In practice many functions of os.path are just string manipulation functions (which just happen to be especially handy for path manipulation) -- and since that's innocuous and occasionally handy, while formally speaking "incorrect", I doubt this will change anytime soon -- for more details, use the following simple one-liner at a shell/command prompt:
$ python -c"import sys; import StringIO; x = StringIO.StringIO(); sys.stdout = x; import this; sys.stdout = sys.__stdout__; print x.getvalue().splitlines()[10][9:]"
Or, for Python 3:
$ python -c"import sys; import io; x = io.StringIO(); sys.stdout = x; import this; sys.stdout = sys.__stdout__; print(x.getvalue().splitlines()[10][9:])"
On windows, look at the source code: C:\Python25\Lib\ntpath.py
def basename(p):
"""Returns the final component of a pathname"""
return split(p)[1]
os.path.split (in the same file) just split "\" (and sth. else)
Beware of URLs with parameters, anchors or anything that isn't a "plain" URL:
>>> import os.path
>>> os.path.basename("protocol://fully.qualifie.host/path/to/file.txt")
'file.txt'
>>> os.path.basename("protocol://fully.qualifie.host/path/to/file.txt?param1¶m1#anchor")
'file.txt?param1¶m1#anchor'
Use the source Luke:
def basename(p):
"""Returns the final component of a pathname"""
i = p.rfind('/') + 1
return p[i:]
Edit (response to clarification):
It works for URLs by accident, that's it. Because of that, exploiting its behaviour could be considered code smell by some.
Trying to "fix" it (check if passed path is not url) is also surprisingly difficult
www.google.com/test.php
me#other.place.com/12
./src/bin/doc/goto.c
are at the same time correct pathnames and URLs (relative), so is the http:/hello.txt (one /, and only on linux, and it's kinda stupid :)). You could "fix" it for absolute urls but relative ones will still work. Handling one special case in differently is a big no no in the python world.
To sum it up: import this
Forward slash is also an acceptable path delimiter in Windows.
It is merely that the command line does not accept paths that begin with a / because that character is reserved for args switches.
Why? Because it's useful for parsing URLs as well as local file paths. Why not?
What is the best way to do cross-platform handling of hidden files?
(preferably in Python, but other solutions still appreciated)
Simply checking for a leading '.' works for *nix/Mac, and file attributes work on Windows. However, this seems a little simplistic, and also doesn't account for alternative methods of hiding things (.hidden files, etc.). Is there a standard way to deal with this?
Here's a script that runs on Python 2.5+ and should do what you're looking for:
import ctypes
import os
def is_hidden(filepath):
name = os.path.basename(os.path.abspath(filepath))
return name.startswith('.') or has_hidden_attribute(filepath)
def has_hidden_attribute(filepath):
try:
attrs = ctypes.windll.kernel32.GetFileAttributesW(unicode(filepath))
assert attrs != -1
result = bool(attrs & 2)
except (AttributeError, AssertionError):
result = False
return result
I added something similar to has_hidden_attribute to jaraco.windows. If you have jaraco.windows >= 2.3:
from jaraco.windows import filesystem
def has_hidden_attribute(filepath):
return filesystem.GetFileAttributes(filepath).hidden
As Ben has pointed out, on Python 3.5, you can use the stdlib:
import os, stat
def has_hidden_attribute(filepath):
return bool(os.stat(filepath).st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)
Though you may still want to use jaraco.windows for the more Pythonic API.
Jason R. Coombs's answer is sufficient for Windows. And most POSIX GUI file managers/open dialogs/etc. probably follow the same "dot-prefix-means-hidden" convention as ls. But not Mac OS X.
There are at least four ways a file or directory can be hidden in Finder, file open panels, etc.:
Dot prefix.
HFS+ invisible attribute.
Finder Info hidden flag.
Matches a special blacklist built into CoreFoundation (which is different on each OS version—e.g., ~/Library is hidden in 10.7+, but not in 10.6).
Trying to write your own code to handle all of that is not going to be easy. And you'll have to keep it up-to-date, as I'm willing to bet the blacklist will change with most OS versions, Finder Info will eventually go from deprecated to completely unsupported, extended attributes may be supported more broadly than HFS+, …
But if you can require pyobjc (which is already included with recent Apple-supplied Python, and can be installed via pip otherwise), you can just call Apple's code:
import Foundation
def is_hidden(path):
url = Foundation.NSURL.fileURLWithPath_(path)
return url.getResourceValue_forKey_error_(None, Foundation.NSURLIsHiddenKey, None)[0]
def listdir_skipping_hidden(path):
url = Foundation.NSURL.fileURLWithPath_(path)
fm = Foundation.NSFileManager.defaultManager()
urls = fm.contentsOfDirectoryAtURL_includingPropertiesForKeys_options_error_(
url, [], Foundation.NSDirectoryEnumerationSkipsHiddenFiles, None)[0]
return [u.path() for u in urls]
This should work on any Python that pyobjc supports, on OS X 10.6+. If you want 10.5 or earlier, directory enumeration flags didn't exist yet, so the only option is something like filtering something like contentsOfDirectoryAtPath_error_ (or just os.listdir) on is_hidden.
If you have to get by without pyobjc, you can drop down to the CoreFoundation equivalents, and use ctypes. The key functions are CFURLCopyResourcePropertyForKey for is_hidden and CFURLEnumeratorCreateForDirectoryURL for listing a directory.
See http://pastebin.com/aCUwTumB for an implementation.
I've tested with:
OS X 10.6, 32-bit python.org 3.3.0
OS X 10.8, 32-bit Apple 2.7.2
OS X 10.8, 64-bit Apple 2.7.2
OS X 10.8, 64-bit python.org 3.3.0
It works as appropriate on each (e.g., it skips ~/Library on 10.8, but shows it on 10.6).
It should work on any OS X 10.6+ and any Python 2.6+. If you need OS X 10.5, you need to use the old APIs (or os.listdir) and filter on is_hidden. If you need Python 2.5, change the bytes checks to str checks (which of course breaks 3.x) and the with to an ugly try/finally or manual releasing.
If anyone plans on putting this code into a library, I would strongly suggest checking for pyobjc first (import Foundation and, if you don't get an ImportError you win), and only using the ctypes code if it's not available.
One last note:
Some people looking for this answer are trying to reinvent a wheel they don't need to.
Often, when people are doing something like this, they're building a GUI and want to, e.g., show a file browsers with an option to hide or show hidden files. Many of the popular cross-platform GUI frameworks (Qt, wx, etc.) have this support built in. (Also, many of them are open source, so you can read their code to see how they do it.)
That may not answer your question—e.g., they may just be passing a "filter hidden files" flag to the platform's native file-browser dialog, but you're trying to build a console-mode file-browser and can't do that. But if it does, just use it.
We actually address this in a project we write. What we do is have a number of different "hidden file checkers" that are registered with a main checker. We pass each file through these to see if it should be hidden or not.
These checkers are not only for different OS's etc, but we plug into version control "ignored" files, and optional user overrides by glob or regular expression.
It mostly amounts to what you have done, but in a pluggable, flexible and extensible way.
See source code here: https://bitbucket.org/aafshar/pida-main/src/tip/pida/services/filemanager/filemanager.py
Incorporating my previous answer as well as that from #abarnert, I've released jaraco.path 1.1 with cross-platform support for hidden file detection. With that package installed, to detect the hidden state of any file, simply invoke is_hidden:
from jaraco import path
path.is_hidden(file)
"Is there a standard way to deal with this?" Yes. Use a standard (i.e., POSIX-compliant) OS.
Since Windows is non-standard -- well -- there's no applicable standard. Wouldn't it be great if there was? I feel your pain.
Anything you try to do that's cross-platform like that will have Win32 oddities.
Your solution is -- for the present state of affairs -- excellent. At some point in the future, Microsoft may elect to write a POSIX-compliant OS. Until then, you're coping well with the situation.