We Keep Coding

Is it possible to display file size in a directory served using http.server in python?

I've served a directory using
python -m http.server
It works well, but it only shows file names. Is it possible to show created/modified dates and file size, like you see in ftp servers?
I looked through the documentation for the module but couldn't find anything related to it.
Thanks!

http.server is meant for dead-simple use cases, and to serve as sample code.1 That's why the docs link right to the source.
That means that, by design, it doesn't have a lot of configuration settings; instead, you configure it by reading the source and choosing what methods you want to override, then building a subclass that does that.
In this case, what you want to override is list_directory. You can see how the base-class version works, and write your own version that does other stuff—either use scandir instead of listdir, or just call stat on each file, and then work out how you want to cram the results into the custom-built HTML.
Since there's little point in doing this except as a learning exercise, I won't give you complete code, but here's a skeleton:
class StattyServer(http.server.HTTPServer):
def list_directory(self, path):
try:
dirents = os.scandir(path)
except OSError:
# blah blah blah
# etc. up to the end of the header-creating bit
for dirent in dirents:
fullname = dirent.path
displayname = linkname = dirent.name
st = dirent.stat()
# pull stuff out of st
# build a table row to append to r
1. Although really, it's sample code for an obsolete and clunky way of building servers, so maybe that should be "to serve as sample code to understand legacy code that you probably won't ever need to look at but just in case…".

How can I test the standard input and standard output in Python Script with a Unittest test?

I'm trying to test a Python script (2.7) where I work with the standar input (readed with raw_input() and writed with a simple print) but I don't find how do this and I'm sure that this issue is very simple.
This is a very very very resume code of my script:
def example():
number = raw_input()
print number
if __name__ == '__main__':
example()
I want to write a unittest test to check this, but I don't find how. I've trying with StringIO and other things but I don't find the solution to do this really simple.
Somebody have a idea?
PD: Of course in the real script I use data blocks with several lines and other kind of data.
Thank you so much.
EDIT:
Thank you so much for the first really specific answer, it works perfectly, only I've had a little problem importing StringIO, because I was doing import StringIO and I needed to import like from StringIO import StringIO (I don't understand really why), but be that as It may, it works.
But I I've found another problem using this way, in my project I need test a scripts with this way (that work perfectly thanks to your support) but I want do this:
I have a file with a lot of test to pass over a script, so I open the file and read blocks of info with their result blocks and I would like to do that the code will be able to process a block checking their result and do the same with other and another...
Something like this:
class Test(unittest.TestCase):
...
#open file and process saving data like datablocks and results
...
allTest = True
for test in tests:
stub_stdin(self, test.dataBlock)
stub_stdouts(self)
runScrip()
if sys.stdout.getvalue() != test.expectResult:
allTest = False
self.assertEqual(allTest, True)
I know that maybe unittest doesn't has sense now, but you can do a idea about I want. So, this way fails and I don't know why.

Typical techniques involve mocking the standard sys.stdin and sys.stdout with your desired items. If you do not care for Python 3 compatibility you can just use the StringIO module, however if you want forward thinking and is willing to restrict to Python 2.7 and 3.3+, supporting for this both Python 2 and 3 in this way becomes possible without too much work through the io module (but requires a bit of modification, but put this thought on hold for now).
Assuming you already have a unittest.TestCase going, you can create a utility function (or method in the same class) that will replace sys.stdin/sys.stdout as outlined. First the imports:
import sys
import io
import unittest
In one of my recent projects I've done this for stdin, where it take a str for the inputs that the user (or another program through pipes) will enter into yours as stdin:
def stub_stdin(testcase_inst, inputs):
stdin = sys.stdin
def cleanup():
sys.stdin = stdin
testcase_inst.addCleanup(cleanup)
sys.stdin = StringIO(inputs)
As for stdout and stderr:
def stub_stdouts(testcase_inst):
stderr = sys.stderr
stdout = sys.stdout
def cleanup():
sys.stderr = stderr
sys.stdout = stdout
testcase_inst.addCleanup(cleanup)
sys.stderr = StringIO()
sys.stdout = StringIO()
Note that in both cases, it accepts a testcase instance, and calls its addCleanup method that adds the cleanup function call that will reset them back to where they were when the duration of a test method is concluded. The effect is that for the duration from when this was invoked in the test case until the end, sys.stdout and friends will be replaced with the io.StringIO version, meaning you can check its value easily, and don't have to worry about leaving a mess behind.
Better to show this as an example. To use this, you can simply create a test case like so:
class ExampleTestCase(unittest.TestCase):
def test_example(self):
stub_stdin(self, '42')
stub_stdouts(self)
example()
self.assertEqual(sys.stdout.getvalue(), '42\n')
Now, in Python 2, this test will only pass if the StringIO class is from the StringIO module, and in Python 3 no such module exists. What you can do is use the version from the io module with a modification that makes it slightly more lenient in terms of what input it accepts, so that the unicode encoding/decoding will be done automatically rather than triggering an exception (such as print statements in Python 2 will not work nicely without the following). I typically do this for cross compatibility between Python 2 and 3:
class StringIO(io.StringIO):
"""
A "safely" wrapped version
"""
def __init__(self, value=''):
value = value.encode('utf8', 'backslashreplace').decode('utf8')
io.StringIO.__init__(self, value)
def write(self, msg):
io.StringIO.write(self, msg.encode(
'utf8', 'backslashreplace').decode('utf8'))
Now plug your example function plus every code fragment in this answer into one file, you will get your self contained unittest that works in both Python 2 and 3 (although you need to call print as a function in Python 3) for doing testing against stdio.
One more note: you can always put the stub_ function calls in the setUp method of the TestCase if every single test method requires that.
Of course, if you want to use various mocks related libraries out there to stub out stdin/stdout, you are free to do so, but this way relies on no external dependencies if this is your goal.
For your second issue, test cases have to be written in a certain way, where they must be encapsulated within a method and not at the class level, your original example will fail. However you might want to do something like this:
class Test(unittest.TestCase):
def helper(self, data, answer, runner):
stub_stdin(self, data)
stub_stdouts(self)
runner()
self.assertEqual(sys.stdout.getvalue(), answer)
self.doCleanups() # optional, see comments below
def test_various_inputs(self):
data_and_answers = [
('hello', 'HELLOhello'),
('goodbye', 'GOODBYEgoodbye'),
]
runScript = upperlower # the function I want to test
for data, answer in data_and_answers:
self.helper(data, answer, runScript)
The reason why you might want to call doCleanups is to prevent the cleanup stack from getting as deep as all the data_and_answers pairs are there, but that will pop everything off the cleanup stack so if you had any other things that need to be cleaned up at the end this might end up being problematic - you are free to leave that there as all of the stdio related objects will be restored at the end in the same order, so the real one will always be there. Now the function I wanted to test:
def upperlower():
raw = raw_input()
print (raw.upper() + raw),
So yes, a bit of explanation for what I did might help: remember within a TestCase class, the framework relies strictly on the instance's assertEqual and friends for it to function. So to ensure testing being done at the right level you really want to call those asserts all the time so that helpful error messages will be shown at the moment the error occurred with the inputs/answers that didn't quite show up right, rather than until the very end like what you did with the for loop (that will tell you something was wrong, but not exactly where out of the hundreds and now you are mad). Also the helper method - you can call it anything you want, as long as it doesn't start with test because then the framework will try to run it as one and it will fail terribly. So just follow this convention and you can basically have templates within your test case to run your test with - you can then use it in a loop with a bunch of inputs/outputs like what I did.
As for your other question:
only I've had a little problem importing StringIO, because I was doing import StringIO and I needed to import like from StringIO import StringIO (I don't understand really why), but be that as It may, it works.
Well, if you look at my original code I did show you how did import io and then overrode the io.StringIO class by defining class StringIO(io.StringIO). However it works for you because you are doing this strictly from Python 2, whereas I do try to target my answers to Python 3 whenever possible given that Python 2 will (probably definitely this time) not be supported in less than 5 years. Think of the future users that might be reading this post who had similar problem as you. Anyway, yes, the original from StringIO import StringIO works, as that's the StringIO class from the StringIO module. Though from cStringIO import StringIO should work as that imports the C version of the StringIO module. It works because they all offer close enough interfaces, and so they will basically work as intended (until of course you try to run this under Python 3).
Again, putting all this together along with my code should result in a self-contained working test script. Do remember to look at documentation and follow the form of the code, and not invent your own syntax and hoping things to work (and as for exactly why your code didn't work, because the "test" code was defined at where the class was being constructed, so all of that was executed while Python was importing your module, and since none of the things that are needed for the test to run are even available (namely the class itself doesn't even exist yet), the whole thing just dies in fits of twitching agony). Asking questions here help too, even though the issue you face is something really common, not having a quick and simple name to search for your exact problem does make it difficult to figure out where you went wrong, I supposed? :) Anyway good luck, and good on you for taking the effort to test your code.
There are other methods, but given that the other questions/answers I looked at here at SO doesn't seem to help, I hope this one this. Other ones for reference:
How to supply stdin, files and environment variable inputs to Python unit tests?
python mocking raw input in unittests
Naturally, it bares repeating that all of this can be done using unittest.mock available in Python 3.3+ or the original/rolling backport version on pypi, but given that those libraries hides some of the intricacies on what actually happens, they may end up hiding some of the details on what actually happens (or need to happen) or how the redirection actually happens. If you want, you can read up on unittest.mock.patch and go down slightly to the StringIO patching sys.stdout section.

Making a GDB debugging helper for the QUuid class

I'm using the QUuid class in my project and for testing and debugging purposes it would be very nice to see the QUuid objects in human readable form instead of their low-level form.
For some reason, the people at Qt have not included a dump method for this type so I attempted to create one on my own, following this documentation and this guide.
I'm not familiar with Python so unfortunately, I could not get something running. Could someone help me create such a function that does nothing more than display the output of QUuid::toString() in the value column of Qt Creator?
Edit:
Mitko's solution worked perfectly. I expanded it a bit so the details can still be read if so desired:
from dumper import *
import gdb
def qdump__QUuid(d, value):
this_ = d.makeExpression(value)
finalValue = gdb.parse_and_eval("%s.toString()" % (this_))
d.putStringValue(finalValue)
d.putNumChild(4)
if d.isExpanded():
with Children(d):
d.putSubItem("data1", value["data1"])
d.putSubItem("data2", value["data2"])
d.putSubItem("data3", value["data3"])
d.putSubItem("data4", value["data4"])

The following python script should do the job:
from dumper import *
import gdb
def qdump__QUuid(d, value):
this = d.makeExpression(value)
stringValue = gdb.parse_and_eval("%s.toString()" % this)
d.putStringValue(stringValue)
d.putNumChild(0)
The easiest way to use it with Qt Creator is to just paste these lines at the end of your <Qt-Creator-Install-Dir>/share/qtcreator/debugger/personaltypes.py file. In this case you can skip the first line, as it's already in the file.
As the personaltypes.py file is overwritten when you update Qt Creator you might want to put the script above in its own file. In that case you'll need to configure Qt Creator to use your file. You can do this by going to Tools > Options... > Debugger > GDB > Extra Debugging Helpers > Browse and selecting your file.
Note:
This script will only work inside Qt Creator, since we use its specific dumper (e.g. putStringValue).
We call QUuid::toString() which creates a QString object. I'm not sure exactly how gdb and python handle this, and if there is a need to clean this up in order to avoid leaking memory. It's probably not a big deal for debugging, but something to be aware of.

How to have win32com code completion in IPython?

Via
import win32com.client
wordapp = win32com.client.gencache.EnsureDispatch('Word.Application')
I can get a Word Application object documented e.g. here. However, ipython's autocompletion is not aware of that API, is there any way to add that?

Quick solution
Perhaps the simplest way to achieve code completion in IPython (tested with 6.2.1, see the answer below for a snippet that works with 7.1) and Jupyter is to run the following snippet:
from IPython.utils.generics import complete_object
import win32com.client
#complete_object.when_type(win32com.client.DispatchBaseClass)
def complete_dispatch_base_class(obj, prev_completions):
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
return list(ole_props) + prev_completions
except AttributeError:
pass
Short story long
With some more details being outlined in this guide, win32com ships with a script, makepy.py for generating Python types corresponding to the type library of a given COM object.
In the case of Word 2016, we would proceed as follows:
C:\Users\username\AppData\Local\Continuum\Anaconda3\pkgs\pywin32-221-py36h9c10281_0\Lib\site-packages\win32com\client>python makepy.py -i "Microsoft Word 16.0 Object Library"
Microsoft Word 16.0 Object Library
{00020905-0000-0000-C000-000000000046}, lcid=0, major=8, minor=7
>>> # Use these commands in Python code to auto generate .py support
>>> from win32com.client import gencache
>>> gencache.EnsureModule('{00020905-0000-0000-C000-000000000046}', 0, 8, 7)
The location of makepy.py will of course depend on your Python distribution. The script combrowse.py, available in the same directory, can be used to find the names of available type libraries.
With that in place, win32com.client will automatically make use of the generated types, rather than the raw IPyDispatch, and at this point, auto-completion is available in e.g. IPython or Jupyter, given that the COM object of interest actually publishes its available properties and methods (which is not a requirement).
Now, in your case, by invoking EnsureDispatch instead of Dispatch, the makepy part of the process is performed automatically, so you really should be able to obtain code completion in IPython for the published methods:
Note, though, that while this does give code completion for methods, the same will not be true for properties. It is possible to inspect those using the _prop_map_get_ attribute. For example, wordapp.Selection.Range.Font._prop_map_get_ gives all properties available on fonts.
If using IPython is not a strong requirement, note also that the PythonWin shell (located around \pkgs\pywin32\Lib\site-packages\pythonwin\Pythonwin.exe) has built-in code completion support for both properties and methods.
This, by itself, suggests that the same is achievable in IPython.
Concretely, the logic for auto-completion, which in turn relies on _prop_map_get_, can be found in scintilla.view.CScintillaView._AutoComplete. On the other hand, code completion in IPython 6.2.1 is handled by core.completer.IPCompleter. The API for adding custom code completers is provided by IPython.utils.generics.complete_object, as illustrated in the first solution above. One gotcha is that with complete_object being based on simplegeneric, only one completer may be provided for any given type. Luckily, all types generated by makepy will inherit from win32com.client.DispatchBaseClass.
If this turns out to ever be an issue, one can also circumvent complete_object entirely and simply manually patch IPython by adding the following five lines to core.completer.Completion.attr_matches:
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
words += list(ole_props)
except AttributeError:
pass
Conversely, IPython bases its code-completion on __dir__, so one could also patch gencache, which is where the code generation ultimately happens, to include something to like
def __dir__(self):
return list(set(self._prop_map_get_).union(set(self._prop_map_put_)))
to each generated DispatchBaseClass.

fuglede's answer is great, just want to update it for the newest versions of IPython (7.1+).
Since IPython.utils.generics has changes from using simplegeneric to using functools, the #complete_object.when_type method should be changed to #complete_object.register. So his initial code should changed to:
from IPython.utils.generics import complete_object
import win32com.client
#complete_object.register(win32com.client.DispatchBaseClass)
def complete_dispatch_base_class(obj, prev_completions):
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
return list(ole_props) + prev_completions
except AttributeError:
pass

Global include in restructured text

I'm using reStructuredText for my blog/website and I want to add a global include file. I have access to and am happy to change the settings file I'm using to generate the html output, I just can't figure out the syntax for either:
adding a default include file to the parser
defining directive/inline-roles, etc in python with docutils in python
I tried reading the source code and the documentation and just find it a bit hard to follow. I'm hoping that I just missed something super-obvious, but I'd like to do something like the following (the first part is just what is already there -- you can see the rest of the file in the jekyll-rst plugin source (links right to it)
import sys
from docutils.core import publish_parts
from optparse import OptionParser
from docutils.frontend import OptionParser as DocutilsOptionParser
from docutils.parsers.rst import Parser
# sets up a writer that is then called to parse rst pages repeatedly
def transform(writer=None, part=None):
p = OptionParser(add_help_option=False)
# Collect all the command line options
docutils_parser = DocutilsOptionParser(components=(writer, Parser()))
for group in docutils_parser.option_groups:
p.add_option_group(group.title, None).add_options(group.option_list)
p.add_option('--part', default=part)
opts, args = p.parse_args()
# ... more settings, etc
# then I just tell the parser/writer to process specified file X.rst every time
# (or alternately a python file defining more roles...but nicer if in rst)
Is there a simple way to do this? It'd be great to define a file defaults.rst and have that load each time.
EDIT: Here are some examples of what I'd like to be able to globally include (custom directives would be nice too, but I'd probably write those in code)
.. role:: raw-html(raw)
:format: html
.. |common-substitution| replace:: apples and orange
.. |another common substitution| replace:: etc

I'm not quite sure if I understand the question. Do you want to define a number of, for example, substitutions in some file and have these available in all your other reStructuredText files, or do you want to include some common HTML in your output files? Can you clarify your question?
If it is the former that you want to do you can use the include directive, as I outline in this answer.
Alternatively, if you want some common HTML included in the generated output, try copying and editing the template.txt file which is include in the module path/to/docutils/writers/html4css1/. You can include arbitrary HTML elements in this file and modify the layout of the HTML generated by Docutils. Neither of these methods require you to modify the Docuitls source code, which is always an advantage.
Edit: I don't think it is possible to set a flag to set an include file using Docuitls. However, if you can use Sphinx, which is based on Docuitls but has a load of extensions, then this package has a setting rst_prolog which does exactly what you need (see this answer). rst_prolog is:
A string of reStructuredText that will be included at the beginning of every source file that is read.

I needed the exact same thing: A way to have some global reStructuredText files being automatically imported into every reStructuredText article without having to specify them each time by hand.
One solution to this problem is the following plugin:
import os
from pelican import signals
from pelican.readers import RstReader
class RstReaderWrapper(RstReader):
enabled = RstReader.enabled
file_extensions = ['rst']
class FileInput(RstReader.FileInput):
def __init__(self, *args, **kwargs):
RstReader.FileInput_.__init__(self, *args, **kwargs)
self.source = RstReaderWrapper.SourceWrapper(self.source)
# Hook into RstReader
RstReader.FileInput_ = RstReader.FileInput
RstReader.FileInput = FileInput
class SourceWrapper():
"""
Mimics and wraps the result of a call to `open`
"""
content_to_prepend = None
def __init__(self, source):
self.source = source
def read(self):
content = self.source.read()
if self.content_to_prepend is not None:
content = "{}\n{}".format(self.content_to_prepend, content)
return content
def close(self):
self.source.close()
def process_settings(pelicanobj):
include_files = pelicanobj.settings.get('RST_GLOBAL_INCLUDES', []) or []
base_path = pelicanobj.settings.get('PATH', ".")
def read(fn):
with open(os.path.join(base_path, fn), 'r') as res:
content = res.read()
return ".. INLCUSION FROM {}\n{}\n".format(fn, content)
inclusion = "".join(map(read, include_files)) if include_files else None
RstReaderWrapper.SourceWrapper.content_to_prepend = inclusion
def register():
signals.initialized.connect(process_settings)
Usage in short:
Create a plugin from the above code (best clone the repository from GitHub)
Import the plugin (adapt PLUGINS in pelicanconf.py)
Define the list of RST files (relative paths to project root) to include by setting the variable RST_GLOBAL_INCLUDES in pelicanconf.py
Please note that pelican and docutils are both not designed to allow this. Neither a signal is provided which provides a clean access to the raw contents of a source file before processing begins, nor is there a possibility to intercept the framework reading the file in "a normal way" (like subclassing, changing hardcoded configuration, etc).
This plugin subclasses the internal class FileInput of RstReader and sets the class reference of RstReader.FileInput to the subclass. Also python file objects are emulated through SourceWrapper.
Nevertheless, this approach works for me and is not cumbersome in the daily workflow.
I know this question is from 2012 but I think this answer can still be helpful to others.