We Keep Coding

PyInstaller: Executable to access user-specified Sourcecode

I'm using pyinstaller to distribute my code as executable within my team as most of them are not coding/scripting people and do not have Python Interpreter installed.
For some advanced usage of my tool, I want to make it possible for the user to implement a small custom function to adjust functionality slightly (for the few experienced people). Hence I want to let them input a python file which defines a function with a fixed name and a string as return.
Is that possible?
I mean the py-file could be drag/dropped for example, and I'd tell them that their user-defined function needs to have a certain name, e.g. "analyze()" - is it now possible to import that from the drag/dropped pythonfile within my PyInstaller Script and use it as this?
I know, it certainly will not be safe/secure and they could do evil things, delete files and so one... But that are things which we don#t care at this point, please no discussions about it. Thanks!

To answer my own question: yes it does actually work to import a module/function from a given path/pythonfile at runtime (that I knew already) even in PyInstaller (that was new for me).
I used this for my Py2.7 program:
f = r'C:\path\to\userdefined\filewithfunction.py'
if os.path.exists(f):
import imp
userdefined = imp.load_source('', f) # Only Python 2.x, for 3.x see: https://stackoverflow.com/a/67692/701049
print userdefined # just a debugging print
userdefined.imported() # here you should use try/catch; or check whether the function with the desired name really exists in the object "userdefined". This is only a small demo as example how to import, so didnt do it here.
filewithfunction.py:
--------------------
def imported():
print 'yes it worked :-)'
As written in the comments of the example code, you'll need a slightly different approach in Python 3.x. See this link: https://stackoverflow.com/a/67692/701049

How can I test the standard input and standard output in Python Script with a Unittest test?

I'm trying to test a Python script (2.7) where I work with the standar input (readed with raw_input() and writed with a simple print) but I don't find how do this and I'm sure that this issue is very simple.
This is a very very very resume code of my script:
def example():
number = raw_input()
print number
if __name__ == '__main__':
example()
I want to write a unittest test to check this, but I don't find how. I've trying with StringIO and other things but I don't find the solution to do this really simple.
Somebody have a idea?
PD: Of course in the real script I use data blocks with several lines and other kind of data.
Thank you so much.
EDIT:
Thank you so much for the first really specific answer, it works perfectly, only I've had a little problem importing StringIO, because I was doing import StringIO and I needed to import like from StringIO import StringIO (I don't understand really why), but be that as It may, it works.
But I I've found another problem using this way, in my project I need test a scripts with this way (that work perfectly thanks to your support) but I want do this:
I have a file with a lot of test to pass over a script, so I open the file and read blocks of info with their result blocks and I would like to do that the code will be able to process a block checking their result and do the same with other and another...
Something like this:
class Test(unittest.TestCase):
...
#open file and process saving data like datablocks and results
...
allTest = True
for test in tests:
stub_stdin(self, test.dataBlock)
stub_stdouts(self)
runScrip()
if sys.stdout.getvalue() != test.expectResult:
allTest = False
self.assertEqual(allTest, True)
I know that maybe unittest doesn't has sense now, but you can do a idea about I want. So, this way fails and I don't know why.

Typical techniques involve mocking the standard sys.stdin and sys.stdout with your desired items. If you do not care for Python 3 compatibility you can just use the StringIO module, however if you want forward thinking and is willing to restrict to Python 2.7 and 3.3+, supporting for this both Python 2 and 3 in this way becomes possible without too much work through the io module (but requires a bit of modification, but put this thought on hold for now).
Assuming you already have a unittest.TestCase going, you can create a utility function (or method in the same class) that will replace sys.stdin/sys.stdout as outlined. First the imports:
import sys
import io
import unittest
In one of my recent projects I've done this for stdin, where it take a str for the inputs that the user (or another program through pipes) will enter into yours as stdin:
def stub_stdin(testcase_inst, inputs):
stdin = sys.stdin
def cleanup():
sys.stdin = stdin
testcase_inst.addCleanup(cleanup)
sys.stdin = StringIO(inputs)
As for stdout and stderr:
def stub_stdouts(testcase_inst):
stderr = sys.stderr
stdout = sys.stdout
def cleanup():
sys.stderr = stderr
sys.stdout = stdout
testcase_inst.addCleanup(cleanup)
sys.stderr = StringIO()
sys.stdout = StringIO()
Note that in both cases, it accepts a testcase instance, and calls its addCleanup method that adds the cleanup function call that will reset them back to where they were when the duration of a test method is concluded. The effect is that for the duration from when this was invoked in the test case until the end, sys.stdout and friends will be replaced with the io.StringIO version, meaning you can check its value easily, and don't have to worry about leaving a mess behind.
Better to show this as an example. To use this, you can simply create a test case like so:
class ExampleTestCase(unittest.TestCase):
def test_example(self):
stub_stdin(self, '42')
stub_stdouts(self)
example()
self.assertEqual(sys.stdout.getvalue(), '42\n')
Now, in Python 2, this test will only pass if the StringIO class is from the StringIO module, and in Python 3 no such module exists. What you can do is use the version from the io module with a modification that makes it slightly more lenient in terms of what input it accepts, so that the unicode encoding/decoding will be done automatically rather than triggering an exception (such as print statements in Python 2 will not work nicely without the following). I typically do this for cross compatibility between Python 2 and 3:
class StringIO(io.StringIO):
"""
A "safely" wrapped version
"""
def __init__(self, value=''):
value = value.encode('utf8', 'backslashreplace').decode('utf8')
io.StringIO.__init__(self, value)
def write(self, msg):
io.StringIO.write(self, msg.encode(
'utf8', 'backslashreplace').decode('utf8'))
Now plug your example function plus every code fragment in this answer into one file, you will get your self contained unittest that works in both Python 2 and 3 (although you need to call print as a function in Python 3) for doing testing against stdio.
One more note: you can always put the stub_ function calls in the setUp method of the TestCase if every single test method requires that.
Of course, if you want to use various mocks related libraries out there to stub out stdin/stdout, you are free to do so, but this way relies on no external dependencies if this is your goal.
For your second issue, test cases have to be written in a certain way, where they must be encapsulated within a method and not at the class level, your original example will fail. However you might want to do something like this:
class Test(unittest.TestCase):
def helper(self, data, answer, runner):
stub_stdin(self, data)
stub_stdouts(self)
runner()
self.assertEqual(sys.stdout.getvalue(), answer)
self.doCleanups() # optional, see comments below
def test_various_inputs(self):
data_and_answers = [
('hello', 'HELLOhello'),
('goodbye', 'GOODBYEgoodbye'),
]
runScript = upperlower # the function I want to test
for data, answer in data_and_answers:
self.helper(data, answer, runScript)
The reason why you might want to call doCleanups is to prevent the cleanup stack from getting as deep as all the data_and_answers pairs are there, but that will pop everything off the cleanup stack so if you had any other things that need to be cleaned up at the end this might end up being problematic - you are free to leave that there as all of the stdio related objects will be restored at the end in the same order, so the real one will always be there. Now the function I wanted to test:
def upperlower():
raw = raw_input()
print (raw.upper() + raw),
So yes, a bit of explanation for what I did might help: remember within a TestCase class, the framework relies strictly on the instance's assertEqual and friends for it to function. So to ensure testing being done at the right level you really want to call those asserts all the time so that helpful error messages will be shown at the moment the error occurred with the inputs/answers that didn't quite show up right, rather than until the very end like what you did with the for loop (that will tell you something was wrong, but not exactly where out of the hundreds and now you are mad). Also the helper method - you can call it anything you want, as long as it doesn't start with test because then the framework will try to run it as one and it will fail terribly. So just follow this convention and you can basically have templates within your test case to run your test with - you can then use it in a loop with a bunch of inputs/outputs like what I did.
As for your other question:
only I've had a little problem importing StringIO, because I was doing import StringIO and I needed to import like from StringIO import StringIO (I don't understand really why), but be that as It may, it works.
Well, if you look at my original code I did show you how did import io and then overrode the io.StringIO class by defining class StringIO(io.StringIO). However it works for you because you are doing this strictly from Python 2, whereas I do try to target my answers to Python 3 whenever possible given that Python 2 will (probably definitely this time) not be supported in less than 5 years. Think of the future users that might be reading this post who had similar problem as you. Anyway, yes, the original from StringIO import StringIO works, as that's the StringIO class from the StringIO module. Though from cStringIO import StringIO should work as that imports the C version of the StringIO module. It works because they all offer close enough interfaces, and so they will basically work as intended (until of course you try to run this under Python 3).
Again, putting all this together along with my code should result in a self-contained working test script. Do remember to look at documentation and follow the form of the code, and not invent your own syntax and hoping things to work (and as for exactly why your code didn't work, because the "test" code was defined at where the class was being constructed, so all of that was executed while Python was importing your module, and since none of the things that are needed for the test to run are even available (namely the class itself doesn't even exist yet), the whole thing just dies in fits of twitching agony). Asking questions here help too, even though the issue you face is something really common, not having a quick and simple name to search for your exact problem does make it difficult to figure out where you went wrong, I supposed? :) Anyway good luck, and good on you for taking the effort to test your code.
There are other methods, but given that the other questions/answers I looked at here at SO doesn't seem to help, I hope this one this. Other ones for reference:
How to supply stdin, files and environment variable inputs to Python unit tests?
python mocking raw input in unittests
Naturally, it bares repeating that all of this can be done using unittest.mock available in Python 3.3+ or the original/rolling backport version on pypi, but given that those libraries hides some of the intricacies on what actually happens, they may end up hiding some of the details on what actually happens (or need to happen) or how the redirection actually happens. If you want, you can read up on unittest.mock.patch and go down slightly to the StringIO patching sys.stdout section.

Add suggestions to python session autocomplete that appear in IPython notebook too

I'm trying to add bespoke suggestions to the interactive python tab auto complete. I found this toy example on the interweb
import readline, rlcompleter
addrs = ['angela#domain.com', 'michael#domain.com', 'david#test.com']
class mycompleter(rlcompleter.Completer):
def completer(self, text, state):
options = [x for x in addrs if x.startswith(text)]
try:
return options[state]
except IndexError:
return None
readline.set_completer(mycompleter().completer)
readline.parse_and_bind("tab: complete")
This works very nicely in python if I save it in a module and them import it. It also works in IPython if I paste it into an active session using the %paste magic.
However, I can't get it to work in an IPython Notebook, either by loading a module or by running it in a cell. I've found the ipython docs about their extension to the readline module but this hasn't helped. I've tried inheriting from IPCompleter objects, and using rlcompete methods etc, but this doesn't seem to have helped.
Any suggestions about how to add things to the autocomplete suggestions in a way that works in plain python and IPython Notebook
Thanks
Niall
UPDATE:
Ultimately, I'm looking of a way to add functionality to a module so that it can dynamically update the session autocomplete list (ideally for args for a specific set of functions so that it doesn't pollute the suggestions).

There is a way to do it, which is not the recommended one but works.
def my_matches(test):
# might want to be smarter here
return ['angela#domain.com', 'michael#domain.com', 'david#test.com']
ip = get_ipython()
ip.Completer.matchers.append(my_matches)
# it works
The old ways require setting hooks, but I haven't used it and is pretty old
and could be refactored

How to have win32com code completion in IPython?

Via
import win32com.client
wordapp = win32com.client.gencache.EnsureDispatch('Word.Application')
I can get a Word Application object documented e.g. here. However, ipython's autocompletion is not aware of that API, is there any way to add that?

Quick solution
Perhaps the simplest way to achieve code completion in IPython (tested with 6.2.1, see the answer below for a snippet that works with 7.1) and Jupyter is to run the following snippet:
from IPython.utils.generics import complete_object
import win32com.client
#complete_object.when_type(win32com.client.DispatchBaseClass)
def complete_dispatch_base_class(obj, prev_completions):
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
return list(ole_props) + prev_completions
except AttributeError:
pass
Short story long
With some more details being outlined in this guide, win32com ships with a script, makepy.py for generating Python types corresponding to the type library of a given COM object.
In the case of Word 2016, we would proceed as follows:
C:\Users\username\AppData\Local\Continuum\Anaconda3\pkgs\pywin32-221-py36h9c10281_0\Lib\site-packages\win32com\client>python makepy.py -i "Microsoft Word 16.0 Object Library"
Microsoft Word 16.0 Object Library
{00020905-0000-0000-C000-000000000046}, lcid=0, major=8, minor=7
>>> # Use these commands in Python code to auto generate .py support
>>> from win32com.client import gencache
>>> gencache.EnsureModule('{00020905-0000-0000-C000-000000000046}', 0, 8, 7)
The location of makepy.py will of course depend on your Python distribution. The script combrowse.py, available in the same directory, can be used to find the names of available type libraries.
With that in place, win32com.client will automatically make use of the generated types, rather than the raw IPyDispatch, and at this point, auto-completion is available in e.g. IPython or Jupyter, given that the COM object of interest actually publishes its available properties and methods (which is not a requirement).
Now, in your case, by invoking EnsureDispatch instead of Dispatch, the makepy part of the process is performed automatically, so you really should be able to obtain code completion in IPython for the published methods:
Note, though, that while this does give code completion for methods, the same will not be true for properties. It is possible to inspect those using the _prop_map_get_ attribute. For example, wordapp.Selection.Range.Font._prop_map_get_ gives all properties available on fonts.
If using IPython is not a strong requirement, note also that the PythonWin shell (located around \pkgs\pywin32\Lib\site-packages\pythonwin\Pythonwin.exe) has built-in code completion support for both properties and methods.
This, by itself, suggests that the same is achievable in IPython.
Concretely, the logic for auto-completion, which in turn relies on _prop_map_get_, can be found in scintilla.view.CScintillaView._AutoComplete. On the other hand, code completion in IPython 6.2.1 is handled by core.completer.IPCompleter. The API for adding custom code completers is provided by IPython.utils.generics.complete_object, as illustrated in the first solution above. One gotcha is that with complete_object being based on simplegeneric, only one completer may be provided for any given type. Luckily, all types generated by makepy will inherit from win32com.client.DispatchBaseClass.
If this turns out to ever be an issue, one can also circumvent complete_object entirely and simply manually patch IPython by adding the following five lines to core.completer.Completion.attr_matches:
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
words += list(ole_props)
except AttributeError:
pass
Conversely, IPython bases its code-completion on __dir__, so one could also patch gencache, which is where the code generation ultimately happens, to include something to like
def __dir__(self):
return list(set(self._prop_map_get_).union(set(self._prop_map_put_)))
to each generated DispatchBaseClass.

fuglede's answer is great, just want to update it for the newest versions of IPython (7.1+).
Since IPython.utils.generics has changes from using simplegeneric to using functools, the #complete_object.when_type method should be changed to #complete_object.register. So his initial code should changed to:
from IPython.utils.generics import complete_object
import win32com.client
#complete_object.register(win32com.client.DispatchBaseClass)
def complete_dispatch_base_class(obj, prev_completions):
try:
ole_props = set(obj._prop_map_get_).union(set(obj._prop_map_put_))
return list(ole_props) + prev_completions
except AttributeError:
pass

Getting Python and AutoIT to work together using win32com: what's up with those window handles?

I have a program with a GUI interface whose initial set-up I need to do manually. AutoIt has been superbly helpful for that so far, as it provides very easy ways to work even with complex-to-access GUI objects (drop down lists, appear-on-hover menus, etc.).
However, the script that I'll eventually be needing in order to do the program setup will need to be passed a large array/list of variables - there are a lot of different settings that need to be changed.
I've set up the logic for deciding what these set-up variables will be using a Python script. Now I'm trying to figure out how to get Python and AutoIt to talk to each other.
Calling a custom AutoIt script from the command line using Python is mostly out of the question because of the large number of variables that would need to be passed. Doesn't feel pretty. I could try and have Python write an AutoIt "key file", which AutoIt could then read in order to set its initial variables, but I'd like to make sure I've exhausted all options for Python to work directly with AutoIt first.
To that end, I've been trying to use Python along with the win32com library to interface with AutoIt. Things seem to be working well - as long as I reference windows/menus/objects by their string titles rather than their (memory?) handles. This is problematic, as my set-up scripts might be running in parallel, setting up two or more separate files at the same time. If this is the case, opening up a box with a title string "Open file..." in each file at the same time might confuse things.
The obvious way to get around this in AutoIt is to work with the "handles" of the objects in question, which I believe are memory addresses of some kind, rather than their string titles. I'm guessing these are memory addresses as the AutoIt Window Info tool, when pointed to a particular Window/GUI object option lists a hexadecimal number as the object's handle value.
AutoIt has a suite of functions that get the handles of windows, menus, etc. They are implemented in the AutoIt COM dll, but I've not been able to get them to work in Python. The handle functions return a unicode object in Python, not a hex string as in AutoIt. I think this is the reason why functions which then try to use this "handle" in Python don't work.
Example:
autoIt = win32com.client.Dispatch("AutoItX3.Control")
windowHandle = autoIt.WinGetHandle(knownWindowTitle)
returnedWindowTitle = autoIt.WinGetTitle(windowHandle)
Usually, returnedWindowTitle and knownWindowTitle do not match as returnedWindowTitle always seems to be "0". hat's happening here?
Are there other ways for me to call custom AutoIt functions apart from using win32com, the command line, or using an AutoIt keyfile?
Thanks for your help.
EDIT: I forgot to mention that the unicode strings do in fact match the hexadecimal numbers that I get when I print out the handle variable in AutoIt.
For example, in Python, the handle variable when printed out gives me u'000C0326'. In AutoIt it gives me '0x000C0326'.
EDIT: Some trials based on Mat's suggestions:
In: autoIt = win32com.client.Dispatch("AutoItX3.Control")
In: mainWindowTitle = "Untitled"
In: mainWindowHandle = autoIt.WinGetHandle(mainWindowTitle)
In: mainWindowHandle
Out: u'000204AC'
In: testHandle = int(mainWindowHandle, 16)
In: testHandle
Out: 132268
In: autoIt.WinGetTitle(testHandle)
Out: u'0'
EDIT: I found out the type of the window handle object: it's a Microsoft HWND object. AutoIt has a function that can "convert" a base 16 number into an HWND object (i.e. find the HWND object with that base 16 number memory/handle/etc.). Just my luck that they didn't put that function into AutoItX (the COM dll). So, if I really want to run with this, I'll have to try and figure out how to return whatever object it is that's pointed to by the base 16 address, and then transfer it the right way to AutoItX? I'm probably sounding very confused, because all of this is not super clear in my head right now.

For the sake of search, I'll post the solution that I've found.
Here's the code:
In: autoIt = win32com.client.Dispatch("AutoItX3.Control")
In: autoIt.AutoItSetOption("WinTitleMatchMode", 4)
In: mainWindowTitle = "Untitled"
In: mainWindowHandle = autoIt.WinGetHandle(mainWindowTitle)
In: mainWindowHandle
Out: u'000204AC'
In: testHandle = "[HANDLE:%s]" % mainWindowHandle
In: autoIt.WinGetTitle(testHandle)
Out: u'Untitled - Notepad'
autoIt.AutoItSetOption("WinTitleMatchMode", 4) tells autoit to use advanced title matching, which allows us to specify a window handle with the [HANDLE:000204AC] string.
No need for the actual window handle here.
By the way, I discovered this solution by stumbling on this forum post. I've found that it's often helpful to not restrict my searches to the specific language I'm looking for. Most of the time, a solution can be found in a different language which can be easily ported to the language of your choice.

The type of window handles is string. The reason for that is WinList returns both the window handle and the window title. A title cannot be fit into a handle type, but a handle can be fit into a string type (for title). My guess is that they took that design descision and applied it to other functions as well. If you look at the documentation for WinGetHandle it will tell you the return type for a handle: It's a string.
AutoIt has a function that can "convert" a base 16 number into an HWND object.
Exactly! That's the key. AutoIt does this for you. You're trying to convert the handle into something useful for AutoIt, but AutoIt can already use those handles stored as strings.
Your test should be:
In: autoIt = win32com.client.Dispatch("AutoItX3.Control")
In: mainWindowTitle = "Untitled"
In: mainWindowHandle = autoIt.WinGetHandle(mainWindowTitle)
In: mainWindowHandle
Out: u'000204AC'
In: autoIt.WinGetTitle(mainWindowHandle)
Out: u'Untitled - Notepad'
When you want to use that handle in other libraries, you may run into some problems. I would recommend then that you try to parse the number like you did in your tests, and pass it along. AutoIt is 'smart' enough to figure out what to do in most cases, maybe the library isn't though.