We Keep Coding

Why does a python script behaves differently when it is run in pycharm and when it is run from a command prompt?

I'm getting an error when I run a script from the Linux command prompt(bash), but when I run the same script in the Pycharm directly, it works fine.
Here is the code:
class EncodeKey:
def __new__(cls, *args):
if len(args) == 1:
return cls.generate_encode_key(args[0].upper())
return cls.get_encode_key(args)
...
...
if __name__ == '__main__':
encode_key = EncodeKey(*["something"])
print(encode_key)
As I already told, in the Pycharm the script works fine without any errors, but here is what I'm getting when the script is run from the command prompt:
user#machine:~/my/path/to/the/script$ python py.py
Traceback (most recent call last):
File "py.py", line 93, in <module>
encode_key = EncodeKey(*["something"])
TypeError: this constructor takes no arguments
Or:
user#machine:~/my/path/to/the/script$ ls -lh
...
-rwxrwxr-x 1 user user 3.2K Jun 20 19:12 py.py
...
user#machine:~/my/path/to/the/script$ ./py.py
Traceback (most recent call last):
File "py.py", line 93, in <module>
encode_key = EncodeKey(*["something"])
TypeError: this constructor takes no arguments
Of, course, I didn't find any solutions on such an unusual problem. Any ideas why is that happening ? And how to solve it ? Thanks !

If you've installed python in the default way, the python command uses Python 2. To interperet with Python 3, use the command python3:
user#machine:~/my/path/to/the/script$ python3 py.py
The program errors in Python 2 because old-style classes, the default in Python 2, do not support __new__. If you need to support Python 2, make your class new-style by extending object:
class EncodeKey(object):
Your code will still work in Python 3 if you do this.

Running Python's doctest on a file named `types.py`

In both Python 2 and 3 I cannot run doctests in a file named types.py, which is part of a package. This is what I get:
$ cat foo/types.py
def x():
"""do something
>>> x()
1
"""
return 1
$ cp foo/types.py foo/types2.py
$ python -m doctest -v foo/types.py
1 items had no tests:
types
0 tests in 1 items.
0 passed and 0 failed.
Test passed.
$ python -m doctest -v foo/types2.py
Trying:
x()
Expecting:
1
ok
1 items had no tests:
types2
1 items passed all tests:
1 tests in types2.x
1 tests in 2 items.
1 passed and 0 failed.
Test passed.
$ python3 -m doctest -v foo/types.py
37 items had no tests:
types
types.DynamicClassAttribute
types.DynamicClassAttribute.__delete__
types.DynamicClassAttribute.__get__
types.DynamicClassAttribute.__init__
types.DynamicClassAttribute.__set__
types.DynamicClassAttribute.deleter
types.DynamicClassAttribute.getter
types.DynamicClassAttribute.setter
types.SimpleNamespace
types.SimpleNamespace.__delattr__
types.SimpleNamespace.__eq__
types.SimpleNamespace.__ge__
types.SimpleNamespace.__getattribute__
types.SimpleNamespace.__gt__
types.SimpleNamespace.__init__
types.SimpleNamespace.__le__
types.SimpleNamespace.__lt__
types.SimpleNamespace.__ne__
types.SimpleNamespace.__reduce__
types.SimpleNamespace.__repr__
types.SimpleNamespace.__setattr__
types._GeneratorWrapper
types._GeneratorWrapper.__init__
types._GeneratorWrapper.__iter__
types._GeneratorWrapper.__next__
types._GeneratorWrapper.close
types._GeneratorWrapper.cr_await
types._GeneratorWrapper.gi_code
types._GeneratorWrapper.gi_frame
types._GeneratorWrapper.gi_running
types._GeneratorWrapper.send
types._GeneratorWrapper.throw
types._calculate_meta
types.coroutine
types.new_class
types.prepare_class
0 tests in 37 items.
0 passed and 0 failed.
Test passed.
$ python3 -m doctest -v foo/types2.py
Trying:
x()
Expecting:
1
ok
1 items had no tests:
types2
1 items passed all tests:
1 tests in types2.x
1 tests in 2 items.
1 passed and 0 failed.
Test passed.
As you see, all invocations with foo/types2.py work as expected, all invocations with foo/types.py seem to try to load the Python built-in types module.
I'm also not able to fix this by tinkering with PYTHONPATH:
$ PYTHONPATH=.:$PYTHONPATH python -m doctest -v foo/types.py
Traceback (most recent call last):
File "/usr/lib/python2.7/site.py", line 68, in <module>
import os
File "/usr/lib/python2.7/os.py", line 400, in <module>
import UserDict
File "/usr/lib/python2.7/UserDict.py", line 116, in <module>
import _abcoll
File "/usr/lib/python2.7/_abcoll.py", line 70, in <module>
Iterable.register(str)
File "/usr/lib/python2.7/abc.py", line 107, in register
if not isinstance(subclass, (type, types.ClassType)):
AttributeError: 'module' object has no attribute 'ClassType'
Unfortunately, I cannot simply rename foo/types.py.
Is there any possibility to run doctests from this file apart from writing lots of boilerplate code around it?

I don’t think you can use python -m doctest here: the documentation says that it “import[s the module] as a standalone module”, adding
Note that this may not work correctly if the file is part of a package and imports other submodules from that package.
which is a fancy way of saying that it uses the module’s unqualified name. Of course it then conflicts with the standard library module.

import unittest error

This is my first time doing unit testing and i'm trying to run a simple code...
import random
import unittest
class TestSequenceFunctions(unittest.TestCase):
def setUp(self):
self.seq = range(10)
def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))
# should raise an exception for an immutable sequence
self.assertRaises(TypeError, random.shuffle, (1,2,3))
def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)
def test_sample(self):
with self.assertRaises(ValueError):
random.sample(self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)
suite = unittest.TestLoader().loadTestsFromTestCase(TestSequenceFunctions)
unittest.TextTestRunner(verbosity=2).run(suite)
then i get the error message...
Traceback (most recent call last):
File "/Users/s66/Desktop/unittest.py", line 2, in <module>
import unittest
File "/Users/s66/Desktop/unittest.py", line 4, in <module>
class TestSequenceFunctions(unittest.TestCase):
AttributeError: 'module' object has no attribute 'TestCase'
>>>
How do I fix this?

That's because your script name is called unittest.py. The statement import unittest is importing your script rather than the unittest module, hence the error with the non-existant TestCase attribute.
For more info, see the docs for Module Search Path. In short, when you do an import, built-in modules are first searched followed by directories listed in sys.path. This usually starts with the location of the running script followed by PYTHONPATH and THEN then default modules directory
In your case, since unittest is not a built-in, it found your script (and loaded it) before it could search for other installed modules.
How do I fix this?
Rename your script.

Do you have a module in the directory where you're running your script that is called unittest.py?
When I run your code I get the following output.
test_choice (__main__.TestSequenceFunctions) ... ok
test_sample (__main__.TestSequenceFunctions) ... ERROR
test_shuffle (__main__.TestSequenceFunctions) ... ok
======================================================================
ERROR: test_sample (__main__.TestSequenceFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 23, in test_sample
with self.assertRaises(ValueError):
TypeError: failUnlessRaises() takes at least 3 arguments (2 given)
----------------------------------------------------------------------
Ran 3 tests in 0.000s
FAILED (errors=1)

My answer is not intended for this author, but for people that found this stackoverflow page on Google for the same error message but for a different reason.
I was upgrading from django 1.5 to django 1.10, and the "import unittest" in one of my test files was causing a similar error.
The error was due to the app/tests/init.py file which was importing the errored test file (this was necessary in django 1.5 as the automatic detection of test files didn't exist at that time. Now it is no longer required).
So the solution was to simply empty the init.py file and it solved the issue.
Hope it can help some people as I lost 30mn to 1h on this issue.

Python assert -- improved introspection of failure?

This is a rather useless assertion error; it does not tell the values of the expression involved (assume constants used are actually variable names):
$ python -c "assert 6-(3*2)"
[...]
AssertionError
Is there a better assert implementation in Python that is more fancy? It must not introduce additional overhead over execution (except when assert fails) .. and must turn off if -O flag is used.
Edit: I know about assert's second argument as a string. I don't want to write one .. as that is encoded in the expression that is being asserted. DRY (Don't Repeat Yourself).

Install your of function as sys.excepthook -- see the docs. Your function, if the second argument is AssertionError, can introspect to your heart's contents; in particular, through the third argument, the traceback, it can get the frame and exact spot in which the assert failed, getting the failing exception through the source or bytecode, the value of all relevant variables, etc. Module inspect helps.
Doing it in full generality is quite a piece of work, but depending on what constraints you're willing to accept in how you write your asserts it can be lightened substantially (e.g. restricting them to only local or global variables makes introspection easier than if nonlocal variables of a closure could be involved, and so forth).

You can attach a message to an assert:
assert 6-(3*2), "always fails"
The message can also be built dynamically:
assert x != 0, "x is not equal to zero (%d)" % x
See The assert statement in the Python documentation for more information.

As #Mark Rushakoff said nose can evaluate failed asserts. It works on the standard assert too.
# test_error_reporting.py
def test():
a,b,c = 6, 2, 3
assert a - b*c
nosetests' help:
$ nosetests --help|grep -B2 assert
-d, --detailed-errors, --failure-detail
Add detail to error output by attempting to evaluate
failed asserts [NOSE_DETAILED_ERRORS]
Example:
$ nosetests -d
F
======================================================================
FAIL: test_error_reporting.test
----------------------------------------------------------------------
Traceback (most recent call last):
File "..snip../site-packages/nose/case.py", line 183, in runTest
self.test(*self.arg)
File "..snip../test_error_reporting.py", line 3, in test
assert a - b*c
AssertionError:
6,2,3 = 6, 2, 3
>> assert 6 - 2*3
----------------------------------------------------------------------
Ran 1 test in 0.089s
FAILED (failures=1)

The nose testing suite applies introspection to asserts.
However, AFAICT, you have to call their asserts to get the introspection:
import nose
def test1():
nose.tools.assert_equal(6, 5+2)
results in
C:\temp\py>C:\Python26\Scripts\nosetests.exe -d test.py
F
======================================================================
FAIL: test.test1
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line
183, in runTest
self.test(*self.arg)
File "C:\temp\py\test.py", line 3, in test1
nose.tools.assert_equal(6, 5+2)
AssertionError: 6 != 7
>> raise self.failureException, \
(None or '%r != %r' % (6, 7))
Notice the AssertionError there. When my line was just assert 6 == 5+2, I would get:
C:\temp\py>C:\Python26\Scripts\nosetests.exe -d test.py
F
======================================================================
FAIL: test.test1
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line
183, in runTest
self.test(*self.arg)
File "C:\temp\py\test.py", line 2, in test1
assert 6 == 5 + 2
AssertionError:
>> assert 6 == 5 + 2
Also, I'm not sure offhand if their asserts are skipped with -O, but that would be a very quick check.

I coded a replacement for sys.excepthook (which is called for any unhandled exception) which is a bit more fancy than the standard one. It will analyze the line where the exception occured and print all variables which are referred to in this line (it does not print all local variables because that might be too much noise - also, maybe the important var is global or so).
I called it py_better_exchook (perfect name) and it's here.
Example file:
a = 6
def test():
unrelated_var = 43
b,c = 2, 3
assert a - b*c
import better_exchook
better_exchook.install()
test()
Output:
$ python test_error_reporting.py
EXCEPTION
Traceback (most recent call last):
File "test_error_reporting.py", line 12, in <module>
line: test()
locals:
test = <local> <function test at 0x7fd91b1a05f0>
File "test_error_reporting.py", line 7, in test
line: assert a - b*c
locals:
a = <global> 6
b = <local> 2
c = <local> 3
AssertionError
There are a few other alternatives:
(Presented here) https://github.com/albertz/py_better_exchook/
https://github.com/patrys/great-justice
Nose does something similar for assertion failures, see here.
IPython has something similar (this). Do this: from IPython.core import ultratb; sys.excepthook = ultratb.VerboseTB().
Ka-Ping Yee's "cgitb.py", which is part of Python, see here, code here.

Add a message to your assertion, which will be displayed if the assertion fails:
$ python -c "assert 6-(3*2), '6-(3*2)'"
Traceback (most recent call last):
File "<string>", line 1, in <module>
AssertionError: 6-(3*2)
The only way I can think of to provide this automatically would be to contain the assertion in a procedure call, and then inspect the stack to get the source code for that line. The additional call would, unfortunately, introduce overhead into the test and would not be disabled with -O.

It sounds like what you really want to do is to set up a debugger breakpoint just before the assert and inspect from your favorite debugger as much as you like.

Combining nosetests with contracts for Python

I'm using contracts for Python to specify preconditons/postconditions/invariants. I'm also using doctests for doing unit testing.
I'd like to have all of my doctest unit tests run with contracts enabled, and I'd like to run my tests using nose. Unfortunately, if I run the tests with nose, it does not execute the pre/post/invariant assertions. I put a setup function in each .py file to make sure that contract.checkmod gets called
def setup():
import contract
contract.checkmod(__name__)
I can confirm that this function is being executed by nose before it runs the tests, but the contracts still don't get executed.
On the other hand, if I run the doctest by calling doctest.testmod, the pre/post/inv do get called:
def _test():
import contract
contract.checkmod(__name__)
import doctest
doctest.testmod()
if __name__=='__main__':
_test()
Here's an example of a Python script whose test will succeed if called directly, but failed if called with nose:
import os
def setup():
import contract
contract.checkmod(__name__)
def delete_file(path):
"""Delete a file. File must be present.
>>> import minimock
>>> minimock.mock('os.remove')
>>> minimock.mock('os.path.exists', returns=True)
>>> delete_file('/tmp/myfile.txt')
Called os.path.exists('/tmp/myfile.txt')
Called os.remove('/tmp/myfile.txt')
>>> minimock.restore()
pre: os.path.exists(path)
"""
os.remove(path)
if __name__ == '__main__':
setup()
import doctest
doctest.testmod()
When I run the above file standalone, the tests pass:
$ python contracttest.py -v
Trying:
import minimock
Expecting nothing
ok
Trying:
minimock.mock('os.remove')
Expecting nothing
ok
Trying:
minimock.mock('os.path.exists', returns=True)
Expecting nothing
ok
Trying:
delete_file('/tmp/myfile.txt')
Expecting:
Called os.path.exists('/tmp/myfile.txt')
Called os.remove('/tmp/myfile.txt')
ok
Trying:
minimock.restore()
Expecting nothing
ok
2 items had no tests:
__main__
__main__.setup
1 items passed all tests:
5 tests in __main__.delete_file
5 tests in 3 items.
5 passed and 0 failed.
Test passed.
Here it is with nose:
$ nosetests --with-doctest contracttest.py
F
======================================================================
FAIL: Doctest: contracttest.delete_file
----------------------------------------------------------------------
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 2131, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for contracttest.delete_file
File "/Users/lorin/Desktop/contracttest.py", line 10, in delete_file
----------------------------------------------------------------------
File "/Users/lorin/Desktop/contracttest.py", line 17, in contracttest.delete_file
Failed example:
delete_file('/tmp/myfile.txt')
Expected:
Called os.path.exists('/tmp/myfile.txt')
Called os.remove('/tmp/myfile.txt')
Got:
Called os.remove('/tmp/myfile.txt')
----------------------------------------------------------------------
Ran 1 test in 0.055s