We Keep Coding

How to display definition of a class in Python

I was going through this question : How do I return the definition of a class in python?
But I am unable to display the class definition. I am getting the below error:
>>> class A:
... pass
...
>>> import inspect
>>> source_text = inspect.getsource(A)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\**\Python\Python36\lib\inspect.py", line 968, in getsource
lines, lnum = getsourcelines(object)
File "C:\Users\**\Python\Python36\lib\inspect.py", line 955, in getsourcelines
lines, lnum = findsource(object)
File "C:\Users\**\Python\Python36\lib\inspect.py", line 768, in findsource
file = getsourcefile(object)
File "C:\Users\**\Python\Python36\lib\inspect.py", line 684, in getsourcefile
filename = getfile(object)
File "C:\Users\**\Python\Python36\lib\inspect.py", line 654, in getfile
raise TypeError('{!r} is a built-in class'.format(object))
TypeError: <module '__main__' (<_frozen_importlib_external.SourceFileLoader object at 0x0000026A79293F60>)> is a built-in class
>>>
Can someone please advise what am I doing wrong here? Thanks.

The inspect.getsource() function only works if there is a text file available to load the source code.
You typed the definition of the class into the interactive interpreter, which doesn't keep the original source around when compiling that source into in-memory class and code objects.
Put your class definition into a module, import the module, and then use inspect.getsource().
inspect.getsource() works by first finding the module for a given object (for classes, by looking at the ClassObj.__module__ attribute for the module name, then getting the module via sys.modules[modulename]) then seeing if the module has a __file__ attribute from which a readable source file can be determined. If there is such a filename and it can be read, then the inspect module reads that file to then search for the class ClassName: line and giving you all lines from that point on with same or deeper indentation. The interactive interpreter executes everything in the __main__ module and there is no __file__ attribute for the interpreter, so any attempts at loading source code for objects defined there will simply fail.
If you just wanted to know what members the class defines, use dir() or help() on the object instead. You don't need to see the full source code for that information.

How can I get the definition of a class in a module?

Why can't I get the definition of Callable from module collections in the following code?
How can I get the definition of a class in a module? Thanks.
>>> from collections import Callable
>>> inspect.getsource(Callable)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/inspect.py", line 944, in getsource
lines, lnum = getsourcelines(object)
File "/usr/lib/python3.5/inspect.py", line 931, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python3.5/inspect.py", line 788, in findsource
raise OSError('could not find class definition')
OSError: could not find class definition
>>> inspect.getsourcelines(Callable)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/inspect.py", line 931, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python3.5/inspect.py", line 788, in findsource
raise OSError('could not find class definition')
OSError: could not find class definition
>>> inspect.getmodule(Callable)
<module 'collections.abc' from '/usr/lib/python3.5/collections/abc.py'>
>>> inspect.getfile(Callable)
'/usr/lib/python3.5/collections/abc.py'
>>> inspect.getsourcefile(Callable)
'/usr/lib/python3.5/collections/abc.py'

In general, this is easily done with inspect.getsource which accepts a module, a class, a method, a function, a traceback, a frame,
or a code object. The source they represent should of course be written in Python else an error is raised.
In this specific case, you just happen to be unlucky because while Callable is defined in _collections_abc the Callable.__module__ name is callections.abc:
>>> Callable.__module__
'collections.abc'
This throws getsource off because it won't look in _collections_abc that contains Callables definition but, instead, in collections.abc which merely imports all definitions from _collections_abc:
>>> print(getsource(collections.abc))
from _collections_abc import *
from _collections_abc import __all__
Normally, getsource doesn't have an issue finding the source, for example, on itself:
>>> print(getsource(getsource))
def getsource(object):
"""Return the text of the source code for an object.
The argument may be a module, class, method, function, traceback, frame,
or code object. The source code is returned as a single string. An
OSError is raised if the source code cannot be retrieved."""
lines, lnum = getsourcelines(object)
return ''.join(lines)
In this specific case, though, it does (due to Callable.__module__ returning collections.abc.) You could replace __module__ with '_collections_abc' for a tricky way to see the source:
>>> Callable.__module__ = '_collections_abc'
>>> src = getsource(Callable)
>>> print(src)
class Callable(metaclass=ABCMeta):
__slots__ = ()
#abstractmethod
def __call__(self, *args, **kwds):
return False
#classmethod
def __subclasshook__(cls, C):
if cls is Callable:
return _check_methods(C, "__call__")
return NotImplemented
but this doesn't make me feel very comfortable.

get_source(fullname)
Return the source code for the specified module.
So you should return the whole module, as Callable is defined in the module of _collections_abc, so your code should be this:
import _collections_abc
import inspect
print(inspect.getsource(_collections_abc))
and you can see the definition of Callable in the print result.

How to get module source code by a python 'object' of that module? (not inspect.getsource)

How to get module source code by a python 'object' of that module?
class TestClass(object):
def __init__(self):
pass
def testMethod(self):
print 'abc'
return 'abc'
It's well-known that
print inspect.getsource(TestClass)
can be used to get source code of 'TestClass'.
However, the result of
ob = TestClass()
print inspect.getsource(ob)
as below, is not as expected.
Traceback (most recent call last):
File "D:\Workspaces\WS1\SomeProject\src\python\utils\ModuleUtils.py", line 154, in <module>
print inspect.getsource(ob)
File "C:\SciSoft\WinPython-64bit-2.7.10.3\python-2.7.10.amd64\lib\inspect.py", line 701, in getsource
lines, lnum = getsourcelines(object)
File "C:\SciSoft\WinPython-64bit-2.7.10.3\python-2.7.10.amd64\lib\inspect.py", line 690, in getsourcelines
lines, lnum = findsource(object)
File "C:\SciSoft\WinPython-64bit-2.7.10.3\python-2.7.10.amd64\lib\inspect.py", line 526, in findsource
file = getfile(object)
File "C:\SciSoft\WinPython-64bit-2.7.10.3\python-2.7.10.amd64\lib\inspect.py", line 420, in getfile
'function, traceback, frame, or code object'.format(object))
TypeError: <utils.TestClass.TestClass object at 0x0000000003C337B8> is not a module, class, method, function, traceback, frame, or code object
The question is:
If there is an object like 'ob' above, how to check the module source code of ob, or 'TestClass', via a method that takes 'ob' itself as a parameter?
In short, implement the following module
def getSource(obj):
###returns the result which is exactly identical to inspect.getsource(TestClass)
ob = TestClass()
###prints the result which is exactly identical to inspect.getsource(TestClass)
print getSource(ob)
(The scenario of a method like this is much more common than inspect.getsource(); for example, check the source code of an unknown, unpickled object.)

Instances don't have source code.
Use:
print inspect.getsource(type(ob))
or:
print inspect.getsource(ob.__class__)

Using mock to mock an imported module for testing a Pylons controller function

I've been chasing an issue with implementing mock for testing a Pylons controller function.
The particular function I am trying to test can be found here: https://hg.mozilla.org/build/buildapi/file/efe11511f42d/buildapi/controllers/selfserve.py#l210
The Objective:
I am simply trying to mock out the functions that it calls so that I can confirm that it eventually calls get_completeness.
The Issue:
I can mock _ok, _get_stable_delay, _failed, and _branches_cache just fine. But when it comes to g.buildapi_cache.get_builds_for_revision, I cannot seem to mock it correctly, and each time I call revision(), it fails.
g is actually pylons.app_globals and is imported in the selfserve.py controller like so: 'from pylons import app_globals as g'
Testing:
I am using a python interpreter to test all of this. These are the 2 implementations I am trying: http://pastebin.mozilla.org/2931825 and http://pastebin.mozilla.org/2931832
All in all here is the complete run in the interpreter, complete with Traceback for http://pastebin.mozilla.org/2931825
>>> from buildapi.controllers.selfserve import SelfserveController
>>> from mock import patch, Mock
>>> s = SelfserveController()
>>> s._ok = Mock(name='_ok')
>>> s._get_stable_delay = Mock(return_value=180)
>>> s._branches_cache = Mock()
>>> s._branches_cache = {'try': None}
>>> s._failed = Mock(return_value="Failed")
>>> with patch('buildapi.controllers.selfserve.g.buildapi_cache.get_builds_for_revision') as get_builds_for_revision:
... get_builds_for_revision.return_value = ['baz', 'bang']
... result = revision('try', 'bar')
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jzeller/buildapi-test/lib/python2.7/site-packages/mock.py", line 1252, in __enter__
self.target = self.getter()
File "/Users/jzeller/buildapi-test/lib/python2.7/site-packages/mock.py", line 1414, in <lambda>
getter = lambda: _importer(target)
File "/Users/jzeller/buildapi-test/lib/python2.7/site-packages/mock.py", line 1102, in _importer
thing = _dot_lookup(thing, comp, import_path)
File "/Users/jzeller/buildapi-test/lib/python2.7/site-packages/mock.py", line 1089, in _dot_lookup
return getattr(thing, comp)
File "/Users/jzeller/buildapi-test/lib/python2.7/site-packages/paste/registry.py", line 137, in __getattr__
return getattr(self._current_obj(), attr)
File "/Users/jzeller/buildapi-test/lib/python2.7/site-packages/paste/registry.py", line 197, in _current_obj
'thread' % self.____name__)
TypeError: No object (name: app_globals) has been registered for this thread
And here it is for http://pastebin.mozilla.org/2931832
>>> from buildapi.controllers.selfserve import SelfserveController
>>> from mock import patch, Mock
>>> s = SelfserveController()
>>> s._ok = Mock(name='_ok')
>>> s._get_stable_delay = Mock(return_value=180)
>>> s._branches_cache = Mock()
>>> s._branches_cache = {'try': None}
>>> s._failed = Mock(return_value="Failed")
>>> g = Mock()
>>> g.buildapi_cache = Mock(name='buildapi_cache')
>>> g.buildapi_cache.get_builds_for_revision = Mock(name='get_builds_for_revision', return_value=['baz', 'bang'])
>>> s.revision('try', 'bar')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "buildapi/controllers/selfserve.py", line 217, in revision
job_items = g.buildapi_cache.get_builds_for_revision(branch, revision)
File "/Users/jzeller/buildapi-test/lib/python2.7/site-packages/paste/registry.py", line 137, in __getattr__
return getattr(self._current_obj(), attr)
File "/Users/jzeller/buildapi-test/lib/python2.7/site-packages/paste/registry.py", line 197, in _current_obj
'thread' % self.____name__)
TypeError: No object (name: app_globals) has been registered for this thread
This is the same error that I have been getting when I try to run revision. Each and every time it errors out here because my attempt to mock this app_globals module has failed, or so it seems. I am really not sure how else to try this. Does anyone have any ideas?
Here are a few more good details:
Here is the function I am attempting to use, which is in the class I am instantiating: https://hg.mozilla.org/build/buildapi/file/efe11511f42d/buildapi/controllers/selfserve.py#l210
Note 1
I have already looked at the following threads which do not seem to quite apply to this specific scenario, or I am completely missing the point.
Mocking imported modules in Python
Mocking away the url object in Pylons

A controller could not be used as an independent object without an initialized Pylons' environment.
Using the functionality Pylons provides for testing saves you from all the boilerplate code to test a Controller.
I recommend you to use the TestController class in combination with nosetests as shown in documentation.
Specifically for your code, I think Pylons treats private methods (starting with _) differently from public methods because public methods run in the context of a request.

Creating objects in parallel with a multiprocessing pool?

Python 2.7.3
I have a folder containing thousands of data files. Each data file gets fed to a constructor and heavily processed. Right now I am iterating through the files and processing them sequentially:
class Foo:
def __init__(self,file):
self.bar = do_lots_of_stuff_with_numpy_and_scipy(file)
def do_lots_of_stuff_with_numpy_and_scipy(file):
pass
def get_foos(dir):
return [Foo(os.path.join(dir,file)) for file in os.listdir(dir)]
This works beautifully but is so slow. I would like to do this in parallel. I tried:
def parallel_get_foos(dir):
p = Pool()
foos = p.map(Foo, [os.path.join(dir,file) for file in os.listdir(dir)])
p.close()
p.join()
return foos
if __name__ == "__main__":
foos = parallel_get_foos(sys.argv[1])
But it just errors out with lots of these:
Process PoolWorker-7:
Traceback (most recent call last):
File "/l/python2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/l/python2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/l/python2.7/lib/python2.7/multiprocessing/pool.py", line 99, in worker
put((job, i, result))
File "/l/python2.7/lib/python2.7/multiprocessing/queues.py", line 390, in put
return send(obj)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
I have tried making a function to return the object, e.g.:
def get_foo(file):
return Foo(file)
def parallel_get_foos(dir):
...
foos = p.map(get_foo, [os.path.join(dir,file) for file in os.listdir(dir)])
...
but as expected I get the same error.
I have read through a great number of similar threads trying to address problems somewhat like this one but none of the solutions have helped me. So I appreciate any help!
EDIT:
Bakuriu correctly surmised that I am defining a non-top-level function inside of my do_lots_of_stuff method. In particular, I am doing as follows:
def fit_curve(data,degree):
"""Fits a least-square polynomial function to the given data."""
sorted = data[data[:,0].argsort()].T
coefficients = numpy.polyfit(sorted[0],sorted[1],degree)
def eval(val,deg=degree):
res = 0
for coefficient in coefficients:
res += coefficient*val**deg
deg -= 1
return res
return eval
Is there anyway to make this function pickleable?

What you are doing(at least, what you show in the examples), actually works fine:
$mkdir TestPool
$cd TestPool/
$for i in {1..100}
> do
> touch "test$i"
> done
$ls
test1 test18 test27 test36 test45 test54 test63 test72 test81 test90
test10 test19 test28 test37 test46 test55 test64 test73 test82 test91
test100 test2 test29 test38 test47 test56 test65 test74 test83 test92
test11 test20 test3 test39 test48 test57 test66 test75 test84 test93
test12 test21 test30 test4 test49 test58 test67 test76 test85 test94
test13 test22 test31 test40 test5 test59 test68 test77 test86 test95
test14 test23 test32 test41 test50 test6 test69 test78 test87 test96
test15 test24 test33 test42 test51 test60 test7 test79 test88 test97
test16 test25 test34 test43 test52 test61 test70 test8 test89 test98
test17 test26 test35 test44 test53 test62 test71 test80 test9 test99
$vi test_pool_dir.py
$cat test_pool_dir.py
import os
import multiprocessing
class Foo(object):
def __init__(self, fname):
self.fname = fname #or your calculations
def parallel_get_foos(directory):
p = multiprocessing.Pool()
foos = p.map(Foo, [os.path.join(directory, fname) for fname in os.listdir(directory)])
p.close()
p.join()
return foos
if __name__ == '__main__':
foos = parallel_get_foos('.')
print len(foos) #expected 101: 100 files plus this script
$python test_pool_dir.py
101
Version information:
$python --version
Python 2.7.3
$uname -a
Linux giacomo-Acer 3.2.0-39-generic #62-Ubuntu SMP Thu Feb 28 00:28:53 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
My guess is that you are not doing exactly what you show in the code samples you showed. For example I get an error similar to yours when doing this:
>>> import pickle
>>> def test():
... def test2(): pass
... return test2
...
>>> import multiprocessing
>>> p = multiprocessing.Pool()
>>> p.map(test(), [1,2,3])
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
Which is obvious since:
>>> pickle.dumps(test())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <function test2 at 0x7fad15fc2938>: it's not found as __main__.test2
And pickle's documentation states that:
The following types can be pickled:
None, True, and False
integers, long integers, floating point numbers, complex numbers
normal and Unicode strings
tuples, lists, sets, and dictionaries containing only picklable objects
functions defined at the top level of a module
built-in functions defined at the top level of a module
classes that are defined at the top level of a module
instances of such classes whose __dict__ or the result of calling __getstate__() is picklable (see section The pickle protocol
for details).
And continues:
Note that functions (built-in and user-defined) are pickled by “fully
qualified” name reference, not by value. This means that only the
function name is pickled, along with the name of the module the
function is defined in. Neither the function’s code, nor any of its
function attributes are pickled. Thus the defining module must be
importable in the unpickling environment, and the module must contain
the named object, otherwise an exception will be raised.