Why can't I get the definition of Callable from module collections in the following code?
How can I get the definition of a class in a module? Thanks.
>>> from collections import Callable
>>> inspect.getsource(Callable)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/inspect.py", line 944, in getsource
lines, lnum = getsourcelines(object)
File "/usr/lib/python3.5/inspect.py", line 931, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python3.5/inspect.py", line 788, in findsource
raise OSError('could not find class definition')
OSError: could not find class definition
>>> inspect.getsourcelines(Callable)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/inspect.py", line 931, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python3.5/inspect.py", line 788, in findsource
raise OSError('could not find class definition')
OSError: could not find class definition
>>> inspect.getmodule(Callable)
<module 'collections.abc' from '/usr/lib/python3.5/collections/abc.py'>
>>> inspect.getfile(Callable)
'/usr/lib/python3.5/collections/abc.py'
>>> inspect.getsourcefile(Callable)
'/usr/lib/python3.5/collections/abc.py'
In general, this is easily done with inspect.getsource which accepts a module, a class, a method, a function, a traceback, a frame,
or a code object. The source they represent should of course be written in Python else an error is raised.
In this specific case, you just happen to be unlucky because while Callable is defined in _collections_abc the Callable.__module__ name is callections.abc:
>>> Callable.__module__
'collections.abc'
This throws getsource off because it won't look in _collections_abc that contains Callables definition but, instead, in collections.abc which merely imports all definitions from _collections_abc:
>>> print(getsource(collections.abc))
from _collections_abc import *
from _collections_abc import __all__
Normally, getsource doesn't have an issue finding the source, for example, on itself:
>>> print(getsource(getsource))
def getsource(object):
"""Return the text of the source code for an object.
The argument may be a module, class, method, function, traceback, frame,
or code object. The source code is returned as a single string. An
OSError is raised if the source code cannot be retrieved."""
lines, lnum = getsourcelines(object)
return ''.join(lines)
In this specific case, though, it does (due to Callable.__module__ returning collections.abc.) You could replace __module__ with '_collections_abc' for a tricky way to see the source:
>>> Callable.__module__ = '_collections_abc'
>>> src = getsource(Callable)
>>> print(src)
class Callable(metaclass=ABCMeta):
__slots__ = ()
#abstractmethod
def __call__(self, *args, **kwds):
return False
#classmethod
def __subclasshook__(cls, C):
if cls is Callable:
return _check_methods(C, "__call__")
return NotImplemented
but this doesn't make me feel very comfortable.
get_source(fullname)
Return the source code for the specified module.
So you should return the whole module, as Callable is defined in the module of _collections_abc, so your code should be this:
import _collections_abc
import inspect
print(inspect.getsource(_collections_abc))
and you can see the definition of Callable in the print result.
Related
When trying to use the overloading function as described in the pep-3124*, I am having some trouble:
[luca#artix tmp]$ more foo.py
from overloading import overload
from collections import Iterable
def flatten(ob):
"""Flatten an object to its component iterables"""
yield ob
#overload
def flatten(ob: Iterable):
for o in ob:
for ob in flatten(o):
yield ob
#overload
def flatten(ob: basestring):
yield ob
[luca#artix tmp]$ python3 foo.py
/tmp/foo.py:2: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
from collections import Iterable
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/overloading.py", line 63, in overload
return register(__registry[fname], func)
KeyError: '__main__.flatten'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/foo.py", line 9, in <module>
def flatten(ob: Iterable):
File "/usr/lib/python3.9/site-packages/overloading.py", line 65, in overload
__registry[fname] = overloaded(func)
File "/usr/lib/python3.9/site-packages/overloading.py", line 151, in overloaded
return register(dispatcher, func)
File "/usr/lib/python3.9/site-packages/overloading.py", line 199, in register
signature = get_signature(fn)
File "/usr/lib/python3.9/site-packages/overloading.py", line 441, in get_signature
types = tuple(normalize_type(type_hints.get(param, AnyType)) for param in parameters)
File "/usr/lib/python3.9/site-packages/overloading.py", line 441, in <genexpr>
types = tuple(normalize_type(type_hints.get(param, AnyType)) for param in parameters)
File "/usr/lib/python3.9/site-packages/overloading.py", line 468, in normalize_type
if not typing or not isinstance(type_, typing.TypingMeta) or type_ is AnyType:
AttributeError: module 'typing' has no attribute 'TypingMeta'
[luca#artix tmp]$
Note that I uninstalled the typing module.
https://www.python.org/dev/peps/pep-3124/
From to this thread on GitHub:
The downgrade of the typing module helped me (from the python repository) to version 3.5.2.2. You can also forcefully ignore the typing module in the overloading module by assigning typing = None after the 30th line in the file overloading.py, but I think the typing module features will not be available.
UPDATE: As pointed out by Martijn Pieters ♦ and MisterMiyagi in the comments, it would be much more practical to uninstall the overloading package, rather then degrading your python version, as the standard library already includes functools.singledispatch() to cover this functionality.
I was going through this question : How do I return the definition of a class in python?
But I am unable to display the class definition. I am getting the below error:
>>> class A:
... pass
...
>>> import inspect
>>> source_text = inspect.getsource(A)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\**\Python\Python36\lib\inspect.py", line 968, in getsource
lines, lnum = getsourcelines(object)
File "C:\Users\**\Python\Python36\lib\inspect.py", line 955, in getsourcelines
lines, lnum = findsource(object)
File "C:\Users\**\Python\Python36\lib\inspect.py", line 768, in findsource
file = getsourcefile(object)
File "C:\Users\**\Python\Python36\lib\inspect.py", line 684, in getsourcefile
filename = getfile(object)
File "C:\Users\**\Python\Python36\lib\inspect.py", line 654, in getfile
raise TypeError('{!r} is a built-in class'.format(object))
TypeError: <module '__main__' (<_frozen_importlib_external.SourceFileLoader object at 0x0000026A79293F60>)> is a built-in class
>>>
Can someone please advise what am I doing wrong here? Thanks.
The inspect.getsource() function only works if there is a text file available to load the source code.
You typed the definition of the class into the interactive interpreter, which doesn't keep the original source around when compiling that source into in-memory class and code objects.
Put your class definition into a module, import the module, and then use inspect.getsource().
inspect.getsource() works by first finding the module for a given object (for classes, by looking at the ClassObj.__module__ attribute for the module name, then getting the module via sys.modules[modulename]) then seeing if the module has a __file__ attribute from which a readable source file can be determined. If there is such a filename and it can be read, then the inspect module reads that file to then search for the class ClassName: line and giving you all lines from that point on with same or deeper indentation. The interactive interpreter executes everything in the __main__ module and there is no __file__ attribute for the interpreter, so any attempts at loading source code for objects defined there will simply fail.
If you just wanted to know what members the class defines, use dir() or help() on the object instead. You don't need to see the full source code for that information.
How to get module source code by a python 'object' of that module?
class TestClass(object):
def __init__(self):
pass
def testMethod(self):
print 'abc'
return 'abc'
It's well-known that
print inspect.getsource(TestClass)
can be used to get source code of 'TestClass'.
However, the result of
ob = TestClass()
print inspect.getsource(ob)
as below, is not as expected.
Traceback (most recent call last):
File "D:\Workspaces\WS1\SomeProject\src\python\utils\ModuleUtils.py", line 154, in <module>
print inspect.getsource(ob)
File "C:\SciSoft\WinPython-64bit-2.7.10.3\python-2.7.10.amd64\lib\inspect.py", line 701, in getsource
lines, lnum = getsourcelines(object)
File "C:\SciSoft\WinPython-64bit-2.7.10.3\python-2.7.10.amd64\lib\inspect.py", line 690, in getsourcelines
lines, lnum = findsource(object)
File "C:\SciSoft\WinPython-64bit-2.7.10.3\python-2.7.10.amd64\lib\inspect.py", line 526, in findsource
file = getfile(object)
File "C:\SciSoft\WinPython-64bit-2.7.10.3\python-2.7.10.amd64\lib\inspect.py", line 420, in getfile
'function, traceback, frame, or code object'.format(object))
TypeError: <utils.TestClass.TestClass object at 0x0000000003C337B8> is not a module, class, method, function, traceback, frame, or code object
The question is:
If there is an object like 'ob' above, how to check the module source code of ob, or 'TestClass', via a method that takes 'ob' itself as a parameter?
In short, implement the following module
def getSource(obj):
###returns the result which is exactly identical to inspect.getsource(TestClass)
ob = TestClass()
###prints the result which is exactly identical to inspect.getsource(TestClass)
print getSource(ob)
(The scenario of a method like this is much more common than inspect.getsource(); for example, check the source code of an unknown, unpickled object.)
Instances don't have source code.
Use:
print inspect.getsource(type(ob))
or:
print inspect.getsource(ob.__class__)
I meet a method in a class. when i tried to find the source code of the method in ipython, i get information like below:
In [9]: model.elementary_rates??
Type: function
String form: <function elementary_rates at 0x06864170>
File: Dynamically generated function. No source code available.
Definition: model.elementary_rates(rate_constants, theta, p, mpf, matrix)
Docstring: <no docstring>
What is the "dynamically generated function"? Could you provide me some information or examples of it?
You could get a "Dynamically generated function" if the function is created by exec:
In [34]: code = compile('def f():\n return 3', '<string>', 'exec')
In [35]: exec code
In [36]: f?
Type: function
String Form:<function f at 0x7f6db467ac80>
File: Dynamically generated function. No source code available.
Definition: f()
Docstring: <no docstring>
The function wasn't defined in a file but it was created on the fly, for example:
>>> globals()['f'] = lambda: 3
>>>
>>> f
<function <lambda> at 0x10f337cf8>
>>> f()
3
>>> import inspect
>>> inspect.getsource(f)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 701, in getsource
lines, lnum = getsourcelines(object)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 690, in getsourcelines
lines, lnum = findsource(object)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 538, in findsource
raise IOError('could not get source code')
IOError: could not get source code
There are various ways of creating a function but in case the source code can't be retrieved (such as module, filename, location in the file, etc) then the function can be considered a dynamically generated function.
The following line creates a function called f which returns the value 3 when called:
globals()['f'] = lambda: 3
The globals() call means it's added to the global namespace, which is where you're at when the interpreter is running.
Python 2.7.3
I have a folder containing thousands of data files. Each data file gets fed to a constructor and heavily processed. Right now I am iterating through the files and processing them sequentially:
class Foo:
def __init__(self,file):
self.bar = do_lots_of_stuff_with_numpy_and_scipy(file)
def do_lots_of_stuff_with_numpy_and_scipy(file):
pass
def get_foos(dir):
return [Foo(os.path.join(dir,file)) for file in os.listdir(dir)]
This works beautifully but is so slow. I would like to do this in parallel. I tried:
def parallel_get_foos(dir):
p = Pool()
foos = p.map(Foo, [os.path.join(dir,file) for file in os.listdir(dir)])
p.close()
p.join()
return foos
if __name__ == "__main__":
foos = parallel_get_foos(sys.argv[1])
But it just errors out with lots of these:
Process PoolWorker-7:
Traceback (most recent call last):
File "/l/python2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/l/python2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/l/python2.7/lib/python2.7/multiprocessing/pool.py", line 99, in worker
put((job, i, result))
File "/l/python2.7/lib/python2.7/multiprocessing/queues.py", line 390, in put
return send(obj)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
I have tried making a function to return the object, e.g.:
def get_foo(file):
return Foo(file)
def parallel_get_foos(dir):
...
foos = p.map(get_foo, [os.path.join(dir,file) for file in os.listdir(dir)])
...
but as expected I get the same error.
I have read through a great number of similar threads trying to address problems somewhat like this one but none of the solutions have helped me. So I appreciate any help!
EDIT:
Bakuriu correctly surmised that I am defining a non-top-level function inside of my do_lots_of_stuff method. In particular, I am doing as follows:
def fit_curve(data,degree):
"""Fits a least-square polynomial function to the given data."""
sorted = data[data[:,0].argsort()].T
coefficients = numpy.polyfit(sorted[0],sorted[1],degree)
def eval(val,deg=degree):
res = 0
for coefficient in coefficients:
res += coefficient*val**deg
deg -= 1
return res
return eval
Is there anyway to make this function pickleable?
What you are doing(at least, what you show in the examples), actually works fine:
$mkdir TestPool
$cd TestPool/
$for i in {1..100}
> do
> touch "test$i"
> done
$ls
test1 test18 test27 test36 test45 test54 test63 test72 test81 test90
test10 test19 test28 test37 test46 test55 test64 test73 test82 test91
test100 test2 test29 test38 test47 test56 test65 test74 test83 test92
test11 test20 test3 test39 test48 test57 test66 test75 test84 test93
test12 test21 test30 test4 test49 test58 test67 test76 test85 test94
test13 test22 test31 test40 test5 test59 test68 test77 test86 test95
test14 test23 test32 test41 test50 test6 test69 test78 test87 test96
test15 test24 test33 test42 test51 test60 test7 test79 test88 test97
test16 test25 test34 test43 test52 test61 test70 test8 test89 test98
test17 test26 test35 test44 test53 test62 test71 test80 test9 test99
$vi test_pool_dir.py
$cat test_pool_dir.py
import os
import multiprocessing
class Foo(object):
def __init__(self, fname):
self.fname = fname #or your calculations
def parallel_get_foos(directory):
p = multiprocessing.Pool()
foos = p.map(Foo, [os.path.join(directory, fname) for fname in os.listdir(directory)])
p.close()
p.join()
return foos
if __name__ == '__main__':
foos = parallel_get_foos('.')
print len(foos) #expected 101: 100 files plus this script
$python test_pool_dir.py
101
Version information:
$python --version
Python 2.7.3
$uname -a
Linux giacomo-Acer 3.2.0-39-generic #62-Ubuntu SMP Thu Feb 28 00:28:53 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
My guess is that you are not doing exactly what you show in the code samples you showed. For example I get an error similar to yours when doing this:
>>> import pickle
>>> def test():
... def test2(): pass
... return test2
...
>>> import multiprocessing
>>> p = multiprocessing.Pool()
>>> p.map(test(), [1,2,3])
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
Which is obvious since:
>>> pickle.dumps(test())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <function test2 at 0x7fad15fc2938>: it's not found as __main__.test2
And pickle's documentation states that:
The following types can be pickled:
None, True, and False
integers, long integers, floating point numbers, complex numbers
normal and Unicode strings
tuples, lists, sets, and dictionaries containing only picklable objects
functions defined at the top level of a module
built-in functions defined at the top level of a module
classes that are defined at the top level of a module
instances of such classes whose __dict__ or the result of calling __getstate__() is picklable (see section The pickle protocol
for details).
And continues:
Note that functions (built-in and user-defined) are pickled by “fully
qualified” name reference, not by value. This means that only the
function name is pickled, along with the name of the module the
function is defined in. Neither the function’s code, nor any of its
function attributes are pickled. Thus the defining module must be
importable in the unpickling environment, and the module must contain
the named object, otherwise an exception will be raised.