I have to analyse some python file. For this I should analyse all modules (i.e. get source code of this modules if they written on Python) imported in given file.
How I can get paths to files with imported python-modules?
I try to use sys.path, but it gives all paths, where does python interpreter may search modules
For the 3rd party modules, following should print the file path.
module_file = <module_name>.__file__
You can then print the file contents.
Good question. I tried some things and the problem is that many standard modules seem quite inaccessible, like the math module isn't even Python, but a C library (.so file). If you only need to access user-defined modules and not standard ones, something like this could get you files and source:
import inspect
import sys
def main():
# sys.modules contains a mapping between module names and module
# objects, but many more than the one imported in a file. dir()
# returns a list of names available to the local scope (also variables
# functions etc.). Combine those two and you get the modules available
# to the local scope
modules = [sys.modules[i] for i in dir() if i in sys.modules]
files = []
code = []
for module in modules:
try:
# modules may have a __file__ attribute
files.append(module.__file__)
# get's you the actual code
code.append(inspect.getsource(module))
except:
pass
print(files, code)
if __name__ == '__main__':
main()
__file__ is not defined for all standard modules and inspect.getsource() doesn't work for them either, that's why the try-except block, but for non-standard modules this could get you started.
Modules are found in /usr/lib/python3/dist-packages in unix-like systems
and are found in C:\Python34\Lib in windows
Related
I have a file called dns_poison.py that needs to call a package called netscanner. When i try and load the icmpscan module from dns_poison.py I get this message:
ModuleNotFoundError: No module named 'icmpscan'
I've done a sys.path and can confirm that the correct path is in place. The files are located at D:\PythonProjects\Networking\tools and D:\PythonProjects appears when I do a sys.path.
Here is my directory structure:
dns_poison.py
netscanner/
__init__.py
icmpscan.py
Code snippets for the files are as follows:
dns_poison.py
import netscanner
netscanner\__init__.py
from icmpscan import ICMPScan
netscanner\icmpscan.py
class ICMPScan:
def __init__(self, target, count=2, timeout=1):
self.target = target
self.count = count
self.timeout = timeout
self.active_hosts = []
# further code below here....
I don't understand why it cannot find the module, as I've used this exact same method on other python projects without any problems. Any help would be much appreciated.
When you run python dns_poison.py, the importer checks the module path then the local directory and eventually finds your netscanner package that has the following available:
netscanner
netscanner.icmpscan
netscanner.icmpscan.ICMPScan
Now I ask you, where is just icmpscan? The importer cannot find because well, it doesnt exist. The PYTHONPATH exists at wherever dns_poison.py resides, and doesn't append itself to include the absolute path of any imported modules because that simply not how it works. So netscanner can be found because its at the same level as dns_poison.py, but the importer has no clue where icmpscan.py exists because you havent told it. So you have two options to alter your __init__.py:
from .icmpscan import ICMPScan which works with Python 3.x
from netscanner.icmpscan import ICMPScan which works with both Python 2.x/3.x
Couple of references for you:
Python Import System
Python Modules recommend you ref section 6.4.2 Intra-package References
The most simple way to think about this is imports should be handled relative to the program entry-point file. Personally I find this the most simple and fool-proof way of handling import paths.
In your example, I would have:
from netscanner.icmpscan import ICMPScan
In the main file, rather than add it to init.py.
I am trying to set up a regression test suite and incorporate the system configuration at runtime. The following code finds most of the modules used by the system (and their versions):
import pkg_resources
import sys
for m in sys.modules:
try:
ml.append(m+'-'+pkg_resources.get_distribution(m).version)
except:
pass
print "# modules used:",sort(ml)
which returns:
# modules used: ['IPython-4.0.0', 'PIL-1.1.7', 'bottleneck-1.0.0',
'decorator-4.0.2', 'ipykernel-4.0.3', 'ipython_genutils-0.1.0',
'jupyter_client-4.0.0', 'jupyter_core-4.0.4', 'matplotlib-1.4.3',
'numexpr-2.4.3', 'numpy-1.9.2', 'pandas-0.16.2', 'pexpect-3.3',
'pickleshare-0.5', 'pyparsing-2.0.3', 'pysal-1.10.0', 'pytz-2015.4',
'scipy-0.16.0', 'simplegeneric-0.8.1', 'six-1.9.0', 'tornado-4.2.1',
'traitlets-4.0.0', 'xlsxwriter-0.7.3']
But there are not all of them. A quick check of the following:
ml = ['sklearn-'+sklearn.__version__,
'osgeo-'+osgeo.__version__]
print "# package versions:",sort(ml)
returns:
# package versions: ['osgeo-2.0.0' 'sklearn-0.16.1']
which are installed and used in the main scripts. I should also add that some of the reported modules like tornado and xlsxwriter and not called or used directly, while some of the ones which are imported directly into the code were not found.
Does anyone know a trick to report jut the ones which are called from the scripts, or at least one that does not miss important ones?
I've been working on a project that creates its own .py files that store handlers for the method, I've been trying to figure out how to store the Python files in folder and open them. Here is the code I'm using to create the files if they don't already exist, then importing the file:
if os.path.isfile("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py") == False:
TestPy = open("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py","w+")
try:
TestPy.write(StrHandler)
except Exception as Error:
print(Error)
TestPy.close()
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
self.IntBtnID += 1
when I change this line:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
to this:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons\\Btn" + str(self.IntBtnID))
the fill can't be found and ends up throwing an error because it can't find the file in the folder.
Do know why it doesn't work I just don't know how to get around the issue:/
My question is how do I open the .py when its stored in a folder?
There are a couple of unidiomatic things in your code that may be the cause of your issue. First of all, it is generally better to use the functions in os.path to manipulate paths to files. From your backslash usage, it appears you're working on Windows, but the os.path module ensures consistent behaviour across all platforms.
Also there is importlib.import_module, which is usually recommended over __import__. Furthermore, in case you want to load the generated module more than once during the lifetime of your program, you have to do that explicitly using imp.reload.
One last tip: I'd factor out the module path to avoid having to change it in more than one place.
You can't reference a path directory when you are importing files. Instead, you want to add the directory to your path and then import the name of the module.
import sys
sys.path.append( "Buttons" )
__import__("Btn"+str(self.IntBtnId))
See this so question for more information.
The first argument to the __import__() function is the name of the module, not a path to it. Therefore I think you need to use:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons.Btn" + str(self.IntBtnID))
You may also need to put an empty __init__.py file in the Buttons folder to indicate it's a package of modules.
Here is a simple case: I want to define a module in python name robot. So, I have a folder named robot with these two files:
__init__.py:
from test import a
test.py:
a = "hello world"
Now, when I import robot in the interpreter, the robot namespace includes test and a. However, I only want it to include a. Why this odd behavior?
EDIT:
Here's a slightly more representative example of what I want to achieve:
Given the following files:
__init__.py:
from spam import a
import ham
spam.py:
a = "hello world"
ham.py:
b = "foo"
Can I have a robot namespace containing a and ham at its top level but not spam?
You have created not just a module but a package. A package contains its submodules in its namespace (once they have been imported, as you imported test here). This is as it should be, since the usual way of using packages is to provide a grouping of several modules. There's not much use to making a package with only one module (i.e., one contentful .py file) inside it.
If you just want a one-file module, just create a file called robots.py and put your code in there.
Edit: See this previous question. The answer is that you should in general not worry about excluding module names from your package namespace. The modules are supposed to be in the package namespace. If you want to add functions and stuff from submodules as well, for convenience, that's fine, but there's not really anything to be gained by "covering your tracks" and hiding the modules you imported. However, as described in the answers to that question, there are some hackish ways to approximate what you want.
Are you just asking how to import specific modules or functions?
test.py:
import robot.spam.a
import robot.ham
Don't import the entire package.
This snippet is from an earlier answer here on SO. It is about a year old (and the answer was not accepted). I am new to Python and I am finding the system path a real pain. I have a few functions written in scripts in different directories, and I would like to be able to import them into new projects without having to jump through hoops.
This is the snippet:
def import_path(fullpath):
""" Import a file with full path specification. Allows one to
import from anywhere, something __import__ does not do.
"""
path, filename = os.path.split(fullpath)
filename, ext = os.path.splitext(filename)
sys.path.append(path)
module = __import__(filename)
reload(module) # Might be out of date
del sys.path[-1]
return module
Its from here:
How to do relative imports in Python?
I would like some feedback as to whether I can use it or not - and if there are any undesirable side effects that may not be obvious to a newbie.
I intend to use it something like this:
import_path(/home/pydev/path1/script1.py)
script1.func1()
etc
Is it 'safe' to use the function in the way I intend to?
The "official" and fully safe approach is the imp module of the standard Python library.
Use imp.find_module to find the module on your precisely-specified list of acceptable directories -- it returns a 3-tuple (file, pathname, description) -- if unsuccessful, file is actually None (but it can also raise ImportError so you should use a try/except for that as well as checking if file is None:).
If the search is successful, call imp.load_module (in a try/finally to make sure you close the file!) with the above three arguments after the first one which must be the same name you passed to find_module -- it returns the module object (phew;-).
As mentioned, please consider thread safety, if appropriate. I prefer something closer to a solution posted in a similar post. The main differences below: the use of insert to specify priority of the import, correct restoration of sys.path using try...finally, and setting the global namespace.
# inspired by Alex Martelli's solution to
# http://stackoverflow.com/questions/1096216/override-namespace-in-python/1096247#1096247
def import_from_absolute_path(fullpath, global_name=None):
"""Dynamic script import using full path."""
import os
import sys
script_dir, filename = os.path.split(fullpath)
script, ext = os.path.splitext(filename)
sys.path.insert(0, script_dir)
try:
module = __import__(script)
if global_name is None:
global_name = script
globals()[global_name] = module
sys.modules[global_name] = module
finally:
del sys.path[0]
It does feel like a bit of a hack, but at the moment, I can't think of any unintended side effects that are likely to occur, at least not as long as you're just using this for your own scripts. Basically what it does is temporarily add the parent directory of the specified file (in your example, /home/pydev/path1/) to the list of paths that Python checks when it's looking for a module to import.
The only risk I can think of right now would arise in a multithreaded environment, where two or more threads (or processes) are running this function simultaneously. If thread A wants to import module A from path dirA/A.py, and thread B wants to import module B from path dirB/B.py, you'd wind up with both dirA and dirB in sys.path for a short time. And if there is a file named B.py in dirA, it's possible that thread B will find that (dirA/B.py) instead of the file it's looking for (dirB/B.py), thus importing the wrong module. For this reason, I wouldn't use it in production code, or code that you're going to distribute to other people (at least not without warning them that this hack is in here!). In a situation like that, you could write a more complex function that allows you to specify the file to import without messing with the standard set of paths. (That's what mod_python does, for example)
I would be worried that your script name might correspond with a module that shows up earlier in the path. To dispel this fear, I would fully replace the path with a new list containing just the directory containing the module, then put it back once the import has completed. Also, you should wrap this in some sort of lock so that multiple threads trying to do the same thing don't interfere with each other.