In order to simplify my code, I have put various functions into external files with I load via:
from (external_file) import (function_name)
...which works fine.
My question though has to do with other modules, such as cv2 or numpy - do I need those listed in my external file (as well as my main file) or is there a way to just list them in my main file?
Each file you put Python code in is its own module. Each module has its own namespace. If some of your code (in any module) uses some library code, it will need some way to access the library from the namespace it is defined in.
Usually this means you need to import the library in each module it's being used from. Don't worry about duplication, modules are cached when they are first loaded, so additional imports from other modules will quickly find the existing module and just add a reference to it in their own namespaces.
Note that it's generally not a good idea to split up your code too much. There's certainly no need for every function or every class to have its own file. Instead, use modules to group related things together. If you have a couple of functions that interoperate a lot, put them in the same module.
Related
I have a Python package with dozens of subpackages/modules. Nearly each of the modules uses open built-in Python function. I have written an own implementation of the file opening function and would like to "redirect" all open calls I have in the package modules to my_open function.
I am aware that it is possible to write open = my_open_file in the top of the module to shadow the open within the module, but this would imply editing each module. Alternatively, putting open = my_open_file in the __init__.py of the package and then doing from package_name import open which also implies adding a single line of code to each module.
Is it possible to define the my_open_file function for the package scope just in a single place? Or adding a single line of code in each module is my only option?
Think about what you're asking to do with this module: you've written your own package, a personal change to the way a built-in function is linked. You want to redefine a standard language feature. That's okay ... part of the power of a language is the capability to do things the way you define.
The problem comes when you want to do this by default, overriding that standard capability for code that hasn't explicitly requested that change. You cannot do this without some high-powered authority. The most straightforward way is to rebuild your Python system from sources, replacing the standard open with your own.
In reality, the "normal" way is to have each of your application modules "opt-in", declaring explicitly that it wants to use the new open instead of the one defined by the language. This is the approach you've outlined in your second paragraph. How is this a problem for you? Inserting a line in each of a list of parameterized files is a single system command.
I read here about sorting your import statements in Python, but what if the thing you are importing needs dependencies that have not been imported yet? Is this the difference between compiled languages and interpreted? I come from a JavaScript background and the order in which you load your scripts matter, whereas Python appears not to care. Thanks.
Import order does not matter. If a module relies on other modules, it needs to import them itself. Python treats each .py file as a self-contained unit as far as what's visible in that file.
(Technically, changing import order could change behavior, because modules can have initialization code that runs when they are first imported. If that initialization code has side effects it's possible for modules to have interactions with each other. However, this would be a design flaw in those modules. Import order is not supposed to matter, so initialization code should also be written to not depend on any particular ordering.)
Python Import order doesnot matter when you are importing standard python libraries/modules.
But, the order matters for your local application/library specific imports as you may stuck in circular dependency loop, so do look before importing.
No, it doesn't, because each python module should be self-contained and import everything it needs. This holds true for importing whole modules and only specific parts of it.
Order can matter for various nefarious reasons, including monkey patching.
I think the following is bad style:
# In foo_pkg/bla_mod.py:
magic_value=42
# In foo_pkg/bar_mod.py:
from .bla_mod import magic_value
# In doit.py:
from foo_pkg.bar_mod import magic_value
Instead, I'd like that to always import an object from the module where it has been defined, i.e. in this case:
# In doit.py:
from foo_pkg.bla_mod import magic_value
Finding issues of this sort by hand is getting tedious very quickly (for each imported object, you have to open the module and check if it defines the object, or if it imports in from another module).
What is the best way to automate this check? To my surprise, neither pylint nor pyflakes seem to have an appropriate checker, but maybe there's another tool (or even some trick that can be used in Python itself)?
Problem statement in a nutshell: given a bunch of python source files, find every import of an object from a module that does not itself define the object.
I know there are libraries (including the standard library) where one module provides the main external API and imports the necessary symbols from other modules internally. However, that's not the case in the code base I'm working with, here these are artifacts of refactorings that I really want to eliminate.
Here's a draft script that solves the problem less than 100 lines: http://pastebin.com/CFsR6b3s
when I read a file riak-python-client/riak/riak_object.py. At the bottom of the file, I saw this
from mapreduce import *
what's it use for? Why just import at the top of the file.
This is designed to put all of the module mapreduce in the riak_object namespace. If you put this import at the top of riak_object.py, then there would be an error because mapreduce imports RiakObject from riak_object, which is not defined yet.
You can use an import at any point in a file; you just have to make sure you don't try to use stuff from the module until after you have imported it.
If this is actually the last line run in the file, than it wouldn't serve any purpose. If it at the bottom of the file but inside some function or method, it might mean that that function/method is not run frequently and the author didn't want the overhead of importing mapreduce every time the program is run.
I don't know the specific reasoning for that project, but just put the imports at the top of your file. This makes the dependencies far easier to track, and it is recommended by Python's style guide.
Technically, the code that is imported by an import statement is only available after the import statement has been executed. This means that when you put it at the bottom of your file, you can't use anything imported there in that file (on module level). Functions in that file can use the imports... but it's just bad practice.
The only reason would be that riak_object (which doesn't seem to require anything from mapreduce?) is bringing into its namespace all the values (or those specified by __all__) from mapreduce as some kind of convenience.
How does one get (finds the location of) the dynamically imported modules from a python script ?
so, python from my understanding can dynamically (at run time) load modules.
Be it using _import_(module_name), or using the exec "from x import y", either using imp.find_module("module_name") and then imp.load_module(param1, param2, param3, param4) .
Knowing that I want to get all the dependencies for a python file. This would include getting (or at least I tried to) the dynamically loaded modules, those loaded either by using hard coded string objects or those returned by a function/method.
For normal import module_name and from x import y you can do either a manual scanning of the code or use module_finder.
So if I want to copy one python script and all its dependencies (including the custom dynamically loaded modules) how should I do that ?
You can't; the very nature of programming (in any language) means that you cannot predict what code will be executed without actually executing it. So you have no way of telling which modules could be included.
This is further confused by user-input, consider: __import__(sys.argv[1]).
There's a lot of theoretical information about the first problem, which is normally described as the Halting problem, the second just obviously can't be done.
From a theoretical perspective, you can never know exactly what/where modules are being imported. From a practical perspective, if you simply want to know where the modules are, check the module.__file__ attribute or run the script under python -v to find files when modules are loaded. This won't give you every module that could possibly be loaded, but will get most modules with mostly sane code.
See also: How do I find the location of Python module sources?
This is not possible to do 100% accurately. I answered a similar question here: Dependency Testing with Python
Just an idea and I'm not sure that it will work:
You could write a module that contains a wrapper for __builtin__.__import__. This wrapper would save a reference to the old __import__and then assign a function to __builtin__.__import__ that does the following:
whenever called, get the current stacktrace and work out the calling function. Maybe the information in the globals parameter to __import__ is enough.
get the module of that calling functions and store the name of this module and what will get imported
redirect the call the real __import__
After you have done this you can call your application with python -m magic_module yourapp.py. The magic module must store the information somewhere where you can retrieve it later.
That's quite of a question.
Static analysis is about predicting all possible run-time execution paths and making sure the program halts for specific input at all.
Which is equivalent to Halting Problem and unfortunately there is no generic solution.
The only way to resolve dynamic dependencies is to run the code.