So I have built a python library in a file, call it mylib.py, which I then import in various scripts using import mylib. I wish to split mylib.py into multiple files, say mylib1.py and mylib2.py, while still using them as a single library like so: import mylib , mylib.mymethod(), etc...
Is there any way to do this that is perfectly equivalent to having them in a single file? This means that, for instance, mylib1 should be able to use methods from mylib2.
Thank you
The "proper" way to do this is to create a package, and have the package's __init__.py file import the desired symbols. This will assign the imported symbols to the package object.
For example, suppose I have the following file, my_lib.py:
def foo():
print("Hello world, I'm Foo!")
def bar():
print("Hello world, I'm Bar!")
I can split this file into two, foo_module.py and bar_module.py (containing the "foo" and "bar" functions respectively), then place them in a package my_lib:
mylib
├── bar_module.py
├── foo_module.py
└── __init__.py
A regular Python package is a directory with an __init__.py file in it, in addition to the module files it contains.
Now, as it currently stands, this won't make import my_lib act exactly as before. This is because the foo and bar functions are contained in submodules of my_lib, so they are only available if you import those submodules:
from my_lib import foo_module
from my_lib import bar_module
foo_module.foo()
bar_module.bar()
To make them available directly from my_lib, we need to modify its __init__.py to import those functions:
from my_lib.foo_module import foo
from my_lib.bar_module import bar
This imports foo and bar into my_lib's namespace. From then on, whenever some other code imports my_lib, it will be able to directly access foo and bar.
Just put import mylib2 in mylib 1, so when you import mylib1, it can use methods from mylib2.
The main program might not be able to use mylib2 functions, though.
In lib.py, rather than doing import one, you could do from one import * (and same with two). Doing this will put all the objects from one into the lib namespace (or something like that. That means than inside of lib you wouldn't call one.func_one(), but you would just call func_one() straight away.
So then, in main.py (or whatever you have that you are ultimately using) you could import lib and then call lib.func_one() even though func_one() itself comes from one.py and not from lib.py.
In main.py you could even do from lib import * and you could call everything directly without the lib., but I think that is a bad idea - it is usually nice to see throughout the code where things are coming from and to not clutter up the namespace.
When you do from xxx import * you bring in the names of all of those things directly, and you could accidently write over something if two libraries/files have functions or global variables with the same name.
Related
Instead of creating a module.py (containing all the functions), can I create a folder MODULE and collect all the functions in different files?
I'd like to do that in a way that main.py contains import MODULE and, if it's possible, to call the functions directly (fun_1(), fun_2()) without the nomenclature MODULE.fun_1(), MODULE.fun_2(), etc.
I think the only right way is creating __init__.py (import all functions contained in other files) in your MODULE folder. And use statement from MODULE import *. If you wanna use import MODULE and call func in other files then, that never work. Interpreter will raise the NameError, cause there are no the variables.
__init__.py file like this:
from file1 import func_1
from file2 import func_2
Yes.
Create your main file, main.py along with another file where you want to put another function. I will use the other.py file.
In main.py, write
from other import *
Note the missing .py, Python handles that for you.
In other.py, write
def your_function():
return -1
You can have as many functions in here as you want. Now, you can call your_function() in main.py.
Example Repl.it: https://repl.it/repls/MeaslyBurdensomeDesigners
I have a plugins package that contains several modules, each defining one class (each class is a plugin).
My package structure looks like this :
plugins
├ __init__.py
├ first_plugin.py
├ second_plugin.py
└ third_plugin.py
And a plugin file typically looks like this, only containing a class definition (and a few imports if necessary) :
# in first_plugin.py
class MyFirstPlugin:
...
I would like the end user to be able to import a plugin like so :
from plugins import FirstPlugin
instead of having to also type the module name (which is what is currently required to do) :
from plugins.first_plugin import FirstPlugin
Is there a way to achieve this by re-exporting the modules' classes directly in the __init__.py file without having to import everything module by module like so (which becomes cumbersome when there are lots of modules) :
# in __init__.py
from .first_plugin import FirstPlugin
from .second_plugin import SecondPlugin
from .third_plugin import ThirdPlugin
I do not think this is possible in Python. However you can import entire modules so you do not have to import each class individually.
For example
from first_plugin import *
Allowing you to do
from plugin import # Anything in first_plugin
Its kinda a pain but writing libraries is not easy (wait till you use CMake with C/C++, you have to specify every single file in your source tree :D)
I think you could elaborate on answers of this post: How to import all submodules?
For example with pkgutil.walk_packages(__path__) you'd have a list of modules. Then you could use dir on the loaded module and import the results (filtering elements starting with __
I simply want to take all my .py files from a single folder (I don't care about the sub-folders for now) and put them into a single module.
The use case I'm having here is that I'm writing some pretty standard object-oriented code and I'm using a single file for every class, and I don't want to have to write from myClass import myClass for every class into my __init__.py. I can't use Python3, so I'm still working with impand reloadand such.
At the moment I'm using
# this is __init__.py
import pkgutil
for loader, name, is_pkg in pkgutil.walk_packages(__path__):
if not is_pkg:
__import__(__name__ + "." + name)
and it doesn't seem to work, it includes the packages but it includes them as modules, so that I have to write MyClass.MyClass for a class that is defined in a file with it's own name. That's silly and I don't like it.
I've been searching forever and I'm just getting more confused how complicated this seemingly standard use case seems to be. Do python devs just write everything into a single file? Or do they always have tons of imports?
Is this something that should be approached in an entirely different way?
What you really want to do
To do the job you need to bind your class names to namespace of your __init__.py script.
After this step you will be able to just from YourPackageName import * and just use your classes directly. Like this:
import YourPackageName
c = YourPackageName.MyClass()
or
from YourPackageName import *
c = MyClass()
Ways to achieve this
You have multiple ways to import modules dynamically: __import__(), __all__.
But.
The only way to bind names into namespace of current module is to use from myClass import myClass statement. Static statement.
In other words, content of each of your __init__.py scripts should be looking like that:
#!/usr/bin/env python
# coding=utf-8
from .MySubPackage import *
from .MyAnotherSubPackage import *
from .my_pretty_class import myPrettyClass
from .my_another_class import myAnotherClass
...
And you should know that even for a dynamic __all__:
It is up to the package author to keep this list up-to-date when a new version of the package is released.
(https://docs.python.org/2/tutorial/modules.html#importing-from-a-package)
So a clear answers to your questions:
Do python devs just write everything into a single file?
No, they don't.
Or do they always have tons of imports?
Almost. But definitely not tons. You need to import each of your modules just once (into an appropriate __init__.py scripts). And then just import whole package or sub-package at once.
Example
Let's assume that there is next package structure:
MyPackage
|---MySubPackage
| |---__init__.py
| |---pretty_class_1.py
| |---pretty_class_2.py
|---__init__.py
|---sleepy_class_1.py
|---sleepy_class_2.py
Content of the MyPackage/MySubPackage/__init__.py:
#!/usr/bin/env python
# coding=utf-8
from .pretty_class_1 import PrettyClass1
from .pretty_class_2 import PrettyClass2
Content of the MyPackage/__init__.py:
#!/usr/bin/env python
# coding=utf-8
from .MySubPackage import *
from .sleepy_class_1 import SleepyClass1
from .sleepy_class_2 import SleepyClass2
As result, now we are able to write next code in our application:
import MyPackage
p = MyPackage.PrettyClass1()
s = MyPackage.SleepyClass2()
or
from MyPackage import *
p = PrettyClass1()
s = SleepyClass2()
i've run through many posts about this, but still doesn't seem to work. The deal is pretty cut. I've the got the following hierarchy.
main.py
DirA/
__init__.py
hello.py
DirB/
__init__.py
foo.py
bla.py
lol.py
The__init__.py at DirA is empty. The respective one at DirB just contains the foo module.
__all__.py = ["foo"]
The main.py has the following code
import DirA
import DirB
hey() #Def written at hello.py
foolish1() #Def written at foo.py
foolish2() #Def written at foo.py
Long story short, I got NameError: name 'foo' is not defined. Any ideas? Thanks in advance.
You only get what you import. Therefore, in you main, you only get DirA and DirB. You would use them in one of those ways:
import DirA
DirA.something_in_init_py()
# Importing hello:
import DirA.hello
DirA.hello.something_in_hello_py()
# Using a named import:
from DirA.hello import something_in_hello_py
something_in_hello_py()
And in DirB, just make the __init__.py empty as well. The only use of __all__ is for when you want to import *, which you don't want because, as they say, explicit is better than implicit.
But in case you are curious, it would work this way:
from DirB import *
something_in_dirb()
By default the import * will import everything it can find that does not start with an underscore. Specifying a __all__ restricts what it imported to the names defined in __all__. See this question for more details.
Edit: about init.
The __init__.py is not really connected to the importing stuff. It is just a special file with the following properties:
Its existence means the directory is a python package, with several modules in it. If it does not exist, python will refuse to import anything from the directory.
It will always be loaded before loading anything else in the directory.
Its content will be available as the package itself.
Just try it put this in DirA/__init__.py:
foo = 42
Now, in your main:
from DirA import foo
print(foo) # 42
It can be useful, because you can import some of your submodules in the __init__.py to hide the inner structure of your package. Suppose you build an application with classes Author, Book and Review. To make it easier to read, you give each class its own file in a package. Now in your main, you have to import the full path:
from myapp.author import Author
from myapp.book import Book
from myapp.review import Review
Clearly not optimal. Now suppose you put those exact lines above in your __init__.py, you may simplify you main like this:
from myapp import Author, Book, Review
Python will load the __init__.py, which will in turn load all submodules and import the classes, making them available on the package. Now your main does not need to know where the classes are actually implemented.
Have you tried something like this:
One way
from DirA import hello
Another way
from DirA.hello import hey
If those don't work then append a new system path
You need to import the function itself:
How to call a function from another file in Python?
In your case:
from DirA import foolish1, foolish2
I'm taking a look at how the model system in django works and I noticed something that I don't understand.
I know that you create an empty __init__.py file to specify that the current directory is a package. And that you can set some variable in __init__.py so that import * works properly.
But django adds a bunch of from ... import ... statements and defines a bunch of classes in __init__.py. Why? Doesn't this just make things look messy? Is there a reason that requires this code in __init__.py?
All imports in __init__.py are made available when you import the package (directory) that contains it.
Example:
./dir/__init__.py:
import something
./test.py:
import dir
# can now use dir.something
EDIT: forgot to mention, the code in __init__.py runs the first time you import any module from that directory. So it's normally a good place to put any package-level initialisation code.
EDIT2: dgrant pointed out to a possible confusion in my example. In __init__.py import something can import any module, not necessary from the package. For example, we can replace it with import datetime, then in our top level test.py both of these snippets will work:
import dir
print dir.datetime.datetime.now()
and
import dir.some_module_in_dir
print dir.datetime.datetime.now()
The bottom line is: all names assigned in __init__.py, be it imported modules, functions or classes, are automatically available in the package namespace whenever you import the package or a module in the package.
It's just personal preference really, and has to do with the layout of your python modules.
Let's say you have a module called erikutils. There are two ways that it can be a module, either you have a file called erikutils.py on your sys.path or you have a directory called erikutils on your sys.path with an empty __init__.py file inside it. Then let's say you have a bunch of modules called fileutils, procutils, parseutils and you want those to be sub-modules under erikutils. So you make some .py files called fileutils.py, procutils.py, and parseutils.py:
erikutils
__init__.py
fileutils.py
procutils.py
parseutils.py
Maybe you have a few functions that just don't belong in the fileutils, procutils, or parseutils modules. And let's say you don't feel like creating a new module called miscutils. AND, you'd like to be able to call the function like so:
erikutils.foo()
erikutils.bar()
rather than doing
erikutils.miscutils.foo()
erikutils.miscutils.bar()
So because the erikutils module is a directory, not a file, we have to define it's functions inside the __init__.py file.
In django, the best example I can think of is django.db.models.fields. ALL the django *Field classes are defined in the __init__.py file in the django/db/models/fields directory. I guess they did this because they didn't want to cram everything into a hypothetical django/db/models/fields.py model, so they split it out into a few submodules (related.py, files.py, for example) and they stuck the made *Field definitions in the fields module itself (hence, __init__.py).
Using the __init__.py file allows you to make the internal package structure invisible from the outside. If the internal structure changes (e.g. because you split one fat module into two) you only have to adjust the __init__.py file, but not the code that depends on the package. You can also make parts of your package invisible, e.g. if they are not ready for general usage.
Note that you can use the del command, so a typical __init__.py may look like this:
from somemodule import some_function1, some_function2, SomeObject
del somemodule
Now if you decide to split somemodule the new __init__.py might be:
from somemodule1 import some_function1, some_function2
from somemodule2 import SomeObject
del somemodule1
del somemodule2
From the outside the package still looks exactly as before.
"We recommend not putting much code in an __init__.py file, though. Programmers do not expect actual logic to happen in this file, and much like with from x import *, it can trip them up if they are looking for the declaration of a particular piece of code and can't find it until they check __init__.py. "
-- Python Object-Oriented Programming Fourth Edition Steven F. Lott Dusty Phillips