I have been creating a few modules organized by purpose, and each module contain a number number of functions. I would like to bundle these individual modules into a larger "package" that other users can import from a shared location.
Currently, I have all of my modules in one folder, called python_modules and I have appended this path to os.path so I can easily import my individual modules as needed.
However, I would like to instead import a single package, that contains all of my modules, so I don't have to import each one individually. I know that I could put all my modules into one file, but that doesn't seem like a good way to organize my processes.
Currently, I have to following files in my python_modules folder:
__init__.py
load_data_functions.py
parse_data_functions.py
network_functions.py
counting_function.py
math_functions.py
...
...
other_functions.py
The __init__.py file is empty and does not have anything inside of it. The other modules all have various functions inside of them, and some are dependent on others. For example, network_functions.py relies on load_data_functions.py and parse_data_functions.py.
As I said, I want to package all of these modules into a larger package that I can share with others, and so we don't have to import each module independently.
A Package is just a bunch of modules. You already have an __init__.py file, so python_modules is already a package. You should simply be able to import python_modules and then call functions from each individual module as load_data_functions.some_function(), parse_data_functions.some_other_function().
assuming "I would like to instead import a single package" means you want to be able to something like:
import python_modules
port = python_modules.network_functions.get_port()
your __init__.py file should look like:
from . import load_data_functions
from . import parse_data_functions
from . import network_functions
...
if you'd like to be able to do:
import python_modules
port = python_modules.get_port()
your __init__.py file should look like:
from .load_data_functions import *
from .parse_data_functions import *
from .network_functions import *
...
As for modules within the package referring to each other, you can use the same idea, e.g. at the top of network_funtions.py you'd want put from . import load_data_functions. You should structure things as to avoid circular imports.
Related
I have a number of functions that I have written myself over time, over time I have placed each of these functions in their own modules (as I thought this was best practice).
I would like take the next step and organise these modules that I can import in fewer lines of code. However, I'm finding that I need to need to use a lot of lines of 'repeated' code to import the functions I want to have access to.
my_functions
-src
--__init__.py
--func1.py
--func2.py
--func3.py
I have followed this talk tutorial to build this collection of modules into a package thinking that I could use something like
import my_fucntions as mf
but I have to import each module
import func1 as fu1
import func2 as fu2
...
a = fu1.func1()
my question is, what do I have to do to be able to import my functions as a package, the same way that I would import something like pandas
import my_functions as mf
a = mf.func1(arg)
I'm finding it difficult to find either a tutorial or a clear simple example of how to structure this so any guidance at all would be useful. If it's just not doable without building something as complex and pandas or numpy thats ok too, I just said I'd try one last shot at it.
Create a __init__.py file in both my_functions directory and src directory.
Now in the my_functions __init__.py file. No need to edit __init__ file inside src.
from src.func1 import func1
from src.func2 import func2
__all__ = [
'func1',
'func2'
]
Now you can use my_functions like below
import my_functions
my_functions.f1()
__all__ gets called when you try to import my_functions and allow you to import things you have mentioned using dot notation.
To call a python script from the root call it using dot notation as below
python3 -m <some_directory>.<file>
I have a plugins package that contains several modules, each defining one class (each class is a plugin).
My package structure looks like this :
plugins
├ __init__.py
├ first_plugin.py
├ second_plugin.py
└ third_plugin.py
And a plugin file typically looks like this, only containing a class definition (and a few imports if necessary) :
# in first_plugin.py
class MyFirstPlugin:
...
I would like the end user to be able to import a plugin like so :
from plugins import FirstPlugin
instead of having to also type the module name (which is what is currently required to do) :
from plugins.first_plugin import FirstPlugin
Is there a way to achieve this by re-exporting the modules' classes directly in the __init__.py file without having to import everything module by module like so (which becomes cumbersome when there are lots of modules) :
# in __init__.py
from .first_plugin import FirstPlugin
from .second_plugin import SecondPlugin
from .third_plugin import ThirdPlugin
I do not think this is possible in Python. However you can import entire modules so you do not have to import each class individually.
For example
from first_plugin import *
Allowing you to do
from plugin import # Anything in first_plugin
Its kinda a pain but writing libraries is not easy (wait till you use CMake with C/C++, you have to specify every single file in your source tree :D)
I think you could elaborate on answers of this post: How to import all submodules?
For example with pkgutil.walk_packages(__path__) you'd have a list of modules. Then you could use dir on the loaded module and import the results (filtering elements starting with __
Okay, so I'm trying to find a way to get a list of all modules in a package, into a variable.
My file structure is as such:
main/
app.py
__init__.py
modules/
__init.py
mod_1.py
mod2.py
All of the code is running from app.py . The modules folder will have many different modules in it, made by several people with a common interface. The goal is to have app.py import a list of all the modules in main.modules, and then iterate through them, doing calls such as:
for mod in module_list:
mod.add(5,2)
I had a working version where I got a list of filenames, and then used module_list = map(__import__, file_list) to make a list of modules that I was able to loop through and call. The problem was, I was using the horrible method of sys.append('path/to/module/folder/') and filtering by .py extension to get my list of files, but the problem is that this wouldn't work well on other computers. Playing with the init.py files, I am able to now correctly import my main/ package from anywhere by simply:
import main
(with a better name obviously). And am able to do stuff such as:
main.modules.mod_1.add(5,2)
from main.modules import *
mod_1.add(5,2)
mod_2.add(6,2)
But I can't find a way to make a list of all of the modules in the main.modules 'package'
mods = main.modules.*
mods = from main.modules import *
import main.modules.* as mods
All don't work, and was the closest I could come. Any help would be great, thanks!
I simply want to take all my .py files from a single folder (I don't care about the sub-folders for now) and put them into a single module.
The use case I'm having here is that I'm writing some pretty standard object-oriented code and I'm using a single file for every class, and I don't want to have to write from myClass import myClass for every class into my __init__.py. I can't use Python3, so I'm still working with impand reloadand such.
At the moment I'm using
# this is __init__.py
import pkgutil
for loader, name, is_pkg in pkgutil.walk_packages(__path__):
if not is_pkg:
__import__(__name__ + "." + name)
and it doesn't seem to work, it includes the packages but it includes them as modules, so that I have to write MyClass.MyClass for a class that is defined in a file with it's own name. That's silly and I don't like it.
I've been searching forever and I'm just getting more confused how complicated this seemingly standard use case seems to be. Do python devs just write everything into a single file? Or do they always have tons of imports?
Is this something that should be approached in an entirely different way?
What you really want to do
To do the job you need to bind your class names to namespace of your __init__.py script.
After this step you will be able to just from YourPackageName import * and just use your classes directly. Like this:
import YourPackageName
c = YourPackageName.MyClass()
or
from YourPackageName import *
c = MyClass()
Ways to achieve this
You have multiple ways to import modules dynamically: __import__(), __all__.
But.
The only way to bind names into namespace of current module is to use from myClass import myClass statement. Static statement.
In other words, content of each of your __init__.py scripts should be looking like that:
#!/usr/bin/env python
# coding=utf-8
from .MySubPackage import *
from .MyAnotherSubPackage import *
from .my_pretty_class import myPrettyClass
from .my_another_class import myAnotherClass
...
And you should know that even for a dynamic __all__:
It is up to the package author to keep this list up-to-date when a new version of the package is released.
(https://docs.python.org/2/tutorial/modules.html#importing-from-a-package)
So a clear answers to your questions:
Do python devs just write everything into a single file?
No, they don't.
Or do they always have tons of imports?
Almost. But definitely not tons. You need to import each of your modules just once (into an appropriate __init__.py scripts). And then just import whole package or sub-package at once.
Example
Let's assume that there is next package structure:
MyPackage
|---MySubPackage
| |---__init__.py
| |---pretty_class_1.py
| |---pretty_class_2.py
|---__init__.py
|---sleepy_class_1.py
|---sleepy_class_2.py
Content of the MyPackage/MySubPackage/__init__.py:
#!/usr/bin/env python
# coding=utf-8
from .pretty_class_1 import PrettyClass1
from .pretty_class_2 import PrettyClass2
Content of the MyPackage/__init__.py:
#!/usr/bin/env python
# coding=utf-8
from .MySubPackage import *
from .sleepy_class_1 import SleepyClass1
from .sleepy_class_2 import SleepyClass2
As result, now we are able to write next code in our application:
import MyPackage
p = MyPackage.PrettyClass1()
s = MyPackage.SleepyClass2()
or
from MyPackage import *
p = PrettyClass1()
s = SleepyClass2()
Currently, I have a package name (let's say DummyPackage). DummyPackage contains three modules with functions, classes, etc. So the directory structure looks like this:
project_dir/
__init__.py
DummyPackage/
__init__.py
Module1/
__init__.py
module_x.py
module_y.py
Module2/
__init__.py
module_z.py
So importing methods from modules looks like this
from DummyPackage.Module1.module_x import method_x
We are adding new stuff to the project and I would like to create a module, with the name DummyProject, which should be importable like this
from DummyProject import new_method
I assumed, only adding file DummyPackage.py would be enough, but apparently, it's not. I tried to add it to the project_dir/ dir and to DummyPackage/ dir, but neither works.
Is it because of name conflict? Is it possible to have a code like this?
import DummyPackage
from DummyPackage.Module1.module_x import method_x
DummyPackage.new_method
method_x
to put my three comments in an answer:
First let me explain relative imports using the modules you already have, if you wanted to import module_x from module_y you can do this:
module_y.py
from .module_x import method_x
or similarly in module_z.py
from ..Module1.module_x import method_x
so depending on the location of your DummyProject in the package Intra-package References may be all you need.
As for the second part yes it is possible to have (runnable) code like this:
import DummyPackage
from DummyPackage.Module1.module_x import method_x
DummyPackage.new_method
method_x
in this case it looks like you want new_method to be a package level variable. To quote this great answer:
In addition to labeling a directory as a Python package and defining __all__, __init__.py allows you to define any variable at the package level.
I highly recommend taking a look at the source code for json/__init__.py in the standard library if you want an real world example.
Or as an example with your setup to be able to import method_x right from the package you would just need to add this to the top level __init__.py:
from .Module1.module_x import method_x
then from any file importing the package you could do this:
import DummyPackage
DummyPackage.method_x
(Although obviously you would do it for new_method according to where you place DummyProject)