This question already has answers here:
Should I hide imports of dependencies in a python module from the __dir__ method?
(1 answer)
How should I perform imports in a python module without polluting its namespace?
(6 answers)
Closed 1 year ago.
I'm making a python package to run analyses with pandas, and I use pandas objects in most files in the package. How do I import those functions so they're usable in the package but don't clutter the namespace for a user? Say I have this directory structure:
MyThing/
MyThing/
__init__.py
apis.py
MyClass.py
where MyClass.py provides a class I will instantiate to process data in memory and apis.py has interfaces to local and remote databases. As a demonstration, say __init__.py contains
from MyThing.MyClass import MyClass
from MyThing.apis import DBInterface
the contents of MyClass.py are
class MyClass:
def __init__():
pass
and apis.py is
import pandas as pd
class DBInterface:
def __init__():
pass
With complete code I expect the use case to look something like this
import MyThing as mt
# get some data
interface = mt.DBInterface()
some_data = interface.query(parameters)
# load it into MyThing
instance = mt.MyThing(some_data)
# add new data from another source
instance.read(filename)
# make some fancy products
instance.magic(parameters)
# update the database
interface.update_db(instance)
The concern I have is that dir(mt.apis) shows everything I've imported, meaning I can do things like make a pandas DataFrame with df = mt.apis.pd.DataFrame(). Is this how it's supposed to work? Should I be using import differently so the namespace isn't cluttered with dependencies? Should I design the package differently so the dependencies aren't available when I import MyThing?
What you are doing is fine and how it's supposed to work and I wouldn't advise trying hard to hide your pandas import.
The solution to this df = mt.apis.pd.DataFrame() is: don't do that.
If there is a function or variable within Mything.apis that you don't want others to use, you can prefix it with a single underscore (eg. _foo). By convention this is understood to be for "internal use" and is not imported when you do from Mything.apis import *. See this section of the PEP-8 style guide for more information about naming conventions of this sort.
If you'd like to be more explicit about what things your module exports you may define them like so __all__ = ['foo', 'bar']. This also makes it so that if you or someone does from Mything.apis import * (which is generally ill-advised anyway) they will only import foo and bar, but you should treat this as a mere suggestion, just like the leading underscore convention.
Related
I want to define a bunch of attributes for use in a module that should also be accessible from other modules, because they're part of my interface contract.
I've put them in a data class in my module like this, but I want to avoid qualifying them every time, similar to how you use import * from a module:
#dataclass
class Schema:
key1='key1'
key2='key2'
and in the same module:
<mymodule.py>
print(my_dict[Schema.key1])
I would prefer to be able to do this:
print(my_dict[key1])
I was hoping for an equivalent syntax to:
from Schema import *
This would allow me to do this from other modules too:
<another_module.py>
from mymodule.Schema import *
but that doesn't work.
Is there a way to do this?
Short glossary
module - a python file that can be imported
package - a collection of modules in a directory that can also be imported, is technically also a module
name - shorthand for a named value (often just "variable" in other languages), they can be imported from modules
Using import statements allows you to import either packages, modules, or names:
import xml # package
from xml import etree # also a package
from xml.etree import ElementTree # module
from xml.etree.ElementTree import TreeBuilder # name
# --- here is where it ends ---
from xml.etree.ElementTree.TreeBuilder import element_factory # does not work
The dots in such an import chain can only be made after module objects, which packages and modules are, and names are not. So, while it looks like we are just accessing attributes of objects, we are actually relying on a mechanism that normal objects just don't support, so we can't import from within them.
In your particular case, a reasonable solution would be to simply turn the object that you wanted to hold the schema into a top-level module in your project:
schema.py
key1 = 'key1'
key2 = 'key2'
...
Which will give you the option to import them in the way that you initially proposed. Doing something like this to make common constants easily accessible in your project is not unusual, and the django framework for example uses a settings.py in the same manner.
One thing you should keep in mind is that names in python modules are effectively singletons, so their values can't be changed at runtime[1].
[1] They can, but it's so hacky that it should pretty much always be treated as not possible.
Suppose I have a code like this in module a.py
import numpy as np
def sqrt(x):
return np.sqrt(x)
And I have a module b.py written like this:
import a
print(a.sqrt(25))
print(a.np.sqrt(25))
I will see that the code runs fine and that when using autocomplete in most IDEs, I found that a.np is accessible. I want to make a.np private so that only a code can see that variable.
I don't want b to be able to access a.np.
What is a good approach to make this possible?
Why do I want a.np to be inaccessible? Because I want it to not show in the autocomplete when I type a. and press Tab in Jupyter Lab. It hides what the modules can do because there are so many imports that I use in my module.
The solution is the same as for "protected" attributes / methods in a class (names defined in a module are actually - at runtime - attributes of the module object): prefix those names with a single leading underscore, ie
import numpy as _np
def sqrt(x):
return _np.sqrt(x)
Note that this will NOT prevent someone to use a._np.sqrt(x), but at least it makes it quite clear that he is using a protected attribute.
I see 2 approaches here:
more user-friendly solution: change alias names to "underscored" ones
import numpy as _np
...
this will not prevent from importing it, but it will say to user that this are implementation details and one should not depend on them.
preferred-by-me solution: do nothing, leave it as it is, use semver and bump versions accordingly.
I have defined several classes in a single python file. My wish is to create a library with these. I would ideally like to import the library in such a way that I can use the classes without a prefix (like mylibrary.myclass() as opposed to just myclass() ), if that's what you can call them, I am not entirely sure as I am a beginner.
What is the proper way to achieve this, or the otherwise best result? Define all classes in __init __? Define them all in a single file as I currently have like AllMyClasses.py? Or should I have a separate file for every class in the library directory like FirstClass.py, SecondClass.py etc.
I realize this is a question that should be easy enough to google, but since I am still quite new to python and programming in general I haven't quite figured out what the correct keywords are for a problem in this context(such as my uncertainty about "prefix")
More information can be found in the tutorial on modules (single files) or packages (when in a directory with an __init__.py file) on the python site.
The suggested way (according to the style guide) is to spell out each class import specifically.
from my_module import MyClass1, MyClass2
object1 = MyClass1()
object2 = MyClass2()
While you can also shorten the module name:
import my_module as mo
object = mo.MyClass1()
Using from my_module import * is recommended to be avoided as it can be confusing (even if it is the recommended way for some things, like tkinter)
If it's for your personal use, you can just put all your classes Class1, Class2, ... in a myFile.py and to use them call import myFile (without the .py extension)
import myFile
myVar1 = myFile.Class1()
myVar2 = myFile.Class2()
from within another script. If you want to be able to use the classes without the file name prefix, import the file like this:
from myFile import *
Note that the file you want to import should be in a directory where Python can find it (the same where the script is running or a directory in PYTHONPATH).
The _init_ is needed if you want to create a Python module for distribution. Here are the instructions: Distributing Python Modules
EDIT after checking the Python's style guide PEP 8 on imports:
Wildcard imports (from import) should be avoided, as they make it unclear which names are present in the namespace, confusing both readers and many automated tools
So in this example you should have used
from myFile import Class1, Class2
I can make this code work, but I am still confused why it won't work the first way I tried.
I am practicing python because my thesis is going to be coded in it (doing some cool things with Arduino and PC interfaces). I'm trying to import a class from another file into my main program so that I can create objects. Both files are in the same directory. It's probably easier if you have a look at the code at this point.
#from ArduinoBot import *
#from ArduinoBot import ArduinoBot
import ArduinoBot
# Create ArduinoBot object
bot1 = ArduinoBot()
# Call toString inside bot1 object
bot1.toString()
input("Press enter to end.")
Here is the very basic ArduinoBot class
class ArduinoBot:
def toString(self):
print ("ArduinoBot toString")
Either of the first two commented out import statements will make this work, but not the last one, which to me seems the most intuitive and general. There's not a lot of code for stuff to go wrong here, it's a bit frustrating to be hitting these kind of finicky language specific quirks when I had heard some many good things about Python. Anyway I must be doing something wrong, but why doesn't the simple 'import ClassName' or 'import FileName' work?
Thank you for your help.
consider a file (example.py):
class foo(object):
pass
class bar(object):
pass
class example(object):
pass
Now in your main program, if you do:
import example
what should be imported from the file example.py? Just the class example? should the class foo come along too? The meaning would be too ambiguous if import module pulled the whole module's namespace directly into your current namespace.
The idea is that namespaces are wonderful. They let you know where the class/function/data came from. They also let you group related things together (or equivalently, they help you keep unrelated things separate!). A module sets up a namespace and you tell python exactly how you want to bring that namespace into the current context (namespace) by the way you use import.
from ... import * says -- bring everything in that module directly into my namespace.
from ... import ... as ... says, bring only the thing that I specify directly into my namespace, but give it a new name here.
Finally, import ... simply says bring that module into the current namespace, but keep it separate. This is the most common form in production code because of (at least) 2 reasons.
It prevents name clashes. You can have a local class named foo which won't conflict with the foo in example.py -- You get access to that via example.foo
It makes it easy to trace down which module a class came from for debugging.
consider:
from foo import *
from bar import *
a = AClass() #did this come from foo? bar? ... Hmmm...
In this case, to get access to the class example from example.py, you could also do:
import example
example_instance = example.example()
but you can also get foo:
foo_instance = example.foo()
The simple answer is that modules are things in Python. A module has its own status as a container for classes, functions, and other objects. When you do import ArduinoBot, you import the module. If you want things in that module -- classes, functions, etc. -- you have to explicitly say that you want them. You can either import them directly with from ArduinoBot import ..., or access them via the module with import ArduinoBot and then ArduinoBot.ArduinoBot.
Instead of working against this, you should leverage the container-ness of modules to allow you to group related stuff into a module. It may seem annoying when you only have one class in a file, but when you start putting multiple classes and functions in one file, you'll see that you don't actually want all that stuff being automatically imported when you do import module, because then everything from all modules would conflict with other things. The modules serve a useful function in separating different functionality.
For your example, the question you should ask yourself is: if the code is so simple and compact, why didn't you put it all in one file?
Import doesn't work quite the you think it does. It does work the way it is documented to work, so there's a very simple remedy for your problem, but nonetheless:
import ArduinoBot
This looks for a module (or package) on the import path, executes the module's code in a new namespace, and then binds the module object itself to the name ArduinoBot in the current namespace. This means a module global variable named ArduinoBot in the ArduinoBot module would now be accessible in the importing namespace as ArduinoBot.ArduinoBot.
from ArduinoBot import ArduinoBot
This loads and executes the module as above, but does not bind the module object to the name ArduinoBot. Instead, it looks for a module global variable ArduinoBot within the module, and binds whatever object that referred to the name ArduinoBot in the current namespace.
from ArduinoBot import *
Similarly to the above, this loads and executes a module without binding the module object to any name in the current namespace. It then looks for all module global variables, and binds them all to the same name in the current namespace.
This last form is very convenient for interactive work in the python shell, but generally considered bad style in actual development, because it's not clear what names it actually binds. Considering it imports everything global in the imported module, including any names that it imported at global scope, it very quickly becomes extremely difficult to know what names are in scope or where they came from if you use this style pervasively.
The module itself is an object. The last approach does in fact work, if you access your class as a member of the module. Either if the following will work, and either may be appropriate, depending on what else you need from the imported items:
from my_module import MyClass
foo = MyClass()
or
import my_module
foo = my_module.MyClass()
As mentioned in the comments, your module and class usually don't have the same name in python. That's more a Java thing, and can sometimes lead to a little confusion here.
I am building an application in Python and I have my whole package. While I really like the fact that you have to explicitly state every import you need, I was wondering if there is a way to add some function or class to the global scope implicitly.
In my example a want a Factory class that should be available in all files. Classes like dict, str and so on are all available and I thought maybe it is possible to add my own class to the global scope in the same way in my __init__.py.
Is this possible?
For interactive mode, add all your import definitions to a file (say all_my_imports.py) like below:
from abc import xyz
from my_stuff import *
And, point the environment variable PYTHONSTARTUP to it.
From a script, simply import the file that contains the above definitions:
from all_my_imports import *
Remember, it is not good to depend on this functionality, it's (almost) always better to explicitly import all your modules.