I'm new at Python and previously I've been using languages like Swift, where import is not a big deal: you're just defining a new class and can access it from another part of your program.
I can't use this way with Python because here import works in the other way: you can't make cyclic imports, where two files import each other. I understand that I'm facing this problem because of using the language in a wrong way but I don't understand how to avoid it.
I mean, in most cases you just can solve this problem by combining two classes into a single file but it doesn't feel right. Also, I've found advice like "move your import statement to the end of file" but it doesn't feel like a good idea too.
I would like to understand the Python's philosophy, if you will. How should I organize my project and what should I be guided by, when deciding on making a class in a separate file?
You certainly can import child from parent and parent from child. The key to making this work is for child not to probe too deeply into parent from its module-level code, because the parent module is only partly loaded at the time that the Python runs the module-level code in child.
Here's what happens when you import child from parent and parent from child (assuming parent is loaded first):
Code at the module level of parent runs until it reaches a statement that loads child (either import child or from child import something). By "module level", I mean the statements that aren't within a class or function definition. The classes and functions defined at the module level will also be created as objects within the module. However, the functions and class methods themselves will not be run yet.
When Python gets to the import child statement (or equivalent) in parent's module-level code, it will stop running the parent code and begin running the module-level code in child. If child imports parent via import parent or from parent import something, it will get the parent module in its current, partially constructed state. So module-level code in child cannot access objects that are defined below import child in parent.py.
Once child's module-level code finishes running, control will return to parent below the import child statement, and Python will finish running all of the module-level code in parent.
This process will give you trouble if child's module-level code tries to access objects that are defined in parent after the import child statement (since parent is only partially finished when child loads). The solution to this is to import parent at the module level of child but defer accessing objects within parent from child until after child and parent have finished loading. In particular, instead of using from parent import something in child's module-level code, you may need to use import parent, then access parent.something from inside function or method code in child. This is safe to do because those functions and methods won't be run until after child and parent finish running, at which point all the elements of the parent module are correctly defined.
Here's an example of what I mean, using the setup you described in a comment. If you give more information on the code that is giving you problems, I could tailor this more closely.
Version 1 (won't work)
__main__.py:
from user import User
u = User()
user.py:
from data_manager import DataManager
...
class User:
def __init__(self, data_manager=None):
if data_manager is None:
data_manager = DataManager(user=self)
self.data_manager = data_manager
data_manager.py:
# next line will fail because user.py has been partly loaded
# and user.User doesn't exist yet
from user import User
...
class DataManager:
def __init__(self, user=None):
...
if user is None:
user = User(data_manager=self)
self.user = user
Version 2 (will work)
__main__.py:
from user import User
u = User()
user.py:
import data_manager as dm
...
class User:
def __init__(self, data_manager=None):
if data_manager is None:
data_manager = dm.DataManager(user=self)
self.data_manager = data_manager
data_manager.py:
import user as user_module
...
# this defines a class which will eventually create an instance
# of user.User, but it won't try to do that until control returns
# to __main__.py, by which point the `user` and `data_manager`
# modules are fully defined
class DataManager:
def __init__(self, user=None):
...
if user is None:
user = user_module.User(data_manager=self)
self.user = user
Note that references in your classes' __init__ methods don't get resolved until the class is actually instantiated. i.e., the user = user_module.User(data_manager=self) line does something like this: "look in the current module for an object called user_module; look in that object for a User attribute; construct an object of that class". The important thing is that data_manager can safely import the user module early on (the module exists already, even though it is only partly constructed), but the code above won't actually look for anything inside the user module until a DataManager object is instantiated, by which time user.User will be properly defined.
I can't use this way with Python because here import works in the other way: you can't make cyclic imports, where two files import each other.
Yes you can!
Say we have two files in the same working directory:
#file1.py
import file2
x = 22
y = 'spam'
print(x)
print(y)
and
#file2.py
import file1
print("file1")
Notice what happens when we import file1.py
>>> import file1
file1
22
spam
file1.py imports file2.py and file2.py imports file1.py.
This problem happens because you're writing Python program like you would write Swift or Java program. Such approach never works well: every language is different and has different best practices. If you write unpythonic code, not only it looks ugly and undecipherable to other Python developers, you're struggling with the language instead of enjoying it.
Just structure your code like python developers structure it. Group related classes in one module. If there's a rare case where there's hard to avoid import cycles (almost never happens), import offending dependency in the function or method instead of global scope.
Related
I've built a couple projects now using the composite pattern where the objects hierarchy is built from a configuration file. My problem is that I's like to save each subclass in a separate file, to allow extensions without changing the base classes' files or having huge files. Each object once instantiated is then going to instantiate a different subclass, based on a type listed in its configuration. To do this I thought of including a factory function which will live in its own file, import all accessible subclasses, and contain a single function that will just return the appropriate subclass based on the name passed to it. The problem with this, is that each of those subclass modules, in order to use this factory, must import it. This creates a circular import situation since the factory module imports all subclasses which all import it back. How can this be avoided or is there a cleaner way to instantiate a subclass dynamically within another ?
As an example - I wrote a "Pipeline" project, useful for automation of different procedures I often need to repeat. The basic parent class is called "Block", it is inherited from to create blocks that comply with a certain interface (i.e. other projects that perform actions) and from those I inherit to blocks that actually execute specific operations. A block only needs to see the its successor in the pipeline, and does not care weather this is a single block or an entire, separate pipeline. To implement this I want each block to instantiate its successor based on the order defined in the config file that is passed along the chain. If I were to write a file that imports all implemented concrete blocks and returns whichever one requested, then I wouldn't be able to import it for use in any of the concrete blocks' modules, since they are imported into the factory one in order to be available for instantiation themselves.
You know that if you write your import statement itself inside a method or function, it will only be executed after all module-level classes and functions have been defined, right? Your circular-dependency can be fixed as simply as writing a "factory" method in the base class that will contain a from factory import factory_function statement and call it.
# basemodel.py Base file
class Base:
def factory(self, *args, **kw):
from factory import factory_function
# baseblock.py Block class hierarchy base file
from basemodel import Base
class Block(Base):
...
# blockXX.py Other block classes files:
from baseblock import Block
class SpecializedBlock31(Block):
...
# factory.py:
from block import Block
...
from block31 import Block31
...
# (or some dynamic importing using __import__ and looking at the filesystem)
def factory_function(*args, **kw):
# logic to decide which class to use
...
instance = decided_class(...)
return instance
I've written a module called Consumer.py, containing a class (Consumer). This class is initialized using a configuration file thay contains different parameters it uses for computation and the name of a loq que used for logging.
I want to write unit tests for this class so i've made a script called test_Consumer.py with a class called TestConsumerMethods(unittest.TestCase).
Now, what i've done is create a new object of the Consumer class called cons, and then i use that to call on the class methods for testing. For example, Consumer has a simple method that checks if a file exists in a given directory. The test i've made looks like this
import Consumer
from Consumer import Consumer
cons = Consumer('mockconfig.config', 'logque1')
class TestConsumerMethods(unittest.TestCase):
def test_fileExists(self):
self.assertEqual(cons.file_exists('./dir/', 'thisDoesntExist.config), False)
self. assertEqual(cons.file_exists('./dir/', thisDoesExist.config), True)
Is this the correct way to test my class? I mean, ideally i'd like to just use the class methods without having to instantiate the class because to "isolate" the code, right?
Don't make a global object to test against, as it opens up the possibility that some state will get set on it by one test, and affect another.
Each test should run in isolation and be completely independent from others.
Instead, either create the object in your test, or have it automatically created for each test by putting it in the setUp method:
import Consumer
from Consumer import Consumer
class TestConsumerMethods(unittest.TestCase):
def setUp(self):
self.cons = Consumer('mockconfig.config', 'logque1')
def test_fileExists(self):
self.assertEqual(self.cons.file_exists('./dir/', 'thisDoesntExist.config), False)
self. assertEqual(self.cons.file_exists('./dir/', thisDoesExist.config), True)
As far as whether you actually have to instantiate your class at all, that depends on the implementation of the class. I think generally you'd expect to instantiate a class to test its methods.
I'm not sure if that's what you're searching for, but you could add your tests at the end of your file like this :
#!/usr/bin/python
...
class TestConsumerMethods(...):
...
if __name__ == "__main__":
# add your tests here.
This way, by executing the file containing the class definition, you execute the tests you put in the if statement.
This way the tests will only be executed if you directly execute the file itself, but not if you import the class from it.
I have a class-factory F that generates classes. It takes no arguments other than a name. I'd like to be able to wrap this method and use it like this:
from myproject.myfactory.virtualmodule import Foo
"myfactory" is a real module in the project, but I want virtualmodule to be something that pretends to be a module.
Whenever I import something from virtualmodule I want it to build a new class using my factory method and make it look as if that was imported.
Can this be done? Is there a pattern that will allow me to wrap a class-factory as a module?
Thanks!
--
UPDATE0: Why do I need this? It's actually to test a process that will be run on a grid which requires that all classes should be importable.
An instance of the auto-generated class will be serialized on my PC and then unserialized on each of the grid nodes. If the class cannot be imported on the grid node the unserialize will fail.
If I hijack the import mechanism as an interface for making my test-classes, then I know for sure that anything I can import on my PC can be re-created exactly the same on the grid. That will satisfy my test's requirements.
You can stuff an arbitrary object into the sys.modules structure:
import sys
class VirtualModule(object):
def __init__(self, name):
self.__name__ = name.rsplit('.', 1)[-1]
self.__package__ = name
self.__loader__ = None
def __getattr__(self, name):
if name is valid:
# Return dynamic classes here
return VirtualClass(name)
raise AttributeError(name)
virtual_module_name = 'myproject.myfactory.virtualmodule'
sys.modules[virtual_module_name] = VirtualModule(virtual_module_name)
The Python import machinery will look up objects using attribute access, triggering the __getattr__ method on your VirtualModule instance.
Do this in the myproject/__init__.py or myproject/myfactory/__init__.py file and you are all set to go. The myproject.myfactory package does need to exist for the import machinery to find the myproject.myfactory.virtualmodule object.
I am making an IRC bot with Twisted and I have run into a problem. I want to import functions from a seperate file into my main class (the one that inherits irc.IRCClient) so they can be used as methods. I have thought of two possible solutions, but they both seem a bit newbish and they both have problems.
Solution 1: Put the functions in a class in a separate file, import it into my main file and make my main class inherit the class. The problem with this solution is that I will end up inheriting quite a few classes and I have to modify my main class each time I make a new module for the bot.
Solution 2: Put the functions in a separate file, import it into my main file and assign each of the imported functions to a variable. This is annoying because I would have to set a variable in my main class for each of the methods I want the class to import from somewhere else.
Example:
importthis.py
class a():
def somemethod(self):
print "blah"
main.py
import importthis
class mainClass(irc.IRCClient):
thisisnowamethod = importthis.a()
As you can see, both methods (no pun intended) require a lot of stupid work to maintain. Is there a smarter way to do this?
class mainClass(irc.IRCClient):
from othermodule import a, b, c
# othermodule.a, b, and c are now methods of mainClass
from anothermodule import *
# everything in anothermodule is now in mainClass
# if there are name conflicts, last import wins!
This works because import simply imports symbols to a namespace and doesn't much care what kind of namespace it is. Module, class, function -- it's all copacetic to import.
Of course, the functions in othermodule must be written to accept self as their first argument since they will become instance methods of mainClass instances. (You could maybe decorate them using #classmethod or #staticmethod, but I haven't tried that.)
This is a nice pattern for "mix-ins" where you don't necessarily want to use multiple inheritance and the headaches that can cause.
I think the question that needs to be answered for each of these imported methods is, does it act on the state of the object? The answer seems to be no -- they don't know about your main class, and therefore should not be methods on the main class. And I don't think there's an IS-A relationship between your main class and those that it would inherit from.
But is your problem that you want callers to have access to these methods? A good solution is to just make these methods available as a module that can be imported.
I'm having a problem with circular imports. I have three Python test modules: robot_test.py which is my main script, then two auxiliary modules, controller_test.py and servo_test.py. The idea is that I want controller_test.py to define a class for my microcontroller and servo_test.py to define a class for my servos. I then want to instantiate these classes in robot_test.py. Here are my three test modules:
""" robot_test.py """
from pi.nodes.actuators.servo_test import Servo
from pi.nodes.controllers.controller_test import Controller
myController = Controller()
myServo = Servo()
print myController.ID, myServo.ID
""" controller_test.py """
class Controller():
def __init__(self, id="controller"):
self.ID = id
""" servo_test.py """
class Servo():
def __init__(self, id="servo"):
self.ID = id
If I run robot_test.py, I get the expected printout:
controller servo
However, now comes the twist. In reality, the servo_test.py module depends on controller_test.py by way of robot_test.py. This is because my servo definitions require an already-instantiated controller object before they themselves can be instantiated. But I'd like to keep all the initial instantiations in robot_test.py. So I tried modifying my servo_test.py script as follows:
""" servo_test.py """
from pi.nodes.robots.robot_test import myController
class Servo():
def __init__(self, id="servo"):
self.ID = id
print myController.ID
Of course, I could sense that the circularity was going to cause problems and sure enough, when I now try to run robot_test.py I get the error:
ImportError: Cannot import name Servo
which in turn is caused by servo_test.py returning the error:
ImportError: Cannot import name myController
In C# I would define myController and myServo as static objects in robot_test.py and then I could use them in other classes. Is there anyway to do the same in Python? One workaround I have found is to pass the myController object to the Servo class as an argument, but I was hoping to avoid having to do this.
Thanks!
One workaround I have found is to pass
the myController object to the Servo
class as an argument, but I was hoping
to avoid having to do this.
Why ever would you want to avoid it? It's a classic case of a crucial Design Pattern (maybe the most important one that wasn't in the original Gang of Four masterpiece), Dependency Injection.
DI implementation alternatives to having the dependency in the initializer include using a setter method (which completes another crucial non-Gof4 DP, two-phase construction, started in the __init__) -- that avoids another circularity problem not related to imports, when A's instances need a B and B's instances need an A, in each case to complete what's logically the instances "initialization". But when you don't need two-phase construction, initializers are the natural place to inject dependencies.
Beyond the advantages related to breaking circularity, DI facilitates reuse (by generalization: e.g, if and when you need to have multiple controllers and servos rather than just one of each, it allows you to easily control the "pairing up" among them) and testing (it becomes easy to isolate each class for testing purposes by injecting in it a mock of the other, for example).
What's not to like?
servo_test.py doesn't actually need myController, since the access is within a function. Import the module instead, and access myController via that module.
import pi.nodes.robots.robot_test as robot_test
class Servo():
def __init__(self, id="servo"):
self.ID = id
print robot_test.myController.ID
This will work as long as myController exists before Servo is instantiated.
Pass the instantiated controller object as an init argument to the instantiation of the Servo.
""" servo_test.py """
class Servo():
def __init__(self,controller,id="servo"):
self.ID = id
self.ctrl = controller
print self.ctrl.ID
""" robot_test.py """
from pi.nodes.actuators.servo_test import Servo
from pi.nodes.controllers.controller_test import Controller
myController = Controller()
myServo = Servo(myController)