I have a model defined with a field using django-picklefield. Everything was working fine till I refactored our codebase for organizational reasons, but now when I load that model from the db I get back a unicode object instead of the unpickled object. Because the class definition moved pickle cannot find the module it needs to import, I traced the error back to here where pickle tries to import a module that no longer exists.
Is there a way I can resolve this import error by pointing pickle to the correct module or somehow fake the existence of that class definition in the module pickle is looking in with other imports or by creating a class of the same name that inherits from the original class that lives elsewhere?
I was able to resolve this by recreating the old file I had deleted and had it contain only a single line that imports the class that pickle was looking for.
Related
I have the following file structure
src
|-main_file.py
|-utils
| |-util_debug.py
|-classes
| |-class_coupon_generic.py
| |-class_coupon_fatigue_type_1.py
| |-class_coupon_fatigue_type_2.py
The file class_coupon_generic.py contains a class named coupon_generic which will act as a parent class for different types of coupons in future development. The file class_coupon_fatigue_type_1.py contains a class named coupon_fatigue_type_1 which is inherited from the parent class coupon_generic. I am keeping all the class files in the folder classes. Inside the utils folder, I have a driver script for debugging util_debug.py. I want to import the classes coupon_generic and coupon_fatigue_type_1 (depending on which coupon I am working on) in this file to build the final coupon model.
How do I achieve that?
I tried to import the modules using imp.load_source and importlib.import_module; but I couldn't figure out a way to do that. I am getting some error like name 'coupon_generic' is not defined. If the class coupon_generic is defined inside the file class_coupon_fatigue_type_1.py, then the I can make the import work using imp.load_source, but that kind of defeats the whole purpose of having a parent class. In order to add any new method to the parent class later on, I have to go to separate scripts and modify them individually. What is the correct way of making such import work with classes having inheritance?
Also, I prefer a solution where I can make the import of class_coupon_fatigue_type_1.py from a string name because it will be varying for different coupon types and I can control that by just passing a string name of the file during debugging.
I'm using cPickle to serialize and deserialize instances of a class. I've recently moved a few classes into different packages and now I'm noticing that cPickle/pickle stores the class name and the package it comes from.
>>> class A(object):
pass
>>> dumps(A())
'ccopy_reg\n_reconstructor\np1\n(c__main__\nA\np2\nc__builtin__\nobject\np3\nNtRp4\n.'
Notice how __main__ is stored because I ran that code in the main python interpreter.
If I try to unpickle the many objects I have stored this way, I'll get an ImportError complaining that since I moved the classes around, the old class doesn't exist in the location where it is expected.
I haven't changed the format of __getstate__ or __setstate__. I've only changed the location of the class that needs to be deserialized. Is there a way to migrate these objects so that I don't run into problems?
If you want to migrate your data, you must provide a reference to the new object at the old location, and dump the data again:
old_location.A = new_location.A
data = loads(pickle_data)
pickle_data = dumps(data)
and now, pickle_data contains a reference to new_location.A.
I can't seem to figure out how to import two classes to each other. When running the application it simply says
from room import Room
ImportError: cannot import name Room
It might be a design problem but I don't think there's another way to reference the two classes, so this needs to be as it is. The only reason the imports are needed is because they are required by the redisco module in the objects (they need to know the types)
#Room class.
class Room(models.Model):
from player import Player
players = models.ListField(Player, required = True)
#Player class
class Player(models.Model):
from room import Room
room = models.ReferenceField(Room, required = True)
How do I get this to work?
E:
The framework is Redisco (Redis)
Most ORM models either support back references (where the target model of a reference field is given an extra attribute that points back to the referencing object), and / or lets you specify relationships through other means.
Redisco doesn't have back-references that I could discover, but it does support string references. If you pass in a string it'll be interpreted as a model name, matched against the __name__ attribute:
class Room(models.Model):
players = models.ListField('Player', required = True)
This neatly bypasses the import problem altogether.
From the ListField docstring:
target_type -- can be a Python object or a redisco model class.
If target_type is not a redisco model class, the target_type should
also a callable that casts the (string) value of a list element into
target_type. E.g. str, unicode, int, float.
ListField also accepts a string that refers to a redisco model.
The code to resolve the name uses the function get_model_from_key() to resolve the string, which simply searches through all subclasses of models.Model, matching on __name__.
It'll resolve the name when validating new values or when retrieving existing values for the first time, by which time the Player subclass has already been imported.
If they are in the same .py file then you don't need imports. If they are in different files in the same directory just use 'import room' (assuming its in a file called 'room.py'). If they are in separate files in separate directories you will need to use the imp module.
import imp
roomClass = imp.load_source('roomClass', [Path to room.py])
Then it can be called with something like:
aRoom = roomClass.Room(Model)
First of all if you want to import a module to your program, they must be in the same directory.
Follow this for importing modules ;
from filename import classname
Which is the filename= your .py file that you want to import class
class is the specific class that you want to use.
Or;
from filename import *
Which is going to add all functions and classes.
Also check this topic. program is running fine but cannot being import with IndexError
You can reference to this problem.
Pythonic way to resolve circular import statements?
Using import rather than from [...] import [...]
# models/__init__.py
from shared.cache import Cache
class modelA():
pass
class modelB():
pass
class modelC():
pass
# shared/cache.py
class Cache:
def methodA():
modelA.SomeStaticMethod()
Basically what I need is to access modelA from inside the Cache class.
If I try to import the models from cache.py, I get an error due to a circular reference error.
I know it seems a little bit weird but it's a very specific issue.
Is there anyway to do that?
You would usually restructure your files so that there is no circular reference error.
Simply answering your question, and usually seen as a workaround, you can import Cache on demand, only within the functions of models/__init__.py that make use of it. This may not be possible in this case, especially if Cache is used as a decorator at the module level.
See also this question.
I struggled with the title for this question so let me just lay out the code:
File A:
class SomeClass(Base):
__tablename__ = 'some_classes'
id = Column(Integer, primary_key=True)
my_awesome_property = Column(Unicode(255))
other_class = relationship('OtherClass', backref='some_class', uselist=False)
File B:
class OtherClass(Base):
__tablename__ = 'other_classes'
id = Column(Integer, primary_key=True)
my_sweet_property = Column(Unicode(255))
some_class_id = Column(ForeignKey('some_classes.id'))
Now, in many cases I would refer to both of these files from a "higher-order" file containing some functions like so:
Higher Order File:
from model.alpha import SomeClass
from model.bravo import OtherClass
from sqlalchemy.orm import sessionmaker
session = sessionmaker(bind=some_engine)()
def some_random_query():
return session.query(SomeClass).join(OtherClass).filter(OtherClass.my_sweet_property=='Mike Bayer\'s cat speaks SQL.').first()
So that's pretty normal, nothing wrong with that... until... I decide I want to a put a function into one of the lower-level files like File A (and avoid circular imports)
Back to File A:
# pretend I imported a session here
def frustrating_situation():
session.query(SomeClass).join(SomeClass.other_class).filter(SomeClass.other_class.my_sweet_property=='Get ready for an exception!').first()
This will throw this bad boy right here:
AttributeError: Neither 'InstrumentedAttribute' object nor 'Comparator' object associated with SomeClass.other_class has an attribute 'my_sweet_property'
Now I suppose that makes sense given what I know of the internals of SQLAlchemy, but I also think from an API standpoint that statement should really work.
Here is how I worked around it:
session.query(SomeClass).join(SomeClass.other_class).filter(SomeClass.other_class.property.mapper.c.my_sweet_property == 'verbose, yet it works as desired').first()
So after all that, my question is really quite simple: Does anybody know a better / more idiomatic / proper / less dirty feeling way of doing this?
Suggestions welcome.
Side Note
For anyone who is wondering:
"Why not just import the class you want the reference to for the join/filter operation?"
There are a couple reasons why you might not want to / be able to import the class into the module where you are writing the query.
You have split up your class definitions across many files and decided to avoid circular imports by strictly not importing across same-level modules
You have decided to place functions that operate on 1 or more classes not currently defined or imported in the current module and do not wish to import them because they are not used for any other reason in the module (and see reason 1 again).
SomeClass.other_class.my_sweet_property
doesn't work in sqlalchemy. sorry.
You are referring to OtherClass, in this .filter() clause. how you arrive at that name is your business, but the clearest way, from the point of view of what each statement means and where the arguments come from is still just to import things.
edit: A common cause of problems with circular imports occurs when you try to import the names out of modules directly instead of just importing the modules. If you turn code that looks like:
from foo import Bar
def baz():
Bar.quux()
you'll have an import problem if foo is also trying to import this module (say, because it wants to use baz).
Fix it by importing only the module:
import foo
def baz()
foo.Bar.quux()
since foo.Bar is resolved later, only when baz() is called, you don't have any trouble when this module gets imported, since it doesn't actually try to use the contents of any of the modules it imports.