We Keep Coding

Recombining object after pickling-unpickling

I have a situation where one party (Alice) has a complex custom object, whose attributes are complicated and can involve circular references. Alice then sends this object to two separate parties, Bob and Claire, by pickling and sending through an (encrypted) socket. They each then modify one attribute of the object each, but what they change includes complex references to the object they received from Alice. Both Bob and Claire then pickle their own modified object themselves, and send it back to Alice.
The question is, how can Alice combine the changes made by both Bob and Claire? Because object persistence is lost on pickling/unpickling, the naive idea of copying the attribute that Bob or Claire created onto the original object doesn't work. I am aware of how persistent_id() and persistent_load() works in pickling, but I would very much like to avoid having to manually write rules for every single attribute in the object that Alice creates. Partly because its a big pile of nested and circularly referenced objects (some 10,000+ lines of them), and partly because I want flexibility to modify the rest of the code without have to change how I pickle/unpickle every time (and the difficulty of properly testing that).
Can this be done? Or do I have to swallow the bitter pill and deal with the pickling "manually"?
Here's a minimal concrete example. Obviously, the circular references could be easily removed here, or Bob and Claire could just send their value over to Alice, but not so in my real case.
import pickle
class Shared:
pass
class Bob:
pass
class Claire:
pass
class Alice:
def __init__(self):
self.shared = Shared()
self.bob = Bob()
self.claire = Claire()
def add_some_data(self, x, y):
self.shared.bob = self.bob
self.shared.claire = self.claire
self.shared.x = x
self.shared.y = y
def bob_adds_data(self, extra):
self.bob.value = self.shared.x + self.shared.y + extra
def claire_adds_data(self, extra):
self.claire.value = self.shared.x * self.shared.y * extra
# Done on Alice's side
original = Alice()
original.add_some_data(2, 3)
outgoing = pickle.dumps(original)
# Done on Bob's side
bobs_copy = pickle.loads(outgoing)
bobs_copy.bob_adds_data(4)
bobs_reply = pickle.dumps(bobs_copy)
# Done on Claires's side
claires_copy = pickle.loads(outgoing)
claires_copy.claire_adds_data(5)
claires_reply = pickle.dumps(claires_copy)
# Done on Alice's side
from_bob = pickle.loads(bobs_reply)
from_claire = pickle.loads(claires_reply)
original.bob = from_bob.bob
original.claire = from_claire.claire
# If the circularly references were maintained, these two lines would be equal
# instead, the attributes on the bottom line do not exist because the reference is broken
print(original.bob.value, original.claire.value)
print(original.shared.bob.value, original.shared.claire.value)

Partial Solution
I have a partial solution that works with some restrictions on the problem case.
Restrictions
The restriction is that Alice's object has only a single reference to Bob's and Claire's in a known place. Those latter two, however, can have arbitrarily complex references to themselves and to Alice, including circular, nested and recursive structures. The other requirement is that Bob doesn't have any reference to Claire and vice-versa: this is very natural if we require both of those objects to be updated independently and in any order.
In other words, Alice receives something from Bob which gets placed in a single tidy place. The difficulty is in making the references contained within Bob match up to the correct objects that Alice contains, but nothing else in Alice itself needs to be changed. This is the use case I need, and it isn't clear to me that the more general case is possible if Bob and Claire can make arbitrary changes to Alice.
Idea
This works by having a base class that creates a persistent id which does not change over the object's life time, is maintained by pickling/unpickling, and is unique. Any object whose references are to be maintained in this scenario must inherit from this class. When Bob sends his changes to Alice, he uses a dictionary of all the objects he received from Alice and their persistent id to pickle, such that those that all the references to pre-existing objects are encoded by the persistent id. On the other side, Alice does the same. She unpickles what Bob sent her using a dictionary of persistent id to object of everything she sent to Bob earlier. Thus, while Alice and Bob have different instances of everything, the persistent id of some objects are the same so they can be "swapped out" when pickling between different parties.
This can be made to work with existing code pretty easily. It just consists of adding a base class to all custom classes that we want to make persistent, and a small addition every time we pickle/unpickle.
Module
import io
import time
import pickle
class Persistent:
def __init__(self):
"""Both unique and unchanging, even after modifying or pickling/unpickling object
Potential problem if clocks are not in sync"""
self.persistent_id = str(id(self)) + str(time.time())
def make_persistent_memo(obj):
"""Makes two dictionaries (one reverse of other) linking every instance of Persistent found
in the attributes and collections of obj recursively, with the persistent id of that instant.
Can cope with circular references and recursively nested objects"""
def add_to_memo(item, id_to_obj, obj_to_id, checked):
# Prevents checking the same object multiple times
if id(item) in checked:
return id_to_obj, obj_to_id, checked
else:
checked.add(id(item))
if isinstance(item, Persistent):
id_to_obj[item.persistent_id] = item
obj_to_id[item] = item.persistent_id
try: # Try to add attributes of item to memo, recursively
for sub_item in vars(item).values():
add_to_memo(sub_item, id_to_obj, obj_to_id, checked)
except TypeError:
pass
try: # Try to add iterable elements of item to memo, recursively
for sub_item in item:
add_to_memo(sub_item, id_to_obj, obj_to_id, checked)
except TypeError:
pass
return id_to_obj, obj_to_id, checked
return add_to_memo(obj, {}, {}, set())[:2]
class PersistentPickler(pickle.Pickler):
""" Normal pickler, but it takes a memo of the form {obj: persistent id}
any object in that memo is pickled as its persistent id instead"""
#staticmethod # Because dumps is not defined for custom Picklers
def dumps(obj_to_id_memo, obj):
with io.BytesIO() as file:
PersistentPickler(file, obj_to_id_memo).dump(obj)
file.seek(0)
return file.read()
def __init__(self, file, obj_to_id_memo):
super().__init__(file)
self.obj_to_id_memo = obj_to_id_memo
def persistent_id(self, obj):
try:
if obj in self.obj_to_id_memo and obj:
return self.obj_to_id_memo[obj]
except TypeError: # If obj is unhashable
pass
return None
class PersistentUnPickler(pickle.Unpickler):
""" Normal pickler, but it takes a memo of the form {persistent id: obj}
used to undo the effects of PersistentPickler"""
#staticmethod # Because loads is not defined for custom Unpicklers
def loads(id_to_obj_memo, pickled_data):
with io.BytesIO(pickled_data) as file:
obj = PersistentUnPickler(file, id_to_obj_memo).load()
return obj
def __init__(self, file, id_to_obj_memo):
super().__init__(file)
self.id_to_obj_memo = id_to_obj_memo
def persistent_load(self, pid):
if pid in self.id_to_obj_memo:
return self.id_to_obj_memo[pid]
else:
super().persistent_load(pid)
Use Example
class Alice(Persistent):
""" Must have a single attribute saved as bob or claire """
def __init__(self):
super().__init__()
self.shared = Shared()
self.bob = Bob()
self.claire = Claire()
def add_some_data(self, x, y):
self.nested = [self]
self.nested.append(self.nested)
self.shared.x = x
self.shared.y = y
class Bob(Persistent):
""" Can have arbitrary reference to itself and to Alice but must not touch Claire """
def make_changes(self, alice, extra):
self.value = alice.shared.x + alice.shared.y + extra
self.attribute = alice.shared
self.collection = [alice.bob, alice.shared]
self.collection.append(self.collection)
self.new = Shared()
class Claire(Persistent):
""" Can have arbitrary reference to itself and to Alice but must not touch Bob """
def make_changes(self, alice, extra):
self.value = alice.shared.x * alice.shared.y * extra
self.attribute = alice
self.collection = {"claire": alice.claire, "shared": alice.shared}
self.collection["collection"] = self.collection
class Shared(Persistent):
pass
# Done on Alice's side
alice = Alice()
alice.add_some_data(2, 3)
outgoing = pickle.dumps(alice)
# Done on Bob's side
bobs_copy = pickle.loads(outgoing)
# Create a memo of the persistent_id of the received objects that are *not* being modified
_, bob_pickling_memo = make_persistent_memo(bobs_copy)
bob_pickling_memo.pop(bobs_copy.bob)
# Make changes and send everything back to Alice
bobs_copy.bob.make_changes(bobs_copy, 4)
bobs_reply = PersistentPickler.dumps(bob_pickling_memo, bobs_copy.bob)
# Same on Claires's side
claires_copy = pickle.loads(outgoing)
_, claire_pickling_memo = make_persistent_memo(claires_copy)
claire_pickling_memo.pop(claires_copy.claire)
claires_copy.claire.make_changes(claires_copy, 5)
claires_reply = PersistentPickler.dumps(claire_pickling_memo, claires_copy.claire)
# Done on Alice's side
alice_unpickling_memo, _ = make_persistent_memo(alice)
alice.bob = PersistentUnPickler.loads(alice_unpickling_memo, bobs_reply)
alice.claire = PersistentUnPickler.loads(alice_unpickling_memo, claires_reply)
# Check that Alice has received changes from Bob and Claire
print(alice.bob.value == bobs_copy.bob.value == 9,
alice.claire.value == claires_copy.claire.value == 30)
# Check that all references match up as expected
print("Alice:", alice is alice.nested[0] is alice.nested[1][0] is alice.claire.attribute)
print("Bob:", (alice.bob is alice.nested[0].bob is alice.bob.collection[0] is
alice.bob.collection[2][0]))
print("Claire:", (alice.claire is alice.nested[0].claire is alice.claire.collection["claire"] is
alice.claire.collection["collection"]["claire"]))
print("Shared:", (alice.shared is alice.bob.attribute is alice.bob.collection[1] is
alice.bob.collection[2][1] is alice.claire.collection["shared"] is
alice.claire.collection["collection"]["shared"] is not alice.bob.new))
Output
C>python test.py
True True
Alice: True
Bob: True
Claire: True
Shared: True
All exactly as required
Follow up
It feels like I'm reinventing the wheel here by doing my own nested introspection, can this be done better with existing tools?
My code feels rather inefficient, with a lot of introspection, can this be improved?
Can I be sure that add_to_memo() is not missing some references out?
The use of time.time() to create the persistent id feels rather clunky, is there a better alternative?

Pass request to different class based on condition Python

I am designing an API which deals with different type of vehicles for example cars, buses and vans. There is an endpoint POST /vehicle which will take a defined body.
{
'registration': 'str',
'date_of_purchase': 'datetime str',
'vehicle_type': 'car'
...
}
The first thing that we do is load an object based on the vehicle_type.
if body[vehicle_type] == 'car':
passed_in_vehicle = Car(body)
elif body['vehicle_type'] == 'bus':
passed_in_vehicle = Bus(body)
...
Within the python program there are multiple classes including:
vehicles.py
cars.py
vans.py
buses.py
The API entry point goes to vehicles.py which does some logic and then depending on the input will route it to one of the other classes to do more specific logic.
Each class has the same set of base methods.
update_vehicle_specs
register_vehicle
At the moment in the bottom of vehicles.py I have
if is_instance(passed_in_vehicle, Car):
carService = Cars()
carService.register_vehicle(passed_in_vehicle)
elif is_instance(passed_in_vehicle, Van):
vanService = Vans()
vanService.register_vehicle(passed_in_vehicle)
...
However, this cannot be scalable. What is the correct solution to route to specific classes based on a condition?

The way it looks to me, you can improve design a bit. Since each vehicle 'knows' where it is serviced, you should probably do in each something like
def __init__(self, ..., service):
super().__init__(self, ...)
...
...
service.register_vehicle(self)
this is coming from the perspective that "the car drives to the garage to register" rather than the "garage looks for new cars".
When you initialize a vehicle, pass it its servicing class, as in
new_car = Car(..., my_car_service)
where
my_car_service = Cars()
somewhere before. This is since we assume each new car needs to register, and we do not make a new service for each new car. Another approach, is for the class to contain (composition) its service object from the get go:
class Car:
service = Cars()
def __init__(...)
...
Car.service.register_vehicle(self)
so the class contains the servicing object always, and all initialized objects share the same one through a class variable. I actually prefer this.
First initialization:
With regard to your initial initialization of the vehicle, while using locals might be a nifty solution for that, it might be a bit more type safe to have something like what you have. In Python I like to use the "Python switch dictionary" (TM) for stuff like that:
{'car': Car,
'bus': Bus,
...
'jet': Jet
}[body[vehicle_type]](body)

I don't know if using locals would be ideal, but at least I believe it would scale better than your current solution:
use string name to instantiate classes from built-in locals() which will store imports as a dictionary
class Car(object):
def print(self):
print("I'm a car")
class Bus(object):
def print(self):
print("I'm a bus")
vehicle_type = "Car"
currV_class = locals()[vehicle_type]
currV_instance = currV_class()
#currV_instance.register_vehicle
currV_instance.print()
vehicle_type = "Bus"
currV_class = locals()[vehicle_type]
currV_instance = currV_class()
#currV_instance.register_vehicle
currV_instance.print()
OUTPUT:
I'm a car
I'm a bus

How to print actual name of variable class type in function?

I'm trying to return variable name, but i keep getting this:
<classes.man.man object at (some numbers (as example:0x03BDCA50))>
Below is my code:
from classes.man import man
def competition(guy1, guy2, counter1=0, counter2=0):
.......................
some *ok* manipulations
.......................
if counter1>counter2:
return guy1
bob = man(172, 'green')
bib = man(190, 'brown')
print(competition(bob , bib ))
Epilogue
If anyone want to, explain please what I can write instead of __class__ in example below to get variable name.
def __repr__(self):
return self.__class__.__name__
Anyway, thank you for all of your support

There are different ways to approach your problem.
The simplest I can fathom is if you can change the class man, make it accept an optional name in its __init__ and store it in the instance. This should look like this:
class man:
def __init__(number, color, name="John Doe"):
self.name = name
# rest of your code here
That way in your function you could just do with:
return guy1.name
Additionnally, if you want to go an extra step, you could define a __str__ method in your class man so that when you pass it to str() or print(), it shows the name instead:
# Inside class man
def __str__(self):
return self.name
That way your function could just do:
return guy1
And when you print the return value of your function it actually prints the name.
If you cannot alter class man, here is an extremely convoluted and costly suggestion, that could probably break depending on context:
import inspect
def competition(guy1, guy2, counter1=0, counter2=0):
guy1_name = ""
guy2_name = ""
for name, value in inspect.stack()[-1].frame.f_locals.items():
if value is guy1:
guy1_name = name
elif value is guy2:
guy2_name = name
if counter1 > counter2:
return guy1_name
elif counter2 > counter2:
return guy1_name
else:
return "Noone"

Valentin's answer - the first part of it at least (adding a name attribute to man) - is of course the proper, obvious solution.
Now wrt/ the second part (the inspect.stack hack), it's brittle at best - the "variables names" we're interested in might not necessarily be defined in the first parent frame, and FWIW they could as well just come from a dict etc...
Also, it's definitly not the competition() function's responsability to care about this (don't mix domain layer with presentation layer, thanks), and it's totally useless since the caller code can easily solve this part by itself:
def competition(guy1, guy2, counter1=0, counter2=0):
.......................
some *ok* manipulations
.......................
if counter1>counter2:
return guy1
def main():
bob = man(172, 'green')
bib = man(190, 'brown')
winner = competition(bob, bib)
if winner is bob:
print("bob wins")
elif winner is bib:
print("bib wins")
else:
print("tie!")

Python prints the location of class objects in memory if they are passed to the print() function as default. If you want a prettier output for a class you need to define the __repr__(self) function for that class which should return a string that is printed if an object is passed to print(). Then you can just return guy1

__repr__ is the method that defines the name in your case.
By default it gives you the object type information. If you want to print more apt name then you should override the __repr__ method
Check below code for instance
class class_with_overrided_repr:
def __repr__(self):
return "class_with_overrided_repr"
class class_without_overrided_repr:
pass
x = class_with_overrided_repr()
print x # class_with_overrided_repr
x = class_without_overrided_repr()
print x # <__main__.class_without_overrided_repr instance at 0x7f06002aa368>
Let me know if this what you want?

Looking for replacement of these Magic Variable in Python

I designed this OOPs code with magic variable actually it feel little complicated so I want to rectify this code to simplest one without using Magic variable.Below code is Inheritance in Python.
And I'm in learning phase of OOPs concepts please suggest me best OOPs practice and which concepts are important in OOPs on perspective of programmer job.
class Bike():
bmodel = ''
def __init__(self,**model):
self.bmodel = model.get('bmodel')
super(Bike,self).__init__(**model)
def setmodelb(self,bmodel):
self.bmodel = bmodel
def getmodel(self):
return self.bmodel
def tostringb(self):
print("Licence",self.lno,"is Bike and Model is",self.bmodel)
class Car():
cmodel = ''
def __init__(self,**model):
self.cmodel = model.get('cmodel')
super(Car,self).__init__()
def setmodelc(self,cmodel):
self.cmodel = cmodel
def getmodel(self):
return self.cmodel
def tostringc(self):
print("Licence",self.lno,"is Car and Model is",self.cmodel)
class Vehicle(Bike,Car):
lno = ''
def __init__(self,**model):
self.lno = model.get('lno')
super(Vehicle,self).__init__(**model)
def setlno(self,lno):
self.lno = lno
def getlno(self):
return self.lno
def tostringv(self):
print("Vehicle Licence is",self.lno)
v = Vehicle()
v.setlno("CALIFORNIA99")
v.setmodelc("HONDA CITY")
v.tostringc()
v.tostringv()
OUTPUT
Licence CALIFORNIA99 is Car and Model is HONDA CITY
Vehicle Licence is CALIFORNIA99
[Finished in 0.1s]

Welcome to OOP. Your code seems complicated also because you are not following python conventions. Here are some essential reading:
For text: PEP 8
For documentation: PEP 257
Use reStructuredText formatting: PEP 287
And for a more Pythonic code: PEP 20.
Last but not least, avoid common pitfalls: Anti-patterns
In your code, you can replace tostring by __repr__ method. This allows to do print(Car()). Also in python you do not need getter and setter as there are no public or private variables. So only define inside your __init__: self.model. You will be able to do:
car = Car()
car.model = ...

How to access an object in a list by property?

Consider a list of objects and a method that works on one object:
beers = []
beers.append(Beer('Carlsberg', 'tasty'))
beers.append(Beer('Guinness', 'bitter'))
beers.append(Beer('Leffe', 'perfect'))
def howIsBeer (name):
taste = ???
print taste
Class Beer:
def __init__ (self, name, taste):
self.name = name
self.taste = taste
How would the howIsBeer() method go about getting the taste of a beer if it is provided only with the beer name? My first inclination is to iterate the beers list, but being Python I suspect that there is a more direct method. Is there?

Can't really think of any better way, simply use a for-loop.
def howIsBeer(name):
for beer in beers:
if beer.name == name:
return beer.taste
In [1]: howIsBeer("Carlsburg")
Out[1]: tasty
Also notice, that keyword class is written with small c. You will also have to define your class before you use it for creating instances.
Edit:
One way, as suggested in the commments, would be to define dictionaries. If you find it useful, you can use the code below. Notice, however, this is only recommended if you have HUGE amount of Beer objects and performance speed is really important for you. Else use the first code provided
class Beer:
names = {}
def __init__(self, name, taste):
Beer.names[name] = self
self.name = name
self.taste = taste
In [3]: Beer.names["Carlsburg"].taste
Out[3]: tasty

Just loop through the list and check for each one.
You need to move the code around a little though, as the class def needs to be above the use of the class, and also it uses a small c. You should also watch out for the case where it's not recognised.
You also spelt Carlsberg wrong :/
class Beer:
def __init__ (self, name, taste):
self.name = name
self.taste = taste
beers = []
beers.append(Beer('Carlsberg', 'tasty'))
beers.append(Beer('Guinness', 'bitter'))
beers.append(Beer('Lef', 'perfect'))
def howIsBeer (name):
taste = "I have no idea"
for beer in beers:
if beer.name == name:
taste = beer.taste
print taste
howIsBeer("Carlsberg") # tasty
I'd do it like this though (using the dictionaries here allows for the flexibility of having more than one property):
beers = {}
beers["Lef"] = {"taste": "tasty"}
beers["Staropramen"] = {"taste": "tasty"}
beers["Peroni"] = {"taste": "tasty"}
beers["Coors Light"] = {"taste": "What is this?!"}
def howIsBeer (name):
taste = "I have no idea"
if name in beers:
taste = beers[name]["taste"]
print taste
howIsBeer("Lef")
If you just want to store the tastes, then you could do this:
beers = {}
beers["Lef"] = "tasty"
beers["Staropramen"] = "tasty"
beers["Peroni"] = "tasty"
beers["Coors Light"] = "What is this?!"
def howIsBeer (name):
taste = "I have no idea"
if name in beers:
taste = beers[name]
print taste
howIsBeer("Lef")
If you are looking to store a series of objects - as you mention in the question - then you want a dictionary of objects - not a dictionary that is a variable of the class.
i.e.
beers = {}
def add_beer(beer):
beers[beer.name] = beer
then to get data on the beer you're looking at;
if beer in beers:
beers[beer].taste
This can be extended to any object type, and i believe is exactly what you're looking for;
e.g.
cheeses = {}
add_cheese(cheese):
cheeses[cheese.name] = cheese
where
class cheese:
def __init__(self, name, smelliness, hardness, colour, country):
self.name = name
self.smelliness = smelliness
self.hardness = hardness
self.colour = colour
self.country = country

For sake of completeness, I have found a way to use a Linq-like single-line statement which also has the advantage of being a bit easier to read:
taste = [b.taste for b in beers if b.name==name][0]
The [0] at the end is to return the first item, since the query returns a list of matching items. So to get the whole list:
[b.taste for b in beers if b.name==name]
This is why I love Python!