I'm having trouble getting Python's jsonpickle 0.4.0 to "recurse" in to custom objects that contain custom objects. Here's sample code that shows my problem.
import jsonpickle
import jsonpickle.handlers
class Ball(object):
def __init__(self, color):
self.color = color
class Box(object):
def __init__(self, *args):
self.contents = args
class BallHandler(jsonpickle.handlers.BaseHandler):
def flatten(self, obj, data):
data['color'] = obj.color
return data
class BoxHandler(jsonpickle.handlers.BaseHandler):
def flatten(self, obj, data):
data['contents'] = obj.contents
return data
jsonpickle.handlers.registry.register(Ball, BallHandler)
jsonpickle.handlers.registry.register(Box, BoxHandler)
# works OK -- correctly prints: {"color": "white"}
white_ball = Ball('white')
print jsonpickle.encode(white_ball, unpicklable=False)
# works OK -- correctly prints: [{"color": "white"}, {"color": "green"}]
green_ball = Ball('green')
balls = [white_ball, green_ball]
print jsonpickle.encode(balls, unpicklable=False)
# works OK -- correctly prints: {"contents": [1, 2, 3, 4]}
box_1 = Box(1, 2, 3, 4)
print jsonpickle.encode(box_1, unpicklable=False)
# dies with "Ball object is not JSON serializable"
box_2 = Box(white_ball, green_ball)
print jsonpickle.encode(box_2, unpicklable=False)
Balls have "color", Boxes have "contents". If I have a [native] array of Balls, then jsonpickle works. If I have a Box of [native] ints, then jsonpickle works.
But if I have a Box of Balls, jsonpickle bombs with "Ball object is not JSON serializable".
From the stacktrace, I have the hunch that the encoder is leaving jsonpickle and going off to some other JSON library... that apparently doesn't know that I've registered the BallHandler.
How can I fix this up?
By the way, my sample is NOT expressly using any part of Django, but I will be needing this to work in a Django app.
THANKS IN ADVANCE FOR ANY INPUT!
I think you can call back to the pickling context to continue the pickling.
class BoxHandler(jsonpickle.handlers.BaseHandler):
def flatten(self, obj, data):
return [self.context.flatten(x,reset=False) for x in obj.contents]
This seems to be similar to how the built in _list_recurse() function handles this case in pickler.py:44, as flatten() just calls self._flatten (after optionally resetting the state variables).
def _list_recurse(self, obj):
return [self._flatten(v) for v in obj]
I'm just testing on this now, and the _depth seems to be maintained as expected.
First, why are you creating custom handlers in the first place? You're attempting to do exactly the same thing the default handlers already do. Remove those two register lines and call encode with and without unpicklable=False with all of those objects, and you will get the same results—except that it will work exactly the way you want with boxes full of balls, instead of failing.
If you look through the tutorial, API, test cases, and samples, they never create a custom handler to simulate a collection like this. (For example, take a look at the Node/Document/Section classes in the test suites (samples.py and document_test.py).) So, I think you're trying to do something that you weren't expected to do, and isn't intended to be doable.
But, let's look at your actual question: Why doesn't it work?
Well, that one's easy. You're doing it wrong. According to the documentation for BaseHandler.flatten, you're supposed to:
Flatten obj into a json-friendly form.
So, given this:
class BoxHandler(jsonpickle.handlers.BaseHandler):
def flatten(self, obj, data):
data['contents'] = obj.contents
return data
You're effectively promising that obj.contents is in JSON-friendly form. But it's not; it's a list of Ball objects.
So, what's the right answer? Well, you could flatten each element in contents the same way you're being flattened. You'd think there must be some easy way to do that, but honestly, I don't see anything in the API, docs, samples, or unit tests, so I guess there isn't, so you'll have to do it manually. Presumably something like this (untested):
class BoxHandler(jsonpickle.handlers.BaseHandler):
def flatten(self, obj, data):
p = jsonpickle.Pickler()
data['contents'] = [p.flatten(elem) for elem in obj.contents]
return data
But… since you're not getting the same Pickler that's being used to pickle you—and I don't see any way that you can—this is likely going to violate the maxdepth and unpicklable parameters of encode.
So, maybe there is no right way to do this.
Looks like a bug to me and a principle one at that. If jsonpickle is about adding custom objects handling to json, it should integrate into the latter rather than attempting to "preprocess" the content for it. The present state of demanding users to handle this themselves in whatever way as abarnert said is laying blame on another's door IMO.
If I were you, I'd go for fixing this myself or make my objects JSON-friendly as they are - e.g. making them look like native Python data structures (which JSON is an alternative representation of). An easier way is to avoid such constructs which is a kludge, of course.
Related
When I subclass QValue3DAxisFormatter in my application, the application simply hangs and then exits without an exception or explanation. I have tried removing every and all methods in my subclass (even though they are supposed to be implemented, according to the docs) just to test what the problem is, but it always simply exits without explanation. I have looked everywhere for examples of how to achieve this custom formatter in Python, but the only example I can find in any language is this example: https://code.qt.io/cgit/qt/qtdatavis3d.git/tree/examples/datavisualization/qmlaxisformatter?h=5.15, which is the code from the explanation found at https://doc.qt.io/qt-5/qtdatavisualization-qmlaxisformatter-example.html#custom-axis-formatter
. I don't really understand how to translate that to Python code (I'm also not looking to make a calendar-based axis, nor am I using QML), though I've tried the basic setup as follows:
class AxisFormatter(QtDataVisualization.QtDataVisualization.QValue3DAxisFormatter):
def __init__(self):
super().__init__()
print("init")
def createNewInstance(self):
print("creating new instance")
return AxisFormatter()
def populateCopy(self, copy: QtDataVisualization.QtDataVisualization.QValue3DAxisFormatter):
print("populating copy")
super().populateCopy(copy)
def recalculate(self) -> None:
print("recalculating")
def stringForValue(self, value: float, format: str) -> str:
print('stringForValue')
return str(value)
(The only print statement that will print here is the "init" one, then after ~10 seconds of hanging, the application exits.) Ideally I would like to simply map the axis' (integer) value to an array that I provide as an argument to AxisFormatter, but I can't even get this simple boilerplate working.
Apparently there is some bug that requires a persistent reference to the formatter, so the problem is not in the code above, but rather in the assignment of the formatter to the axis. I did have:
my_chart.axisX().setFormatter(AxisFormatter())
But it must be created as follows:
self.my_new_formatter = AxisFormatter()
my_chart.axisX().setFormatter(self.my_new_formatter)
Thanks to Friedemann Kleint, a Qt JIRA admin, for this answer.
(I believe this issue may be addressed in a future Qt release.)
I was building a class that would work like a switch;
such custom class would have been a string, with an inner list, whereas the string identifies the current position of the switch ("on") and the list covers all the possible positions (["on","off","halfway on", "kinda dead but still on", ...];
While initializing an instance of this class, I would accept an indefinite number of possible positions;
Hence the code
class switch(str):
pos=[]
def __init__(self,*positions):
self.pos=[str(el) for el in positions]
self=self.pos[0]
def swap(self):
try:
self=self.pos[self.pos.index(self)+1]
except IndexError:
self=self.pos[0]
and the call
mood=switch("upbeat","depressed","horny")
That quite doesn't work. The error goes like this:
TypeError: decoding str is not supported
Of course there are better ways to make this Switch class (this was just a draft of a quick insertion) and eventually, I already did; still I didn't understand the origins of that error - whose search through Google wasn't useful - and angered enough about it that I finally decided to sign up here on Stack and post about it.
What do you guys think about it?
Googling it up, it seems like the error would turn off when trying to concatenate strings like "str1","str2" and not like "str1"+"str2"; then I think it's safe to keep suspicions around the *positions thing, with it not unpacking the strings as separate variables, but trying to guess I wanted them to be joined as one but failed at it as I wasn't using the + concatenator.
That's what came to my mind.
Otherwise, if I'm wrong, I've seen the error pop up only while playing with formats and stuff. Which is pretty way off in this case.
You can subclass str. But if you are passing more than a single argument, you need to intercept that at __new__() so python doesn't try to interpret the other arguments when creating the object.
Note this is just to demonstrate — it won't work for your problem
class switch(str):
def __new__(cls, *content):
return str.__new__(cls, content[0])
def __init__(self, *positions):
self.pos = [str(el) for el in positions]
mood=switch("upbeat", "depressed", "horny")
# prints as expected and has string methods
print(mood, mood.upper())
# upbeat UPBEAT
# has your instance attribute
print(mood.pos)
['upbeat', 'depressed', 'horny']
The problem is that strings are not mutable. So this is doomed from the beginning if the idea is to change the value of the string in-place. You can instead use collections.UserString for this. This acts like a string but gives you a data property to store the actual value. With this, your idea might work:
from collections import UserString
class Switch(UserString):
def __init__(self, *positions):
self.pos = [str(el) for el in positions]
# store the actual string in .data
self.data = self.pos[0]
def swap(self):
try:
self.data = self.pos[self.pos.index(self)+1]
except IndexError:
self.data = self.pos[0]
mood=Switch("upbeat", "depressed", "horny")
# still acts like a string
print(mood, mood.upper())
# upbeat UPBEAT
# but now you can swap
mood.swap()
print(mood, mood.upper())
# depressed DEPRESSED
I think Python thinks you're using the second constructor of str() (https://docs.python.org/3/library/stdtypes.html#str).
e.g. switch(b"upbeat", "utf-8") would work.
Say I have a class:
class Data():
def __init__(self):
self.scores = []
self.encoding= {1: 'first', 2: 'second', 3:'third'}
def build():
self.scores = [1, 2, 3]
def translate(self):
return [self.encoding[score] for val in self.scores]
Now I want to be able to translate the columns for a given data object...
# I want to be able to do
d= Data()
d.scores.translate()
# AttributeError: 'list' object has no attribute 'translate'
# Instead of
d= Data()
d.translate()
Now I am fully aware that I am trying to access a method that does NOT exist for that list (translate()). I want to be able to make method calls as is mentioned above (d.scores.translate()) because I may have some specific subslice of d.scores I want to translate.
For Example, if d.scores was a nested numpy array (I only want to translate 1st 5 columns but keep all rows)
#this is what I would like to be able to do
d.scores[:, 1:5].translate()
# And I don't want to build a kwarg in the translate method to handle it like
d.scores.translate(indices=[1])
I know this is more of an implementation question, and I'm wondering what the best practice should be.
Am I trying to force a square peg into a round hole at this point? Should I just give up and define a module function or consider the kwargs? Is that more 'pythonic'?
UPDATE
I should have said this sooner but I did try using the kwarg and staticmethod route. I just want to know if there are other ways to accomplish this? Maybe through subclassing? or Python's equivalent of interfacing in java/C# (if it exists?)
Yes, you are trying to "force a square peg into a round hole".
Your translate method works on the whole scores list, full stop. This can not be changed with some trickery, which is simply not supported in Python.
When you want to do subslices, I would recommend doing it explicitly.
Examples:
# Using args/kwargs:
scores = john.translate(10, 15) # translate subslice 10:15
# Using a new static method:
scores = Person.translate_scores(john.scores[10:15])
Looks not that elegant, but works.
(BTW: Since you changed your question, my classes might be a little of, but I will not change my answer with every edit you make)
Your trickery simply does not work, because "scores" is not some part of your main class, but simply an attribute of it, which has its own type. So, when you do "d.scores.translate()" translate is not called on d, but on a list or whatever type scores is. You can not change that, because it is core Python.
You could do it by using a second class and use _scores for the list and a sub-object scores which manipulates _scores:
class DataHelper(object):
def __init__(self, data_obj):
self.data_obj = data_obj
def translate(self, *args):
... # work on self.data_obj._scores
class Data(object):
def __init__(self):
self.scores = DataHelper(self)
self._scores = []
With such a class structure, you might be able to to this:
scores = d.scores.translate(1, 5)
And with more trickery, you might be able to even do:
scores = d.scores[1:5].translate()
But for that, you will need a third class (objects of that will be created temporary on indexing scores objects, so that d.scores[1:5] will not create a list slice but a new object with translate method).
I am trying to set the attribute values of a certain class AuxiliaryClass than is instantiated in a method from MainClass class in the most efficient way possible.
AuxiliaryClass is instantiated within a method of MainClass - see below. However, AuxiliaryClass has many different attributes and I need to set the value of those attributes once the class has been instantiated - see the last 3 lines of my code.
Note: due to design constraints I cannot explain here, my classes only contain methods, meaning that I need to declare attributes as methods (see below).
class AuxiliaryClass(object):
def FirstMethod(self):
return None
...
def NthMethod(self):
return None
class MainClass(object):
def Auxiliary(self):
return AuxiliaryClass()
def main():
obj = MainClass()
obj.Auxiliary().FirstMethod = #some_value
...
obj.Auxiliary().NthMethod = #some_other_value
# ~~> further code
Basically I want to replace these last 3 lines of code with something neater, more elegant and more efficient. I know I could use a dictionary if I was instantiating AuxiliaryClass directly:
d = {'FirstMethod' : some_value,
...
'NthMethod' : some_other_value}
obj = AuxiliaryClass(**d)
But this does not seem to work for the structure of my problem. Finally, I need to set the values of AuxiliaryClass's attributes once MainClass has been instantiated (so I can't set the attribute's values within method Auxiliary).
Is there a better way to do this than obj.Auxiliary().IthMethod = some_value?
EDIT
A couple of people have said that the following lines:
obj.Auxiliary().FirstMethod = #some_value
...
obj.Auxiliary().NthMethod = #some_other_value
will have no effect because they will immediately get garbage collected. I do not really understand what this means, but if I execute the following lines (after the lines above):
print(obj.Auxiliary().FirstMethod())
...
print(obj.Auxiliary().NthMethod())
I am getting the values I entered previously.
To speed things up, and make the customization somewhat cleaner, you can cache the results of the AuxilliaryClass constructor/singleton/accessor, and loop over a dict calling setattr().
Try something like this:
init_values = {
'FirstMethod' : some_value,
:
'NthMethod' : some_other_value,
}
def main():
obj = MainClass()
aux = obj.Auxiliary() # cache the call, only make it once
for attr,value in init_values.items(): # python3 here, iteritems() in P2
setattr(aux, attr, value)
# other stuff below this point
I understand what is happening here: my code has a series of decorators before all methods which allow memoization. I do not know exactly how they work but when used the problem described above - namely, that lines of type obj.Auxiliary().IthMethod = some_value get immediately garbage collected - does not occur.
Unfortunately I cannot give further details regarding these decorators as 1) I do not understand them very well and 2) I cannot transmit this information outside my company. I think under this circumstances it is difficult to answer my question because I cannot fully disclose all the necessary details.
First, if you guys think the way I'm trying to do things is not Pythonic, feel free to offer alternative suggestions.
I have an object whose functionality needs to change based on outside events. What I've been doing originally is create a new object that inherits from original (let's call it OrigObject()) and overwrites the methods that change (let's call the new object NewObject()). Then I modified both constructors such that they can take in a complete object of the other type to fill in its own values based on the passed in object. Then when I'd need to change functionality, I'd just execute myObject = NewObject(myObject).
I'm starting to see several problems with that approach now. First of all, other places that reference the object need to be updated to reference the new type as well (the above statement, for example, would only update the local myObject variable). But that's not hard to update, only annoying part is remembering to update it in other places each time I change the object in order to prevent weird program behavior.
Second, I'm noticing scenarios where I need a single method from NewObject(), but the other methods from OrigObject(), and I need to be able to switch the functionality on the fly. It doesn't seem like the best solution anymore to be using inheritance, where I'd need to make M*N different classes (where M is the number of methods the class has that can change, and N is the number of variations for each method) that inherit from OrigObject().
I was thinking of using attribute remapping instead, but I seem to be running into issues with it. For example, say I have something like this:
def hybrid_type2(someobj, a):
#do something else
...
class OrigObject(object):
...
def hybrid_fun(self, a):
#do something
...
def switch(type):
if type == 1:
self.hybrid_fun = OrigObject.hybrid_fun
else:
self.fybrid_fun = hybrid_type2
Problem is, after doing this and trying to call the new hybrid_fun after switching it, I get an error saying that hybrid_type2() takes exactly 2 arguments, but I'm passing it one. The object doesn't seem to be passing itself as an argument to the new function anymore like it does with its own methods, anything I can do to remedy that?
I tried including hybrid_type2 inside the class as well and then using self.hybrid_fun = self.hybrid_type2 works, but using self.hybrid_fun = OrigObject.hybrid_fun causes a similar error (complaining that the first argument should be of type OrigObject). I know I can instead define OrigObject.hybrid_fun() logic inside OrigObject.hybrid_type1() so I can revert it back the same way I'm setting it (relative to the instance, rather than relative to the class to avoid having object not be the first argument). But I wanted to ask here if there is a cleaner approach I'm not seeing here? Thanks
EDIT:
Thanks guys, I've given points for several of the solutions that worked well. I essentially ended up using a Strategy pattern using types.MethodType(), I've accepted the answer that explained how to do the Strategy pattern in python (the Wikipedia article was more general, and the use of interfaces is not needed in Python).
Use the types module to create an instance method for a particular instance.
eg.
import types
def strategyA(possible_self):
pass
instance = OrigObject()
instance.strategy = types.MethodType(strategyA, instance)
instance.strategy()
Note that this only effects this specific instance, no other instances will be effected.
You want the Strategy Pattern.
Read about descriptors in Python. The next code should work:
else:
self.fybrid_fun = hybrid_type2.__get__(self, OrigObject)
What about defining it like so:
def hybrid_type2(someobj, a):
#do something else
...
def hybrid_type1(someobj, a):
#do something
...
class OrigObject(object):
def __init__(self):
...
self.run_the_fun = hybrid_type1
...
def hybrid_fun(self, a):
self.run_the_fun(self, a)
def type_switch(self, type):
if type == 1:
self.run_the_fun = hybrid_type1
else:
self.run_the_fun = hybrid_type2
You can change class at runtime:
class OrigObject(object):
...
def hybrid_fun(self, a):
#do something
...
def switch(self):
self.__class__ = DerivedObject
class DerivedObject(OrigObject):
def hybrid_fun(self, a):
#do the other thing
...
def switch(self):
self.__class__ = OrigObject