We Keep Coding

Is it really impossible to unpickle a Python class if the original python file has been deleted?

Suppose you have the following:
file = 'hey.py'
class hey:
def __init__(self):
self.you =1
ins = hey()
temp = open("cool_class", "wb")
pickle.dump(ins, temp)
temp.close()
Now suppose you delete the file hey.py and you run the following code:
pkl_file = open("cool_class", 'rb')
obj = pickle.load(pkl_file)
pkl_file.close()
You'll get an error. I get that it's probably the case that you can't work around the problem of if you don't have the file hey.py with the class and the attributes of that class in the top level then you can't open the class with pickle. But it has to be the case that I can find out what the attributes of the serialized class are and then I can reconstruct the deleted file and open the class. I have pickles that are 2 years old and I have deleted the file that I used to construct them and I just have to find out what what the attributes of those classes are so that I can reopen these pickles
#####UPDATE
I know from the error messages that the module that originally contained the old class, let's just call it 'hey.py'. And I know the name of the class let's call it 'you'. But even after recreating the module and building a class called 'you' I still can't get the pickle to open. So I wrote this code on the hey.py module like so:
class hey:
def __init__(self):
self.hey = 1
def __setstate__(self):
self.__dict__ = ''
self.you = 1
But I get the error message: TypeError: init() takes 1 positional argument but 2 were given
#########UPDATE 2:
I Changed the code from
class hey:
to
class hey():
I then got an AttributeError but it doesn't tell me what attribute is missing. I then performed
obj= pickletools.dis(file)
And got an error on the pickletools.py file here
def _genops(data, yield_end_pos=False):
if isinstance(data, bytes_types):
data = io.BytesIO(data)
if hasattr(data, "tell"):
getpos = data.tell
else:
getpos = lambda: None
while True:
pos = getpos()
code = data.read(1)
opcode = code2op.get(code.decode("latin-1"))
if opcode is None:
if code == b"":
raise ValueError("pickle exhausted before seeing STOP")
else:
raise ValueError("at position %s, opcode %r unknown" % (
"<unknown>" if pos is None else pos,
code))
if opcode.arg is None:
arg = None
else:
arg = opcode.arg.reader(data)
if yield_end_pos:
yield opcode, arg, pos, getpos()
else:
yield opcode, arg, pos
if code == b'.':
assert opcode.name == 'STOP'
break
At this line:
code = data.read(1)
saying: AttributeError: 'str' object has no attribute 'read'
I will now try the other methods in the pickletools
########### UPDATE 3
I wanted to see what happened when I saved an object composed mostly of dictionary but some of the values in the dictionaries were classes. This is the class that was saved:
so here is the class in question:
class fss(frozenset):
def __init__(self, *args, **kwargs):
super(frozenset, self).__init__()
def __str__(self):
str1 = lbr + "{}" + rbr
return str1.format(','.join(str(x) for x in self))
Now keep in mind that the object pickled is mostly a dictionary and that class exists within the dictionary. After performing
obj= pickletools.genops(file)
I get the following output:
image
image2
I don't see how I would be able to construct the class referred to with that data if I hadn't known what the class was.
############### UPDATE #4
#AKK
Thanks for helping me out. I am able to see how your code works but my pickled file saved from 2 years ago and whose module and class have long since been deleted, I cannot open it into a bytes-like object which to me seems to be a necessity.
So the path of the file is
file ='hey.pkl'
pkl_file = open(file, 'rb')
x = MagicUnpickler(io.BytesIO(pkl_file)).load()
This returns the error:
TypeError: a bytes-like object is required, not '_io.BufferedReader'
But I thought the object was a bytes object since I opened it with open(file, 'rb')
############ UPDATE #5
Actually, I think with AKX's help I've solved the problem.
So using the code:
pkl_file = open(name, 'rb')
x = MagicUnpickler(pkl_file).load()
I then created two blank modules which once contained the classes found in the save pickle, but I did not have to put the classes on them. I was getting an error in the file pickle.py here:
def load_reduce(self):
stack = self.stack
args = stack.pop()
func = stack[-1]
try:
stack[-1] = func(*args)
except TypeError:
pass
dispatch[REDUCE[0]] = load_reduce
So after excepting that error, everything worked. I really want to thank AKX for helping me out. I have actually been trying to solve this problem for about 5 years because I use pickles far more often than most programmers. I used to not understand that if you alter a class then that ruins any pickled files saved with that class so I ran into this problem again and again. But now that I'm going back over some code which is 2 years old and it looks like some of the files were deleted, I'm going to need this code a lot in the future. So I really appreciate your help in getting this problem solved.

Well, with a bit of hacking and magic, sure, you can hydrate missing classes, but I'm not guaranteeing this will work for all pickle data you may encounter; for one, this doesn't touch the __setstate__/__reduce__ protocols, so I don't know if they work.
Given a script file (so72863050.py in my case):
import io
import pickle
import types
from logging import Formatter
# Create a couple empty classes. Could've just used `class C1`,
# but we're coming back to this syntax later.
C1 = type('C1', (), {})
C2 = type('C2', (), {})
# Create an instance or two, add some data...
inst = C1()
inst.child1 = C2()
inst.child1.magic = 42
inst.child2 = C2()
inst.child2.mystery = 'spooky'
inst.child2.log_formatter = Formatter('heyyyy %(message)s') # To prove we can unpickle regular classes still
inst.other_data = 'hello'
inst.some_dict = {'a': 1, 'b': 2}
# Pickle the data!
pickle_bytes = pickle.dumps(inst)
# Let's erase our memory of these two classes:
del C1
del C2
try:
print(pickle.loads(pickle_bytes))
except Exception as exc:
pass # Can't get attribute 'C1' on <module '__main__'> – yep, it certainly isn't there!
we now have successfully created some pickle data that we can't load anymore, since we forgot about those two classes. Now, since the unpickling mechanism is customizable, we can derive a magic unpickler, that in the face of certain defeat (or at least an AttributeError), synthesizes a simple class from thin air:
# Could derive from Unpickler, but that may be a C class, so our tracebacks would be less helpful
class MagicUnpickler(pickle._Unpickler):
def __init__(self, fp):
super().__init__(fp)
self._magic_classes = {}
def find_class(self, module, name):
try:
return super().find_class(module, name)
except AttributeError:
return self._create_magic_class(module, name)
def _create_magic_class(self, module, name):
cache_key = (module, name)
if cache_key not in self._magic_classes:
cls = type(f'<<Emulated Class {module}:{name}>>', (types.SimpleNamespace,), {})
self._magic_classes[cache_key] = cls
return self._magic_classes[cache_key]
Now, when we run that magic unpickler against a stream from the aforebuilt pickle_bytes that plain ol' pickle.loads() couldn't load...
x = MagicUnpickler(io.BytesIO(pickle_bytes)).load()
print(x)
print(x.child1.magic)
print(x.child2.mystery)
print(x.child2.log_formatter._style._fmt)
prints out
<<Emulated Class __main__:C1>>(child1=<<Emulated Class __main__:C2>>(magic=42), child2=<<Emulated Class __main__:C2>>(mystery='spooky'), other_data='hello', some_dict={'a': 1, 'b': 2})
42
spooky
heyyyy %(message)s
Hey, magic!

The error in function load_reduce(self) can be re-created by:
class Y(set):
pass
pickle_bytes = io.BytesIO(pickle.dumps(Y([2, 3, 4, 5])))
del Y
print(MagicUnpickler(pickle_bytes).load())
AKX's answer do not solve cases when the class inherit from base classes as set, dict, list,...

Subscriber receives String with 'Data: ""'

I have a publisher that sends a number from 0 to 6, or None. So I convert it to a string before sending it to make sure it can send the "None" value.
self.pub = rospy.Publisher("jugada", String, queue_size=1)
self.pub.publish(str(col))
and then i want the subscriber to receive it:
class clicker(object):
def __init__(self):
self.jugada_sub = rospy.Subscriber("jugada",String, self.callback)
def callback(self, data):
jugada = data
print(jugada)
and instead of printing:
3
instead it prints
Data: "3"
which breaks up the rest of the code I need that number (or None) for.
I tried to str(data) and then try to edit the string to remove the 'Data:' part, but that doesn't work.
I tried to google it for like an hour, but can't figure out how to get rid of the "Data:" or how to change the message type to send only the string.
PS Later in the code i do:
if jugada != "None":
jugada = int(jugada)
and I get the error: int() argument must be a string
and if i do jugada = str(data) at the beginning, i get the error: Invalid literal for int() with base 10: 'data: "3"'

ROS messages are defined as classes, not built in types. For std_msg types you need to retrieve the data directly using the .data attribute. Take the following example:
def callback(self, data):
jugada = data.data
print(jugada)
Another couple of notes. When publishing data it is usually a best practice to not pass the raw type directly to .publish and instead use a message type as follows:
output_msg = String()
output_msg.data = 'some string val'
pub.publish(output_msg)
Lastly, if you're publishing integer data you should use one of the integer types: Int8, Int16, Int32, etc

Getting TypeError: object of type 'int' has no len() when using struct.pack(), but only in a class

I'm getting this 'TypeError: object of type 'int' has no len()' for a data conversion function that I am trying to integrating into a class. Outside of the class it works with no errors, put it into the class, or try to access it inside of the class, and I'm getting the same error.
The program I'm writing sends int/float type data between two rf/lora nodes. (This is just a portion of a larger telemetry program) I am converting the data to byte arrays using struct.pack() in order to send them. Code is written in CircuitPython using Adafruit hardware (Feather m4 & lora breakout board) and libraries.
Any suggestion on how to fix the issue or what's causing it would be greatly appreciated!
Code Below:
class TransmitState(State):
def __init__(self):
super().__init__()
self.entered = time.monotonic()
self.rf = adafruit_rfm9x.RFM9x(spi, cs, reset, radio_freq)
self.rf.tx_power = 23
self.rf.enable_crc = True
self.rf.node = 1
self.rf.destination = 2
led.value = True
# converts data to byte array - line error on ba = ...
def pack_data(self, data_to_pack):
# First encode the number of data items, then the actual items
ba = struct.pack("!I" + "d" * len(data_to_pack), # <--- Line that's causing the error
len(data_to_pack), *data_to_pack)
return ba
# sendData sends the converted data to our 2nd node
def sendData(self, data):
# checks to see if data is null
if data:
ts = self.pack_data(data) # <--- Call to convert fcn
self.rf.send(ts)
#property
def name(self):
return 'transmit'
def enter(self, machine):
State.enter(self, machine)
self.entered = time.monotonic()
led.value = True
print("Entering Transmit")
def exit(self, machine):
print("Exiting Transmit")
State.exit(self, machine)
def update(self, machine):
if State.update(self, machine):
now = time.monotonic()
if now - self.entered >= 1.0:
led.value = False
print("Transmitting...")
self.sendData(3) # <-- FCN Call, 3 is arbitrary
machine.go_to_state('idle')
Note: Import and pin assignments not shown

That error is because you are trying to get the length of an int.
len() attribute is only for iterable types like dict, list, strings,sets and tuples.
Try :
ba = struct.pack("!I" + "d" * len(str(data_to_pack)), len(str(data_to_pack)), *data_to_pack):
What this will do is convert the data into string and then get the length.

How to emulate data from a serial port using Python 3 unittest mocks?

I want to create unit test for a Python 3 method that reads unicode text strings from a serial port. I want to test the method's response to various strings. The line of code that I want to emulate is:
comm_buffer = serial_port.readline().decode("utf-8").strip()
Where 'serial_port' is an instance of a serial port that is passed to the method. I would like to use the unittest.mock module to set the comm_buffer variable to a unicode string, but I have been struggling all day with no success. My first time with trying to use mock, and I'm out of my depth.
The code for the whole method is:
def wait_for_data(serial_port, comm_flag = ""):
"""Read lines of data from the serial_port
Receives optional comm_flag (single character to check for at beginning of string)"""
logging.debug("Start wait_for_data " + comm_flag)
timeout_count = 0
while True:
# Get a line from the buffer and convert to string and strip line feed
logging.debug("Waiting for data…")
comm_buffer = serial_port.readline().decode("utf-8").strip()
if len(comm_buffer) == 0:
timeout_count += 1
logging.warning("Serial port timeout - no data received. Timeout count = " + str(timeout_count))
if timeout_count > 10:
raise TimeoutError(["Too many timeouts"])
# If no id character was specified, just return the string
elif comm_flag == "":
logging.debug("Returning no comm_flag")
return comm_buffer
# If an id character was specified, return the string if it's present (strip id), otherwise wait for another string
elif comm_buffer[0] == comm_flag:
logging.debug("Returning with comm_flag")
return comm_buffer[1:]

Serial_port is not an instance of a serial port but an object with a readline() method. So don’t care about such thing as a serial port, your mock object is an object with the readline() method which provides the kind of value you want to test.
So you just have to crate something like :
port = Mock()
port.readline = Mock(return_value="my string")
This is the first parameter for your call.
So, if I copy your function in a module named test.port this tests are ok :
class TestWaitData(unittest.TestCase):
def testFunc(self):
port = mock.Mock()
port.readline = mock.Mock(return_value="my string".encode('utf-8')) # as you use a decode call
self.assertEqual(test.port.wait_for_data(port), "my string")
def testBufferEmpty(self):
port = mock.Mock()
port.readline = mock.Mock(return_value="".encode('utf-8'))
with self.assertRaises(TimeoutError):
test.port.wait_for_data(port)
def testWithFlag(self):
port = mock.Mock()
port.readline = mock.Mock(return_value="fmy string".encode('utf-8'))
self.assertEqual(test.port.wait_for_data(port, 'f'), "my string")

Strange Code Indenting Issue

I'm probably missing something obvious, but I've been trying to solve this problem for about an hour, without any success. Probably going to feel really stupid when the solution is found. Here's the error I'm getting:
File "xpc_connection.py", line 77
def remoteObjectProxy():
^
IndentationError: unexpected indent
If I delete that portion of code, I get an indent error for the next line. There are some super strange problems going on with my indenting ...
Here's my code (yes, I know it's rather incomplete, and may have a few issues):
from collections import namedtuple;
import socket;
import sys;
from recvTimeout import recv_timeout
from recvTimeout import recv_end
from recvTimeout import sendAllWithEnd
# Named tuple that defines struct-like structure.
# field1 - body length, field2 - headerChecksum, field3 - checksum
XPCMessageHeader = namedtuple("XPCMessageHeader", "field1 field2 field3");
class XPCConnection(object):
"""An class that represents a connection made between processes using XPC.
Attributes:
"""
def __init__(self, serviceName):
self._serviceName = serviceName;
self._exportedObject = None; # This process's "representation" of itself.
self._remoteObjectProxy = None; # This process's "representation" of the remote process.
self._exportedObjectInterface = None; # Methods allowed to be received by exported object on this connection.
self._remoteObjectInterface = None; # Methods allowed to be received by object that has been "imported"
# to this connection.
self._connectionSocket = None # Domain socket that is endpoint of connection between processes.
def connect():
# Create a UDS socket
_connectionSocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
# Change this to port where remote process is listening.
print >>sys.stderr, 'connecting to %s' % _serviceName
try:
_connectionSocket.connect(_serviceName)
except socket.error, msg:
print >>sys.stderr, msg
sys.exit(1)
print >>sys.stderr, 'Attempting to connect.'
try:
# Send data
sendAllWithEnd(_connectionSocket,'Connected Successfully.')
data = recv_end(_connectionSocket);
print >>sys.stderr, 'received "%s"' % data;
except socket.error:
print >>sys.stderr, 'Connection Failed.';
def disconnect():
print >>sys.stderr, 'closing socket';
_connectionSocket.close();
def serviceName():
return serviceName;
#TODO
'''
def readMessage():
def readMessage():
def resume():
def invalidate():
'''
def remoteObjectProxy():
return _remoteObjectProxy;
#classmethod
def setRemoteObjectProxy(cls):
_remoteObjectProxy = CreateWithConnection(cls); # Create a remoteObjectProxy object with connection
# field that is this connection object.
def exportedObject():
return exportedObject;
def setExportedObject(exportedObject):
_exportedObject = exportedObject;
'''
# PRIVATE Helper Methods
# invocation of type XPCInvocation, completionHandler of type XPCInvocationCompletionHandler
# return type void
def _invokeOnRemoteObject(invocation, completionHandler):
# invocation of type XPCInvocation
# return type XPCValue
def _invokeOnExportedObject(invocation):
# May be unnecessary for now.
# fd of type int (represents a file descriptor)
# return type bool (indicates success?)
def _setFileDescriptor(int fd):
# data of type DataRef
# return type void
def _write(data):
# Probably not necessary
# estimatedBytesToRead of type int
# return type void
def _readDataOnReadQueue(estimatedBytesToRead):
# Likely necessary
# bytes of type string, length of type int
# return type void
def _processDataOnReadQueue(bytes, length):
# Likely unecessary
# data of type DataRef
# return type void
def _writeDataOnWriteQueue(data):
# error of type ErrorRef
# return type void
def _terminateWithErrorSync(error):
# error of type Error Ref
# return type void
def _terminateWithErrorOnUserQueue(error):
# return type bool. Returns true if connected, false otherwise
def _isConnected():
# delayInSecond of type int. Not sure if you can pass in an argument like this.
# return type void
def _connectWithExponentialBackoff(delayInSeconds = 0.0625):
# return type bool. Returns true iff connected successfully
def _attemptToConnect():
# Likely unecessary
# return type void
def _flushRequestQueue():
# closes connection
# return type void
def _disconnect():
'''
#TODO: Invocation handler equivalent.
# "This" process's representation of the other process.
class XPCRemoteObjectProxy(object):
def __init__(self, connection):
# Reference to the connection Remote Object is a part of. Necessary so that when
# you invoke a method and get stuff back, reomte object proxy knows where to send stuff back to.
self._connection = connection;
def invoke(invocation):
_connection.invokeOneRemoteObject(invocation); # TODO: invokeOneRemoteObject
# Used to represent "this" process.
class XPCExportedObject(object):
def __init__(self):
self._invocationHandlersByMethodSignature = {}; # Invocation handlers stored in dictionary. Keyed by method signatures.
# invocation is XPCInfocation object, returnValue XPCValue object
def invoke(invocation, returnValue):
try:
# We directly modify returnValue here. Unsure if this acutally works.
returnValue = _invocationHandlersByMethodSignature[methodSignature()];
return True;
except KeyError:
return False;
# Handler is of type XPCInvocationHandler and methodName is of type string.
# Come back to this
def registerInvocationHandlerForMethodSignature(handler, methodName):
return True
# Used to call a method across an XPC connection.
class XPCInvocation(object):
def __init__(self):
self._methodSignature = ""; # Signature of method to be called.
self._arguments = []; # List of arguments for the called method. Elements are XPCValue objects.
# TODO: This is definitely incorrect.
# Make this a classmethod?
def createFromSerializedRepresentation(serializedRepresentation):
invocation = self.__class__();
invocation.setMethodSignature(serializedRepresentation.methodsignature());
for serializedValue in serializedRepresentation.values():
invocation._arguments.append(FromSerializedRepresentation(serializedValue));
# Note: FromSerializedRepresentation function defined in XPCValue class.
# TODO: XPCValue Class
return invocation;
def getMethodSignature():
return _methodSignature;
def setMethodSignature(methodSignature):
_methodSignature = methodSignature;
def getArguments():
return _arguments
# Takes in an XPCValue as an argument.
def appendArgument(value):
_arguments.append(value);
# TODO: This is definitely incorrect.
# NOTE: XPCInvocationMessage has yet to be written. It is provided by protobuf.
def serializedRepresentation():
message = XPCInvocationMessage();
message.set_methodsignature(_methodSignature);
for value in _arguments:
message.add_values().CopyFrom(value.serializedRepresentation());
return message

You're using multi-line strings to substitute for multi-line comments. They are not comments, however, and when they're dedented all the way out like that, then it terminates the class scope and you can't get back into it.
Quick fix, indent the opening ''' to match the class scope.