We Keep Coding

TypeError: can't pickle _thread.RLock objects in python 3

I have a very large project of a Web API using Flask and Python. It is used for testing some electronic hardware automatically.
The program uses some threading in order to run a web UI while a server runs some services (SSH, serial, VISA) among others.
The program was originally coded in python 2.7 and works just fine with this version. Right now, I am trying to update it to python 3.8 for obvious reasons.
As I am updating the project, I'm having trouble with the copy library. It is supposed to serialize a _thread.RLock object and to send it to another thread, but it keeps giving me an error. Here is the traceback that I get:
Traceback (most recent call last):
File "c:\git_files\[...]\nute\route_config\flask_api_testbench.py", line 208, in _hook_run
super(FlaskAPITestbench, self).hook_run()
File "c:\git_files\[...]\nute\core\testbench\base.py", line 291, in hook_run
while self.state_machine():
File "c:\git_files\[...]\nute\core\testbench\base.py", line 304, in state_machine
on_input=self.state_testrun
File "c:\git_files\[...]\nute\core\testbench\base.py", line 380, in wait_for_input_or_testrun
self.hook_load_testrun(config_with_input)
File "c:\git_files\[...]\nute\core\testbench\base.py", line 428, in hook_load_testrun
self.interface.load_testrun(self.load_testrun(config))
File "c:\git_files\[...]\nute\core\testbench\base.py", line 461, in load_testrun
testrun = self.test_loader.load_testrun(config, context_type=self.TestRunContext)
File "c:\git_files\[...]\nute\core\testrun\loader.py", line 89, in load_testrun
testrun_template = process_all_loaders(self.batchers, _process_batcher)
File "c:\git_files\[...]\nute\core\config\loader.py", line 127, in process_all_loaders
return fn(loader)
File "c:\git_files\[...]\nute\core\testrun\loader.py", line 85, in _process_batcher
batcher.batch_testrun(testrun_template, config, context)
File "c:\git_files\[...]\nute\batcher\python_module_batcher.py", line 21, in batch_testrun
batch_module.main(testrun, context)
File "C:\GIT_Files\[...]\pyscripts\script\patest\_batch.py", line 168, in main
test.suite(ImpedanceTest)
File "c:\git_files\[...]\nute\core\testrun\base.py", line 213, in suite
testsuite = testsuite_instance_or_class()
File "c:\git_files\[...]\nute\core\functions\helpers.py", line 233, in __new__
cls._attach_nodes_to(template)
File "c:\git_files\[...]\nute\core\functions\helpers.py", line 271, in _attach_nodes_to
node = root.import_testcase(testcase)
File "c:\git_files\[...]\nute\core\functions\specific.py", line 307, in import_testcase
test_node = testcase.copy(cls=self.__class__)
File "c:\git_files\[...]\nute\core\functions\base.py", line 645, in copy
value = copy(value)
File "c:\users\[...]\.conda\envs\py37\lib\copy.py", line 96, in copy
rv = reductor(4)
TypeError: can't pickle _thread.RLock objects
It works fine in Python 2.7, but not with Python 3.x. I've tried it on 3.7.10, 3.8.9 and 3.9.6 with the same result.
Here's the implementation of my wrap method of copy:
from copy import copy
...
def copy(self, cls=None): # class method
if cls is None:
cls = self.__class__
new_self = cls()
for key, value in self.__dict__.items():
# if key == "multithread_lock":
# continue
if self.should_copy_attribute(key, value):
# Handle recursion by pointing to the new object instead of copying.
if value is self:
value = new_self
else:
value = copy(value) # This is where it fails
new_self.__dict__[key] = value
return new_self
As you can see with the commented part, skipping the pickling of any _thread.RLock object makes the program work, but I need to refresh the web UI manually to see it running since the thread doesn't work.
Any idea why it's working on python 2.7 but not on newer versions? Thanks in advance.

So I found out that a _thread.RLock() object cannot be copied. I just added a condition to skip an object like this to be copied, and it works fine.
For the web UI not refreshing, I changed to a lower version of Flask-SocketIO and it worked just fine.

pickle error when serialize cython class with lambda

I use pickle and dill for follow lambda function and work fine :
import dill
import pickle
f = lambda x,y: x+y
s = pickle.dumps(f)
or even when used in class, for example:
file
foo.py
class Foo(object):
def __init__(self):
self.f = lambda x, y: x+y
file
test.py
import dill
import pickle
from foo import Foo
f = Foo()
s = pickle.dumps(f) # or s = dill.dumps(f)
but when build same file with format .pyx (foo.pyx) using cython, can't serialize with dill, pickle or cpickle, get this error :
Traceback (most recent call last):
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_cod
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
a = pickle.dumps(c)
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 1380, in dumps
Pickler(file, protocol).dump(obj)
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 425, in save_reduce
save(state)
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/dill/_dill.py", line 912, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 669, in _batch_setitems
save(v)
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 317, in save
self.save_global(obj, rv)
File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 754, in save_global
(obj, module, name))
PicklingError: Can't pickle . at 0x7f9ab1ff07d0>: it's not found as foo.lambda
setup.py file for build cython
setup.py
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize("foo.pyx"))
then run in terminal:
python setup.py build_ext --inplace
Is there a way ?

I'm the dill author. Expanding on what #DavidW says in the comments -- I believe there are (currently) no known serializers that can pickle cython lambdas, or the vast majority of cython code. Indeed, it is much more difficult for python serializers to be able to pickle objects with C-extensions unless the authors of the C-extension code specifically build serialization instructions (as did numpy and pandas). In that vein... instead of a lambda, you could build a class with a __call__ method, so it acts like a function... and then add one or more of the pickle methods (__reduce__, __getstate__, __setstate__, or something similar)... and then you should be able to pickle instances of your class. It's a bit of work, but since this path has been used to pickle classes written in C++ -- I believe you should be able to get it to work for cython-built classes.

In this code
import dill
import pickle
f = lambda x,y: x+y
s = pickle.dumps(f)
f is a function,
But in another code
import dill
import pickle
from foo import Foo
f = Foo()
s = pickle.dumps(f)
# or
s = dill.dumps(f)
f is a class

Pyspark got TypeError: can’t pickle _abc_data objects

I’m trying to generate predictions from a pickled model with pyspark, I get the model with the following command
model = deserialize_python_object(filename)
with deserialize_python_object(filename) defined as:
import pickle
def deserialize_python_object(filename):
try:
with open(filename, ‘rb’) as f:
obj = pickle.load(f)
except:
obj = None
return obj
the error log looks like:
File “/Users/gmg/anaconda3/envs/env/lib**strong text**/python3.7/site-packages/pyspark/sql/udf.py”, line 189, in wrapper
return self(*args)
File “/Users/gmg/anaconda3/envs/env/lib/python3.7/site-packages/pyspark/sql/udf.py”, line 167, in __call__
judf = self._judf
File “/Users/gmg/anaconda3/envs/env/lib/python3.7/site-packages/pyspark/sql/udf.py”, line 151, in _judf
self._judf_placeholder = self._create_judf()
File “/Users/gmg/anaconda3/envs/env/lib/python3.7/site-packages/pyspark/sql/udf.py”, line 160, in _create_judf
wrapped_func = _wrap_function(sc, self.func, self.returnType)
File “/Users/gmg/anaconda3/envs/env/lib/python3.7/site-packages/pyspark/sql/udf.py”, line 35, in _wrap_function
pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command)
File “/Users/gmg/anaconda3/envs/env/lib/python3.7/site-packages/pyspark/rdd.py”, line 2420, in _prepare_for_python_RDD
pickled_command = ser.dumps(command)
File “/Users/gmg/anaconda3/envs/env/lib/python3.7/site-packages/pyspark/serializers.py”, line 600, in dumps
raise pickle.PicklingError(msg)
_pickle.PicklingError: Could not serialize object: TypeError: can’t pickle _abc_data objects

Seems that you are having the same problem like in this issue:
https://github.com/cloudpipe/cloudpickle/issues/180
What is happening is that pyspark's cloudpickle library is outdated for python 3.7, you should fix the problem with this crafted patch by now until pyspark gets that module updated.
Try using this workaround:
Install cloudpickle pip install cloudpickle
Add this to your code:
import cloudpickle
import pyspark.serializers
pyspark.serializers.cloudpickle = cloudpickle
monkeypatch credit https://github.com/cloudpipe/cloudpickle/issues/305

Python, how to handle the "ValueError: unsupported pickle protocol: 4" error?

I'm new to Python.
I've to run this TargetFinder script ("Custom Analyses").
I installed all the required python packages, and copied the code into a script I named main.py, and ran it.
I got this error:
[davide#laptop]$ python main.py
Traceback (most recent call last):
File "main.py", line 8, in <module>
training_df = pd.read_hdf('./paper/targetfinder/K562/output-epw/training.h5', 'training').set_index(['enhancer_name', 'promoter_name'])
File "/usr/lib64/python2.7/site-packages/pandas/io/pytables.py", line 330, in read_hdf
return store.select(key, auto_close=auto_close, **kwargs)
File "/usr/lib64/python2.7/site-packages/pandas/io/pytables.py", line 680, in select
return it.get_result()
File "/usr/lib64/python2.7/site-packages/pandas/io/pytables.py", line 1364, in get_result
results = self.func(self.start, self.stop, where)
File "/usr/lib64/python2.7/site-packages/pandas/io/pytables.py", line 673, in func
columns=columns, **kwargs)
File "/usr/lib64/python2.7/site-packages/pandas/io/pytables.py", line 2786, in read
values = self.read_array('block%d_values' % i)
File "/usr/lib64/python2.7/site-packages/pandas/io/pytables.py", line 2327, in read_array
data = node[:]
File "/usr/lib64/python2.7/site-packages/tables/vlarray.py", line 677, in __getitem__
return self.read(start, stop, step)
File "/usr/lib64/python2.7/site-packages/tables/vlarray.py", line 817, in read
outlistarr = [atom.fromarray(arr) for arr in listarr]
File "/usr/lib64/python2.7/site-packages/tables/atom.py", line 1211, in fromarray
return cPickle.loads(array.tostring())
ValueError: unsupported pickle protocol: 4
I've no idea about what this pickle protocol means, and also my colleagues know nothing about it.
How can I solve this problem?
I'm using Python 2.7.5 on a CentOS Linux release 7.2.1511 (Core) operating system

The Pickle protocol is basically the file format. From the documentation,
The higher the protocol used, the more recent the version of Python needed to read the pickle produced. ... Pickle protocol version 4 was added in Python 3.4, your python version (2.7.5) does not support this.
Either upgrade to Python 3.4 or later (current is 3.5) or create the pickle using a lower protocol (2) in the third parameter to pickle.dump().

This sometimes happens due to incorrect data in the redis database. Try it:
sudo redis-cli flushall

It's a python version issue, upgrade it to the latest python version and try.

tkinter: How to serialize a treeview?

I am working on a python tkinter application that reads initial data from yaml file into a hierarchical TreeView to be edited further by the user.
To implement "save data" and "undo" functions, should I walk the treeview and reconstruct the data into a python object to be serialized (pickle)?
Or is there a python module allowing, for example, to specify the treeview and the output file to be saved on?

I doubt there's any Python module that does what you want, and even if there was, I don't think you'd want to structure your application around using it. Instead you would probably be better off decoupling things and storing the primary data in something independent of the human interface (which may or may not be graphical and might vary or otherwise be changed in the future). This is sometimes called the application "Model".
Doing so will allow you to load and save it regardless of what constitutes the current human interface. So, for example, you would then be free to use pickle if the internal Model is comprised of one or more Python objects. Alternatively you could save the data back into a yaml format file which would make loading it back in again later a cinch since the program can already do that.
Likewise, as the user edits the TreeView, equivalent changes should be made to the Model to keep the two in sync.
Take some time out from coding and familiarize yourself with the Model–View–Controller (MVC) design pattern.

Out of the box, the answer is no, you can't serialize a TreeView. dill is probably your best bet at serialization out of the box… and it fails to pickle a TreeView object.
>>> import ttk
>>> import Tkinter as tk
>>>
>>> f = tk.Frame()
>>> t = ttk.Treeview(f)
>>>
>>> import dill
>>> dill.pickles(t)
False
>>> dill.detect.errors(t)
PicklingError("Can't pickle 'tkapp' object: <tkapp object at 0x10eda75e0>",)
>>>
You might be able to figure out how to pickle a TreeView, and then add that method to the pickle registry… but, that could take some serious work on your part to chase down how things fail to pickle.
You can see what happens, it hits the __dict__ of the Tkinter.Tk object, and dies trying to pickle something.
>>> dill.detect.trace(True)
>>> dill.dumps(t)
C2: ttk.Treeview
D2: <dict object at 0x1147f5168>
C2: Tkinter.Frame
D2: <dict object at 0x1147f1050>
C2: Tkinter.Tk
D2: <dict object at 0x1148035c8>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mmckerns/lib/python2.7/site-packages/dill-0.2.3.dev0-py2.7.egg/dill/dill.py", line 194, in dumps
dump(obj, file, protocol, byref, fmode)#, strictio)
File "/Users/mmckerns/lib/python2.7/site-packages/dill-0.2.3.dev0-py2.7.egg/dill/dill.py", line 184, in dump
pik.dump(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 725, in save_inst
save(stuff)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/mmckerns/lib/python2.7/site-packages/dill-0.2.3.dev0-py2.7.egg/dill/dill.py", line 678, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 725, in save_inst
save(stuff)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/mmckerns/lib/python2.7/site-packages/dill-0.2.3.dev0-py2.7.egg/dill/dill.py", line 678, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 725, in save_inst
save(stuff)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/mmckerns/lib/python2.7/site-packages/dill-0.2.3.dev0-py2.7.egg/dill/dill.py", line 678, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 313, in save
(t.__name__, obj))
pickle.PicklingError: Can't pickle 'tkapp' object: <tkapp object at 0x10eda7648>
>>>
That something is a tkapp object.
So, if you'd like to dig further, you can use the methods in dill.detect to help you uncover exactly why it's not pickling… and try to get around it.
I'm doubtful that pickling a widget is the right way to go. You probably also don't want to go the route of pulling the state out of a treeview into a shadow class, and saving that class. The problem is that the treeview is not really built with a good separation in mind for saving state.
If you can redesign to cleanly separate the state of your application from the widgets themselves, then that's more likely to do what you want. So, when you ask How to serialize a treeview, this is really not what you are asking. You want to know how to save the state of your application.
There are packages that can do something like that very easily. I'd suggest you looking at enaml and/or traits. enaml is a declarative markup that asks you to build a class that describes how your application interface works. It forces you to separate the inner workings of the thing your are displaying from the code that's necessary top operate the user interface… and it does it in a very easy to build way -- where the state of the application is separate from the user interface wiring. Thus, an instance of the class you build contains the state of the application at any time -- regardless of whether it has a UI on it or not, or two or three UIs for that matter. It makes saving the state of the application very easy because you never have to
worry about saving the state of the UI -- the UI has no state -- it's just layout painted on top of the application. Then you won't have to worry about pickling widgets…
Check out enaml here: https://github.com/nucleic/enaml
and traits here: http://docs.enthought.com/traits

Another Q&A shows how to pickle a treeview on exit and reload it on startup:
How to save information from the program then use it to show in program again (simple programming)
The OP has information laid out thusly:
#----------TreeViewlist----------------------
Header =['Website','Username','Password','etc']
The gist of the treeview is a record of each website the OP visits, what user ID is used and the password used.
To summarize the accepted answer:
Save treeview to pickle on exit
x=[tree.item(x)['values'] for x in tree.get_children()]
filehandler = open('data.pickle', 'wb')
pickle.dump(x,filehandler)
filehandler.close()
Load pickle to build treeview on startup
items = []
try:
filehandler = open('data.pickle', 'rb')
items = pickle.load(filehandler)
filehandler.close()
except:
pass
for item in items:
tree.insert('','end',values=item)
The answer appears straight forward (to me) but if you have any questions post a comment below. If you see a flaw or bug in the code post a comment in the link above.