good afternoon!
saving list(dict(),dict(),dict()) struct with pickle module, but when reading I get: <class 'function'>, and <function lesson at 0x00000278BA3A0D30>
what am I doing wrong?
def lesson(user, date):
with open(user+"_"+date+".data", 'wb') as file:
pickle.dump(lesson, file)
file.close()
def read(user, date):
with open(user+"_"+date+".data", 'rb') as file:
lesson = pickle.load(file)
file.close()
return(lesson)
I am using python 3.10.7
"saving list(dict(),dict(),dict()) struct with pickle module". No, you're not. You're saving the lesson function. See line 3 of your code.
Related
I created some data and stored it several times like this:
with open('filename', 'a') as f:
pickle.dump(data, f)
Every time the size of file increased, but when I open file
with open('filename', 'rb') as f:
x = pickle.load(f)
I can see only data from the last time.
How can I correctly read file?
Pickle serializes a single object at a time, and reads back a single object -
the pickled data is recorded in sequence on the file.
If you simply do pickle.load you should be reading the first object serialized into the file (not the last one as you've written).
After unserializing the first object, the file-pointer is at the beggining
of the next object - if you simply call pickle.load again, it will read that next object - do that until the end of the file.
objects = []
with (open("myfile", "rb")) as openfile:
while True:
try:
objects.append(pickle.load(openfile))
except EOFError:
break
There is a read_pickle function as part of pandas 0.22+
import pandas as pd
obj = pd.read_pickle(r'filepath')
The following is an example of how you might write and read a pickle file. Note that if you keep appending pickle data to the file, you will need to continue reading from the file until you find what you want or an exception is generated by reaching the end of the file. That is what the last function does.
import os
import pickle
PICKLE_FILE = 'pickle.dat'
def main():
# append data to the pickle file
add_to_pickle(PICKLE_FILE, 123)
add_to_pickle(PICKLE_FILE, 'Hello')
add_to_pickle(PICKLE_FILE, None)
add_to_pickle(PICKLE_FILE, b'World')
add_to_pickle(PICKLE_FILE, 456.789)
# load & show all stored objects
for item in read_from_pickle(PICKLE_FILE):
print(repr(item))
os.remove(PICKLE_FILE)
def add_to_pickle(path, item):
with open(path, 'ab') as file:
pickle.dump(item, file, pickle.HIGHEST_PROTOCOL)
def read_from_pickle(path):
with open(path, 'rb') as file:
try:
while True:
yield pickle.load(file)
except EOFError:
pass
if __name__ == '__main__':
main()
I developed a software tool that opens (most) Pickle files directly in your browser (nothing is transferred so it's 100% private):
https://pickleviewer.com/ (formerly)
Now it's hosted here: https://fire-6dcaa-273213.web.app/
Edit: Available here if you want to host it somewhere: https://github.com/ch-hristov/Pickle-viewer
Feel free to host this somewhere.
I'm using python3.3 and I'm having a cryptic error when trying to pickle a simple dictionary.
Here is the code:
import os
import pickle
from pickle import *
os.chdir('c:/Python26/progfiles/')
def storvars(vdict):
f = open('varstor.txt','w')
pickle.dump(vdict,f,)
f.close()
return
mydict = {'name':'john','gender':'male','age':'45'}
storvars(mydict)
and I get:
Traceback (most recent call last):
File "C:/Python26/test18.py", line 31, in <module>
storvars(mydict)
File "C:/Python26/test18.py", line 14, in storvars
pickle.dump(vdict,f,)
TypeError: must be str, not bytes
The output file needs to be opened in binary mode:
f = open('varstor.txt','w')
needs to be:
f = open('varstor.txt','wb')
Just had same issue. In Python 3, Binary modes 'wb', 'rb' must be specified whereas in Python 2x, they are not needed. When you follow tutorials that are based on Python 2x, that's why you are here.
import pickle
class MyUser(object):
def __init__(self,name):
self.name = name
user = MyUser('Peter')
print("Before serialization: ")
print(user.name)
print("------------")
serialized = pickle.dumps(user)
filename = 'serialized.native'
with open(filename,'wb') as file_object:
file_object.write(serialized)
with open(filename,'rb') as file_object:
raw_data = file_object.read()
deserialized = pickle.loads(raw_data)
print("Loading from serialized file: ")
user2 = deserialized
print(user2.name)
print("------------")
pickle uses a binary protocol, hence only accepts binary files. As the document said in the first sentence, "The pickle module implements binary protocols for serializing and de-serializing".
I am facing this error:
File "C:\Python27\lib\site-packages\gensim\utils.py", line 1334, in unpickle
return _pickle.load(f, encoding='latin1')
UnpicklingError: could not find MARK
while my utils.py code is:
with smart_open(fname, 'rb') as f:
f.seek(0)
# Because of loading from S3 load can't be used (missing readline in smart_open)
if sys.version_info > (3, 0):
return _pickle.load(f, encoding='latin1')
else:
return _pickle.loads(f.read())
def pickle(obj, fname, protocol=2):
"""Pickle object `obj` to file `fname`.
Parameters
----------
obj : object
Any python object.
fname : str
Path to pickle file.
protocol : int, optional
Pickle protocol number, default is 2 to support compatible across python 2.x and 3.x.
"""
with smart_open(fname, 'wb') as fout: # 'b' for binary, needed on Windows
_pickle.dump(obj, fout, protocol=protocol)
Anyone please help me I suffering on it few days.....
You're probably trying to load a model that was trained and saved in Python 3, but you're using Python 2. See
https://github.com/RaRe-Technologies/gensim/issues/853
I am having trouble trying to save a dictionary using pickle in python:
my code is the following:
import re
import os
def save_obj(outputFolder,obj, name ):
directory = outputFolder + ' obj/'
if not os.path.exists(directory):
os.makedirs(directory)
with open(directory + name + '.pkl', 'wb') as f:
pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)
The Error message is the following:
File "CleanPEPS.py", line 14, in save_obj
pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)
NameError: name 'pickle' is not defined
It is a problem of library or something like that? Or there is something that I have forgotten?
You're missing import pickle. Other than that your code looks OK. If you have any more issues check out the pickle docs.
when trying to apply some code i found on the internet i ran into a dataset that was pickled. Now to insert my own dataset into that i need to reverse the pickling myself. The piece of code that reads the pickle is:
import cPickle, gzip, numpy
# Load the dataset
f = gzip.open('mnist.pkl.gz', 'rb')
train_set, valid_set, test_set = cPickle.load(f)
f.close()
And i want to write the pickle myself now:
with open(outfile) as f:
train_set = allfiles[:len(allfiles)/3]
valid_set = allfiles[len(allfiles)/3:(len(allfiles)/3)*2]
test_set = allfiles[(len(allfiles)/3)*2:]
cPickle.dump((train_set,valid_set,test_set), outfile,0)
However i get :
TypeError: argument must have 'write' attribute
What could be my problem? How would a good pickling code look like?
You want to use the file object, not the filename:
cPickle.dump((train_set,valid_set,test_set), f, 0)
However, your input was GZIP-compressed as well:
with gzip.open(outfile, 'wb') as f:
# ...
cPickle.dump((train_set,valid_set,test_set), f, 0)
Note that your own code forgot to state the correct mode for the opened file object as well; open(outfile) without arguments opens the file in read-modus, and writes would fail with an IOError: File not open for writing exception.
cPickle.dump((train_set,valid_set,test_set), outfile,0)
outfile is just a file name. You should use:
cPickle.dump((train_set,valid_set,test_set), f,0)