We Keep Coding

EOFError: Ran out of input, using torch.load()

I saw this error being posted a lot and often it was due to the file not being closed properly after opening. But since I'm using the integrated torch.load() function, I'm not sure what I could do different.
First the saving part:
torch.save({
'model_state_dict': agent.dqn.state_dict(),
...
'loss_history': agent.losshistory
}, modelpath)
and here the loading part, where I also get the error message:
if os.path.exists(modelpath):
checkpoint = torch.load(modelpath)
agent.dqn.load_state_dict(checkpoint['model_state_dict'])
...
agent.losshistory = checkpoint['loss_history']
and here the error:
Traceback (most recent call last):
File "c:/Users/levin/Desktop/programming/main.py", line 33, in <module>
checkpoint = torch.load(modelpath)
File "C:\Users\levin\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 529, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "C:\Users\levin\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 702, in _legacy_load
result = unpickler.load()
EOFError: Ran out of input
One more thing I want to mention is that I used this exact code several times without a problem. I can't remember changing anything that could have caused the error.

According to this thread it seems to raise an exception when reading an empty file, so please check the size of the document before reading it and post a response if it is not solved.

Getting Dill Memory Error when loading serialized object, how to fix?

I am getting a dill/pickle memory error when loading a serialized object file. I am not quite sure what is happening and I am unsure on how to fix it.
When I call:
stat_bundle = train_batch_iterator(clf, TOTAL_TRAINED_EVENTS)
The code traces to the train_batch_iterator function in which it loads a serialized object and trains the classifier with the data within the object. This is the code:
def train_batch_iterator(clf, tte):
plot_data = [] # initialize plot data array
for file in glob.glob('./SerializedData/Batch8172015_19999/*'):
with open(file, 'rb') as stream:
minibatch_train = dill.load(stream)
clf.partial_fit(minibatch_train.data[1], minibatch_train.target,
classes=np.array([11, 111]))
tte += len(minibatch_train.target)
plot_data.append((test_batch_iterator(clf), tte))
return plot_data
Here is the error:
Traceback (most recent call last):
File "LArSoftSGD-version2.0.py", line 154, in <module>
stat_bundle = train_batch_iterator(clf, TOTAL_TRAINED_EVENTS)
File "LArSoftSGD-version2.0.py", line 118, in train_batch_iterator
minibatch_train = dill.load(stream)
File "/home/jdoe/.local/lib/python3.4/site-packages/dill/dill.py", line 199, in load
obj = pik.load()
File "/home/jdoe/.local/lib/python3.4/pickle.py", line 1038, in load
dispatch[key[0]](self)
File "/home/jdoe/.local/lib/python3.4/pickle.py", line 1184, in load_binbytes
self.append(self.read(len))
File "/home/jdoe/.local/lib/python3.4/pickle.py", line 237, in read
return self.file_read(n)
MemoryError
I have no idea what could be going wrong. The error seems to be in the line minibatch_train = dill.load(stream) and the only thing I can think of is that the serialized data file is too large, however the file is exactly 1161 MB which doesn't seem to big/big enough to cause a memory error.
Does anybody know what might be going wrong?

Cannot determine type of file

Hi i have just started learning image processing using python.
When i tried to open an image that i downloaded from the net, I keep getting this error and I have no idea about how to resolve it. Can anyone please help me with this?
>>> dna=mahotas.imread('dna.jpeg')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\mahotas\io\freeimage.py", line 773, in imread
img = read(filename)
File "C:\Python27\lib\site-packages\mahotas\io\freeimage.py", line 444, in read
bitmap = _read_bitmap(filename, flags)
File "C:\Python27\lib\site-packages\mahotas\io\freeimage.py", line 490, in _read_bitmap
'mahotas.freeimage: cannot determine type of file %s' % filename)
ValueError: mahotas.freeimage: cannot determine type of file dna.jpeg

Hello this looks like a pretty old thread but I found it recently because I had the same problem.
I think that the error message is misleading because it implies that the type of file is incorrect.
I fixed the problem by including the full path to the image file. For example, it could look something like:
dna = mahotas.imread('C:\Documents\dna.jpeg')

EOFError whilst pickling in Python

Here's the problem - I'm trying to pickle, and then unpickle hiscores. When I use pickle.load, Python seems to think that I'm trying to load a file called 'Pickle' that I have. Here's the code:
def recieve_hiscores():
hiscores_file = open("hiscores_file.dat", "rb")
for i in hiscores_file:
hiscores = pickle.load(hiscores_file)
hiscores = str(hiscores)
print(hiscores)
Here's the pickling code:
def send_hiscores(score):
hiscores_file = open("hiscores_file.dat", "ab")
pickle.dump(score, hiscores_file)
hiscores_file.close()
And here's the error message:
Traceback (most recent call last):
File "C:\Python31\My Updated Trivia Challenge.py", line 106, in <module>
main()
File "C:\Python31\My Updated Trivia Challenge.py", line 104, in main
recieve_hiscores()
File "C:\Python31\My Updated Trivia Challenge.py", line 56, in recieve_hiscores
hiscores = pickle.load(hiscores_file)
File "C:\Python31\lib\pickle.py", line 1365, in load
encoding=encoding, errors=errors).load()
EOFError
Don't worry if there's any other mistakes, I'm still learning, but I can't work this out.

When you iterate over the file, you get newline separated lines. This is NOT how you get a series of pickles. The end of file error is raised because the first line has a partial pickle.
Try this:
def recieve_hiscores():
highscores = []
with open("hiscores_file.dat", "rb") as hiscores_file:
try:
while True:
hiscore = pickle.load(hiscores_file)
hiscore = str(hiscore)
print(hiscore)
highscores.append(hiscore)
except EOFError:
pass
return highscores

EOFError in Python script

I have the following code fragment:
def database(self):
databasename=""
host=""
user=""
password=""
try:
self.fp=file("detailing.dat","rb")
except IOError:
self.fp=file("detailing.dat","wb")
pickle.dump([databasename,host,user,password],self.fp,-1)
self.fp.close()
selffp=file("detailing.dat","rb")
[databasename,host,user,password]=pickle.load(self.fp)
return
It has the error:
Traceback (most recent call last):
File "detailing.py", line 91, in ?
app=myApp()
File "detailing.py", line 20, in __init__
wx.App.__init__(self,redirect,filename,useBestVisual,clearSigInt)
File "/usr/lib64/python2.4/site-packages/wx-2.6-gtk2-unicode/wx/_core.py", line 7473, in __init__
self._BootstrapApp()
File "/usr/lib64/python2.4/site-packages/wx-2.6-gtk2-unicode/wx/_core.py", line 7125, in _BootstrapApp
return _core_.PyApp__BootstrapApp(*args, **kwargs)
File "detailing.py", line 33, in OnInit
self.database()
File "detailing.py", line 87, in database
[databasename,host,user,password]=pickle.load(self.fp)
File "/usr/lib64/python2.4/pickle.py", line 1390, in load
return Unpickler(file).load()
File "/usr/lib64/python2.4/pickle.py", line 872, in load
dispatch[key](self)
File "/usr/lib64/python2.4/pickle.py", line 894, in load_eof
raise EOFError
EOFError
What am I doing wrong?

Unless you've got a typo, the issue may be in this line where you assign the file handle to selffp not self.fp:
selffp=file("detailing.dat","rb")
If that is a typo, and your code actually opens the file to self.fp, then you may wish to verify that the file actually has contents (ie: that the previous pickle worked)... the error suggests that the file is empty.
Edit: In the comments to this answer, S. Lott has a nice summary of why the typo generated the error you saw that I'm pasting here for completeness of the answer: "selffp will be the unused opened file, and self.fp (the old closed file) will be used for the load".

Here's the version that I would recommend using:
def database(self):
databasename=""
host=""
user=""
password=""
try:
self.fp=open("detailing.dat","rb")
except IOError:
with open("detailing.dat", "wb") as fp:
pickle.dump([databasename,host,user,password],fp,-1)
self.fp=open("detailing.dat","rb")
[databasename,host,user,password]=pickle.load(self.fp)
return
As has been pointed out, there was a typo on self.fp. But here are a few other things that I notice that can cause problems.
First of all, you shouldn't be using the file constructor directly. You should instead use the built-in open function.
Secondly, you should avoid calling a file's close method outside a finally block. In this case, I've used python 2.6's with block. You can use this in Python 2.5 with the following command:
from __future__ import with_statement
This will prevent the file from being stuck open if an exception is thrown anywhere (as it will close the file when the with block is exited). Although this isn't the cause of your problem, it is an important thing to remember because if one of the file object's methods throws an exception, the file will get held open in sys.traceback indefinitely.
(note that you should probably accept Jarret Hardie's answer though, he caught the bug :-) )

I got this error when I didn't chose the correct mode to read the file (wb instead of rb). Changing back to rb was not sufficient to solve the issue. However, generating again a new clean pickle file solved the issue. It seems that not choosing the correct mode to open the binary file somehow "damages" the file which is then not openable whatsoever afterward.
But I am quite a beginner with Python so I may have miss something too.

While this is not a direct answer to the OP's question -- I happened upon this answer while searching for a reason for an EOFError when trying to unpickle a binary file with : pickle.load(open(filename, "r")).
import cPickle as pickle
A = dict((v, i) for i, v in enumerate(words))
with open("words.pkl", "wb") as f:
pickle.dump(A, f)
#...later open the file -- mistake:trying to read a binary with non-binary method
with open("words.pkl", "r") as f:
A =pickle.load(f) # EOFError
# change that to
with open ("words.pkl", "rb") as f: # notice the "rb" instead of "r"
A = pickle.load(f)