I'm currently trying to mock out the open() built in method in Python for a test. However, I always end up getting a crash and this resulting message:
File "/opt/home/venv/lib/python2.7/site-packages/nose-1.3.0-py2.7.egg/nose/result.py", line 187, in _exc_info_to_string
return _TextTestResult._exc_info_to_string(self, err, test)
File "/opt/python-2.7.3/lib/python2.7/unittest/result.py", line 164, in _exc_info_to_string
msgLines = traceback.format_exception(exctype, value, tb)
File "/opt/python-2.7.3/lib/python2.7/traceback.py", line 141, in format_exception
list = list + format_tb(tb, limit)
File "/opt/python-2.7.3/lib/python2.7/traceback.py", line 76, in format_tb
return format_list(extract_tb(tb, limit))
File "/opt/python-2.7.3/lib/python2.7/traceback.py", line 101, in extract_tb
line = linecache.getline(filename, lineno, f.f_globals)
File "/opt/home/venv/lib/python2.7/linecache.py", line 14, in getline
lines = getlines(filename, module_globals)
File "/opt/home/venv/lib/python2.7/linecache.py", line 40, in getlines
return updatecache(filename, module_globals)
File "/opt/home/venv/lib/python2.7/linecache.py", line 127, in updatecache
with open(fullname, 'rU') as fp:
AttributeError: __exit__
Here is my test code:
m = mox.Mox()
m.StubOutWithMock(__builtin__, 'open')
mock_file = m.CreateMock(__builtin__.file)
open(mox.IgnoreArg(), mox.IgnoreArg()).AndReturn(mock_file)
mock_file.write(mox.IgnoreArg()).MultipleTimes()
mock_file.close()
write_file_method()
__exit__ is the method that gets called when you try to close a file. Your mock file does not handle mock_file.close(), just open(). You'll need to mock the close method too.
Edit:
On second though, why do you want to mock open? AFAIK you shouldn't be doing that method. The method under test should take an open stream (instead of a filename, for instance). In production code, clients are responsible for opening a file (e.g. pickle.dump). In your tests, you pass in a StringIO, or a mock object that supports writing.
Edit 2:
I would split your method in two and test each bit separately.
creating a file: check that prior to calling this method the file does not exist, and it does after that. One might argue such a one-line method isn't worth testing.
writing to a file: see above. Create a StringIO and write to that, so your tests can then verify the correct thing has been written.
Related
I have a very large project of a Web API using Flask and Python. It is used for testing some electronic hardware automatically.
The program uses some threading in order to run a web UI while a server runs some services (SSH, serial, VISA) among others.
The program was originally coded in python 2.7 and works just fine with this version. Right now, I am trying to update it to python 3.8 for obvious reasons.
As I am updating the project, I'm having trouble with the copy library. It is supposed to serialize a _thread.RLock object and to send it to another thread, but it keeps giving me an error. Here is the traceback that I get:
Traceback (most recent call last):
File "c:\git_files\[...]\nute\route_config\flask_api_testbench.py", line 208, in _hook_run
super(FlaskAPITestbench, self).hook_run()
File "c:\git_files\[...]\nute\core\testbench\base.py", line 291, in hook_run
while self.state_machine():
File "c:\git_files\[...]\nute\core\testbench\base.py", line 304, in state_machine
on_input=self.state_testrun
File "c:\git_files\[...]\nute\core\testbench\base.py", line 380, in wait_for_input_or_testrun
self.hook_load_testrun(config_with_input)
File "c:\git_files\[...]\nute\core\testbench\base.py", line 428, in hook_load_testrun
self.interface.load_testrun(self.load_testrun(config))
File "c:\git_files\[...]\nute\core\testbench\base.py", line 461, in load_testrun
testrun = self.test_loader.load_testrun(config, context_type=self.TestRunContext)
File "c:\git_files\[...]\nute\core\testrun\loader.py", line 89, in load_testrun
testrun_template = process_all_loaders(self.batchers, _process_batcher)
File "c:\git_files\[...]\nute\core\config\loader.py", line 127, in process_all_loaders
return fn(loader)
File "c:\git_files\[...]\nute\core\testrun\loader.py", line 85, in _process_batcher
batcher.batch_testrun(testrun_template, config, context)
File "c:\git_files\[...]\nute\batcher\python_module_batcher.py", line 21, in batch_testrun
batch_module.main(testrun, context)
File "C:\GIT_Files\[...]\pyscripts\script\patest\_batch.py", line 168, in main
test.suite(ImpedanceTest)
File "c:\git_files\[...]\nute\core\testrun\base.py", line 213, in suite
testsuite = testsuite_instance_or_class()
File "c:\git_files\[...]\nute\core\functions\helpers.py", line 233, in __new__
cls._attach_nodes_to(template)
File "c:\git_files\[...]\nute\core\functions\helpers.py", line 271, in _attach_nodes_to
node = root.import_testcase(testcase)
File "c:\git_files\[...]\nute\core\functions\specific.py", line 307, in import_testcase
test_node = testcase.copy(cls=self.__class__)
File "c:\git_files\[...]\nute\core\functions\base.py", line 645, in copy
value = copy(value)
File "c:\users\[...]\.conda\envs\py37\lib\copy.py", line 96, in copy
rv = reductor(4)
TypeError: can't pickle _thread.RLock objects
It works fine in Python 2.7, but not with Python 3.x. I've tried it on 3.7.10, 3.8.9 and 3.9.6 with the same result.
Here's the implementation of my wrap method of copy:
from copy import copy
...
def copy(self, cls=None): # class method
if cls is None:
cls = self.__class__
new_self = cls()
for key, value in self.__dict__.items():
# if key == "multithread_lock":
# continue
if self.should_copy_attribute(key, value):
# Handle recursion by pointing to the new object instead of copying.
if value is self:
value = new_self
else:
value = copy(value) # This is where it fails
new_self.__dict__[key] = value
return new_self
As you can see with the commented part, skipping the pickling of any _thread.RLock object makes the program work, but I need to refresh the web UI manually to see it running since the thread doesn't work.
Any idea why it's working on python 2.7 but not on newer versions? Thanks in advance.
So I found out that a _thread.RLock() object cannot be copied. I just added a condition to skip an object like this to be copied, and it works fine.
For the web UI not refreshing, I changed to a lower version of Flask-SocketIO and it worked just fine.
I have a script that is constantly updating a data-frame and saving it to disk (overwriting the old csv-file). I found out that if interrupt the program right at the saving call, df.to_csv("df.csv"), all data is losed, and the df.csv is empty only containing the column-index.
I can perhaps do a workaround by temporarily saving the data to df.temp.csv, and then replacing df.csv. But is there a pythonic, short way to make the saving "Atomary" and prevent data-loss? This is the stack trace I get when interrupting right at the saving call.
Traceback (most recent call last):
File "/opt/homebrew-cask/Caskroom/pycharm/2016.1.3/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1531, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "/opt/homebrew-cask/Caskroom/pycharm/2016.1.3/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 938, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Users/user/test.py", line 49, in <module>
d.to_csv("out.csv", index=False)
File "/usr/local/lib/python2.7/site-packages/pandas/core/frame.py", line 1344, in to_csv
formatter.save()
File "/usr/local/lib/python2.7/site-packages/pandas/formats/format.py", line 1551, in save
self._save()
File "/usr/local/lib/python2.7/site-packages/pandas/formats/format.py", line 1652, in _save
self._save_chunk(start_i, end_i)
File "/usr/local/lib/python2.7/site-packages/pandas/formats/format.py", line 1666, in _save_chunk
quoting=self.quoting)
File "/usr/local/lib/python2.7/site-packages/pandas/core/internals.py", line 1443, in to_native_types
return formatter.get_result_as_array()
File "/usr/local/lib/python2.7/site-packages/pandas/formats/format.py", line 2171, in get_result_as_array
formatted_values = format_values_with(float_format)
File "/usr/local/lib/python2.7/site-packages/pandas/formats/format.py", line 2157, in format_values_with
for val in values.ravel()[imask]])
File "/usr/local/lib/python2.7/site-packages/pandas/formats/format.py", line 2108, in base_formatter
return str(v) if notnull(v) else self.na_rep
File "/usr/local/lib/python2.7/site-packages/pandas/core/common.py", line 250, in notnull
res = isnull(obj)
File "/usr/local/lib/python2.7/site-packages/pandas/core/common.py", line 73, in isnull
def isnull(obj):
File "_pydevd_bundle/pydevd_cython.pyx", line 937, in _pydevd_bundle.pydevd_cython.ThreadTracer.__call__ (_pydevd_bundle/pydevd_cython.c:15522)
File "/opt/homebrew-cask/Caskroom/pycharm/2016.1.3/PyCharm.app/Contents/helpers/pydev/_pydev_bundle/pydev_is_thread_alive.py", line 14, in is_thread_alive
def is_thread_alive(t):
KeyboardInterrupt
You can create a context manager to handle your atomic overwriting:
import os
import contextlib
#contextlib.contextmanager
def atomic_overwrite(filename):
temp = filename + '~'
with open(temp, "w") as f:
yield f
os.rename(temp, filename) # this will only happen if no exception was raised
The to_csv method on a Pandas DataFrame will accept a file object instead of a path, so you can use:
with atomic_overwrite("df.csv") as f:
df.to_csv(f)
The temporary filename I chose is the requested filename with a tilde at the end. You can of course change the code to use something else if you want. I'm also not exactly sure what mode the file should be opened with, you may need "wb" instead of just "w".
The best you can do is to implement a signal handler (signal module) which waits with terminating the program until the last write operation has finished.
Something along the lines (pseudo-code):
import signal
import sys
import time
import pandas as pd
lock = threading.Lock()
def handler(signum, frame):
# ensure that latest data is written
sys.exit(1)
signal.signal(signal.SIGTERM, handler)
signal.signal(signal.SIGINT, handler)
while True:
# might exit any time.
pd.to_csv(...)
time.sleep(1)
I'm working on a Python-programm that use the modul "pptx" (edit and manipulate powerpoint). The programm works without problem in the interpretor, but once I start it as a .exe file (built with cx_freeze or py2exe) I have this error-message when I click on the main-button :
Exception in Tkinter callback
Traceback (most recent call last):
File "Tkinter.pyc", line 1470, in __call__
File "pptx_02.py", line 187, in ButtonErstellen
File "pptx\api.pyc", line 29, in __init__
File "pptx\presentation.pyc", line 87, in __init__
File "pptx\presentation.pyc", line 170, in __open
File "pptx\packaging.pyc", line 88, in open
File "pptx\packaging.pyc", line 671, in __new__
PackageNotFoundError: Package not found at 'C:\Users\Moi\Programmation\Python\build\exe.win32-2.7\library.zip\pptx\templates\default.pptx'
The problem is that the file "default.pptx" actually exists at this adress.
While looking in the functions of "pptx", I may had an idea about the reason of this problem (but no idea of the solution).
For the last error : File "pptx\packaging.pyc", line 671, in new
is the following code:
class FileSystem(object):
"""
Factory for filesystem interface instances.
A FileSystem object provides access to on-disk package items via their URI
(e.g. ``/_rels/.rels`` or ``/ppt/presentation.xml``). This allows parts to
be accessed directly by part name, which for a part is identical to its
item URI. The complexities of translating URIs into file paths or zip item
names, and file and zip file access specifics are all hidden by the
filesystem class. |FileSystem| acts as the Factory, returning the
appropriate concrete filesystem class depending on what it finds at *path*.
"""
def __new__(cls, file):
# if *file* is a string, treat it as a path
if isinstance(file, basestring):
path = file
if is_zipfile(path):
fs = ZipFileSystem(path)
elif os.path.isdir(path):
fs = DirectoryFileSystem(path)
else:
raise PackageNotFoundError("Package not found at '%s'" % path)
else:
fs = ZipFileSystem(file)
return fs
The problem may be that the final file (default.pptx) isnot a zip but is in a zip. But I have no idea how I could fix it or if it realy is the problem.
If somebody as an idea ...
Thank you !
I am writing a small script that needs to merge many one-page pdf files. I want the script to run with Python3 and to have as few dependencies as possible.
For the PDF merging part, I tried using PyPdf. However, the Python 3 support seems to be buggy; It can't handle inkscape generated PDF files (which I need). I have the current git version of PyPdf installed, and the following test script doesn't work:
import PyPDF2
output_pdf = PyPDF2.PdfFileWriter()
with open("testI.pdf", "rb") as input:
input_pdf = PyPDF2.PdfFileReader(input)
output_pdf.addPage(input_pdf.getPage(0))
with open("test.pdf", "wb") as output:
output_pdf.write(output)
It throws the following stack trace:
Traceback (most recent call last):
File "test.py", line 7, in <module>
output.addPage(input.getPage(0))
File "/usr/lib/python3.3/site-packages/pyPdf/pdf.py", line 420, in getPage
self._flatten()
File "/usr/lib/python3.3/site-packages/pyPdf/pdf.py", line 574, in _flatten
self._flatten(page.getObject(), inherit)
File "/usr/lib/python3.3/site-packages/pyPdf/generic.py", line 165, in getObject
return self.pdf.getObject(self).getObject()
File "/usr/lib/python3.3/site-packages/pyPdf/pdf.py", line 616, in getObject
retval = readObject(self.stream, self)
File "/usr/lib/python3.3/site-packages/pyPdf/generic.py", line 66, in readObject
return DictionaryObject.readFromStream(stream, pdf)
File "/usr/lib/python3.3/site-packages/pyPdf/generic.py", line 526, in readFromStream
value = readObject(stream, pdf)
File "/usr/lib/python3.3/site-packages/pyPdf/generic.py", line 57, in readObject
return ArrayObject.readFromStream(stream, pdf)
File "/usr/lib/python3.3/site-packages/pyPdf/generic.py", line 152, in readFromStream
obj = readObject(stream, pdf)
File "/usr/lib/python3.3/site-packages/pyPdf/generic.py", line 86, in readObject
return NumberObject.readFromStream(stream)
File "/usr/lib/python3.3/site-packages/pyPdf/generic.py", line 231, in readFromStream
return FloatObject(name.decode("ascii"))
File "/usr/lib/python3.3/site-packages/pyPdf/generic.py", line 207, in __new__
return decimal.Decimal.__new__(cls, str(value), context)
TypeError: optional argument must be a context
The same script, however, works flawlessly with Python 2.7.
What am I doing wrong here? Is it a bug in the library? Can I work around it without touching the PyPDF library?
So I found the answer. The decimal.Decimal module in Python3.3 shows some weird behaviour. This is the corresponding StackOverflow question: Instantiate Decimal class I added some workaround to the PyPDF2 library and submitted a pull request.
Just to make sure you are aware of already existing tools that do exactly this:
PDFtk
PDFjam (my favourite, requires LaTeX though)
Directly with GhostScript:
gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=finished.pdf file1.pdf file2.pdf
I have the following code fragment:
def database(self):
databasename=""
host=""
user=""
password=""
try:
self.fp=file("detailing.dat","rb")
except IOError:
self.fp=file("detailing.dat","wb")
pickle.dump([databasename,host,user,password],self.fp,-1)
self.fp.close()
selffp=file("detailing.dat","rb")
[databasename,host,user,password]=pickle.load(self.fp)
return
It has the error:
Traceback (most recent call last):
File "detailing.py", line 91, in ?
app=myApp()
File "detailing.py", line 20, in __init__
wx.App.__init__(self,redirect,filename,useBestVisual,clearSigInt)
File "/usr/lib64/python2.4/site-packages/wx-2.6-gtk2-unicode/wx/_core.py", line 7473, in __init__
self._BootstrapApp()
File "/usr/lib64/python2.4/site-packages/wx-2.6-gtk2-unicode/wx/_core.py", line 7125, in _BootstrapApp
return _core_.PyApp__BootstrapApp(*args, **kwargs)
File "detailing.py", line 33, in OnInit
self.database()
File "detailing.py", line 87, in database
[databasename,host,user,password]=pickle.load(self.fp)
File "/usr/lib64/python2.4/pickle.py", line 1390, in load
return Unpickler(file).load()
File "/usr/lib64/python2.4/pickle.py", line 872, in load
dispatch[key](self)
File "/usr/lib64/python2.4/pickle.py", line 894, in load_eof
raise EOFError
EOFError
What am I doing wrong?
Unless you've got a typo, the issue may be in this line where you assign the file handle to selffp not self.fp:
selffp=file("detailing.dat","rb")
If that is a typo, and your code actually opens the file to self.fp, then you may wish to verify that the file actually has contents (ie: that the previous pickle worked)... the error suggests that the file is empty.
Edit: In the comments to this answer, S. Lott has a nice summary of why the typo generated the error you saw that I'm pasting here for completeness of the answer: "selffp will be the unused opened file, and self.fp (the old closed file) will be used for the load".
Here's the version that I would recommend using:
def database(self):
databasename=""
host=""
user=""
password=""
try:
self.fp=open("detailing.dat","rb")
except IOError:
with open("detailing.dat", "wb") as fp:
pickle.dump([databasename,host,user,password],fp,-1)
self.fp=open("detailing.dat","rb")
[databasename,host,user,password]=pickle.load(self.fp)
return
As has been pointed out, there was a typo on self.fp. But here are a few other things that I notice that can cause problems.
First of all, you shouldn't be using the file constructor directly. You should instead use the built-in open function.
Secondly, you should avoid calling a file's close method outside a finally block. In this case, I've used python 2.6's with block. You can use this in Python 2.5 with the following command:
from __future__ import with_statement
This will prevent the file from being stuck open if an exception is thrown anywhere (as it will close the file when the with block is exited). Although this isn't the cause of your problem, it is an important thing to remember because if one of the file object's methods throws an exception, the file will get held open in sys.traceback indefinitely.
(note that you should probably accept Jarret Hardie's answer though, he caught the bug :-) )
I got this error when I didn't chose the correct mode to read the file (wb instead of rb). Changing back to rb was not sufficient to solve the issue. However, generating again a new clean pickle file solved the issue. It seems that not choosing the correct mode to open the binary file somehow "damages" the file which is then not openable whatsoever afterward.
But I am quite a beginner with Python so I may have miss something too.
While this is not a direct answer to the OP's question -- I happened upon this answer while searching for a reason for an EOFError when trying to unpickle a binary file with : pickle.load(open(filename, "r")).
import cPickle as pickle
A = dict((v, i) for i, v in enumerate(words))
with open("words.pkl", "wb") as f:
pickle.dump(A, f)
#...later open the file -- mistake:trying to read a binary with non-binary method
with open("words.pkl", "r") as f:
A =pickle.load(f) # EOFError
# change that to
with open ("words.pkl", "rb") as f: # notice the "rb" instead of "r"
A = pickle.load(f)