I have a program that uses the Python fileinput module, and I am trying to write unittests for the main() function. They work find when using an actual file, but raise OSError: reading from stdin while output is captured when I try to pass data via stdin. What is the correct way to mock the stdin input when using fileinput?
Example my_fileinput.py:
"""
$ echo "42" | python3.8 my_fileinput.py -
answer: 42
"""
import fileinput
import sys
def main():
for line in fileinput.input(sys.argv[1:]):
sys.stdout.write(f"answer #{fileinput.lineno()}: {line}")
if __name__ == "__main__":
main()
Example test_my_fileinput.py:
"""
$ python3.10 -m pytest test_my_fileinput.py
OSError: reading from stdin while output is captured
"""
import io
from unittest import mock
import my_fileinput
def test_stdin():
"""Test fileinput with stdin."""
with mock.patch.object(my_fileinput, "raw_input", create=True, return_value="42"):
with mock.patch("sys.stdout", new=io.StringIO()) as stdout:
with mock.patch("sys.argv", ["my_fileinput.py", "-"]):
# Raises OSError: reading from stdin while output is captured
my_fileinput.main()
assert stdout.getvalue() == "answer #1: 42"
I have tried various ways of mocking stdin, all with the same results. All result in the same OSError.
Update: A different version patching sys.stdin instead of inputfile.input
import io
from unittest import mock
import my_fileinput
def test_stdin():
"""Test fileinput with stdin."""
with mock.patch("sys.stdin", new=io.StringIO("42\n")):
with mock.patch("sys.stdout", new=io.StringIO()) as stdout:
with mock.patch("sys.argv", ["my_fileinput.py", "-"]):
my_fileinput.main()
assert stdout.getvalue() == "answer: 42\n"
Warning: The original answer gets rid of the OSError, but renders other functions in the inputfile module unusable (see comments).
Original:
Changing the first two arguments of mock.patch.object to fileinput and "input" seems to fix the OSError.
with mock.patch.object(fileinput, "input", create=True, return_value="42"):
The first argument is the target object you want to patch, which is the fileinput module. The second argument is the attribute to be changed in target object, which is input.
import io
import fileinput
from unittest import mock
import my_fileinput
def test_stdin():
"""Test fileinput with stdin."""
with mock.patch.object(fileinput, "input", create=True, return_value="42"):
with mock.patch("sys.stdout", new=io.StringIO()) as stdout:
with mock.patch("sys.argv", ["my_fileinput.py", "-"]):
my_fileinput.main()
assert stdout.getvalue() == "answer: 42\n"
It is not necessary for you to test fileinput itself, since that will be tested by CPython's own test suite: Lib/test/test_fileinput.py. The pytesthonic way to test your code would be like this, using fixtures:
import my_fileinput
def test_stdin(mocker, capsys):
mocker.patch("fileinput.lineno", return_value=1)
mocker.patch("fileinput.input", return_value=["42\n"])
my_fileinput.main()
out, err = capsys.readouterr()
assert out == "answer #1: 42\n"
The capsys fixture is included with pytest, and the mocker fixture is provided by plugin pytest-mock.
Related
I have two python files, the first python file which is triggered in server will dynamically fetch the variable result based on the environment the script is triggered . for an example , when script is triggered in dev environment. ideally, ${RPM_ENVIRONMENT} will return as 'DEV'
I have two files, one is my main file and one is my unit test cases
import os
import json
import subprocess
import logging
from os import listdir
from os.path import isfile, join
_ENV = os.popen("echo ${RPM_ENVIRONMENT}").read().split('\n')[0]
SERVER_URL = {
'DEV':{'ENV_URL':'https://dev.net'},
'UAT':{'ENV_URL':'https://uat.net'},
'PROD':{'ENV_URL':'https://prod.net'}
}[_ENV]
inside my test cases script below, i wanted to mimic as dev environment using unitest mock . i have tried below script but it was returning RPM_ENVIROMENT as key error .
test_env.py
import unittest , sys , tempfile, os , json , shutil
from unittest import mock
## i wanted to mock all the required variables before running import env_test so that it wont return any error.
with mock.patch.object(os, 'popen') as mock_popen:
sys.path.insert(1, 'C:/home/test/conf')
import env_test as conf
class test_tbrp_case(unittest.TestCase):
def test_port(self):
#function yet to be created
pass
if __name__=='__main__':
unittest.main()
I have tried using os.popen to mimic , but i am confused on how i can assign 'DEV' to _ENV variable .
when i tried to run this script, it was returning error as
SERVER_URL = {
KeyError: <MagicMock name='popen().read().split().__getitem__()' id='1893950325424'
**Approach 2 i have tried **
What i am trying to mock is the import , when im importing my main.py , it should dynamically replace/mock _ENV as 'DEV' , and SERVER_URL variable should automatically call Dev.
In a scenario where i call conf._ENV after i have implemented the mock below. it should return the value as "DEV"
def rpm_environment():
return os.popen("echo ${RPM_ENVIRONMENT}").read()
def test_rpm_environment():
with mock.patch("os.popen") as popen_mock:
popen_mock().read.return_value = "DEV"
actual = rpm_environment()
assert actual == "DEV"
## When i import env_test , RPM_ENVIROMENT wont be able to mock as DEV on what was declared in our test_rpm_enviromnet
rpm_environment()
test_rpm_environment()
# How can we safely import our env_test files with having variables been mocked so that i can call server_url variable
sys.path.insert(1, 'C:/home/test/conf')
import env_test import conf
I didn't quite understand if your code is inside a function or not.
If it is, the best way to do so is not patch.object. It's just a normal patch:
Consider this example:
def question():
return os.popen("what_ever").read()
def test_question():
with patch("os.popen") as popen_mock:
popen_mock().read.return_value = "DEV"
actual = question()
assert actual == "DEV"
In my opinion, patching os.popen and adding read to it's structure is the best practice.
Good luck !
When you mock popen it will return a MagickMock object and that object does not have a defined read response. You need to define what happens when someone calls read() on a MagickMock object that you have returned. Although it is not the most elegant solution, you can do this by adding this line in the with block:
mock_popen.return_value.read.return_value = "DEV"
This will instruct the MagickMock object to return the string "DEV" when read() is called on it.
i'm writing a test of this function
def create_folder_if_not_exists(
sdk: looker_sdk,
folder_name: str,
parent_folder_name: str) -> dict:
folder = sdk.search_folders(name=folder_name)[0]
try:
parent_id = sdk.search_folders(name=parent_folder_name)[0].id
logger.info(f'Creating folder "{folder_name}"')
folder = sdk.create_folder(
body=models.CreateFolder(
name=folder_name,
parent_id=parent_id
)
)
return folder
except looker_sdk.error.SDKError as err:
logger.error(err.args[0])
return folder
This is my current test, using the python pytest library, but i keep getting this for my test Failed: DID NOT RAISE <class 'looker_sdk.error.SDKError'>
def test_create_folder_if_not_exists_parent1(mocker):
# Tests if a folder has parent id of 1 we raise an exception
sdk = fake_methods_data.MockSDK()
sf_data = fake_methods_data.MockSearchFolder(
name='goog', parent_id=1, id=3)
mocker.patch.object(sdk, "search_folders")
mocker.patch.object(sdk, "create_folder",
side_effect=[looker_sdk.error.SDKError])
sdk.search_folders.return_value = [sf_data]
with pytest.raises(looker_sdk.error.SDKError) as err:
test = fc.create_folder_if_not_exists(
sdk=sdk, folder_name='googn', parent_folder_name='1')
assert str(err.value) == 'test'
assert test.parent_id == 1
assert test.name == 'googn'
Does anyone know how to force a function to return a class error using pytest ? I've been looking at this [stackoverflow] (Mocking a function to raise an Exception to test an except block) but am struggling to get it to work. Hoping for some other thoughts.
This sounds like something I have done for work (open-source software dev stuff). In my case, I needed to test an except block raised when an executable file could not be run on a particular OS version. In our testing framework we use pytest and monkeypatch to test things. I've included the relevant bits of code below, along with some explanation about what is happening. I think this is probably what you mean by 'patch the sdk error', and I believe that is probably what you need to do. If anything is unclear, or you have more questions, let me know.
In conftest.py I define pytest fixtures that get used for tests in more than one test file. Here, I mock the scenario I want to test, using monkeypatch to fake the results I want from the parts of the get_version() function I'm not trying to test.
# conftest.py
import subprocess
import shutil
import os
import re
import platform
from pathlib import Path
import pytest
#pytest.fixture
def executable_incompatible_with_os(monkeypatch):
"""
Mocks an executable file that is incompatible with the OS.
(This situation likely only applies to blastall.)
"""
def mock_which(*args, **kwargs):
"""Mock an absolute file path."""
return args[0]
def mock_isfile(*args, **kwargs):
"""Mock a call to `os.path.isfile()`."""
return True
def mock_access(*args, **kwargs):
"""Mock a call to `os.access()`."""
return True
def mock_subprocess(*args, **kwargs):
"""Mock a call to `subprocess.run()` with an incompatible program."""
raise OSError
# Replace calls to existing methods with my mocked versions
monkeypatch.setattr(shutil, "which", mock_which)
monkeypatch.setattr(Path, "is_file", mock_isfile)
monkeypatch.setattr(os.path, "isfile", mock_isfile)
monkeypatch.setattr(os, "access", mock_access)
monkeypatch.setattr(subprocess, "run", mock_subprocess)
In test_aniblastall.py I test parts of aniblastall.py. In this case, I'm testing the behaviour when an OSError is raised; the code that raises the error in the test is in conftest.py. The entire pytest fixture I defined there is passed as a parameter to the test.
# test_aniblastall.py
from pathlib import Path
import unittest
# Test case 4: there is an executable file, but it will not run on the OS
def test_get_version_os_incompatible(executable_incompatible_with_os):
"""Test behaviour when the program can't run on the operating system.
This will happen with newer versions of MacOS."""
test_file_4 = Path("/os/incompatible/blastall")
assert (
aniblastall.get_version(test_file_4)
== f"blastall exists at {test_file_4} but could not be executed"
)
aniblastall.py contains the function the error should be raised from.
# aniblastall.py
import logging
import os
import platform
import re
import shutil
import subprocess
from pathlib import Path
def get_version(blast_exe: Path = pyani_config.BLASTALL_DEFAULT) -> str:
"""
The following circumstances are explicitly reported as strings
- no executable at passed path
- non-executable file at passed path (this includes cases where the user doesn't have execute permissions on the file)
- no version info returned
- executable cannot be run on this OS
"""
logger = logging.getLogger(__name__)
try:
blastall_path = Path(shutil.which(blast_exe)) # type:ignore
except TypeError:
return f"{blast_exe} is not found in $PATH"
if not blastall_path.is_file(): # no executable
return f"No blastall at {blastall_path}"
# This should catch cases when the file can't be executed by the user
if not os.access(blastall_path, os.X_OK): # file exists but not executable
return f"blastall exists at {blastall_path} but not executable"
if platform.system() == "Darwin":
cmdline = [blast_exe, "-version"]
else:
cmdline = [blast_exe]
try:
result = subprocess.run(
cmdline, # type: ignore
shell=False,
stdout=subprocess.PIPE, # type: ignore
stderr=subprocess.PIPE,
check=False, # blastall doesn't return 0
)
except OSError:
logger.warning("blastall executable will not run", exc_info=True)
return f"blastall exists at {blastall_path} but could not be executed"
version = re.search( # type: ignore
r"(?<=blastall\s)[0-9\.]*", str(result.stderr, "utf-8")
).group()
if 0 == len(version.strip()):
return f"blastall exists at {blastall_path} but could not retrieve version"
return f"{platform.system()}_{version} ({blastall_path})"
This is super valuable #baileythegreen, however my problem was far simpler. I had an if/else and the else had the try/catch error code piece. I was so focused on that I didn't check the simple part of if it was even getting to the else. :(
Building a GUI for users to select Python scripts they want to run. Each script has its own docstring explaining inputs and outputs for the script. I want to display that information in the UI once they've highlighted the script, but not selected to run it, and I can't seem to get access to the docstrings from the base program.
ex.
test.py
"""this is a docstring"""
print('hello world')
program.py
index is test.py for this example, but is normally not known because it's whatever the user has selected in the GUI.
# index is test.py
def on_selected(self, index):
script_path = self.tree_view_model.filePath(index)
fparse = ast.parse(''.join(open(script_path)))
self.textBrowser_description.setPlainText(ast.get_docstring(fparse))
Let's the docstring you want to access belongs to the file, file.py.
You can get the docstring by doing the following:
import file
print(file.__doc__)
If you want to get the docstring before you import it then the you could read the file and extract the docstring. Here is an example:
import re
def get_docstring(file)
with open(file, "r") as f:
content = f.read() # read file
quote = content[0] # get type of quote
pattern = re.compile(rf"^{quote}{quote}{quote}[^{quote}]*{quote}{quote}{quote}") # create docstring pattern
return re.findall(pattern, content)[0][3:-3] # return docstring without quotes
print(get_docstring("file.py"))
Note: For this regex to work the docstring will need to be at the very top.
Here's how to get it via importlib. Most of the logic has been put in a function. Note that using importlib does import the script (which causes all its top-level statements to be executed), but the module itself is discarded when the function returns.
If this was the script docstring_test.py in the current directory that I wanted to get the docstring from:
""" this is a multiline
docstring.
"""
print('hello world')
Here's how to do it:
import importlib.util
def get_docstring(script_name, script_path):
spec = importlib.util.spec_from_file_location(script_name, script_path)
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
return foo.__doc__
if __name__ == '__main__':
print(get_docstring('docstring_test', "./docstring_test.py"))
Output:
hello world
this is a multiline
docstring.
Update:
Here's how to do it by letting the ast module in the standard library do the parsing which avoids both importing/executing the script as well as trying to parse it yourself with a regex.
This looks more-or-less equivalent to what's in your question, so it's unclear why what you have isn't working for you.
import ast
def get_docstring(script_path):
with open(script_path, 'r') as file:
tree = ast.parse(file.read())
return ast.get_docstring(tree, clean=False)
if __name__ == '__main__':
print(repr(get_docstring('./docstring_test.py')))
Output:
' this is a multiline\n docstring.\n'
I wrote little wrapper for urllib (python3). Is it proper and safe to import module in if?
if self.response_encoding == 'gzip':
import gzip
I didn't find any PEP about this code. However, it bothers me.
The Python standard library uses it, so it is most definitely proper and safe. See the os module source for an excellent example:
if 'posix' in _names:
name = 'posix'
linesep = '\n'
from posix import *
try:
from posix import _exit
except ImportError:
pass
import posixpath as path
import posix
__all__.extend(_get_exports_list(posix))
del posix
It's quite common to conditionally import modules in python. Instead of if, you'll often see a try:/except ImportError: combination too:
try:
from subprocess import check_output
except ImportError:
# Python 2.6 and before
def check_output(*popenargs, **kwargs):
from subprocess import Popen
if 'stdout' in kwargs:
raise ValueError('stdout argument not allowed, it will be '
'overridden.')
process = Popen(stdout=PIPE, *popenargs, **kwargs)
output, unused_err = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise CalledProcessError(retcode, cmd)
return output
Here, we basically use the moral equivalent of an if test: If you can import check_output, do so, otherwise define the full function here.
An import statement is just a re-binding of an external piece of code to a local name. Using an if control flow to control the import is no different from assigning a variable in an if statement in that regard. You need to make sure you don't end up using the name without it being defined either way.
This is a reasonably common idiom actually. You'll sometimes see it to pick between different modules:
if system == 'linux':
import linuxdeps as deps
elif system == 'win32':
import win32deps as deps
Then, assuming both linuxdeps and win32deps have the same functions, you can just use it:
deps.func()
This is even used to get os.path in the standard library (some of the source code for os follows):
if 'posix' in _names:
name = 'posix'
linesep = '\n'
from posix import *
try:
from posix import _exit
except ImportError:
pass
import posixpath as path
import posix
__all__.extend(_get_exports_list(posix))
del posix
elif 'nt' in _names:
name = 'nt'
linesep = '\r\n'
from nt import *
try:
from nt import _exit
except ImportError:
pass
import ntpath as path
import nt
__all__.extend(_get_exports_list(nt))
del nt
Sure, that's fine. It can even be necessary in cases where the module has initialization code that you don't always want to run.
Is it safe? Yes. As Martijin's answer pointed out that Official Python use this.
Is it proper? Depends. Python performance docs points out that even though python can avoid import the same module, there is still overhead.
So i believe you should ask yourself, how often the if statement is true. If very often, then there will be large overhead, and you should import it at the beginning of the file. If not often, then import in if statement is a wise choice.
I have an application that reads test scripts in python and sends them across the network for execution on a remote python instance. As the controlling program does not need to run these scripts I do not want to have all the modules the test scripts use installed on the controller's python environment. However the controller does need information from the test script to tell it how to run the test.
Currently what I do to read and import test script data is something like
with open( 'test.py', 'r' ) as f:
source = f.read()
m = types.ModuleType( "imported-temp", "Test module" )
co = compile( source, 'test.py', 'exec' )
exec co in m.__dict__
which yields a new module that contains the test. Unfortunately exec will raise ImportErrors if the test tries to import something the controller does not have. And worse, the module will not be fully imported.
If I can guarantee that the controller will not use the missing modules, is there someway I can ignore these exceptions? Or some other way to find out the names and classes defined in the test?
Examples test:
from controller import testUnit
import somethingThatTheControllerDoesNotHave
_testAttr = ['fast','foo','function']
class PartOne( testUnit ):
def run( self ):
pass
What the controller needs to know is the data in _testAttr and the name of all class definitions inheriting from testUnit.
Write an import hook that catches the exception and returns a dummy module if the module doesn't exist.
import __builtin__
from types import ModuleType
class DummyModule(ModuleType):
def __getattr__(self, key):
return None
__all__ = [] # support wildcard imports
def tryimport(name, globals={}, locals={}, fromlist=[], level=-1):
try:
return realimport(name, globals, locals, fromlist, level)
except ImportError:
return DummyModule(name)
realimport, __builtin__.__import__ = __builtin__.__import__, tryimport
import sys # works as usual
import foo # no error
from bar import baz # also no error
from quux import * # ditto
You could also write it to always return a dummy module, or to return a dummy module if the specified module hasn't already been loaded (hint: if it's in sys.modules, it has already been loaded).
I think, based on what you're saying, that you can just say:
try:
exec co in m.__dict__
except ImportError: pass
Does that help?
You could use python ast module, and parse the script to an AST tree and then scan through the tree looking for elements of interest. That way you don't have to execute the script at all.