We Keep Coding

Create a gzip file like object for unit testing

I want to test a Python function that reads a gzip file and extracts something from the file (using pytest).
import gzip
def my_function(file_path):
output = []
with gzip.open(file_path, 'rt') as f:
for line in f:
output.append('something from line')
return output
Can I create a gzip file like object that I can pass to my_function? The object should have defined content and should work with gzip.open()
I know that I can create a temporary gzip file in a fixture but this depends on the filesystem and other properties of the environment. Creating a file-like object from code would be more portable.

You can use the io and gzip libraries to create in-memory file objects. Example:
import io, gzip
def inmem():
stream = io.BytesIO()
with gzip.open(stream, 'wb') as f:
f.write(b'spam\neggs\n')
stream.seek(0)
return stream

You should never try to test outside code in a unit test. Only test the code you wrote. If you're testing gzip, then gzip is doing something wrong (they should be writing their own unit tests). Instead, do something like this:
from unittest import mock
#mock.Mock('gzip', return_value=b'<whatever you expect to be returned from gzip>')
def test_my_function(mock_gzip):
file_path = 'testpath'
output = my_function(file_path=file_path)
mock_gzip.open.assert_called_with(file_path)
assert output == b'<whatever you expect to be returned from your method>'
That's your whole unit test. All you want to know is that gzip.open() was called (and you assume it works or else gzip is failing and that's their problem) and that you got back what you expected from the method being tested. You specify what gzip returns based on what you expect it to return, but you don't actually call the function in your test.

It's a bit verbose but I'd do something like this (I have assumed that you saved my_function to a file called patch_one.py):
import patch_one # this is the file with my_function in it
from unittest.mock import patch
from unittest import TestCase
class MyTestCase(TestCase):
def test_my_function(self):
# because you used "with open(...) as f", we need a mock context
class MyContext:
def __enter__(self, *args, **kwargs):
return [1, 2] # note the two items
def __exit__(self, *args, **kwargs):
return None
# in case we want to know the arguments to open()
open_args = None
def f(*args, **kwargs):
def my_open(*args, **kwargs):
nonlocal open_args
open_args = args
return MyContext()
return my_open
# patch the gzip.open in our file under test
with patch('patch_one.gzip.open', new_callable=f):
# finally, we can call the function we want to test
ret_val = patch_one.my_function('not a real file path')
# note the two items, corresponding to the list in __enter__()
self.assertListEqual(['something from line', 'something from line'], ret_val)
# check the arguments, just for fun
self.assertEqual('rt', open_args[1])
If you want to try anything more complicated, I would recommend reading the unittest mock docs because how you import the "patch_one" file matters as does the string you pass to patch().
There will definitely be a way to do this with Mock or MagicMock but I find them a bit hard to debug so I went the long way round.

Unable to create a formatted JSON file in python

I am a newbie in python (Using python 2.7) and I am trying to write a JSON file like this
import os;
import json;
BUILDNUMBER = "1.0.0"
class Foo(object):
def __init__(self):
self.buildNumber = BUILDNUMBER;
foo = Foo()
s = json.dumps(foo.__dict__)
os.system("echo {0} > ./build.json".format(s));
The contents of build.json looks like this
{buildNumber: 1.0.0}
I want it to look like this
{"buildNumber" : "1.0.0"}
Any help is appreciated.

No, you do not use os.system to call echo to redirect to a file. Never. In Python. Like ever.
Since no one showed how to do it right, this is how you write a JSON file in Python:
with open('./build.json', 'w') as f:
json.dump(foo.__dict__, f)

How to design a command line interface (CLI) which accepts Python functions or code?

I have a function that parses a given string with specific rules. I would like to design a CLI interface for this function. But the problem is I want that a user should be able to call this function via CLI using a READER & WRITER function of its own. To make it clear, here is a sample code and a demonstration of what I'm trying to explain.
# mylib.py
# piece of code that belongs to my lib
def parser(_id, text):
# parse the text & do some magic
return (_id, parsed_text)
# user-side code
def reader():
# read from a database
# or file or network or who knows where
yield (_id, text)
# user-side code
def writer(_id, text):
# write to somewhere
return True # or false depends on write action
A sample call should be something like this:
$ python mylib.py --reader <something-that-I-dont-know>
I don't want to use eval tricks but also I want that the user should be flexible while passing data to my library. Does this possible? Or should I try another approach?

With the help of #AlexHall, I've come up with the following solution:
import pathlib
import importlib.util
def load_module(filepath):
module_path = pathlib.Path(filepath)
abs_path = module_path.resolve()
module_name = module_path.stem
spec = importlib.util.spec_from_file_location(module_name, abs_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
Using this function, I am be able to import any valid python module exists in the filesystem even if the module is not in the path.
Here is a sample usage:
parser = make_parser(prog="tokenizer")
args = parser.parse_args()
module = load_module(args.writer) # if nothing is passed, default action defined in the parser
writer = module.writer
module = load_module(args.reader)
reader = module.reader
# do what you want to do with them

python: How to mock helper method?

Can you please help me out to figure what I did wrong? I have the following unit test for a python lambdas
class Tests(unittest.TestCase):
def setUp(self):
//some setup
#mock.patch('functions.tested_class.requests.get')
#mock.patch('functions.helper_class.get_auth_token')
def test_tested_class(self, mock_auth, mock_get):
mock_get.side_effect = [self.mock_response]
mock_auth.return_value = "some id token"
response = get_xml(self.event, None)
self.assertEqual(response['statusCode'], 200)
The problem is that when I run this code, I get the following error for get_auth_token:
Invalid URL '': No schema supplied. Perhaps you meant http://?
I debugged it, and it doesn't look like I patched it correctly. The Authorization helper file is in the same folder "functions" as the tested class.
EDIT:
In the tested_class I was importing get_auth_token like this:
from functions import helper_class
from functions.helper_class import get_auth_token
...
def get_xml(event, context):
...
response_token = get_auth_token()
After changing to this, it started to work fine
import functions.helper_class
...
def get_xml(event, context):
...
response_token = functions.helper_class.get_auth_token()
I still don't fully understand why though

In your first scenario
in tested_class.py, get_auth_token is imported
from functions.helper_class import get_auth_token
The patch should be exactly the get_auth_token at tested_class
#mock.patch('functions.tested_class.get_auth_token')
Second scenario
With the following usage
response_token = functions.helper_class.get_auth_token()
The only way to patch is this
#mock.patch('functions.helper_class.get_auth_token')
alternative
With import like this in tested_class
from functions import helper_class
helper_class.get_auth_token()
patch could be like this:
#mock.patch('functions.tested_class.helper_class.get_auth_token')

patch() works by (temporarily) changing the object that a name points to with another one. There can be many names pointing to any individual object, so for patching to work, you must ensure that you patch the name used by the system under test.
The basic principle is that you patch where an object is looked up, which is not necessarily the same place as where it is defined.
Python documentation has a very good example. where to patch

Line profiling python code running as background service

I am aware of the way independent scripts are profiled using kerprof/profile/cProfile. But how can I profile python web application running as background service/long running application

After some drill down and exploring potential solutions; I came up with following solution:
Add the following function into source file and decorate the original function to profile with #do_cprofile
import cProfile
def do_cprofile(func):
def profiled_func(*args, **kwargs):
profile = cProfile.Profile()
try:
profile.enable()
result = func(*args, **kwargs)
profile.disable()
return result
finally:
profile.dump_stats('/tmp/profile_bin.prof')
return profiled_func
Convert the generated /tmp/profile_bin.prof to human readable file
import pstats
f = open('/tmp/human_readable_profile.prof', 'w')
stats = pstats.Stats('/tmp/profile_bin.prof', stream=f)
stats.sort_stats('cumulative').print_stats()
f.close()