We Keep Coding

Why unit tests fil whereas the program runs?

I'm asked to develop unit tests for a program which is such badly developed that the tests don't run... but the program does. Thus, I need to explain the reason why and I actually don't know!
Here is a piece of code that intends to represent the code I need to test:
from services import myModule1
from services.spec1 import importedFunc
from services.spec2 import getTool
from services.spec3 import getDict
class myClass(object):
def __init__(self, param1, param2):
self.param1 = param1
self.param2 = param2
self.param3 = 0
self.param4 = 0
def myMethod(self):
try:
myVar1 = globalDict['key1']
myVar2 = globalDict['key2']
newVar = importedFunc(par1=myVar1, par2=myVar2, par3=extVar3)
calcParam = myModule1.methodMod1(self.param1)
self.param3 = calcParam["keyParam3"]
self.param4 = newVar.meth1(self.param2)
globTools.send_message(self.param3, self.param4)
except:
globTools.error_message(self.param3, self.param4)
return
class myClass2(object):
def __init__(self, *myclass2_params):
# some piece of code to intialize dedicated attributes
self.add_objects()
def add_objects(self):
# Some piece of code
my_class = myClass(**necessary_params)
# Some piece of code
return
if __name__ == '__main__':
globTools = getTool("my_program")
globalDict = getDict(some_params)
# Some piece of code
my_class2 = myClass2(**any_params)
# Some piece of code
As you can see, the problem is that the class and its methods uses global variables, defined in the main scope. And it's just a quick summary because it's actually a bit more complicated, but I hope it's enough to give you an overview of the context and help me understand why the unit test fail.
I tried to mock the imported modules, but I did not manage to a successful result, so I first tried to make it simple and just initialize all parameters.
I went to this test file:
import unittest
from my_module import myClass
from services import myModule1
from services.spec1 import importedFunc
from services.spec2 import getTool
from services.spec3 import getDict
def test_myClass(unittest.TestCase):
def setUp(self):
globTools = getTool("my_program")
globalDict = getDict(some_params)
def test_myMethod(self):
test_class = myClass(*necessary_parameters)
test_res = test_class.myMethod()
self.assertIsNotNone(test_res)
if __name__ == '__main__':
unittest.main()
But the test fail, telling me 'globTools is not defined' when trying to instantiate myClass
I also tried to initialize variables directly in the test method, but the result is the same
And to be complete about the technical environment, I cannot run python programs directly and need to launch a docker environment via a Jenkins pipeline - I'm not very familiar with this but I imagine it should not have an impact on the result
I guess the problem comes from the variable's scopes, but I'm not able to explain it in this case: why the test fail where as the method itself works (yes, it actually works, or at least the program globally runs without)

It's not as bad as you think. Your setUp method just needs to define the appropriate top-level globals in your module, rather than local variables.
import unittest
import my_module
from my_module import myClass
from services import myModule1
from services.spec1 import importedFunc
from services.spec2 import getTool
from services.spec3 import getDict
class test_myClass(unittest.TestCase):
def setUp(self):
my_module.globTools = getTool("my_program")
my_module.globalDict = getDict(some_params)
def test_myMethod(self):
test_class = myClass(*necessary_parameters)
test_res = test_class.myMethod()
self.assertIsNotNone(test_res)
if __name__ == '__main__':
unittest.main()
Depending on how the code uses the two globals, setUpClass might be a better place to initialize them, but it's probably not worth worrying about. Once you have tests for the code, you are in a better position to remove the dependency on these globals from the code.

attributeError for #classmethod generated when python script is imported into another script but not when tested stand alone

I am trying to re-use some code from the CoderBot project to build my own web controlled robot.
I am using python3 on a raspberry PI and I have simplified the code just to generate the error.
'''
class myclass():
objClass = None
def init(self):
print("initialised")
#classmethod
def get_it(cls):
if not cls.objClass:
print("objClass does not exist")
cls.objClass = myclass()
print("created")
return cls.objClass
def main(args):
method_list = [method for method in dir(myclass) if method.startswith('_') is False]
print(method_list)
makeone = myclass.get_it()
print(makeone)
print(myclass.get_it())
return 0
if __name__ == '__main__':
import sys
sys.exit(main(sys.argv))
'''
when I run this the following output is produced
'''
['get_it', 'objClass']
objClass does not exist
initialised
created
<__main__.myclass object at 0x7fbe6e1fa0>
<__main__.myclass object at 0x7fbe6e1fa0>
'''
Everything is as I expected.
When I run this code in a separate script
'''
import ct
def main(args):
method_list = [attribute for attribute in dir(ct.myclass) if
callable(getattr(ct.myclass, attribute)) and attribute.startswith('__') is
False]
print(method_list)
two = ct.get_it()
return 0
if __name__ == '__main__':
import sys
sys.exit(main(sys.argv))
'''
The following output is produced:
'''
['get_it']
Traceback (most recent call last):
File "/home/pi/classTest/mainct.py", line 35, in <module>
sys.exit(main(sys.argv))
File "/home/pi/classTest/mainct.py", line 30, in main
two = ct.get_it()
AttributeError: module 'ct' has no attribute 'get_it'
'''
This output is telling me that 'get_it' exists in the class, but not when I try and create one.
I am very confused and have looked at lots of tutorials but can't spot what I am doing wrong.
It is probably one of those errors where if I sat down and spoke to someone it would be obvious, but I can't spot it!
Steve
PS I hope the formatting is ok,
As jasonharper said, I was not referencing the class!
When I went back and looked at the original code I had not spotted the change in case.
so with my example the class MyClass should have been defined in a file myclass.py and then referenced as
from myclass import MyClass
then in the code
two = MyClass.get_it()
I have hopefully sorted out the formatting, the post was originally created on my PI and I spent 40 minutes trying to format it correctly.
This problem is definitely closed, thanks for replies

Is "ct" your class?
two = ct()
two.get_it()

Multiprocessing in a class and use it's instance in another class

I am stuck in using the multiprocessing module of Python. I wanted to create multiprocessing workers in a class and use the object of this class as a composite in another class. Below is the dummy code where the structure looks the same as my original code.
a.py
import concurrent.futures
class A:
def __init__(self, basic_input):
self.initial = basic_input
def _worker(self, work):
print(f"Doing some {work} using {self.initial}")
def print_my_work(self, set_of_works):
with concurrent.futures.ProcessPoolExecutor(max_workers=5) as executor:
executor.map(self._worker, set_of_works)
b.py
class B:
def __init__(self, basic_input):
# A is a composite object
self.a = A(basic_input)
def get_all_works(self):
return {'A', 'B', 'C'}
def processingB(self):
works = self.get_all_works()
self.a.print_my_work(works)
Here I'm trying to use class B in another module as below
check.py
import b
obj = B('Test')
obj.processingB()
Getting below error
Python multiprocessing RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.......
Can somebody help me. Thank you for reading this question.

Googling your problem, youn find a similar post : It seems that you are using Windows and you should then add if __name__ == '__main__': at the start of you main script (to avoid creating subprocesses recursively) :
import B
if __name__ == '__main__':
obj = B('Test')
obj.processingB()
Using your code I finally get :
Doing some A using Test
Doing some B using Test
Doing some C using Test

Wrote script in OSX, with multiprocessing. Now windows won't play ball

The program/script I've made works on OSX and linux. It uses selenium to scrape data from some pages, manipulates the data and saves it. In order to be more efficient, I included the multiprocessing pool and manager. I create a pool, for each item in a list, it calles the scrap class, starts a phantomjs instance and scrapes. Since I'm using multiprocessing.pool, and I want a way to pass data between the threads, I read that multiprocessing.manager was the way forward. If I wrote
manager = Manager()
info = manager.dict([])
it would create a dict that could be accessed by all threads. It all worked perfectly.
My issue is that the client wants to run this on a windows machine (I wrote the entire thing on OSX) I assumed, it would be as simple as installing python, selenium and launching it. I had errors which later lead me to writing if __name__ == '__main__: at the top of my main.py file, and indenting everything to be inside. The issue is, when I have class scrape(): outside of the if statement, it cannot see the global info, since it is declared outside of the scope. If I insert the class scrape(): inside the if __name__ == '__main__': then i get an attribute error saying
AttributeError: 'module' object has no attribute 'scrape'
And if I go back to declaring manager = manager() and info = manager.dict([]) outside of the if __name__ == '__main__' then I get the error in windows about making sure I use if __name__ == '__main__' it doesn't seem like I can win with this project at the moment.
Code Layout...
Imports...
from multiprocessing import Pool
from multiprocessing import Manager
manager = Manager()
info = manager.dict([])
date = str(datetime.date.today())
class do_scrape():
def __init__():
def...
def scrape_items():#This contains code which creates a pool and then pool.map(do_scrape, s) s = a list of items
def save_scrape():
def update_price():
def main():
main()
Basically, the scrape_items is called by main, then scrape_items uses pool.map(do_scrape, s) so it calls the do_scrape class and passes the list of items to it one by one. The do_scrape then scrapes a web page based on the item url in "s" then saves that info in the global info which is the multiprocessing.manager dict. The above code does not show any if __name__ == '__main__': statements, it is an outline of how it works on my OSX setup. It runs and completes the task as is. If someone could issue a few pointers, I would appreciate it. Thanks

It would be helpful to see your code, but its sounds like you just need to explicitly pass your shared dict to scrape, like this:
import multiprocessing
from functools import partial
def scrape(info, item):
# Use info in here
if __name__ == "__main__":
manager = multiprocessing.Manager()
info = manager.dict()
pool = multiprocessing.Pool()
func = partial(scrape, info) # use a partial to make it easy to pass the dict to pool.map
items = [1,2,3,4,5] # This would be your actual data
results = pool.map(func, items)
#pool.apply_async(scrape, [shared_dict, "abc"]) # In case you're not using map...
Note that you shouldn't put all your code inside the if __name__ == "__main__": guard, just the code that's actually creating processes via multiprocessing, this includes creating the Manager and the Pool.
Any method you want to run in a child process must be declared at the top level of the module, because it has to be importable from __main__ in the child process. When you declared scrape inside the if __name__ ... guard, it could no longer be imported from the __main__ module, so you saw the AttributeError: 'module' object has no attribute 'scrape' error.
Edit:
Taking your example:
import multiprocessing
from functools import partial
date = str(datetime.date.today())
#class do_scrape():
# def __init__():
# def...
def do_scrape(info, s):
# do stuff
# Also note that do_scrape should probably be a function, not a class
def scrape_items():
# scrape_items is called by main(), which is protected by a`if __name__ ...` guard
# so this is ok.
manager = multiprocessing.Manager()
info = manager.dict([])
pool = multiprocessing.Pool()
func = partial(do_scrape, info)
s = [1,2,3,4,5] # Substitute with the real s
results = pool.map(func, s)
def save_scrape():
def update_price():
def main():
scrape_items()
if __name__ == "__main__":
# Note that you can declare manager and info here, instead of in scrape_items, if you wanted
#manager = multiprocessing.Manager()
#info = manager.dict([])
main()
One other important note here is that the first argument to map should be a function, not a class. This is stated in the docs (multiprocessing.map is meant to be equivalent to the built-in map).

Find the starting point of your program, and make sure you wrap only that with your if statement. For example:
Imports...
from multiprocessing import Pool
from multiprocessing import Manager
manager = Manager()
info = manager.dict([])
date = str(datetime.date.today())
class do_scrape():
def __init__():
def...
def scrape_items():#This contains code which creates a pool and then pool.map(do_scrape, s) s = a list of items
def save_scrape():
def update_price():
def main():
if __name__ == "__main__":
main()
Essentially the contents of the if are only executed if you called this file directly when running your python code. If this file/module is included as an import from another file, all attributes will be defined, so you can access various attributes without actually beginning execution of the module.
Read more here:
What does if __name__ == "__main__": do?

How do I execute a module's code from another?

I know it's a greenhorn question. But. I have a very simple module that contains a class, and I want to call the module to run from another. Like so:
#module a, to be imported
import statements
if __name__ == '__main__':
class a1:
def __init__(self, stuff):
do stuff
def run_proc():
do stuff involving 'a1' when called from another module
#Module that I'll run, that imports and uses 'a':
if __name__ == '__main__':
import a
a.run_proc()
However, for reasons that are likely obvious to others, I get the error Attribute Error: 'Module' object has no attribute 'run_proc' Do I need a static method for this class, or to have my run_proc() method within a class, that I initialize an instance of?

Move the
if __name__ == '__main__':
in module a to the end of the file and add pass or some test code.
Your problems are that:
Any thing in the scope of if __name__ == '__main__': is only considered in the top level file.
You are defining a class but not creating a class instance.
module a, to be imported
import statements
class a1:
def __init__(self, stuff):
do stuff
def run_proc():
#do stuff involving 'a1' when called from another module
if __name__ == '__main__':
pass # Replace with test code!
Module that I'll run, that imports and uses 'a':
import a
def do_a():
A = a.a1() # Create an instance
A.run_proc() # Use it
if __name__ == '__main__':
do_a()