We Keep Coding

Mocking Subproces Calls in Python

I am trying to use mock.patch decorator in the mock library but am having problems. I have this init function where I make two subprocess calls. The first time in self.setPackageQuota() and the second time in self.setBandwidthLimit().
class User():
def __init__(self, username, plan, domain, owner, diskQuota, diskUsed):
self.setUsername(username)
self.setPlan(plan)
self.setDomain(domain)
self.setOwner(owner)
self.setDiskQuota(diskQuota)
self.setDiskUsed(diskUsed)
self.setPackageQuota()
self.setBandwidthLimit()
self.setPackageQuota()
def setBandwidthLimit(self):
whmapicall = subprocess.Popen(["whmapi1" , "showbw", 'searchtype=user', 'search=^%s$' % self.username], stdout=subprocess.PIPE)
whmapireturn = whmapicall.stdout.read().split("\n")
for line in whmapireturn:
self.setPackageQuota()
def setPackageQuota(self):
whmapicall = subprocess.Popen(["whmapi1" , "getpkginfo", "pkg=%s" % self.plan], stdout=subprocess.PIPE)
whmapireturn = whmapicall.stdout.read().split("\n")
for line in whmapireturn:
I would much rather patch whmapireturn to be something else. I also would not like to not run subprocess.Popen. My inital thought was to patch out #mock.patch('subprocess.Popen', MockedClass) and #mock.patch('whmapireturn', OtherMockedClass) but I cant seem to get it to work. How would I test an init fucntion like this while patching out things that I can't have in my enviroment? Thanks in advance for any asistance.

I comprehend from your question that you want to mock-patch the Subprocess.Popen() call from your file. For that you should use the following approach:-
Let take the scenario where your file name is for which you want to create the unittest. Therefore:- in the unittest file, you should write:-
import library
#mock.patch("library.subprocess")
def test_subprocess_call(mock_subprocess):
mock_subprocess.Popen.return_value = None
I hope you find your answer.

Set Spark Config property for spark-testing-base

While I was trying to use spark-testing-base in Python, I needed to test a function which writes on a Postgres DB.
To do so is necessary to provide to the Spark Session the Driver to connect to Posgtres; to achieve that I first tried to override the getConf() method (as reported in the comment Override this to specify any custom configuration.). But apparently it doesn't work. Probably I'm not passing the value with the required syntax or whatever but after many attempts I anyway get the error java.lang.ClassNotFoundException: org.postgresql.Driver (typical of when the Driver Jar was not correctly downloaded through the conf parameter).
Attempted getConf override:
def getConf(self):
return ("spark.jars.packages", "org.postgresql:postgresql:42.1.1")
def getConf(self):
return {"spark.jars.packages", "org.postgresql:postgresql:42.1.1"}
def getConf(self):
return SparkConf()\
.setMaster("local[*]")\
.setAppName("test")\
.set("spark.jars.packages", "org.postgresql:postgresql:42.1.1")
So I even tried to Override the setUp() method like that:
def setUp(self):
try:
from pyspark.sql import Session
self.session = Session.Builder.config("spark.jars.packages", "org.postgresql:postgresql:42.1.1")
self.sqlCtx = self.session._wrapped
except Exception:
self.sqlCtx = SQLContext(self.sc)
But still no luck. So what I am doing wrong? How am I supposed to override the getConf() method?

Not exactly sure how to do this in python. In scala, using sbt, it is quite straight forward. But anyways, the System.setProperty("spark.jars.packages", "org.postgresql:postgresql:42.1.1") method found here: https://github.com/holdenk/spark-testing-base/issues/187 worked for me.
So I would rec looking up how to do that with python + spark.

It was necessary to override the setUpClass method:
#classmethod
def setUpClass(cls):
"""Setup a basic Spark context for testing"""
class_name = cls.__name__
conf = SparkConf().set("spark.jars.packages", "org.postgresql:postgresql:42.1.1")
cls.sc = SparkContext(cls.getMaster(), appName=class_name, conf=conf)
quiet_py4j()
And in this way is then possible to pass to the Spark test library external jars.
Credits to Leonardo Noleto: https://github.com/holdenk/spark-testing-base/issues/281#event-2200108290

python: How to mock helper method?

Can you please help me out to figure what I did wrong? I have the following unit test for a python lambdas
class Tests(unittest.TestCase):
def setUp(self):
//some setup
#mock.patch('functions.tested_class.requests.get')
#mock.patch('functions.helper_class.get_auth_token')
def test_tested_class(self, mock_auth, mock_get):
mock_get.side_effect = [self.mock_response]
mock_auth.return_value = "some id token"
response = get_xml(self.event, None)
self.assertEqual(response['statusCode'], 200)
The problem is that when I run this code, I get the following error for get_auth_token:
Invalid URL '': No schema supplied. Perhaps you meant http://?
I debugged it, and it doesn't look like I patched it correctly. The Authorization helper file is in the same folder "functions" as the tested class.
EDIT:
In the tested_class I was importing get_auth_token like this:
from functions import helper_class
from functions.helper_class import get_auth_token
...
def get_xml(event, context):
...
response_token = get_auth_token()
After changing to this, it started to work fine
import functions.helper_class
...
def get_xml(event, context):
...
response_token = functions.helper_class.get_auth_token()
I still don't fully understand why though

In your first scenario
in tested_class.py, get_auth_token is imported
from functions.helper_class import get_auth_token
The patch should be exactly the get_auth_token at tested_class
#mock.patch('functions.tested_class.get_auth_token')
Second scenario
With the following usage
response_token = functions.helper_class.get_auth_token()
The only way to patch is this
#mock.patch('functions.helper_class.get_auth_token')
alternative
With import like this in tested_class
from functions import helper_class
helper_class.get_auth_token()
patch could be like this:
#mock.patch('functions.tested_class.helper_class.get_auth_token')

patch() works by (temporarily) changing the object that a name points to with another one. There can be many names pointing to any individual object, so for patching to work, you must ensure that you patch the name used by the system under test.
The basic principle is that you patch where an object is looked up, which is not necessarily the same place as where it is defined.
Python documentation has a very good example. where to patch

Using Magic mock to test Github Api

I am basically using magic mock and context manager to test my code, I was successfully able to mock my get_urls function, But I am having trouble mocking out my access_all_repos_pr(): function which contains data of PR newer than 7 days, can anyone help me out on how to mock that data.
Here is the test code for my get_urls():
import unittest
from mock import MagicMock, patch
from contextlib2 import ExitStack
from GithubAPIpackage.GithubAPI import get_urls
class Test_GithubApi(unittest.TestCase):
def test_get_urls_returns_valid_urls(self):
with ExitStack() as stack:
mock_get_urls = stack.enter_context(
patch("GithubAPIpackage.GithubAPI._fetch_url")
)
fake_data = {"current_user_repositories_url": "http://FAKEURL.com"}
mock_get_urls.return_value = fake_data
print(type(fake_data))
result = get_urls()
self.assertEqual(result, "http://FAKEURL.com")
I want to mock out the response for the function access_all_repo_pr, can anyone help me out in what I need to do exactly to create a mock for my access_all_repo_pr function. Do I need to refactor my code in some way? (relatively new to python)
what I am trying is:
class Test_GithubApi_newer_than_7_days(unittest.TestCase):
def test_access_all_repo_pr_returns_valid_response(self):
with ExitStack() as stack:
mock_access_all_repo_pr = stack.enter_context(
patch("GithubAPIpackage.GithubAPI._fetch_url")
)
fake_data = {"current_user_repositories_url": "http://myfakeurl.com"}
mock_access_all_repo_pr.return_value = fake_data
result = access_all_repo_pr()
self.assertEqual(result, "")

Since you are using requests under the hood, may I suggest using responses for your testing? Not trying to skirt the question, but in my experience, I have found this to be the path of least resistance when it comes to writing tests that deal with the requests module. The tests end up being a lot cleaner, safer, and easier to write.

Python Mock not correctly setting return value

I am attempting to build unit tests and have been using mock, However upon using two patch statements, I was not able to set the proper return values.
#patch('pulleffect.lib.google.gcal_helper.validate_and_refresh_creds')
#patch('pulleffect.lib.google.gcal_helper.get_google_creds')
def test_get_calendar_list_for_gcalhelper_without_credentials(self,
mock_get_google_creds,
mock_validate_and_refresh_creds):
mock_validate_and_refresh_creds = "redirect"
mock_get_google_creds = "credentials"
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
however the assert fails and instead of the expected string redirect I instead get
<MagicMock name = "validate_and_refresh_creds() id = 14054613955344>
I was wondering what is necessary to have redirect returned instead. I have not encountered this issue when only patching a single method.

I was able to fix the issue of
<MagicMock name = "foo()" id = number>
incorrectly appearing by replacing my earlier code with:
from mock import MagicMock
def test_get_calendar_list_for_gcalhelper_without_credentials(self):
rtn = { "redirect": "/gcal/authenticate"}
pulleffect.lib.google.gcal_helper.validate_and_refresh_creds = MagicMock(name = "sup", return_value = rtn)
pulleffect.lib.google.gcal_helper.get_google_creds = MagicMock(name = "sup2", return_value = "redirect")
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
this allowed the return values to be properly set.

mock_get_google_creds and mock_validate_and_refresh_creds created with patch decorator are ordinary mock objects (Mock or MagicMock). Direct assignment is not the correct way to set return values. Use return_value attribute:
mock_validate_and_refresh_creds.return_value = "redirect"
Also you can set it during patching:
patch takes arbitrary keyword arguments. These will be passed to the
Mock (or new_callable) on construction.
#patch('pulleffect.lib.google.gcal_helper.get_google_creds', return_value="redirect")

I recommend you to use this solution. You should move your functions to helper class and instead static methods user class methods, because it's possible to mock class in this way.
class GCallHelper(object):
#classmethond
def validate_and_refresh(cls):
...
return result
def test_get_calendar_list_for_gcalhelper_without_credentials(self):
with patch('pulleffect.lib.google.gcal_helper') as mocked_gcal:
mocked_gcal.return_value.validate_and_refresh_creds.return_value = 'redirect'
mocked_gcal.return_value.get_google_creds.return_value = 'credentials'
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
p.s. And you forgot 'return_value' in your example.