We Keep Coding

how to import function in pytest

I am trying to create a pytest for a Python 2.x script executable with dashes included in its name. I tried to import it the usual way but I can't figure out how to make it work with the dashes.
My project structure is as follows:
package
-- tests
-- bin
-- subpackage
-- ...py
Specifically, I need to test a function called master_disaster() which exists inside bin/let-me-out (yes with -). let-me-out is an executable .py file and my folder has no setup.py file or anything similar.
How can I import this function inside my test? My test is going to be a simple fixture that checks the time with:
#pytest.fixture
def now():
return timezone.now()
It then uses the now() function to create a new file which let-me-out will delete after a specific amount of time.

First of all, dashes make let-me-out word to an invalid identifier in Python. To work around it, you have to invoke the imp (Python 2.7)
or importlib (Python 3.5+) machinery.
Python 3.5+
Here is an example of importing a new module having a qualified name let_me_out, but using bin/let-me-out as source file:
import importlib
def test_master_disaster():
loader = importlib.machinery.SourceFileLoader('let_me_out', 'bin/let-me-out')
spec = importlib.util.spec_from_loader(loader.name, loader)
let_me_out = importlib.util.module_from_spec(spec)
loader.exec_module(let_me_out)
# this is only a stub, to show an example of calling the master_disaster function
assert let_me_out.master_disaster() == 'spam'
You can extract this code into a fixture to make it reusable:
import importlib
import pytest
#pytest.fixture(scope='session')
def let_me_out():
loader = importlib.machinery.SourceFileLoader('let_me_out', 'bin/let-me-out')
spec = importlib.util.spec_from_loader(loader.name, loader)
let_me_out = importlib.util.module_from_spec(spec)
loader.exec_module(let_me_out)
return let_me_out
def test_master_disaster(let_me_out):
assert let_me_out.master_disaster() == 'spam'
Python 2.7
Things are even easier with Python 2.7:
import imp
import pytest
#pytest.fixture(scope='session')
def let_me_out():
return imp.load_source('let_me_out', 'bin/let-me-out')
def test_master_disaster(let_me_out):
assert let_me_out.master_disaster() == 'spam'

Unable to import module 'lambda_function': No module named 'error'

I have a simple Python Code that uses Elasticsearch module "curator" to make snapshots.
I've tested my code locally and it works.
Now I want to run it in an AWS Lambda but I have this error :
Unable to import module 'lambda_function': No module named 'error'
Here is how I proceeded :
I created manually a Lambda and gave it a "AISA-BasicLambdaExecutionRole" role. Then I created my package with my function and the dependencies that I installed with the command :
pip install elasticsearch-curator -t /<path>/myRepository
I zipped the content (not the folder) and uploaded it in my Lambda.
I changed the Handler name to "lambda_function.lambda_handler" (my function's name is "lambda_function.py").
Did I miss something ? This is my first time working with Lambda and Python
I've seen the other questions about this error :
"errorMessage": "Unable to import module 'lambda_function'"
But nothing works for me.
EDIT :
Here is my lambda_function :
from __future__ import print_function
import curator
import time
from curator.exceptions import NoIndices
from elasticsearch import Elasticsearch
def lambda_handler(event, context):
es = Elasticsearch()
index_list = curator.IndexList(es)
index_list.filter_by_regex(kind='prefix', value="logstash-")
Number = 1
try:
while Number <= 3:
Name="snapshotLmbd_n_"+ str(Number) +""
curator.Snapshot(index_list, repository="s3-backup", name= Name , wait_for_completion=True).do_action()
Number += 1
print('Just taking a nap ! will be back soon')
time.sleep(30)
except KeyboardInterrupt:
print('My bad ! I interrupted this')
return
Thank you for your time.

Ok, since you have everything else correct, check for the permissions of the python script.
It must have executable permissions (755)

pytest can't access django classes

I have a Django application, working fine. When I run the tests with pytest, it only works with utility classes (so not Django related).
For example, a test from package A calling an utility class from this package, or another, works fine.
However, I'm facing an error as soon as I import a django class.
example 1 :
I import my model (in the test), starting the test class with :
from app.common.models import Country
--> ImportError: No module named django.db
[django.db is called in models.py]
example 2 : I import an url resolver (in the test), starting the test class with :
from django.core.urlresolvers import reverse
--> ImportError: No module named django.core.urlresolvers
first try of fix
Following another topic, I set the content of PYTHONPATH :
/home/user/pyenv/lib/python3.5/site-packages
This folder contains installed packages in the virtualenv : django, pytest, psycopg2, and more.
1) If I set DJANGO_SETTINGS_MODULE to the same file as the application, "py.test" gives this error ending with :
File "/home/user/pyenv/lib/python3.5/site-packages/django/db/backends/postgresql/base.py", line 24, in
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
2) If I set DJANGO_SETTINGS_MODULE to a smaller test setting file, containing only the database infos, I face a different error (self.client failing in a test) :
class Test(unittest.TestCase):
def testUrls(self):
response = self.client.get('/countries/all/get')
self.assertEqual(response.status_code, 200)
--> AttributeError: 'Test' object has no attribute 'client'
more infos :
1) The python interpreter, for the application, is located in a virtualenv.
2) conftest.py is located in application root folder, so same place as manage.py, and has the following content:
import os
import sys
sys.path.append(os.path.dirname(__file__))
3) pytest.ini is located in same folder as manage.py too, with this content :
[pytest]
python_files = test_*.py test*.py
4) most important : the application works fine, so db settings are valid
If you have any idea of what's wrong, and how to test django classes, any idea will be welcomed. Thank you in advance.

I use pytest-django and I also explicitly set DJANGO_SETTINGS_MODULE=project.my_settings in pytest.ini. Works great every time.

I faced other issues trying the standard Django testing tool + coverage.
Using Pytest with "--cov" provided me what I needed, in one command only.
I finally found a fix for the issue. After trying again, and so restarting everything (pc, virtualenv), it worked fine.
So :
1) start virtualenv and move to the application folder
2) don't set PYTHONPATH
3) set DJANGO_SETTINGS_MODULE to application settings in pytest.ini
(so only one settings file in the project)
4) run the test with : py.test --cov=my_application_name
Now I can test models or forms without error.
For the previously mentioned test about urls, it works now with the following syntax:
import unittest
from django.test import Client
class Test(unittest.TestCase):
def setUp(self):
# global variable
self.client = Client()
def testUrls(self):
reponse = self.client.get('/countries/all/get')
self.assertEqual(reponse.status_code, 200)

Python Selenium Exception AttributeError: "'Service' object has no attribute 'process'" in selenium.webdriver.ie.service.Service

I have a Selenium Python test suite. It starts to run but after a few mins the following error is thrown:
Exception AttributeError: "'Service' object has no attribute 'process'" in <bound method Service.__del__ of <selenium.webdriver.ie.service.Service object at 0x0000000002610DD8>> ignored
My test suite implementation is:
import unittest
from HTMLTestRunner2 import HTMLTestRunner
import os
import Regression_TestCase.RegressionProject_TestCase2
# get the directory path to output report file
#result_dir = os.getcwd()
result_dir = r"E:\test_runners\selenium_regression_test_5_1_1\ClearCore - Regression Test\TestReport"
# get all tests from SearchProductTest and HomePageTest class
search_tests = unittest.TestLoader().loadTestsFromTestCase(Regression_TestCase.RegressionProject_TestCase2.RegressionProject_TestCase2)
# create a test suite combining search_test
re_tests = unittest.TestSuite([search_tests])
# open the report file
outfile = open(result_dir + "\TestReport.html", "w")
# configure HTMLTestRunner options
runner = HTMLTestRunner.HTMLTestRunner(stream=outfile,
title='Test Report',
description='Smoke Tests')
# run the suite using HTMLTestRunner
runner.run(re_tests)
Can anyone help why this error is stopping my test suite from running? How do I solve this?

Provided you have installed selenium, and assuming that earlier in the console's traceback log you also got something like "'chromedriver' executable needs to be in PATH" in your script, you should be able to do:
from selenium import webdriver
driver = webdriver.Chrome("/path/to/chromedriver")
This should tell your script where to find chromedriver. On a Mac you can usually use: /usr/local/bin/chromedriver

Download chromium driver from https://sites.google.com/a/chromium.org/chromedriver/downloads
Unzip the file and then from your code, write something like:
from selenium import webdriver
driver = webdriver.Chrome("/path/to/chromedriver")
where /path/to/chromedriver is the location of your chromedriver.
This is the class declaration for Chrome Webdriver: selenium.webdriver.chrome.webdriver.WebDriver(executable_path='chromedriver', ...
taken from https://seleniumhq.github.io/selenium/docs/api/py/webdriver_chrome/selenium.webdriver.chrome.webdriver.html#module-selenium.webdriver.chrome.webdriver

Given what #CubeBot88 had already written, another way to get the chromedriver executable in PATH is to do as follow:
from os
from selenium import webdriver
os.environ['PATH'] += "/path/to/chromedriver"
driver = webdriver.Chrome()
The above way puts the path to chromedriver to environment variable PATH only in this program, allowing independent PATH in different situations.

This page comes up first on Google, but only suggests that you are declaring you path incorrectly.
If you are reading this, try making sure the chromedriver is an executable:
run this command in the path of the driver:
sudo chmod a+x chromedriver

How to split python 3 unittests into separate file with script to control which one to run?

I want to split my Python 3.4 unit tests in separate modules and still be able to control which tests to run or skip from the command line, as if all tests were located in the same file. I'm having trouble doing so.
According to the docs, command line arguments can be used to select which tests to run. For example:
TestSeqFunc.py:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import random
import unittest
class TestSequenceFunctions(unittest.TestCase):
def setUp(self):
self.seq = list(range(10))
def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, list(range(10)))
# should raise an exception for an immutable sequence
self.assertRaises(TypeError, random.shuffle, (1,2,3))
def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)
def test_sample(self):
with self.assertRaises(ValueError):
random.sample(self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)
if __name__ == '__main__':
unittest.main()
can be controlled with either:
./TestSeqFunc.py
to run all tests in the file,
./TestSeqFunc.py TestSequenceFunctions
to run all tests defined in the TestSequenceFunctions class, and finally:
./TestSeqFunc.py TestSequenceFunctions.test_sample
to run the specific test_sample() method.
The problem I have is that I cannot find an organization of files that will allow me to:
Have multiple modules containing multiple classes and methods in separate files
Use a kind of wrapper script that will give the same kind of control over which tests (module/file, class, method) to run.
The problem I have is I cannot find a way to emulate the python3 -m unittest behaviour using a run_tests.py script. For example, I want to be able to do:
Run all the tests in the current directory
So ./run_tests.py -v should do do the same as python3 -m unittest -v
Run one module (file):
./run_tests.py -v TestSeqFunc being equivalent to python3 -m unittest -v TestSeqFunc
Run one class:
./run_tests.py -v TestSeqFunc.TestSequenceFunctions being equivalent to python3 -m unittest -v TestSeqFunc.TestSequenceFunctions
Run specific methods from a class:
./run_tests.py -v TestSeqFunc.TestSequenceFunctions.test_sample being equivalent to python3 -m unittest -v TestSeqFunc.TestSequenceFunctions.test_sample
Note that I want to:
be able to pass arguments to unittests, for example the verbose flag used previously;
allow running specific modules, classes and even methods.
As of now, I use a suite() function in my run_all.py script which loads manually the modules and add their class to a suite using addTest(unittest.makeSuite(obj)). Then, my main() is simple:
if __name__ == '__main__':
unittest.main(defaultTest='suite')
But using this I cannot run specific tests. In the end, I might just execute python3 -m unittest <sys.argv> from inside the run_all.py script, but that would be inelegant...
Any suggestions?!
Thanks!

Here's my final run_all.py:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import unittest
import glob
test_pattern = 'validate_*.py'
if __name__ == '__main__':
# Find all files matching pattern
module_files = sorted(glob.glob(test_pattern))
module_names = [os.path.splitext(os.path.basename(module_file))[0] for module_file in module_files]
# Iterate over the found files
print('Importing:')
for module in module_names:
print(' ', module)
exec('import %s' % module)
print('Done!')
print()
unittest.main(defaultTest=module_names)
Notes:
I use exec() to simulate 'import modulename'. The issue is that using importlib (explained here for example) will import the module but will not create a namespace for the module content. When I type import os, an "os" namespace is created and I can then access os.path. By using importlib, I couldn't figure out a way to do create that namespace. Having such a namespace is required for unittest; you get these kind of errors:
Traceback (most recent call last):
File "./run_all.py", line 89, in <module>
unittest.main(argv=sys.argv)
File "~/usr/lib/python3.4/unittest/main.py", line 92, in __init__
self.parseArgs(argv)
File "~/usr/lib/python3.4/unittest/main.py", line 139, in parseArgs
self.createTests()
File "~/usr/lib/python3.4/unittest/main.py", line 146, in createTests
self.module)
File "~/usr/lib/python3.4/unittest/loader.py", line 146, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "~/usr/lib/python3.4/unittest/loader.py", line 146, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "~/usr/lib/python3.4/unittest/loader.py", line 114, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'validate_module1'
Hence the use of exec().
I have to add defaultTest=module_names or else main() defaults to all test classes inside the current file. Since there is no test class in run_all.py, nothing gets executed. So defaultTest must point to a list of all the modules name.

You can pass command-line arguments to unittest.main using the argv parameter:
The argv argument can be a list of options passed to the program, with
the first element being the program name. If not specified or None,
the values of sys.argv are used. (my emphasis)
So you should be able to use
if __name__ == '__main__':
unittest.main(defaultTest='suite')
without any change and be able to call your script with command-line arguments as desired.