Can the nose testing framework be instructed to only run tests in test_*.py files?
In fact, doing nosetests A with the following directory structure:
A/
test_A.py
B/
__init__.py
imports B, which I want to avoid.
The reason for this is that the B module starts with import numpy because it is only meant to be used when the user has the optional NumPy module installed. However, users who did not install NumPy do not want nosetests to process B/__init__.py, because it necessarily fails on import numpy even though NumPy is optional. How can this be achieved?
Simply wrap the import with a try:..except ImportError:... block. In that case you could even set a variable letting you know if numpy is available or not.
Sure, just use the --match and the --exclude command line options to limit what nose will discover as a test program.
Nose can accept/reject tests at the directory, file, module, class and method levels. You need to reject the B directory. There is no way to ignore the __init__.py file; Python never sees it. It shows up when B is imported, so you need to ignore B.
Try:
nosetests --exclude=B
I think the nose-exclude plugin may help. If you install this plugin and then run:
nosetests --exclude-dir B
It may work for you. I have the same problem and have achieved some passable results. Now my next issue is that the exclude-dir doesn't seem to be a usable option when creating a nosetests config file.
Related
I wrote a custom python package for Ansible to handle business logic for some servers I manage. I have multiple files and they reference each other by re-importing the package.
So my package named <MyCustomPackage> has functions <Function1> <Function2> <Function3>, etc all in their own files... Some of these functions reference functions in the same package, so to do that the file has:
import MyCustomPackage
at the top. I did it this way instead of a relative import because I'm also unit testing these and mocking would not work with relative paths because of a __init__ file in the test directory which was needed for test discovery. The only way I could mock was through importing the package itself. Seemed simple enough.
The problem is with Ansible. These packages are in module_utils. I import them with:
from ansible.module_utils.MyCustomPackage import MyCustomPackage
but when I use the commands I get module not found errors - and traced it back to the import MyCustomPackage statement in the package itself.
So - how should I be structuring my package? Should I try again with relative file imports, or have the package modify the path so it's found with the friendly name?
Any tips would be helpful! Or if someone has a module they've written with Python modules in module_utils and unit tests that they'd be willing to share, that'd be great also!
Many people have problems with relative imports and imports in general in Python because they are ambiguous and surprisingly depend on your current working directory (and other things).
Thus I've created an experimental, new import library: ultraimport
It gives you more control over your imports and lets you do file system based, relative imports.
Given that you have a file function1.py, to import a function from function2.py, you would then write:
import ultraimport
Function2 = ultraimport('__dir__/function2.py', 'Function2')
This will always work, no matter how you run your code. It also does not force you to a specific package structure. You can just have any files you like.
First, I need to describe the environment I'm writing for. I'm writing Python code that will be loaded and executed by a Python runtime running within a CAD application. The CAD application uses Python as its scripting engine. As a result, I don't have access to the Python runtime and as a good citizen to all other scripts shouldn't modify any system settings. My script is just one of many that are loaded and running.
This all works fine except when I want to use non-standard libraries. In that case, I need to install a local copy of the library for my script to access. The problem I'm having is that most libraries expect to be installed and added to the sys path which is something I shouldn't do because it could create conflicts with what other scripts are doing. What I'm attempting to do instead is to set up a local copy of the library(s) and then edit their source so their imports are relative and they don't depend on the sys path. That way my program will have its own local copy of the libraries and not depend on anything else and won't disturb any other scripts.
I'm using the -t option of PIP to install Requests and PyOpenSSL into a "Packages" subfolder in my script folder. Here's an abbreviated list of what I have.
RequestsTest/
RequestsTest.py
Packages/
OpenSSL/
cryptography/
x509/
__init__.py
base.py
hazmat/
__init__.py
backends/
__init__.py
interfaces.py
openssl/
__init__.py
backend.py
x509.py
OpenSSL/
__init__.py
SSL.py
Requests/
chardet/
__init__.py
requests/
__init__.py
urllib3/
__init__.py
request.py
contrib/
__init__.py
pyopenssl.py
util/
__init__.py
request.py
ssl_.py
Although it's tedious to track down the various import statements and make then relative, it does seem to work. However, I'm having problems with one particular set of imports.
In Packages/Requests/urllib3/contrib/pyopenssl.py it contains the following imports, which I've modified:
from ....OpenSSL.OpenSSL import SSL
from ....OpenSSL.cryptography import x509
They were originally:
from OpenSSL import OpenSSL.SSL
from cryptography import x509
I get the error "ImportError: No module named 'OpenSSL'" for the first line and "ImportError: No module named 'cryptography'" for the second line. I'm fairly certain the path is correct because if I change the number of dots I get the no module named error but it lists the full path of the what it's trying to load and not just the name of the module.
I would appreciate some help with this specific issue but can also use some overall advice of how to set up and use private copies of libraries. Remember that my program is just one of many that the system is loading changing the system or setting up a virtual environment is not an option.
Check out the localimport module, which seems to be a solution for your particular use case. From the README:
Given your Python script, application or plugin comes with a directory that contains modules for import, you can use localimport to keep the global importer state clean.
app.py
res/modules/
some_package/
__init__.py
# app.py
with localimport('res/modules') as _importer:
import some_package
assert 'some_package' not in sys.modules
The tagline is "Isolated import of Python Modules for embedded applications." so it seems pretty relevant.
When using that module, the following may help keep things neat:
Put your actual script logic into its own file.
Have a wrapper script (which will be the one loaded by the CAD software) which does localimport as mentioned in the README then does a relative import of your module. If your module gets big enough maybe put it into its own package and consume it in the same way as everything else (just do from RequestsTest import * in the body of the with localimport(): ....
Try to have a clear boundary between the source code you write and the final organized set of files required to use that source code in the context of the CAD Python runtime. It is OK to have a build/packaging step that creates the localimport script, downloads the required packages, etc. It's better even because then it is automated and not something that was done manually that someone in the future may have to recreate.
I have a project subdivided into packages, in the following structure:
/project
/pkg
__init__.py
engine.y
ai.py
__init__.py
test_script.py
engine.py has an import statement to use ai.py which looks like
import pkg.ai as ai
This means that test_script.py can be run in the command line as python test_script.py and it has no issues. However for debugging purposes, engine.py is also often run. When running in PyCharm it has no problems but when using python engine.py I get errors saying No module named pkg.
Is there any way I can run engine.py in the command line so that it does not have import errors the way PyCharm does it?
Interestingly the way PyCharm works is that if I do not put that pkg. in front of the import module, it underlines it in red saying it can't find the module (but still runs). I've looked everywhere for a solution to this but have only got more confused.
The best explanation to this issue was explained here. In short, keep the absolute imports with import pkg.ai in engine.py but when I want to run the engine module for testing reasons it should be run on the top level (when in project dir) with the module flag, such as:
cd project/
python -m pkg.engine
Make something similar to what you've done with test_script.py In the same script or in another one in the same level and invoke the scripts from there.
You may also want to read my two answers here and here, they contain two examples on how to use import packages. That should help you understand their usage.
I am using the following command to run tests:
nosetests --with-coverage --cover-html --cover-package mypackage
I would like the coverage report to be updated, even if a developer adds new, untested, code to the package.
For example, imagine a developer adds a new module to the package but forgets to write tests for it. Since the tests may not import the new module, the code coverage may not reflect the uncovered code. Obviously this is something which could be prevented at the code review stage but it would be great to catch it even earlier.
My solution was to write a simple test which dynamically imports all modules under the top-level package. I used the following code snippet to do this:
import os
import pkgutil
for loader, name, is_pkg in pkgutil.walk_packages([pkg_dirname]):
mod = loader.find_module(name).load_module(name)
Dynamically importing sub-packages and sub-modules like this does not get picked up by the code coverage plugin in nose.
Can anyone suggest a better way to achieve this type of thing?
The problem seems to be the method for dynamically importing all packages/modules under the top-level package.
Using the method defined here seems to work. The key difference being the use of importlib instead of pkgutil. However, importlib was introduced in python 2.7 and 3.1 so this solution is not appropriate for older versions of python.
I have updated the original code snippet to use __import__ instead of the ImpLoader.load_module method. This also seems to do the trick.
import os
import pkgutil
for loader, name, is_pkg in pkgutil.walk_packages([pkg_dirname]):
mod = loader.find_module(name)
__import__(mod.fullname)
Okay, so in the past, I've made my own Python packages with Python 2.x (most recently, 2.7.5). It has worked fine. Let me explain how I did that, for reference:
Make a directory within the working directory. We'll call it myPackage.
Make a file called __init__.py in the directory myPackage.
Make sure all the modules that you want to be part of the package are imported within __init__.py. These modules are typically in the myPackage folder.
From a Python program in the working directory, type import myPackage (and it imports fine, and is usable).
However, in Python 3, I get errors with that. (ImportError: No module named 'Whatever the first imported module is')
I researched the problem and found the following:
Starred imports don't work in Python 3.3.
The __init__.py file is not required in Python 3.3.
So, I removed the stars from my imports, and leaving the __init__.py file in, I still got errors (ImportError: No module named 'Whatever the first imported module is'). So, I removed the __init__.py file, and I don't get any errors, but my package doesn't include any of my modules.
Okay, so I discovered by doing a web search for python3 __init__.py or some such that I can do the following, although I don't have any clue if this is the standard way of doing things:
In the modules in the package, make sure there are no plain imports (not just no starred ones). Only do from myModule import stuff. However, you need to put a . in front of myModule: e.g. from .myModule import stuff. Then, I can import myPackage.oneOfMyModules
I found that by following this rule in the __init__.py file, it also works.
Once again, I don't know if this is how it's supposed to work, but it seems to work.
I found this page that is supposed to have something to do with the changes in packages in Python 3.something, but I'm not sure how it relates to what I'm doing:
http://legacy.python.org/dev/peps/pep-0420/
So, what is the standard way to do this? Where is it documented (actually saying the syntax)? Is the way I'm doing it right? Can I do regular imports instead of from package import module?
After analyzing some Python 3 packages installed on my system (I should have tried that to start with!) I discovered that they often seem to do things a little differently. Instead of just doing from .myModule import stuff they would do from myPackage.myModule import stuff (inside the modules in the package). So, that works, too, I suppose, and seems to be more frequently used.