My folder tree:
project/
app/
__init__.py
models.py
dir/test1.py
dir/__init__.py
run.py
dir/test2.py
dir/__init__.py
If I want to do a
from app.models import Whatever
from test1 and test2 the only thing that works is to manually sys.path.append something like
os.path.join(os.path.dirname(__file__), "../..")
However there are a ton of answers on SO saying messing up with sys.path will give me troubles down the line (why?); Sadly, after 1+ hour of googling I still haven't figured out the right way to do import stuff and I'm getting really confused.
It is enormously better to test than not test, so if you need to append paths to sys.path to make it work--and in this directory configuration, you will--that's a reasonable and pragmatic step.
However, in general it is better not to fiddle with module load paths manually. It assumes that your code will will always be loaded in a directory right outside your test folder, which might not always be true. But "you will run into problems down the line" is pretty weak tea.
The bigger issue is that you cannot use that little path-patch to accomplish the kind of automated testing and test management you will ultimately want/need. Much better is to learn to use a real test harness / test runner such as pytest or nose. Even better if you also use a macro test runner such as tox. The combination will automatically install your software (including any declared dependencies) in a completely new, pristine virtual environment. This will help you test not just the normal operability of your module, but also its installability. It will also help you easily run tests across different versions of Python with very little additional effort. (I currently test across a range of 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, and several versions of PyPy, for example--not because I use all those versions, but because it doesn't cost me anything extra to make sure my software runs across a large swath of the entire Python universe.)
Fair warning: Setting up your testing environment is a bit of a pill the first time out, requiring a fair amount of effort and learning of "mechanic" skills. But this is a one-time investment that will pay dividends across all of your Python work.
So long story short, patching sys.path is a fair place to start. But you'll ultimately need more generality, breadth, depth, test fixtures, and automation in your testing--and path patching can't help you with those. Upgrading to real testing tools can.
Related
We have been using Django for a long time. Some old code is not being used now. How can I find which code is not being used any more and remove them.
I used coverage.py with unit tests, which works fine and shows which part of code is never used, but the test covered is very low. Is there any way to use it with WSGI server to find which code have never served any web requests?
It depends on what you mean by unused code.
For unreachable dead code, like functions are were never called, classes that are never instantiated, you can use a pure static code analyzer to find them. Pylint is a good option. Bear in mind that this is not 100% accurate, false positive is possible:
# static analysis can't detect methods called this way
func = getattr(obj, "func_name")
func()
For code that are reachable, but never reached. You have to rely on tools like coverage.py, and improve your test coverage.
On a well tested project, coverage would be ideal but with some untested legacy code I don't think there is a magical tool.
You could write a big test loading all the pages and run coverage to get some indication.
Cowboy style:
If it's not some critical code and you're fairly sure it's unused (i.e. not handling payments, etc.). Comment it out, check that the tests pass, deploy and wait a week or so before removing it definitely (or putting it back if you got a notification).
As other answers indicate coverage.py accurately finds out which parts of the code are never executed, but coverage requires your code to be actually run to perform the analysis. Vulture on the other hand, runs static analysis for finding dead (unused code) for Python Programs. Also, if you run vulture on both your library and test suite, you might be able to find untested code.
Vulture is a standard PyPI package and can be installed using pip:
$ pip install vulture
Run vulture using the command:
$ vulture apps/ tests/ --exclude settings
Bear in mind that due to Python's dynamic nature, there may be some false positives, but they can be dealt with by the means of "Whitelists" - Please refer to this answer by Rahul for more information on how to use Vulture with django and tackle false positives.
I'm in the process of redesigning/refactoring my Python quantum chemistry package (pyquante). One of the things I don't like about the existing release is that I have to install the package to run the test suite. That is, the test suite has statements like from PyQuante import SCF, and, of course, this PyQuante could refer to the installed version or a local version.
I know about virtualenv, and realize this is an option for me. But I was wondering whether anything else might be appropriate. In the past I've hacked sys.path for things like this, and have been told by better Python programmers that I shouldn't ever to this.
Does anyone have any suggestions for how I can do this? The point is that I want to test the current version of the code without installing it.
Thanks in advance for anyone who can see through my babbling and offer suggestions!
Create a proper package for your stuff and use
python setup.py develop
to make it a proper dev-package.
See:
https://stackoverflow.com/a/19048754/548039
http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
I would honestly insist on using virtualenv, its designed for this exact reason in mind. very small overhead, and if you ever mess up just delete directory. I am sure as you grow, things won't be as simple as they are now for your current situation. Take it as an opportunity to learn.
Altering sys.path much in production environment may be unwise. Altering it for testing is usually OK.
If you don't want to tinker with the variable from sys, use an environment variable named PYTHONPATH, it's a clean and documented way.
We've been doing a fair amount of Python scripting, and now we have a
directory with almost a hundred loosely related scripts. It's
obviously time to organize this, but there's a problem. These scripts
import freely from each other and although code reuse is generally a
good thing it makes it quite complicated to organize them into
directories.
There's a few things that you should know about our corporate environment:
I don't have access to the users'
environment. Editing the PYTHONPATH
is out, unless it happens in the
script itself.
Users don't install things. Systems
are expected to be already
installed and working on, so setup.py
is not a solution unless I can run it once for all users.
I'm quite willing to edit my import statements and do some minor refactoring, but the solutions I see currently require me to divide all the code strictly between "user runnable scripts" and "libraries", which isn't feasible, considering the amount of code.
Has anyone out there solved a similar problem? Are you happy with it?
--Buck
Another way to state the same question:
When looking at google code search, this kind of code is rampant (below). Is everyone happy with this? Is there a good alternative?
sys.path.insert(0, os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__))
)))
Is there a standard convention, or even a growing one, around where and how to invoke the tests associated with a project? In many projects, I'm seeing it bundled into a Make, a separate test.py script at the top level of the project, etc to do the work.
I looked around for some common thing with setup.py, but didn't spot anything there (granted, I didn't look hard). What's common and best practice?
The short answer is yes, there's a simple convention built-in to the unittest module. See this previous question.
I've recently been having some problems with my imports in Django (Python)... It's better to explain using a file diagram:
- project/
- application/
- file.py
- application2/
- file2.py
In project/application/file.py I have the following:
def test_method():
return "Working"
The problem occurs in project/application2/file2.py, when I try to import the method from above:
from application.file import test_method
Usually works, but sometimes not.
from project.application.file import test_method
Does work, but it goes against Django's portability guidelines as the project folder must always be called the same.
I wouldn't mind, but it's the fact that this issue is occurring inconsistently, most of the time omitting project is fine, but occasionally not (and as far as I can see, with no reason).
I can pretty much guarantee I'm doing something stupid, but has anyone experienced this? Would I be better just putting the project in front of all relevant imports to keep things consistent? Honestly, it's unlikely the project folder name will ever change, I just want things to stick with guidelines where possible.
For import to find a module, it needs to either be in sys.path. Usually, this includes "", so it searches the current directory. If you load "application" from project, it'll find it, since it's in the current directory.
Okay, that's the obvious stuff. A confusing bit is that Python remembers which modules are loaded. If you load application, then you load application2 which imports application, the module "application" is already loaded. It doesn't need to find it on disk; it just uses the one that's already loaded. On the other hand, if you didn't happen to load application yet, it'll search for it--and not find it, since it's not in the same directory as what's loading it ("."), or anywhere else in the path.
That can lead to the weird case where importing sometimes works and sometimes doesn't; it only works if it's already loaded.
If you want to be able to load these modules as just "application", then you need to arrange for project/ to be appended to sys.path.
(Relative imports sound related, but it seems like application and application2 are separate packages--relative imports are used for importing within the same package.)
Finally, be sure to consistently treat the whole thing as a package, or to consistently treat each application as their own package. Do not mix and match. If package/ is in the path (eg. sys.path includes package/..), then you can indeed do "from package.application import foo", but if you then also do "from application import foo", it's possible for Python to not realize these are the same thing--their names are different, and they're in different paths--and end up loading two distinct copies of it, which you definitely don't want.
If you dig into the django philosophy, you will find, that a project is a collection of apps. Some of these apps could depend on other apps, which is just fine. However, what you always want is to make your apps plug-able so you can move them to a different project and use them there as well. To do this, you need to strip all things in your code that's related to your project, so when doing imports you would do.
from aplication.file import test_method
This would be the django way of doing this. Glenn answered why you are getting your errors so I wont go into that part. When you run the command to start a new project: django-admin.py startproject myproject
This will create a folder with a bunch of files that django needs, manage.py settings,py ect, but it will do another thing for you. It will place the folder "myproject" on your python path. In short this means that what ever application you put in that folder, you would be able to import like shown above. You don't need to use django-admin.py to start a project as nothing magical happens, it's just a shortcut. So you can place you application folders anywhere really, you just need to have them on a python path, so you can import from them directly and make your code project independent so it easily can be used in any future project, abiding to the DRY principle that django is built upon.
It is better to always import using the same way - say, using project.app.models - because otherwise, you may find your module is imported twice, which sometimes leads to obscure errors as discussed in this question.