Ok, I think whatever I'm doing wrong, it's probably blindingly obvious, but I can't figure it out. I've read and re-read the tutorial section on packages and the only thing I can figure is that this won't work because I'm executing it directly. Here's the directory setup:
eulerproject/
__init__.py
euler1.py
euler2.py
...
eulern.py
tests/
__init__.py
testeulern.py
Here are the contents of testeuler12.py (the first test module I've written):
import unittest
from .. import euler12
class Euler12UnitTests(unittest.TestCase):
def testtriangle(self):
"""
Ensure that the triangle number generator returns the first 10
triangle numbers.
"""
self.seq = [1,3,6,10,15,21,28,36,45,55]
self.generator = euler12.trianglegenerator()
self.results = []
while len(self.results) != 10:
self.results.append(self.generator.next())
self.assertEqual(self.seq, self.results)
def testdivisors(self):
"""
Ensure that the divisors function can properly factor the number 28.
"""
self.number = 28
self.answer = [1,2,4,7,14,28]
self.assertEqual(self.answer, euler12.divisors(self.number))
if __name__ == '__main__':
unittest.main()
Now, when I execute this from IDLE and from the command line while in the directory, I get the following error:
Traceback (most recent call last):
File "C:\Documents and Settings\jbennet\My Documents\Python\eulerproject\tests\testeuler12.py", line 2, in <module>
from .. import euler12
ValueError: Attempted relative import in non-package
I think the problem is that since I'm running it directly, I can't do relative imports (because __name__ changes, and my vague understanding of the packages description is that __name__ is part of how it tells what package it's in), but in that case what do you guys suggest for how to import the 'production' code stored 1 level up from the test code?
I had the same problem. I now use nose to run my tests, and relative imports are correctly handled.
Yeah, this whole relative import thing is confusing.
Generally you would have a directory, the name of which is your package name, somewhere on your PYTHONPATH. For example:
eulerproject/
euler/
__init__.py
euler1.py
...
tests/
...
setup.py
Then, you can either install this systemwide, or make sure to set PYTHONPATH=/path/to/eulerproject/:$PYTHONPATH when invoking your script.
An absolute import like this will then work:
from euler import euler1
Edit:
According to the Python docs, "modules intended for use as the main module of a Python application should always use absolute imports." (Cite)
So a test harness like nose, mentioned by the other answer, works because it imports packages rather than running them from the command line.
If you want to do things by hand, your runnable script needs to be outside the package hierarchy, like this:
eulerproject/
runtests.py
euler/
__init__.py
euler1.py
...
tests/
__init__.py
testeulern.py
Now, runtests.py can do from euler.tests.testeulern import TestCase and testeulern.py can do from .. import euler1
Related
This has to be simple, but I swear I've been searching and experimenting for 2 days with no end of errors. The best I've managed to achieve is either the IDE indicates no errors, but fails when running. Or indicates errors, but runs successfully. I'm on Windows 10. Python 3.8.3 conda. I have VS Code, Spyder, and PyCharm--all of which I've uninstalled and reinstalled multiple times hoping one of them would give me useful default behavior.
I think I have a minimally reproducible anomaly that hopefully serves to identify issues, which I'll explain below. As a preface, I'll say that I wish that I could find a complete and direct guide to identifying, declaring, repairing, etc packages and modules. Every source I look at has a different set of suggestions, it seems like. And I'm lost between PYTHONPATH, sys.path, .env workspace files, .vscode json files.
c:\myproject
__init__.py
file1.py
\myenums
__init__.py
category.py
functions.py
general.py
That's leaving out a lot of files and folders. And I won't belabor a long list of variations of errors I've had. Hopefully this pair of examples is telling.
I have a function in \myproject\file1.py and another function in \myproject\myenums\functions.py. Each of those .py files has import references to other .py files. Pretty much all those import statements are red-underlined in VS Code. But depending on how I qualify the import statments--how high up the folder hierarchy I go--one or the other will run but not both. To wit (reduced and simplified indication of actual code):
\myproject\file1.py
import myenums.category # red underline for error
import myenums.functions # red underline for error
import myenums.general # red underline for error
import myenums.results # red underline for error
print(myenums.category) # works
print(myenums.functions) # works
print(myenums.general) # works
print(myenums.results) # works
\myproject\myenums\functions.py
from myenums.category import A, B, C # red underline, note mutual parent folder
from myenums.general import X, Y, Z # red underline, note mutual parent folder
def testfunc(value):
if value is A.x:
return X.a
elif value is A.y:
return Y.a
print(testfunc(A.y)) # doesn't work. ModuleNotFoundError: No module named 'myenums'.
I get the working/not working results by Run Without Debugging in either file.
\myproject\file1.py
import myenums.category # red underline for error
import myenums.functions # red underline for error
import myenums.general # red underline for error
import myenums.results # red underline for error
print(myenums.category) # doesn't work, ModuleNotFoundError: No module named 'category' (on import functions)
print(myenums.functions)
print(myenums.general)
print(myenums.results)
\myproject\myenums\functions.py
from category import A, B, C # red underline, note parent folder qualification removed
from general import X, Y, Z # red underline, note parent folder qualification removed
def testfunc(value):
if value is A.x:
return X.a
elif value is A.y:
return Y.a
print(testfunc(A.y)) # Now works!
So depending on whether I overqualify with the containing folder name in functions.py, I can get functions.py or file1.py to run--but never both. And I can never make error indicators go away on the imports.
I can't figure out at all how to diagnose this problem. Could I have too many or the wrong init.py files? Could they be bad files? They look empty in Notepad. And are not .py.bin. When I run print(p) in sys.path, I do see c:\myproject--but it's a lower case 'c' where all the Anaconda references have upper case 'C'. Hmmm. It's been a long time since I've felt this lost and helpless trying to write some code. I wouldn't have thought that creating references between the files in a single folder would be such a challenge. I have no idea what to try.
Edit: Originally I thought my problem was specific to references between subfolders next to each other in hierarchy, and I thought I would try to fix it by rearranging code to only ever reference to direct subfolders--but this anomaly seems to shoot even that down. Also, I originally posted saying that my anomaly went away. Now I think it's still there, I just had a typo.
Here is my suggestion. Organize your file structure like this for testing purposes:
myproject\
tests.py
myproject\
__init__.py
file1.py
myenums\
__init__.py
category.py
functions.py
general.py
What I did was create a topmost folder for testing purposes that doesn't include a __init__.py file, i.e., it is not part of the main module myproject. You can do all of your testing from here.
The general rule of thumb is that, when creating a Python package, you should not run the scripts directly from inside the package.
To illustrate this, let's write a function inside of myproject/myproject/file1.py:
# myproject/myproject/file1.py
def file1_function():
print("Inside file1.py!")
Now, we have to import it in the topmost __init__.py so that we can access it from outside the function.
# myproject/myproject/__init__.py
from myproject import file1
Established earlier, we are going to run our tests from tests.py, which is outside of the package. In tests.py, we can say:
# myproject/tests.py
import myproject
myproject.file1.file1_function()
Which outputs:
Inside file1.py!
Of course, you could always replace from myproject import file1 with from myproject.file1 import file1_function, which allows us to call file1_function a little more easily:
# myproject/tests.py
import myproject
myproject.file1_function()
Which gives us the same output as before.
What happens when we need to import files/functions/classes/etc. to use elsewhere in the package? Well, we can use absolute imports.
For demonstration purposes, let's say there is an important piece of code in functions.py that file1.py needs to use.
# myproject/myproject/myenums/functions.py
def really_important_function():
print("Inside myproject/myenums/functions.py!")
Now we have to do the same thing that we did above. The only difference is that we are trying to access it from inside the package. Let's move into the __init__.py that is located in myenums:
# myproject/myproject/myenums/__init__.py
from myproject.myenums.functions import really_important_function
Here we use an absolute import, which is an import that stems from the root of the package (myproject is the root).
To use this in file1.py, we then have to import it into file1.py:
# myproject/file1.py
from myenums import really_important_function
def use_important_function():
really_important_function()
Now that we are able to access really_important_function from myproject/myenums/functions.py in myproject/file1.py, we can import it in tests.py the same way we did in the above example:
# myproject/tests.py
import myproject
myproject.file1.use_important_function()
Which outputs:
Inside myproject/myenums/functions.py!
I think it should be worth noting that you don't necessarily need to traverse the package upwards, updating every __init__.py just to test a function that won't be imported directly anyway. If you want to test really_important_function from tests.py, you can do this:
# myproject/tests.py
import myproject
myproject.myenums.functions.really_important_function()
Which gives the correct output.
Conclusion
Another way to look at this is as follows. Let's say you define "function" in functions.py and you want to use it in a function defined in file1.py:
myproject\
tests.py -> import myproject
-> myproject.use_function()
myproject\
__init__.py -> from myproject.file1 import use_function
file1.py -> from myproject.myenums import function
-> def use_function():
function()
myenums\
__init__.py -> from myproject.myenums.functions import function
category.py
functions.py -> def function():
...
general.py
I have some difficulties constructing my project structure.
This is my project directory structure :
MusicDownloader/
__init__.py
main.py
util.py
chart/
__init__.py
chart_crawler.py
test/
__init__.py
test_chart_crawler.py
These are codes :
1.main.py
from chart.chart_crawler import MelonChartCrawler
crawler = MelonChartCrawler()
2.test_chart_crawler.py
from ..chart.chart_crawler import MelonChartCrawler
def test_melon_chart_crawler():
crawler = MelonChartCrawler()
3.chart_crawler.py
import sys
sys.path.append("/Users/Chois/Desktop/Programming/Project/WebScrape/MusicDownloader")
from .. import util
class MelonChartCrawler:
def __init__(self):
pass
4.util.py
def hi():
print("hi")
In MusicDownloader, when I execute main.py by python main.py, it shows errors:
File "main.py", line 1, in <module>
from chart.chart_crawler import MelonChartCrawler
File "/Users/Chois/Desktop/Programming/Project/WebScrape/MusicDownloader/chart/chart_crawler.py", line 4, in <module>
from .. import util
ValueError: attempted relative import beyond top-level package
But when I execute my test code in test directory by py.test test_chart_crawler.py, it works
When I first faced with absolute, relative imports, it seems like very easy and intuitive. But it drives me crazy now. Need your helps. Thanks
The first problem is MusicDownloader not being a package. Add __init__.py to MusicDownloader along with main.py and your relative import ..chart should work. Relative imports work only inside packages, so you can't .. to non-package folder.
Editing my post to provide you with more accurate answer to your answer edit.
It's all about the __name__. Relative imports use __name__ of the module they are used in and the from .(.) part to form a full package/module name to import. Explaining in simple terms importer's __name__ is concatenated with from part, with dots showing how many components of name to ignore/remove, i.e.:
__name__='packageA.packageB.moduleA' of the file containing line: from .moduleB import something, leads to combined value for import packageA.packageB.moduleB, so roughly from packageA.packageB.moduleB import something(but not absolute import as it would be if typed like that directly).
__name__='packageA.packageB.moduleA' of the file containing line: from ..moduleC import something, leads to combined value for import packageA.moduleC, so roughly from packageA.moduleC import something(but not absolute import as it would be if typed like that directly).
Here if it's a moduleB(C) or a packageB(C) doesn't really matter. What's important is that we still have that packageA part which works as an 'anchor' for relative import in both cases. If there will be no packageA part, relative import won't be resolved, and we'll get an error like "Attempted relative import beyond toplevel package".
One more note here, when a module is run it gets a special __name__ value of __main__, which obviously prevents it from solving any relative imports.
Now regarding your case try adding print(__name__) as the very first line to every file and run your files in different scenarios and see how the output changes.
Namely if you run your main.py directly, you'll get:
__main__
chart.chart_crawler
Traceback (most recent call last):
File "D:\MusicDownloader\main.py", line 2, in <module>
from chart.chart_crawler import MelonChartCrawler
File "D:\MusicDownloader\chart\chart_crawler.py", line 2, in <module>
from .. import util
ValueError: Attempted relative import beyond toplevel package
What happened here is... main.py has no idea about MusicDownloader being a package (even after previous edit with adding __init__.py). In your chart_crawler.py: __name__='chart.chart_crawler' and when running relative import with from .. the combined value for package will need to remove two parts (one for every dot) as explained above, so the result will become '' as there're just two parts and no enclosing package. This leads to exception.
When you import a module the code inside it is run, so it's almost the same as executing it, but without the __name__ becoming __main__ and the enclosing package, if there's any, being 'noticed'.
So, the solution is to import main.py as part of the MusicDownloader package. To accomplish the described above, create a module, say named launcher.py on the same level of hierarchy as MusicDownloader folder (near it, not inside it near main.py) with the following code:
print(__name__)
from MusicDownloader import main
Now run launcher.py and see the changes. The output:
__main__
MusicDownloader.main
MusicDownloader.chart.chart_crawler
MusicDownloader.util
Here __main__ is the __name__ inside launcher.py. Inside chart_crawler.py: __name__='MusicDownloader.chart.chart_crawler' and when running relative import with from .. the combined value for package will need to remove two parts (one for every dot) as explained above, so the result will become 'MusicDownloader' with import becoming from MusicDownloader import util. And as we see on the next line when util.py is imported successfully it prints its __name__='MusicDownloader.util'.
So that's pretty much it - "it's all about that __name__".
P.S. One thing not mentioned is why the part with test package worked. It wasn't launched in common way, you used some additional module/program to lauch it and it probably imported it in some way, so it worked. To understand this it's best to see how that program works.
There's a note in official docs:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
I have written a package (http://github.com/anntzer/parsedcmd) that runs with both Python2 and Python3. However, I had to write separate (py.test) unit tests for Python2 and Python3 (mainly because I want to test extra features of Python3, in particular keyword-only arguments), so I have a test_py2.py and a test_py3.py in a test subpackage. Now, if I run, say py.test2 mypkg, test_py2 passes, but test_py3 fails with a SyntaxError. Likewise, for py.test3 mypkg, test_py3 passes but test_py2 fails (I could make this one work though, it's just an issue of StringIO having moved to io).
I can design the test subpackage so that import mypkg.test only imports the proper version of the tests, but apparently py.test doesn't care -- it just sees two files matching test_* and grabs all tests in both of them, ignoring what __init__.py tells him to import.
So right now I have to do both py.test2 mypkg/test/test_py2.py and py.test3 mypkg/test/test_py3.py. Is there a way to set up the whole thing so that py.test2 mypkg and py.test3 mypkg would "just work"?
Thanks.
If you can then making your modules importable on all interpreters and skipping tests as appropriate is a common solution. Otherwise you can put the following as "conftest.py" into the test directory:
import sys
py3 = sys.version_info[0] >= 3
class DummyCollector(pytest.collect.File):
def collect(self):
return []
def pytest_pycollect_makemodule(path, parent):
bn = path.basename
if "py3" in bn and not py3 or ("py2" in bn and py3):
return DummyCollector(path, parent=parent)
This gets picked up a project-specific plugin and will properly ignore a test module with a filename containing a "py2" or "py3" substring on the wrong interpreter version. Of course you can refine it to rather have an explicit list directly in the conftest.py file instead of checking the filename etc.pp.
HTH, holger
You can put your tests in different packages and run only the tests in the appropriate package. Or you can load the appropriate test module in a script:
import sys, unittest
cur_version = sys.version_info
if cur_version[0] < 3:
import myApp.test.test_py2
unittest.TestLoader().loadTestsFromModule(myApp.test.test_py2).run()
else:
import myApp.test.test_py3
unittest.TestLoader().loadTestsFromModule(myApp.test.test_py3).run()
Alternatively, use a setup.py file so you can run:
python setup.py test
and put the versioning logic in there:
versionedTestSuite = "parsedcmd.test.test_py2" # do something as above here
setup(name='parsedcmd',
...
test_suite=versionedTestSuite,
)
A problem I continue to have it "bootstrapping" my tests.
The problem that I have is exactly what this guy has.
The top solution talks about creating a "boostrap" script. I presume that I must then enumerate all of the tests to be run, or use test manifests in the __init__.py files using the __all__ keyword. However, I noticed that the most recent Python documentation on unittest does not talk about __all__ anymore.
In 2.7, we have the python command called "discovery"
python -m unittest discover
That works even nicer. Because:
1) There's no need for Nose
2) There's no need for test manifests
But it doesn't seem to have a way to "bootstrap"
Do I need to use another test runner? One that allows bootstrapping AND discovery?
Do I need py.test?
http://pytest.org/
The reason that I need bootstrapping, is the problem that this guy has. Basically, my import statements don't work right if I run the test directly. I want to execute my suite of tests from the top of my project, just like the app would when it runs normally.
After all, import statements are always relative to their physical location. (BTW, I think this is a hindrance in Python)
Definition: What is Bootstrapping?
Bootstrapping means that I want to do some setup before running any tests at all in the entire project. This is sort of like me asking for a "test setup" at the whole project level.
Update
Here is another posting about the same thing. Using this 2.7 command, we can avoid Nose. But how does one add bootstrapping?
I got it!
Using this one script that I wrote and called it "runtests.py" and placed in my project root, I was able to "bootstrap" that is to run some initialization code AND use discovery. Woot!
In my case, the "bootstrap" code is the two lines that say:
import sys
sys.path.insert(0, 'lib.zip')
Thanks!
#!/usr/bin/python
import unittest
import sys
sys.path.insert(0, 'lib.zip')
if __name__ == "__main__":
all_tests = unittest.TestLoader().discover('.')
unittest.TextTestRunner().run(all_tests)
Here's what I do, and I think it works quite well. For a file/directory structure similar to this:
main_code.py
run_tests.py
/Modules
__init__.py
some_module1.py
some_module2.py
/Tests
__init__.py
test_module1.py
test_module2.py
It's fairly easy to organize your run_tests.py file to bootstrap the tests. First every file with test (test_module1.py, etc.) should implement a function that generates a test suite. Something like:
def suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(Test_Length))
suite.addTest(unittest.makeSuite(Test_Sum))
return suite
at the end of your test code. Then, in the run_tests.py file, you aggregate these into an additional test_suite, and run that:
import unittest
import Tests.test_module1 as test_module1
import Tests.test_module2 as test_module2
module1_test_suite = test_module1.suite()
module2_test_suite = test_module2.suite()
aggregate_suite = unittest.TestSuite()
aggregate_suite.addTest(module1_test_suite)
aggregate_suite.addTest(module2_test_suite)
unittest.TextTestsRunner(verbosity = 2).run(aggregate_suite
Then to run all of these tests, from the command line, simply run
python run_tests.py
Given the following package:
testpackage
__init__.py
testmod.py
testmod2.py
Contents of __init__.py
from . import testmod
from . import testmod2
Contents of testmod.py
# relative import without conditional logic, breaks when run directly
from . import testmod2
...
if __name__ == '__main__':
# do my module testing here
# this code will never execute since the relative import
# always throws an error when run directly
Contents of testmod2.py
if __name__ == 'testpackage.testmod2':
from . import testmod
else:
import testmod
...
if __name__ == '__main__':
# test code here, will execute when the file is run directly
# due to conditional imports
Is this bad? Is there a better way?
That will definitely become a maintenance headache in the future. Not just the conditional imports... more the reason why you're having to do conditional imports, namely that running testpackage/testmod2.py as the main script is causing the first entry of sys.path to be ./testpackage instead of ., which makes the very existence of testpackage as a package go away.
Instead, I'd recommend running testmod2 via python -m testpackage.testmod2, and doing it from outside testpackage. testpackage.testmod2 will still show up as __main__, but the conditional imports will always work because testpackage will always be a package.
The catch with -m is that it requires Python 2.5 or newer.