I have a Python project that looks like this:
my_project
|- my_dir
| |- part_1.py
| |- part_2.py
| |- ...
|- main.py
Every file (main.py and part_1.py, part_2.py, ...) should be able to write to the same log file, with the following two requirements:
The file name should appear in the log message: a line in the log file should look like
ERROR - my_dir.part_1 - Something went wrong in part_1.py
The log level for all loggers should be set programatically from main.py:
# main.py
from my_dir import part_1, part_2, ...
...
logger.setLevel('DEBUG')
result_1 = part_1.my_function() # Logger in part_1.py should be set to DEBUG level
logger.setLevel('WARNING')
result_2 = part_2.my_other_function() # Logger in part_2.py should be set to WARNING level
I accomplish the first requirement by beginning each file in my_dir with
import logging
logger = logging.getLogger(__name__)
Setting the same level for all loggers works if I use a config file and do logging.config.fileConfig('config.ini'). But changes to the log level made in main.py are not propagated to the other loggers. What is the best way to handle this situation?
Related
I want to import constants from a constants module from two different modules, but I get the following error:
Traceback (most recent call last):
File "C:\Temp\tmp\pycircular\pycircular\pycircular.py", line 2, in <module>
from my_classes.foo import Foo
File "C:\Temp\tmp\pycircular\pycircular\my_classes\foo.py", line 1, in <module>
from pycircular.constants import ANOTHER_CONSTANT
File "C:\Temp\tmp\pycircular\pycircular\pycircular.py", line 2, in <module>
from my_classes.foo import Foo
ImportError: cannot import name 'Foo' from partially initialized module 'my_classes.foo' (most likely due to a circular import) (C:\Temp\tmp\pycircular\pycircular\my_classes\foo.py)
My project structure is the following:
|-constants.py
|-my_classes
| |-foo.py
| |-__init__.py
|-pycircular.py
|-__init__.py
# =============
# pycircular.py
# =============
from constants import SOME_CONSTANT
from my_classes.foo import Foo
def main():
print(SOME_CONSTANT)
my_foo = Foo()
my_foo.do_something()
if __name__ == "__main__":
main()
# =============
# foo.py
# =============
from pycircular.constants import ANOTHER_CONSTANT
class Foo:
def do_something(self):
print(ANOTHER_CONSTANT)
# =============
# constants.py
# =============
ANOTHER_CONSTANT = "ANOTHER"
SOME_CONSTANT = "CONSTANT"
I assume that it is the same problem as solved here https://stackoverflow.com/a/62303448/2021763.
But I really do not get why from my_classes.foo import Foo in pycircular.py is called a second time.
Update:
After renaming the package pycircular to pycircular_pack it worked in PyCharm.
But it only works because in Pycharm the option Add content roots to to PYTHONPATH is automatically set.
The output of sys.path is ['C:\\Temp\\tmp\\pycircular\\pycircular_pack', 'C:\\Temp\\tmp\\pycircular', 'C:\\Tools\\miniconda\\envs\\my_env\\python39.zip', 'C:\\Tools\\miniconda\\envs\\my_env\\DLLs', 'C:\\Tools\\miniconda\\envs\\my_env\\lib', 'C:\\Tools\\miniconda\\envs\\my_env', 'C:\\Tools\\miniconda\\envs\\my_env\\lib\\site-packages']
Without the option the output is ['C:\\Temp\\tmp\\pycircular\\pycircular_pack', 'C:\\Tools\\miniconda\\envs\\my_env\\python39.zip', 'C:\\Tools\\miniconda\\envs\\my_env\\DLLs', 'C:\\Tools\\miniconda\\envs\\my_env\\lib', 'C:\\Tools\\miniconda\\envs\\my_env', 'C:\\Tools\\miniconda\\envs\\my_env\\lib\\site-packages']
And without the option I only get it to work with absolute imports.
# pycircular.py
from constants import SOME_CONSTANT
from my_classes.foo import Foo
...
# foo.py
from constants import ANOTHER_CONSTANT
To elaborate based on the comments and edit:
After renaming the package pycircular to pycircular_pack it worked in PyCharm. But it only works because in Pycharm the option Add content roots to to PYTHONPATH is automatically set.
You should make sure the package directory is not set as a content root or source root. The directory hosting the package directory should be set as source root.
C:\Temp\tmp\pycircular # <- source root
|- pycircular_pack # <- not set as anything
| |- constants.py
| |- my_classes
| | |- foo.py
| | |- __init__.py
| |- pycircular.py
| |- __init__.py
|- other_file.py # <- for illustration's sake
Now your sys.path will be set to include C:\Temp\tmp\pycircular only and there will be exactly one way to import things from your module.
Namely,
other_file.py (outside the package) will be able to use the package as pycircular_pack
pycircular_pack/*.py can refer to modules in the pycircular_pack package by either
(e.g.) from .constants import ... (relative import from current package), or
(e.g.) from pycircular_pack.constants import ... (absolute import)
pycircular_pack/my_classes/*.py can refer to modules in the pycircular_pack package by either
(e.g.) from ..constants import ... (relative import from parent package), or
(e.g.) from pycircular_pack.constants import ... (absolute import)
If your pycircular_pack package would contain a runnable script, e.g. a CLI as pycircular_pack/cli.py, then the correct way to run that script on the command line would be to use python -m pycircular_pack.cli; this has Python set up the path just like we want here, where python pycircular_pack/cli.py would not do the right thing.
This is a scenario where a module contains a Logger and it is being imported into another module. When main.py is called however, no LogRecords are written to the log. What can be revised in order for the log to be called?
#objects.py
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
formatter = logging.Formatter()
log_handler = logging.FileHandler('log_objects.log')
log_handler.setFormatter(formatter)
logger.addHandler(log_handler)
class Person:
def __init__(self):
pass
logger.info(f"A new {self.__class__.__name__} is created.")
#main.py
from my_objects import objects
if __name__ == '__main__':
objects.Person()
-dir/
-my_objects/
__init__.py
log_objects.log
objects.py
-main.py
I'm expecting to see this in the log:
"A new Person is created."
Upon execution nothing shows in the log
Since you do not specify a path name for log_objects.log, the file will be created under the directory where the Python interpreter is run, which in this case, is where main.py is located, rather than under the my_objects sub-directory.
If you want log_objects.log to be created under the my_objects directory, you can specify a relative path:
log_handler = logging.FileHandler('my_objects/log_objects.log')
It works as expected. 'log_objects.log' is got created with valid line.
Are you expecting the text to be printed on the console as well? Once specific file handler is added the default handler will be overwritten.
(or) You possibly missing that how module load happens in python
$ python main.py
$ tree .
.
├── log_objects.log
├── main.py
└── my_objects
├── __init__.py
└── objects.py
$ cat log_objects.log
A new Person is created.
Say there is a file tree of:
sample_module
|___ file1.py
|___ file2.py
How can I get file2's __file__, from a function in file1?
file1.py:
def get_file_path():
return __file__
file2.py
from sample_module.file1 import get_file_path
print(get_file_path())
# C:/.../sample_module/file1.py
How can I make it give me C:/.../file2.py?
I understand that the function get_file_path is saving the __file__ into its namespace, and is not generating a new __file__. Is there a way to make __file__ callable, or get the __file__ at runtime?
Why I need it?
I have a library, with a logging module, that generates a logger.
I want the location of that logger to be e.g. 2 folders above file importing generate_logger.
I'd rather not the user have to specify __file__ each time they want to use it.
Is there a better way?
i.e. in mylibrary/logs.py
def generate_logger(logname=__name__, level=logging.DEBUG, path=''):
"""Creates a logger instance. Set level of file handler.
"""
logger = logging.getLogger(logname)
if path == '':
path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../logs'))
fh = logging.FileHandler(f'{path}/{logname}.log')
## rest of logging code.
return logger
-
project
|___ project
| |__ log
| |__ loads_generate_logger (code that does the import)
|_______logs
|_logname.log (creates a log folder 2 folders above where it was imported).
in loads_generate_logger
from mylibrary import generate_logger()
logger = generate_logger()
# logger path = C:/.../venv/lib/site-packages/mylib-0.10/logs
you can try this:
import inspect
current_frame = inspect.currentframe()
current_frame.f_back.f_globals['__file__']
this will get the __file__ global from the function that called your function
I am at my wits end with trying to get all my unittest to run in Python. I have searched about 30 different posts and the unit test documentation but still cannot figure it out.
First I have two test classes that I can run each individually and all the tests pass:
File: unittest.subfolder1.TestObject1.py
class TestObject1(unittest.TestCase):
def test_case1(self):
...some code...
...some assertions...
if __name__ == '__main__':
unittest.main()
File: unittest.subfolder2.TestObject2.py
class TestObject2(unittest.TestCase):
def test_case1(self):
...some code...
...some assertions...
if __name__ == '__main__':
unittest.main()
Starting in the top level directory above 'unittest' I am trying to us unittest.discover to find and run all my tests:
import unittest
loader = unittest.TestLoader()
suite = loader.discover('unittest')
unittest.TextTestRunner().run(suite)
When I do this I get the error `ModuleNotFoundError: No module named 'subfolder1.TestObject1'
What am I doing wrong?
A good approach is to run all the tests in a subdirectory from the command line. In order to find the following files "TestObject1.py, TestObject2.py, ..." in subdirectories, you can run the following command in the command line:
python -m unittest discover -p 'Test*.py'
Additionally, the __init__.py is required within the import and module directories: Python unittest discovery with subfolders
The import unittest is required in the files unittest.subfolder1.TestObject1.py and unittest.subfolder2.TestObject2.py
It is also possible to define explicitly the directory where the discovery starts with the -s parameter:
python -m unittest discover [options]
-s directory Directory to start discovery ('.' default)
-p pattern Pattern to match test files ('test*.py' default)
In case you are using unittest2, it comes with a script unit2. The command line usage is:
unit2 discover unit2 -v test_module
Do not name your directory unittest, it may conflict with the standard library.
You also need to create a file named __init__.py in all of your directories (subfolder1, etc.), so they become packages and their content can be imported.
So I had to do my own workaround but at least I can get them all to run with the above file structure. It requires that I reinstantiate the TestLoader and the TestSuite each time I give it a new file path, so first I need to collect all relevant file paths in the unittest directory.
import os
import unittest
import traceback
class UnitTestLauncher(object):
def runTests(self):
#logging.INFO("Running unit tests...")
lsPaths = []
#Find all relevant subdirectories that contain unit tests
#Exclude 'unittest' directory, but include subdirectories, with code `path != 'unittest'`
for path,subdirs,files in os.walk('unittest'):
if "pycache" not in path and path != 'unittest':
lsPaths.append(path)
#loop through subdirectories and run individually
for path in lsPaths:
loader = unittest.TestLoader()
suite = unittest.TestSuite()
suite = loader.discover(path)
unittest.TextTestRunner().run(suite)
This solution is not perfect and each different directory comes out as a line of output so you have to look through each line manually for failed tests.
Old question but, oh, so current. I am new to Python, coming from strong typed languages and while the language itself is ok(ish), the conventions, tools and workarounds to make everything work in the ecosystem can drive you nuts. I struggled myself with running unit tests from separate subdirectories and this is the way I solved it.
First, the code you test, package it into a package. Organize your directories like this:
Work
|
+---PkToTest
| |
| +--- __init__.py
| +--- a.py
| +--- <other modules>.py
|
+---Tests (for PKToTest)
|
+--- test_a.py
PkToTest becomes a package due to the init.py file. In test_a.py make sure your sys.path will contain the path to PkToTest (absolute path not relative). I did that by:
import sys
sys.path.insert(0, "<absolute path to parent of PkTotest directory>")
import unittest
from PkToTest import a
class aTestSuite(unittest.TestCase):
def test1(self):
self.assertEqual(a.fnToTest(), ...)
Testing All Subdirectories
Given a structure of:
my_package
|
|
controller
|-- validator.py
|
validator
|-- controller.py
|
test
|-- controller
|
|-- __init__.py (MAKE SURE THIS EXISTS OR unittest MODULE WOULD NOT KNOW)
|-- test_controller.py
|
|-- validator
|
|-- __init__.py (MAKE SURE THIS EXISTS OR unittest MODULE WOULD NOT KNOW)
|-- test_validator.py
|
then just run
python -m unittest discover -s my_package/test
What this does is to test and -s means to start with the my_package/test as the starting directory
In my project all folders are folders (not modules) and they have the structure:
Folder > Subfolder > Subfolder > Tests > test_xxxx.py
Folder > Subfolder > Subfolder > xxxx.py
So i modified the answer from here, and also took a part from How do I run all Python unit tests in a directory? and came up with this:
import os, unittest
testFolderPaths = []
for path, subdirs, files in os.walk(os.getcwd()):
for file in files:
if file.startswith("test_") and file.endswith(".py"):
testFolderPaths.append(path)
for path in testFolderPaths:
print(f"Running tests from {path}...")
loader = unittest.TestLoader()
suite = loader.discover(path)
runner = unittest.TextTestRunner()
result = runner.run(suite)
print(f"RUN {result.testsRun} Tests. PASSED? {result.wasSuccessful()}")
If any tests fail it will throw and error showing which one exactly failed.
Let us say that we have the following directory structure ...
+-- main.py
|
+--+ ./web
| |
| +--- ./web/bottleApp.py
Currently, I want to organize the files so that the I can separate different functionality in different areas. Template main.py and ./web/bottleApp.py look like the following ...
This is the ./web/bottleApp.py file:
import bottle
app = bottle.Bottle()
#app.route('/')
def root():
return 'This is the root application'
# some additional functions here ...
And this is the main.py file ...
from web import bottleApp as app
with app.app as report:
# Some random routes here ...
report.run(host = 'localhost', port=8080)
Now I want to add another folder which can handle some functions which I may optionally use is a bunch of my projects, (for example configuration file handling via the web interface just created)
Let us say we want to insert the following folder/file configuration ...
+-- main.py
|
+--+ ./web
| |
| +--- ./web/bottleApp.py
|
+--+ ./configure
|
+--- ./configure/config.py
Given the original app = bottle.Bottle() I want to create the following sample route in the file ./configure/config.py:
#app.route('/config/config1')
def config1():
return 'some config data'
How do I even go about doing this? Once I run the main.py file, how do I make sure that the other routes are available?
Bottle can run multiple bottle apps as a single instance.
You can use something like this on main.py
import bottle
from web.bottleApp import app
from configure.config import configure_app
main = bottle.Bottle()
main.mount("/config/",configure)
main.mount("/",app)
main.run(host = 'localhost', port=8080)
and on configure/config.py something like this:
import bottle
config_app = bottle.Bottle()
#config_app.route('/config1')
def config1():
return 'some config data'