We Keep Coding

Group Django commands to a folder inside the same app

Is it allowed to group custom Django commands to separate folders inside the same Django app?
I have a lot of them and wanted to group them logically by purpose. Created folders but Django can't find them.
Maybe I'm trying to run them wrong. Tried:
python manage.py process_A_related_data
the same plus imported all commands in __init__.py
python manage.py folderA process_A_related_data
python manage.py folderA.process_A_related_data
python manage.py folderA/process_A_related_data
Got following error:
Unknown command: 'folderA/process_A_related_data'
Type 'manage.py help' for usage.

I think you can create a basic custom command which will run other commands from relevent folders. Here is an approach you can take:
First make a folder structure like this:
management/
commands/
folder_a/
process_A_related_data.py
folder_b/
process_A_related_data.py
process_data.py
Then inside process_data.py, update the command like this:
from django.core import management
from django.core.management.base import BaseCommand
import importlib
class Command(BaseCommand):
help = 'Folder Process Commands'
def add_arguments(self, parser):
parser.add_argument('-u', '--use', type=str, nargs='?', default='folder_a.process_A_related_data')
def handle(self, *args, **options):
try:
folder_file_module = options['use'] if options['use'].startswith('.') else '.' + options['use']
command = importlib.import_module(folder_file_module, package='your_app.management.commands')
management.call_command(command.Command())
except ModuleNotFoundError:
self.stderr.write(f"No relevent folder found: {e.name}")
Here I am using call_command method to call other managment commands.
Then run commands like this:
python manage.py process_data --use folder_a.process_A_related_data
Finally, if you want to run commands like python manage.py folder_a.process_A_related_data, then probably you need to change in manage.py. Like this:
import re
...
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
if re.search('folder_[a-z].*', sys.argv[-1]):
new_arguments = sys.argv[:-1] + ['process_data','--use', sys.argv[-1]]
execute_from_command_line(new_arguments)
else:
execute_from_command_line(sys.argv)

You should be able to partition the code by using mixins (I have not tried this in this context, though)
A standard management command looks like
from django.core.management.base import BaseCommand
class Command(BaseCommand):
help = 'FIXME A helpful comment goes here'
def add_arguments(self, parser):
parser.add_argument( 'name', ...)
# more argument definitions
def handle(self, *args, **options):
# do stuff
Which can probably be replaced by a "stub" in app/management/commands:
from wherever.commands import FooCommandMixin
from django.core.management.base import BaseCommand
class Command(FooCommandMixin, BaseCommand):
# autogenerated -- do not put any code in here!
pass
and in wherever/commands
class FooCommandMixin( object):
help = 'FIXME A helpful comment goes here'
def add_arguments(self, parser):
parser.add_argument( 'name', ...)
# more argument definitions
def handle(self, *args, **options):
# do the work
It would not be hard to write a script to go through a list of file names or paths (using glob.glob) using re.findall to identify appropriate class declarations, and to (re)generate a matching stub for each in the app's management/commands folder.
Also/instead Python's argparse allows for the definition of sub-commands. So you should be able to define a command that works like
./manage.py foo bar --aa --bb something --cc and
./manage.py foo baz --bazzy a b c
where the syntax after foo is determined by the next word (bar or baz or ...). Again I have no experience of using subcommands in this context.

I found no mention of support for this feature in the release notes. It looks to be that this is still not supported as of version Django 3.0. I would suggest that you use meaningful names for your files that help you specify. You could always come up w/ a naming convention!

A workaround could be: create a specific Django "satellite" app for each group of management commands.
In recent version of Django, the requirements for a Python module to be an app are minimal: you won't need to provide any fake models.py or other specific files as happened in the old days.
While far from perfect from a stylistic point of view, you still gain a few advantages:
no need to hack the framework at all
python manage.py will list the commands grouped by app
you can control the grouping by providing suitable names to the apps
you can use these satellite apps as container for specific unit tests
I always try to avoid fighting against the framework, even when this means to compromise, and sometimes accept it's occasional design limitations.

python nose test reporting in teamcity

I have a script call run_test.py, here's the content:-
if __name__ == '__main__':
nose.main(argv=sys.argv)
Running all my tests is as simple as doing this:
run_test.py unittests/test_*.py
I'm trying to now incorperate the output reporting for this into teamcity.
I'm referring to this https://github.com/JetBrains/teamcity-messages
I tried changing all my unittests/test_*.py program following the documentation. It works if running the test individually like this:-
unittest/test_one.py
But it does not work when running it thru nose, like this:
run_test.py unittest/test_one.py
According to the documentation link, it says that nose reporting is enabled automatically under TeamCity build. I don't quite get what that means.
Is there anything that i'm missing out here?
Any help is greatly appreciated. Thanks.

have a look at the xunit plugin of nose.
it will generate an xml file with the results => which jenkins and teamcity can use.
some documentation for teamcity
this post tells you how to enable the plugin in your test script
if __name__ == '__main__':
argv = sys.argv[:]
argv.insert(1, "--with-xunit")
nose.main(argv=argv)

I finally found out the way to achieve that.
Here's what i modified in my run_test.py
#!/usr/bin/env python
import os
import sys
import unittest
from teamcity import is_running_under_teamcity
from teamcity.unittestpy import TeamcityTestRunner
loader = unittest.TestLoader()
start_dir = sys.argv[1]
suite = loader.discover(start_dir, pattern='test_*.py')
#runner = unittest.TextTestRunner()
runner = TeamcityTestRunner(verbosity=2)
runner.run(suite)

Why is my twisted plugin not appearing when I run the `twistd` command with no options?

Here is the current state of my twistd plugin, which is located in project_root/twisted/plugins/my_plugin.py:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from zope.interface import implements
from twisted.plugin import IPlugin
from twisted.python.usage import Options
from twisted.application import internet, service
from mylib.io import MyFactory
class Options(Options):
"""Flags and options for the plugin."""
optParameters = [
('sock', 's', '/tmp/io.sock', 'Path to IO socket'),
]
class MyServiceMaker(object):
implements(service.IServiceMaker, IPlugin)
tapname = "myplugin"
description = "description for my plugin"
options = Options
def makeService(self, options):
return internet.UNIXServer(options['sock'], MyFactory())
There is no __init__.py file in project_root/twisted/plugins/
The output of twistd doesn't show my plugin when run from the project's root directory
I installed my library via python setup.py develop --user and it is importable from anywhere
Any ideas?

As suspected, it was something very simple: I needed to instantiate an instance of MyServiceMaker, so simply adding service_maker = MyServiceMaker() at the bottom of the script fixes the issue.

How to add a custom nose plugin to the `nosetests` command

So I'm very noob in dealing with nose plugins.
I've been searching a lot but docs regarding nose plugins seem scarce.
I read and tried what's in the following links to try to write a simple nose plugin
and run it with nosetests, without success:
https://nose.readthedocs.org/en/latest/doc_tests/test_init_plugin/init_plugin.html
https://nose.readthedocs.org/en/latest/plugins/writing.html
I don't want to write my own test-runner or run the tests from any other script (via run(argv=argv, suite=suite(), ...)),
like they do in the first link.
I wrote a file myplugin.py with a class like this:
import os
from nose.plugins import Plugin
class MyCustomPlugin(Plugin):
name = 'myplugin'
def options(self, parser, env=os.environ):
parser.add_option('--custom-path', action='store',
dest='custom_path', default=None,
help='Specify path to widget config file')
def configure(self, options, conf):
if options.custom_path:
self.make_some_configs(options.custom_path)
self.enabled = True
def make_some_configs(self, path):
# do some stuff based on the given path
def begin(self):
print 'Maybe print some useful stuff...'
# do some more stuff
and added a setup.py like this:
try:
from setuptools import setup, find_packages
except ImportError:
import distribute_setup
distribute_setup.use_setuptools()
from setuptools import setup, find_packages
setup(
name='mypackage',
...
install_requires=['nose==1.3.0'],
py_modules=['myplugin'],
entry_points={
'nose.plugins.1.3.0': [
'myplugin = myplugin:MyCustomPlugin'
]
}
)
Both files are in the same directory.
Every time I run nosetests --custom-path [path], I get:
nosetests: error: no such option: --custom-path
From the links mentioned above, I thought that's all that was required to register and enable a custom plugin.
But it seems that, either I'm doing something really wrong, or nose's docs are outdated.
Can someone please point me the correct way to register and enable a plugin, that I can use with nosetests?
Thanks a lot!! :)

You don't want the nose version in entry_points in setup.py. Just use nose.plugins.0.10 as the docs say. The dotted version in the entry point name is not so much a nose version as a plugin API version.

Packaging a Python library

I have a few Munin plugins which report stats from an Autonomy database. They all use a small library which scrapes the XML status output for the relevant numbers.
I'm trying to bundle the library and plugins into a Puppet-installable RPM. The actual RPM-building should be straightforward; once I have a distutils-produced distfile I can make it into an RPM based on a .spec file pinched from the Dag or EPEL repos [1]. It's the distutils bit I'm unsure of -- in fact I'm not even sure my library is correctly written for packaging. Here's how it works:
idol7stats.py:
import datetime
import os
import stat
import sys
import time
import urllib
import xml.sax
class IDOL7Stats:
cache_dir = '/tmp'
def __init__(self, host, port):
self.host = host
self.port = port
# ...
def collect(self):
self.data = self.__parseXML(self.__getXML())
def total_slots(self):
return self.data['Service:Documents:TotalSlots']
Plugin code:
from idol7stats import IDOL7Stats
a = IDOL7Stats('db.example.com', 23113)
a.collect()
print a.total_slots()
I guess I want idol7stats.py to wind up in /usr/lib/python2.4/site-packages/idol7stats, or something else in Python's search path. What distutils magic do I need? This:
from distutils.core import setup
setup(name = 'idol7stats',
author = 'Me',
author_email = 'me#example.com',
version = '0.1',
py_modules = ['idol7stats'])
almost works, except the code goes in /usr/lib/python2.4/site-packages/idol7stats.py, not a subdirectory. I expect this is down to my not understanding the difference between modules/packages/other containers in Python.
So, what's the rub?
[1] Yeah, I could just plonk the library in /usr/lib/python2.4/site-packages using RPM but I want to know how to package Python code.

You need to create a package to do what you want. You'd need a directory named idol7stats containing a file called __init__.py and any other library modules to package. Also, this will affect your scripts' imports; if you put idol7stats.py in a package called idol7stats, then your scripts need to "import idol7stats.idol7stats".
To avoid that, you could just rename idol7stats.py to idol7stats/__init__.py, or you could put this line into idol7stats/__init__.py to "massage" the imports into the way you expect them:
from idol7stats.idol7stats import *