We Keep Coding

How to modify pytest arguments?

I found out for this purpose I can use PyTest function pytest_load_initial_conftests()
https://docs.pytest.org/en/latest/example/simple.html#dynamically-adding-command-line-options
But I can't implement this example (see link) correctly.
pytest_load_initial_conftests() doesn't even start (looked through debug).
Tests run ordinary without any params (one thread), but I expected "-n" param.
I installed pytest and xdist.
Only two file in project. There are no pytest.ini.
What am I doing wrong? Please help run it.
conftest.py
import pytest
import os
import sys
def pytest_addoption(parser):
parser.addoption('--some_param', action='store', help='some_param', default='')
def pytest_configure(config):
some_param = config.getoption('--some_param')
def pytest_load_initial_conftests(args):
if "xdist" in sys.modules:
import multiprocessing
num = max(multiprocessing.cpu_count() / 2, 1)
args[:] = ["-n", str(num)] + args
test_t1.py
import inspect
from time import sleep
import os
import pytest
class Test_Run:
def test_1(self):
body()
def test_2(self):
body()
def test_3(self):
body()
def test_4(self):
body()
def setup(self):
pass
def teardown(self):
pass
def body():
sleep(5)

According to the docs on pytest_load_initial_conftests:
Note: This hook will not be called for conftest.py files, only for
setuptools plugins.
https://docs.pytest.org/en/latest/reference/reference.html#pytest.hookspec.pytest_load_initial_conftests
Probably it shouldn't be mentioned on that page that you found.
Edit: update docs url

Add extra plugin to make pytest arguments dynamic
As per the API documentation, pytest_load_initial_conftests hook will not be called in conftest.py file and it can be used in plugins only.
Further, pytest documentation mentions how to write custom plugin for pytest and make it installable.
Following this:
create following files in root directory
- ./setup.py
- ./plugin.py
- ./tests/conftest.py
- ./pyproject.toml
# contents of ./setup.py
from setuptools import setup
setup(
name='my_project',
version='0.0.1',
entry_points={
'console_scripts': [
], # feel free to add if you have any
"pytest11": ["custom_args = plugin"]
},
classifiers=["Framework :: Pytest"],
)
notice python11 here, it is reserved for adding pytest plugins as I have read.
# contents of ./plugin.py
import sys
def pytest_load_initial_conftests(args):
if "xdist" in sys.modules:
import multiprocessing
num = max(multiprocessing.cpu_count() / 2, 1)
args[:] = ["-n", str(num)] + args
# contents of ./tests/conftest.py
pytest_plugins = ["custom_args"] # allows to load plugin while running tests
# ... other fixtures and hooks
finally, pyproject.toml file for the project
# contents of ./pyproject.toml
[tool.setuptools]
py-modules = []
[tool.setuptools]
py-modules = []
[build-system]
requires = [
"setuptools",
]
build-backend = "setuptools.build_meta"
[project]
name = "my_package"
description = "My package description"
readme = "README.md"
requires-python = ">=3.8"
classifiers = [
"Framework :: Flask",
"Programming Language :: Python :: 3",
]
dynamic = ["version"]
This will Dynamically add -n argument with value which enables parallel running based on the number of CPUs your system has.
Hope this helps, feel free to comment.

How can I share matplotlib style?

One can load a custom plotting style in matplotlib with something like:
>>> import matplotlib.pyplot as plt
>>> plt.style.use('ggplot')
And I know that I can create my own, http://matplotlib.org/users/style_sheets.html explains how.
Let's say that I create an amazing matplotlib style -- how can I share this with other people? Is there a way to do it with pip/conda or something else appropriate?
The docs include the suggestion to "create custom styles and use them by calling style.use with the path or URL to the style sheet." -- so I guess that I could maintain a link w/ that on some public git repository, and people would just get the most recent style if they put that URL?

You could organize your code in a structure like this:
|
└─── setup.py
└─── mplstyles
style_01.mplstyle
style_02.mplstyle
Then, in your file setup.py write something like the following:
# -*- coding: utf-8 -*-
import matplotlib as mpl
import glob
import os.path
import shutil
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-install', action='store_true', default=True)
parser.add_argument('-upgrade', action='store_true')
options = parser.parse_args()
#~ # ref -> matplotlib/style/core
BASE_LIBRARY_PATH = os.path.join(mpl.get_data_path(), 'stylelib')
STYLE_PATH = os.path.join(os.getcwd(),'mplstyles')
STYLE_EXTENSION = 'mplstyle'
style_files = glob.glob(os.path.join(STYLE_PATH,"*.%s"%(STYLE_EXTENSION)))
for _path_file in style_files:
_, fname = os.path.split(_path_file)
dest = os.path.join(BASE_LIBRARY_PATH, fname)
if not os.path.isfile(dest) and options.install:
shutil.copy(_path_file, dest)
print("%s style installed"%(fname))
elif options.upgrade:
shutil.copy(_path_file, dest)
print("%s style upgraded"%(fname))
elif os.path.isfile(dest):
print("%s style already exists (use -upgrade to upgrade)"%(fname))
else:
pass # ¿?
The code above copy each .mplstyle (or stylesheet) file from "mplstyles" folder to Matplotlib installation directory.
"Install" styles
>> python setup.py -install
"Upgrade" styles
>> python setup.py -upgrade

I just had this exact same question. A petty that was not been solved yet. I have found a solution to be able to distribute the style(s) using PyPi (in my case goosempl, also on GitHub).
I have created a Python module of which the mplstyle-files are a part:
|-- setup.py
|-- package_name
| |-- __init__.py
| |-- styles
| | |-- example.mplstyle
The idea is now:
The .mplstyle file(s) gets packaged with the module.
The module gets installed.
At the end of the installation a small script runs that extracts the .mplstyle file(s) from the newly installed package and writes them to the matplotlib config directory.
Here are the essentials
setup.py
import atexit
from setuptools import setup
from setuptools.command.install import install
def _post_install():
import goosempl
package_name.copy_style()
class new_install(install):
def __init__(self, *args, **kwargs):
super(new_install, self).__init__(*args, **kwargs)
atexit.register(_post_install)
__version__ = '0.1.0'
setup(
name = 'package_name',
version = __version__,
...
install_requires = ['matplotlib>=2.0.0'],
packages = ['package_name'],
cmdclass = {'install': new_install},
package_data = {'package_name/styles':[
'package_name/styles/example.mplstyle',
]},
)
init.py
def copy_style():
import os
import matplotlib
from pkg_resources import resource_string
files = [
'styles/example.mplstyle',
]
for fname in files:
path = os.path.join(matplotlib.get_configdir(),fname)
text = resource_string(__name__,fname).decode()
open(path,'w').write(text)
For future reference see these related questions / documentation:
Why use package_data and not data_files.
Running a post-installation script/command that relies on the installed package
Uploading to PyPi

Why is my twisted plugin not appearing when I run the `twistd` command with no options?

Here is the current state of my twistd plugin, which is located in project_root/twisted/plugins/my_plugin.py:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from zope.interface import implements
from twisted.plugin import IPlugin
from twisted.python.usage import Options
from twisted.application import internet, service
from mylib.io import MyFactory
class Options(Options):
"""Flags and options for the plugin."""
optParameters = [
('sock', 's', '/tmp/io.sock', 'Path to IO socket'),
]
class MyServiceMaker(object):
implements(service.IServiceMaker, IPlugin)
tapname = "myplugin"
description = "description for my plugin"
options = Options
def makeService(self, options):
return internet.UNIXServer(options['sock'], MyFactory())
There is no __init__.py file in project_root/twisted/plugins/
The output of twistd doesn't show my plugin when run from the project's root directory
I installed my library via python setup.py develop --user and it is importable from anywhere
Any ideas?

As suspected, it was something very simple: I needed to instantiate an instance of MyServiceMaker, so simply adding service_maker = MyServiceMaker() at the bottom of the script fixes the issue.

How to extract dependencies information from a setup.py

I have a python project, let's call it foobar, there is a setup.py script in project root directory like all Python projects. For example
foobar
setup.py
setup.py file content:
from ez_setup import use_setuptools
use_setuptools()
from setuptools import setup, find_packages
setup(
name='foobar',
version='0.0.0',
packages=find_packages(),
install_requires=[
'spam==1.2.3',
'eggs>=4.5.6',
],
)
I need to get dependencies information from that setup.py file using Python. The part I want would be
[
'spam==1.2.3',
'eggs>=4.5.6',
]
in the example above. I don't want to install this package, all I need is the dependencies information. Certainly, I can use regular expression to parse it, but that would be ugly, I can also use Python AST to parse it, but I think there should already be some tool can do this. What is the best way to do so?

It seems to me that you could use mock to do the work (assuming that you have it installed and that you have all the setup.py requirements...). The idea here is to just mock out setuptools.setup and inspect what arguments it was called with. Of course, you wouldn't really need mock to do this -- You could monkey patch setuptools directly if you wanted ...
import mock # or `from unittest import mock` for python3.3+.
import setuptools
with mock.patch.object(setuptools, 'setup') as mock_setup:
import setup # This is setup.py which calls setuptools.setup
# called arguments are in `mock_setup.call_args`
args, kwargs = mock_setup.call_args
print kwargs.get('install_requires', [])

You can use distutils.core's run_setup:
from distutils.core import run_setup
result = run_setup("./setup.py", stop_after="init")
result.install_requires
['spam==1.2.3', 'eggs>=4.5.6']
This way there is no need to mock anything and you can potentially extract more information about the project than would be possible by mocking the setup() call.
Note that this solution might be problematic as there apparently is active work being done to deprecate distutils. See comments for details.

Pretty similar idea to #mgilson's solution, I use ast, parse setup.py module, insert a mock setup method before setup call, and collect the args and kwargs.
import ast
import textwrap
def parse_setup(setup_filename):
"""Parse setup.py and return args and keywords args to its setup
function call
"""
mock_setup = textwrap.dedent('''\
def setup(*args, **kwargs):
__setup_calls__.append((args, kwargs))
''')
parsed_mock_setup = ast.parse(mock_setup, filename=setup_filename)
with open(setup_filename, 'rt') as setup_file:
parsed = ast.parse(setup_file.read())
for index, node in enumerate(parsed.body[:]):
if (
not isinstance(node, ast.Expr) or
not isinstance(node.value, ast.Call) or
node.value.func.id != 'setup'
):
continue
parsed.body[index:index] = parsed_mock_setup.body
break
fixed = ast.fix_missing_locations(parsed)
codeobj = compile(fixed, setup_filename, 'exec')
local_vars = {}
global_vars = {'__setup_calls__': []}
exec(codeobj, global_vars, local_vars)
return global_vars['__setup_calls__'][0]

Great answers here already, but I had to modify the answer from #mgilson just a bit to get it to work for me because there are (apparently) incorrectly configured projects in my source tree, and the wrong setup was being imported. In my solution I am temporarily creating a copy of the setup.py file by another name so that I can import it and the mock can intercept the correct install_requires data.
import sys
import mock
import setuptools
import tempfile
import os
def import_and_extract(parent_dir):
sys.path.insert(0, parent_dir)
with tempfile.NamedTemporaryFile(prefix="setup_temp_", mode='w', dir=parent_dir, suffix='.py') as temp_fh:
with open(os.path.join(parent_dir, "setup.py"), 'r') as setup_fh:
temp_fh.write(setup_fh.read())
temp_fh.flush()
try:
with mock.patch.object(setuptools, 'setup') as mock_setup:
module_name = os.path.basename(temp_fh.name).split(".")[0]
__import__(module_name)
finally:
# need to blow away the pyc
try:
os.remove("%sc"%temp_fh.name)
except:
print >> sys.stderr, ("Failed to remove %sc"%temp_fh.name)
args, kwargs = mock_setup.call_args
return sorted(kwargs.get('install_requires', []))
if __name__ == "__main__":
if len(sys.argv) > 1:
thedir = sys.argv[1]
if not os.path.isdir(thedir):
thedir = os.path.dirname(thedir)
for d in import_and_extract(thedir):
print d
else:
print >> sys.stderr, ("syntax: %s directory"%sys.argv[0])

If you need only to parse setup.cfg, you can use the configparser module in the standard library:
>>> import configparser
>>> config = configparser.ConfigParser()
>>> config.read('setup.cfg')
['setup.cfg']
>>> config.sections()
['metadata', 'options', 'options.extras_require', 'options.package_data', 'options.packages.find']
>>> config['options']['install_requires']
'pandas'

Using run_setup from distutils.core is great, but some setup.py files do some additional actions before running setup(...), so parsing the code seems the only solution.
import ast
import sys
path = sys.argv[1]
parsed = ast.parse(open(path).read())
for node in parsed.body:
if not isinstance(node, ast.Expr):
continue
if not isinstance(node.value, ast.Call):
continue
if node.value.func.id != "setup":
continue
for keyword in node.value.keywords:
if keyword.arg == "install_requires":
requirements = ast.literal_eval(keyword.value)
print("\n".join(requirements))

How to add a custom nose plugin to the `nosetests` command

So I'm very noob in dealing with nose plugins.
I've been searching a lot but docs regarding nose plugins seem scarce.
I read and tried what's in the following links to try to write a simple nose plugin
and run it with nosetests, without success:
https://nose.readthedocs.org/en/latest/doc_tests/test_init_plugin/init_plugin.html
https://nose.readthedocs.org/en/latest/plugins/writing.html
I don't want to write my own test-runner or run the tests from any other script (via run(argv=argv, suite=suite(), ...)),
like they do in the first link.
I wrote a file myplugin.py with a class like this:
import os
from nose.plugins import Plugin
class MyCustomPlugin(Plugin):
name = 'myplugin'
def options(self, parser, env=os.environ):
parser.add_option('--custom-path', action='store',
dest='custom_path', default=None,
help='Specify path to widget config file')
def configure(self, options, conf):
if options.custom_path:
self.make_some_configs(options.custom_path)
self.enabled = True
def make_some_configs(self, path):
# do some stuff based on the given path
def begin(self):
print 'Maybe print some useful stuff...'
# do some more stuff
and added a setup.py like this:
try:
from setuptools import setup, find_packages
except ImportError:
import distribute_setup
distribute_setup.use_setuptools()
from setuptools import setup, find_packages
setup(
name='mypackage',
...
install_requires=['nose==1.3.0'],
py_modules=['myplugin'],
entry_points={
'nose.plugins.1.3.0': [
'myplugin = myplugin:MyCustomPlugin'
]
}
)
Both files are in the same directory.
Every time I run nosetests --custom-path [path], I get:
nosetests: error: no such option: --custom-path
From the links mentioned above, I thought that's all that was required to register and enable a custom plugin.
But it seems that, either I'm doing something really wrong, or nose's docs are outdated.
Can someone please point me the correct way to register and enable a plugin, that I can use with nosetests?
Thanks a lot!! :)

You don't want the nose version in entry_points in setup.py. Just use nose.plugins.0.10 as the docs say. The dotted version in the entry point name is not so much a nose version as a plugin API version.