I build netifaces module with "python setup.py buiild". But during two build process, the output path is different.
First build, the output path is lib.linux-x86_64-3.9
Second build, the output path is lib.linux-x86_64-cpython-39
I want to know how python setup.py determines output path？
Thanks.
I don't know the reason why the build path changes. Unfortunately, I'm unable to replicate your experience. I created this dockerfile in an attempt to replacate:
FROM jaraco/multipy-tox
RUN pipx install httpie
RUN http GET https://files.pythonhosted.org/packages/a6/91/86a6eac449ddfae239e93ffc1918cf33fd9bab35c04d1e963b311e347a73/netifaces-0.11.0.tar.gz | tar xz
WORKDIR netifaces-0.11.0
RUN pip-run -q setuptools -- setup.py build > build1.txt
RUN pip-run -q setuptools -- setup.py build > build2.txt
CMD diff --unified build1.txt build2.txt
It produces this output:
$ docker run -it #$(docker build -q .)
--- build1.txt 2022-09-05 14:54:42.639229009 +0000
+++ build2.txt 2022-09-05 14:54:43.869229009 +0000
## -1,16 +1,12 ##
running build
running build_ext
-checking for getifaddrs...found.
-checking for getnameinfo...found.
-checking for IPv6 socket IOCTLs...not found.
-checking for optional header files...netash/ash.h netatalk/at.h netax25/ax25.h neteconet/ec.h netipx/ipx.h netpacket/packet.h netrose/rose.h linux/atm.h linux/llc.h linux/tipc.h linux/dn.h.
-checking whether struct sockaddr has a length field...no.
-checking which sockaddr_xxx structs are defined...at ax25 in in6 ipx un rose ash ec ll atmpvc atmsvc dn llc.
-checking for routing socket support...no.
-checking for sysctl(CTL_NET...) support...no.
-checking for netlink support...yes.
+checking for getifaddrs...found. (cached)
+checking for getnameinfo...found. (cached)
+checking for IPv6 socket IOCTLs...not found. (cached)
+checking for optional header files...netash/ash.h netatalk/at.h netax25/ax25.h neteconet/ec.h netipx/ipx.h netpacket/packet.h netrose/rose.h linux/atm.h linux/llc.h linux/tipc.h linux/dn.h. (cached)
+checking whether struct sockaddr has a length field...no. (cached)
+checking which sockaddr_xxx structs are defined...at ax25 in in6 ipx un rose ash ec ll atmpvc atmsvc dn llc. (cached)
+checking for routing socket support...no. (cached)
+checking for sysctl(CTL_NET...) support...no. (cached)
+checking for netlink support...yes. (cached)
will use netlink to read routing table
-building 'netifaces' extension
-aarch64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DNETIFACES_VERSION=0.11.0 -DHAVE_GETIFADDRS=1 -DHAVE_GETNAMEINFO=1 -DHAVE_NETASH_ASH_H=1 -DHAVE_NETATALK_AT_H=1 -DHAVE_NETAX25_AX25_H=1 -DHAVE_NETECONET_EC_H=1 -DHAVE_NETIPX_IPX_H=1 -DHAVE_NETPACKET_PACKET_H=1 -DHAVE_NETROSE_ROSE_H=1 -DHAVE_LINUX_ATM_H=1 -DHAVE_LINUX_LLC_H=1 -DHAVE_LINUX_TIPC_H=1 -DHAVE_LINUX_DN_H=1 -DHAVE_SOCKADDR_AT=1 -DHAVE_SOCKADDR_AX25=1 -DHAVE_SOCKADDR_IN=1 -DHAVE_SOCKADDR_IN6=1 -DHAVE_SOCKADDR_IPX=1 -DHAVE_SOCKADDR_UN=1 -DHAVE_SOCKADDR_ROSE=1 -DHAVE_SOCKADDR_ASH=1 -DHAVE_SOCKADDR_EC=1 -DHAVE_SOCKADDR_LL=1 -DHAVE_SOCKADDR_ATMPVC=1 -DHAVE_SOCKADDR_ATMSVC=1 -DHAVE_SOCKADDR_DN=1 -DHAVE_SOCKADDR_LLC=1 -DHAVE_PF_NETLINK=1 -I/usr/include/python3.11 -c netifaces.c -o build/temp.linux-aarch64-cpython-311/netifaces.o
-creating build/lib.linux-aarch64-cpython-311
-aarch64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-aarch64-cpython-311/netifaces.o -L/usr/lib/aarch64-linux-gnu -o build/lib.linux-aarch64-cpython-311/netifaces.cpython-311-aarch64-linux-gnu.so
If I remove the build file between builds, there's no diff.
The output path is determined by setuptools or possibly by distutils, but I also observe that netifaces has a giant setup.py file, so it's entirely possible the behavior affecting the output directory is there.
What I recommend is to:
(a) Determine if the behavior you're observing is unique to netifaces or happens to other (simpler) packages with an extension module.
(b) Using a simpler package with an extension module, create a set of steps that replicate the behavior you observe.
(c) Put breakpoints or print statements at various points in the distutils/setuptools code to trace the behavior and determine where the output dir is set and how it changes.
Related
I'm trying to compile both C and C++ sources at the same time in Cython. This is my current setup:
-setup.py
from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
import os
language = "c++"
extra_compile_flags = ["-std=c++17"]
os.environ["CC"] = "clang++"
ext_modules = [
Extension(
name="Dummy",
sources=["mydummy.pyx", "source1.cpp","source2.c"],
language=language,
extra_compile_args=extra_compile_flags,
)
]
ext_modules = cythonize(ext_modules)
setup(
name="myapp",
ext_modules=ext_modules,
)
-compile command:
python3 setup.py build_ext --inplace --verbose
In the log I get the following message:
clang++ -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/usr/include/python3.6m -I/usr/include/python3.6m -c /path/source2.c -o build/temp.linux-x86_64-3.6/./path/source2.o -std=c++17 -O3
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
The compilation goes trough, but the warning looks pretty nasty. How can I get rid of it?
The naive solution
os.environ["CC"] = "clang"
os.environ["CXX"] = "clang++"
is failing with error: invalid argument '-std=c++17' not allowed with 'C'
gcc (or clang) is just a frontend and it calls the appropriate compiler (c- or c++-compiler) given the file-extension of the compiled file (.c means it should be compiled with c-compiler and .cpp means it should be compiled with c++-compiler), see for example this similar SO-issue.
Thus there is no need to set the compiler to "clang++", as it will happen automatically.
However, cython needs to know that it has to produce a cpp-file (along with accepting c++-syntax) and not a c-file. Also the linker has to know, that it has to link against cpp-libaries (it is done automatically if g++/clang++ is used for linking). Thus we need to add language = "c++" to the Extension's definition, as suggested in DavidW's answer - this will take care of the last two problems.
The remaining problem is, that we would to like use different compiler options (-std=c++17 only for cpp-files) for different source-files. This can be achieved using a less known command build_clib:
#setup.py:
from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
ext_modules = [
Extension(
name="mydummy",
sources=["mydummy.pyx", "source1.cpp"]
language="c++",
extra_compile_args=["-std=c++17"],
)
]
ext_modules = cythonize(ext_modules)
myclib = ('myclib', {'sources': ["source2.c"]})
setup(
name="mydummy",
libraries=[myclib],
ext_modules=ext_modules,
)
And now building either with
python setup.py build --verbose
or
python setup.py build_clib --verbose build_ext -i --verbose
will first build a simple static library consisting of C-code and link it to the resulting extension consisting of c++-code.
The above code uses the default compilation flags when building the static library, which should be enough in your case.
distutils doesn't offer the possibility to specify additinal flags, so if it is necessary we have either to switch to setuptools which offers this functionality, or to patch the build_clib command.
For the second alternative we have to add the following to the above setup.py-file:
#setup.py
...
# adding cflags to build_clib
from distutils import log
from distutils.command.build_clib import build_clib
# use original implementation but with tweaked build_libraries!
class build_clib_with_cflags(build_clib):
def build_libraries(self, libraries):
for (lib_name, build_info) in libraries:
sources = build_info.get('sources')
if sources is None or not isinstance(sources, (list, tuple)):
raise DistutilsSetupError(
"in 'libraries' option (library '%s'), "
"'sources' must be present and must be "
"a list of source filenames" % lib_name)
sources = list(sources)
log.info("building '%s' library", lib_name)
macros = build_info.get('macros')
include_dirs = build_info.get('include_dirs')
cflags = build_info.get('cflags') # HERE we add cflags
objects = self.compiler.compile(sources,
output_dir=self.build_temp,
macros=macros,
include_dirs=include_dirs,
extra_postargs=cflags, # HERE we use cflags
debug=self.debug)
self.compiler.create_static_lib(objects, lib_name,
output_dir=self.build_clib,
debug=self.debug)
...
setup(
...
cmdclass={'build_clib': build_clib_with_cflags}, # use our class instead of built-in!
)
and now we can add additional compile flags to the library-definitions (the previous step can be skipped if setuptools is used):
...
myclib = ('myclib', {'sources': ["source2.c"], 'cflags' : ["-O3"]})
...
Intro
As a follow-up to my earlier question here, i still try to:
add new functionality to one of scipy's modules (e.g. scipy.optimize)
based on C++-code
which defines an interface-function to be called with ctypes
automatically setup through setup.py in scipy/optimize
Code incorporation
To keep it simple, let's assume we got the following (dumb and) simple C++-code:
extern "C" void hallo()
{
int a = 0;
}
where we use the extern keyword to handle name-mangling.
We introduce new files and folders in scipy.optimize:
scipy/optimize/__init__py
scipy/optimize/lbfgsb/...
scipy/optimize/lbfgsb.py
...
scipy/optimize/_mylib/README.md # new
scipy/optimize/_mylib/LICENSE # new
scipy/optimize/_mylib/src/debug.cpp # new
scipy/optimize/mylib.py # new
Setup preparation
We prepare the module-setup in scipy/optimize/setup.py:
from __future__ import division, print_function, absolute_import
from os.path import join
from scipy._build_utils import numpy_nodepr_api
def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
from numpy.distutils.system_info import get_info
config = Configuration('optimize',parent_package, top_path)
# MODIFICATION START
# OTHER EXTENSIONS OMITTED
# INSPIRED BY scipy.spatial: ckdtree/src & setup.py
mylib_src = ['debug.cpp']
mylib_src = [join('_mylib', 'src', x) for x in mylib_src]
mylib_headers = []
mylib_headers = [join('_mylib', 'src', x) for x in mylib_headers]
mylib_dep = mylib_headers + mylib_src
config.add_extension('_mylib',
sources=mylib_src,
depends=mylib_dep,
include_dirs=[join('_mylib', 'src')])
# MODIFICATION END
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration(top_path='').todict())
Run scipy install
Installing scipy now (from base-dir) with:
python3 setup.py build_ext --inplace
works and we will see those shared-libraries in scipy/optimize/:
_lbfgsb.cpython-35m-x86_64-linux-gnu.so
...
_mylib.cpython-35m-x86_64-linux-gnu.so
Let's try to use it / a look into scipy/optimize/mylib.py
As seen in the other question, we can get the lib with (we are in scipy/optimize/mylib.py) and get the function we want to use:
import scipy as scp
import numpy.ctypeslib as ctl
lib = ctl.load_library('_mylib', scp.optimize.__file__)
myfunc = lib.hallo
Now here is the problem: this fails with:
AttributeError: /.../_mylib.cpython-35m-x86_64-linux-gnu.so: undefined symbol: hallo
Trying it per hand:
import ctypes
lib = ctypes.CDLL('full path to above so')
myfun = lib.hallo
fails too.
But, checking (on my linux OS): nm --defined-only _mylib.cpython-35m-x86_64-linux-gnu.so outputs:
...
...
0000000000000530 t hallo
...
which should be ok. (Disclaimer: i absolutely miss any knowledge about C++ linking). Edit: maybe it's not okay. See later observation in regards to t vs. T!
Doing it manually: this works
Going to scipy/optimize/_mylib/src:
g++ -shared -fPIC debug.cpp -o mylib.so
followed by nm --defined-only mylib.so shows the same functions, but some t became T.
...
...
0000000000000600 T hallo
...
Probably this is the reason, and there is some related general question here.
As mentioned, this works:
# in src
import numpy.ctypeslib as ctl
lib = ctl.load_library('mylib.so', '.')
lib.hallo
# < FuncPtr object at 0x....
But what needs to be done in setup.py or my sources to make it work?
During running scipy-install, output looks like:
building 'scipy.optimize._mylib' extension
compiling C++ sources
C compiler: x86_64-linux-gnu-g++ -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
creating build/temp.linux-x86_64-3.5/scipy/optimize/_mylib
creating build/temp.linux-x86_64-3.5/scipy/optimize/_mylib/src
compile options: '-Iscipy/optimize/_mylib/src -I/usr/local/lib/python3.5/dist-packages/numpy-1.15.0.dev0+e4d678a-py3.5-linux-x86_64.egg/numpy/core/include -I/usr/include/python3.5m -c'
x86_64-linux-gnu-g++: scipy/optimize/_mylib/src/debug.cpp
scipy/optimize/_mylib/src/debug.cpp: In function ‘void hallo()’:
scipy/optimize/_mylib/src/debug.cpp:3:9: warning: unused variable ‘a’ [-Wunused-variable]
int a = 0;
^
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/scipy/optimize/_mylib/src/debug.o -Lbuild/temp.linux-x86_64-3.5 -o scipy/optimize/_mylib.cpython-35m-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.5/link-version-scipy.optimize._mylib.map
I have a Makefile to build some simple Python bindings. Unfortunately, upon a plain make or make all it will rebuild every time, even when py11_bindings.cpp was not changed. I checked whether maybe the source file would accidentally be touched or something, but that's not the case as far as I see.
PYTHON = /Library/Frameworks/Python.framework/Versions/3.5/bin
CPP = c++
INC = -I/software/pybind11/include -I/software/eigen
PYTHONCFG = `$(PYTHON)/python3.5-config --cflags --ldflags`
SRC = py11_bindings.cpp
TARGET = _chain.so
all: $(SRC)
$(CPP) -O3 -shared -std=c++11 $(INC) $(PYTHONCFG) $^ -o $(TARGET)
clean:
rm $(TARGET)
I have absolutely no clue why this should happen.
I'm not a makefile expert, so maybe I am not using the correct terms.
However: Your all: defines to create the source file, when it should list the targets to create. Then you should, for each target, list the dependencies and describe, how to create it.
So, your makefile should look like this:
all: $(TARGET)
$(TARGET) : $(SRC)
$(CPP) -O3 -shared -std=c++11 $(INC) $(PYTHONCFG) $^ -o $(TARGET)
The output produced by running perl -V is packed with useful information (see example below). Is there anything like it for Python?
Example output:
% perl -V
Summary of my perl5 (revision 5 version 10 subversion 1) configuration:
Platform:
osname=linux, osvers=2.6.32-5-amd64, archname=x86_64-linux-gnu-thread-multi
uname='linux brahms 2.6.32-5-amd64 #1 smp tue jun 14 09:42:28 utc 2011 x86_64 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.1 -Dsitearch=/usr/local/lib/perl/5.10.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.1 -Dd_dosuid -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.4.5', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=/lib/libc-2.11.2.so, so=so, useshrplib=true, libperl=libperl.so.5.10.1
gnulibc_version='2.11.2'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector'
Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_ALL
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_PERLIO USE_REENTRANT_API
Locally applied patches:
DEBPKG:debian/arm_thread_stress_timeout - http://bugs.debian.org/501970 Raise the timeout of ext/threads/shared/t/stress.t to accommodate slower build hosts
DEBPKG:debian/cpan_config_path - Set location of CPAN::Config to /etc/perl as /usr may not be writable.
<snip-- iow patches galore --you get the picture>
DEBPKG:fixes/safe-reval-rdo-cve-2010-1447 - [PATCH] Wrap by default coderefs returned by rdo and reval
DEBPKG:patchlevel - http://bugs.debian.org/567489 List packaged patches for 5.10.1-17squeeze2 in patchlevel.h
Built under linux
Compiled at Jun 30 2011 22:28:00
#INC:
/etc/perl
/usr/local/lib/perl/5.10.1
/usr/local/share/perl/5.10.1
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.10
/usr/share/perl/5.10
/usr/local/lib/site_perl
/usr/local/lib/perl/5.10.0
/usr/local/share/perl/5.10.0
.
Not to be confused with the much less informative perl -v:
% perl -v
This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi
(with 53 registered patches, see perl -V for more detail)
Copyright 1987-2009, Larry Wall
Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl". If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.
python -c 'import sysconfig, pprint; pprint.pprint(sysconfig.get_config_vars())'
Although this is incredibly hackish, impractical, and not as detailed as perl -V, this is a one-liner that can get decent information about the environment.
python -c "import platform as p;exec('for x in vars(p):\n try:\n print ({x:vars(p)[x]()})\n except:\n pass')"
Since this is not your typical easy-to-remember command, you could save this line to Python's Lib directory as sys_info.py and then you could just run:
python -m sys_info
I'm trying to embed Python into a MATLAB mex function on OS X. I've seen references that this can be done (eg here) but I can't find any OS X specific information. So far I can successfully build an embedded Python (so my linker flags must be OK) and I can also build example mex files without any trouble and with the default options:
jm-g26b101:mex robince$ cat pytestnomex.c
#include <Python/Python.h>
int main() {
Py_Initialize();
PyRun_SimpleString("print 'hello'");
Py_Finalize();
return 0;
}
jm-g26b101:mex robince$ gcc -arch i386 pytestnomex.c -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -L/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config -ldl -lpython2.5
jm-g26b101:mex robince$ ./a.out
hello
But when I try to build a mex file that embeds Python I run into a problem with undefined symbol main. Here is my mex function:
#include <Python.h>
#include <mex.h>
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray*prhs[])
{
mexPrintf("hello1\n");
Py_Initialize();
PyRun_SimpleString("print 'hello from python'");
Py_Finalize();
}
Here are the mex compilation steps:
jm-g26b101:mex robince$ gcc -c -I/Applications/MATLAB_R2009a.app/extern/include -I/Applications/MATLAB_R2009a.app/simulink/include -DMATLAB_MEX_FILE -arch i386 -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -DMX_COMPAT_32 -O2 -DNDEBUG "pytest.c"
jm-g26b101:mex robince$ gcc -O -arch i386 -L/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config -ldl -lpython2.5 -o "pytest.mexmaci" pytest.o -L/Applications/MATLAB_R2009a.app/bin/maci -lmx -lmex -lmat -lstdc++
Undefined symbols:
"_main", referenced from:
start in crt1.10.6.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
I've tried playing around with arch settings (I added -arch i386 it to try and keep everything 32bit - I am using the python.org 32 bit 2.5 build), and the order of the linker flags, but haven't been able to get anywhere. Can't find much online either. Does anyone have any ideas of how I can get this to build?
[EDIT: should probably add I'm on OS X 10.6.1 with MATLAB 7.8 (r2009a), Python 2.5.4 (python.org) - I've tried both gcc-4.0 and gcc-4.2 (apple)]
I think I found the answer - by including the mysterious apple linker flags:
-undefined dynamic_lookup -bundle
I was able to get it built and it seems to work OK. I'd be very interested if anyone has any references about these flags or library handling on OS X in general. Now I see them I remember being bitten by the same thing in the past - yet I'm unable to find any documentation on what they actually do and why/when they should be needed.