We Keep Coding

Linting classes created at runtime in Python

For context, I am using the Python ctypes library to interface with a C library. It isn't necessary to be familiar with C or ctypes to answer this question however. All of this is taking place in the context of a python module I am creating.
In short, my question is: how can I allow Python linters (e.g. PyCharm or plugin for neovim) to lint objects that are created at runtime? "You can't" is not an answer ;). Of course there is always a way, with scripting and the like. I want to know what I would be looking at for the easiest way.
First I introduce my problem and the current approach I am taking. Second, I will describe what I want to do, and ask how.
Within this C library, a whole bunch of error codes are defined. I translated this information from the .h header file into a Python enum:
# CustomErrors.py
from enum import Enum
class CustomErrors(Enum):
ERROR_BROKEN = 1
ERROR_KAPUTT = 2
ERROR_BORKED = 3
Initially, my approach is to have a single exception class containing a type field which described the specific error:
# CustomException.py
from CustomErrors import CustomErrors
class CustomException(Exception):
def __init__(self, customErr):
assert type(customErr) is CustomError
self.type = customErr
super().__init__()
Then, as needed I can raise CustomException(CustomErrors.ERROR_KAPUTT).
Now, what I want to do is create a separate exception class corresponding to each of the enum items in CustomErrors. I believe it is possible to create types at runtime with MyException = type('MyException', (Exception,), {'__doc__' : 'Docstring for ABC class.'}).
I can create the exception classes at runtime like so:
#CustomException.py
from CustomErrors import CustomErrors
...
for ce in CustomErrors:
n = ce.name
vars()[n] = type(n, (Exception,), {'__doc__' : 'Docstring for {0:s} class.'.format(n)})
Note: the reason I want to create these at runtime is to avoid hard-coding of an Exception list that change in the future. I already have the problem of extracting the C enum automatically on the backburner.
This is all well and good, but I have a problem: static analysis cannot resolve the names of these exceptions defined in CustomException. This means PyCharm and other editors for Python will not be able to automatically resolve the names of the exceptions as a suggested autocomplete list when the user types CustomException.. This is not acceptable, as this is code for the end user, who will need to access the exception names for use in try-except constructs.
Here is the only solution I have been able to think of: writing a script which generates the .py files containing the exception names. I can do this using bash. Maybe people will tell me this is really the only option. But I would like to know what other approaches are suggested for solving this problem. Thanks for reading.

You can add a comment to tell mypy to ignore dynamically defined attribute errors. Perhaps the linters that you use share a similar way to silence such errors.
mypy docs on silencing errors based on error codes
This example shows how to ignore an error about an imported name mypy thinks is undefined:
# 'foo' is defined in 'foolib', even though mypy can't see the
# definition.
from foolib import foo # type: ignore[attr-defined]

How to log the name of the test class, if the test method resides in a class common for all tests?

I have the following project structure:
/root
/tests
common_test_case.py
test_case_1.py
test_case_2.py
...
project_file.py
...
Every test test_case_... is inherited from both unittest.TestCase and common_test_case.CommonTestCase. Class CommonTestCase contains test methods that should be executed by all the tests (though using data unique to each test, stored and accessed in self.something of the test). If some specific tests are needed for an exact test case, they are added directly to that particular class.
Currently I am working on adding logging to my tests. Among other things I would like to log the class the method was run from (since the approach above implies the same test method name for different classes). I would like to stick with the built-in logging module to achieve this.
I have tried the following LogRecord attributes:%(filename)s, %(module)s, %(pathname)s. Though, for methods defined in common_test_case.py they all return path/name to the common_test_case.py and not the test module they were actually run from.
My questions are:
Is there a way to achieve what I am trying to, using only built-in logging module?
Using some third-party/other module (I was thinking maybe some "hacky" solution with inspect)?
Is it possible to achieve (in Python) at all?

Your question appears similar to this one, and solved by:
self.id()
See the function definition here, which calls self.__class__ for the instance of the TestCase class that is instantiated. Given that you are using multiple inheritance the multiple inheritance rules from Python apply:
For most purposes, in the simplest cases, you can think of the search for attributes inherited from a parent class as depth-first, left-to-right, not searching twice in the same class where there is an overlap in the hierarchy.
Which means that common_test_case.CommonTestCase will be searched then unittest.TestCase. If there is no id function in common_test_case.CommonTestCase things should work as if it is only derived from unittest.TestCase. If you feel the need to add an id function to the CommonTestCase, something like this (if really necessary):
def id(self):
if issubclass(self,unittest.TestCase):
return super(unittest.TestCase,self).id()

The solution I've found (that does the trick, so far):
import inspect
class_called_from = inspect.stack()[1][0].f_locals['self'].__class__.__name__
I'm still wondering, though, if there is a "clearer" method, or if this is possible to achieve using logging module.
Recipes, based on West's answer (tested on Python 3.6.1):
test_name = self.id().split('.')[-1]
class_called_from = self.id().split('.')[-2]

Python abstract module possible?

I've built a module in Python in one single file without using classes. I do this so that using some api module becomes easier. Basically like this:
the_module.py
from some_api_module import some_api_call, another_api_call
def method_one(a, b):
return some_api_call(a + b)
def method_two(c, d, e):
return another_api_call(c * d * e)
I now need to built many similar modules, for different api modules, but I want all of them to have the same basic set of methods so that I can import any of these modules and call a function knowing that this function will behave the same in all the modules I built. To ensure they are all the same, I want to use some kind of abstract base module to build upon. I would normally grab the Abstract Base Classes module, but since I don't use classes at all, this doesn't work.
Does anybody know how I can implement an abstract base module on which I can build several other modules without using classes? All tips are welcome!

You are not using classes, but you could easily rewrite your code to do so.
A class is basically a namespace which contains functions and variables, as is a module.
Should not make a huge difference whether you call mymodule.method_one() or mymodule.myclass.method_one().
In python there is no such thing as interfaces which you might know from java.
The paradigm in python is Duck typing, that means more or less that for a given module you can tell whether it implements your API if it provides the right methods.
Python does this i.e. to determine what to do if you call myobject[i] on an instance of your class myclass. It looks whether the class has the method __getitem__ and if it does so, it replaces myobject[i] by myobject.__getitem__(i).
Yout don't have to tell python that your class supports this kind of access, python just figures it out from the way you defined your class.
The same way you should determine whether your module implements your API.
Maybe you want to look inside the hidden dictionary mymodule.__dict__ after import mymodulewhich contains all function names and pointers to them of your module. You could then check whether the right functions are present and raise an error otherwise
import my_module_4
#check if my_module_4 implements api
if all(func in my_module_4.__dict__ for func in ("method_one","method_two"):
print "API implemented"
else:
print "Warning: Not all API functions found in my_module_4"

When instancing an imported "class" from a package, how can i avoid using the package.class() declaration?

I'm kinda new to Python, so i'm still lost in the whole namespace thing.
I've created a package, with the init file in it and also a classname.py file, with the class, obviously.
For instance:
from parachute import container, emitter
I tried to instance the class Container directly, but it gave me an error, so i had to instance it as container.Container(). How can i avoid doing this?
Basically, what i want to do is to import a class from a package and avoid typing the package name and/or the file name.
Thanks in advance, and please let me know if the question isn't clear enough.
UPDATE
The structure i have is:
- parachute
-- init.py
-- container.py
Serves as a controller, i'd say, instancing, calling and glueing all the other parts together.
-- sparkles.py
Has two classes: Sparkle and Sparkles. Sparkle is a single element, with only one property so far, and Sparkles serves as a collection. Sparkles() belongs to the Emitter, and Sparkle() belongs to Sparkles().
-- emitter.py
Emitter could be seen as the user entity. It has a name and an uid, it belongs to a Container and the Container belongs to it.
Now, from outside the package i'm calling Container and passing some arguments, and the Container instances and distributes the arguments as it needs.
I have the impression that this isn't the best way to do what i need to do, which is: Create a collection of sparkles, owned by the emitter.

Don't put the class in it's own file. Put Container and Emitter directly in parachute.py.
You can then do
from parachute import Container, Emitter
or
import parachute
container = parachute.Container()
This essentially boils down to "Python isn't Java so for best results, don't treat it like it is" ;)

from module import Class
classInst = Class()
This will work if your class is in module.py

Problem using super(python 2.5.2)

I'm writing a plugin system for my program and I can't get past one thing:
class ThingLoader(object):
'''
Loader class
'''
def loadPlugins(self):
'''
Get all the plugins from plugins folder
'''
from diones.thingpad.plugin.IntrospectionHelper import loadClasses
classList=loadClasses('./plugins', IPlugin)#Gets a list of
#plugin classes
self.plugins={}#Dictionary that should be filled with
#touples of objects and theirs states, activated, deactivated.
classList[0](self)#Runs nicelly
foo = classList[1]
print foo#prints <class 'TestPlugin.TestPlugin'>
foo(self)#Raise an exception
The test plugin looks like this:
import diones.thingpad.plugin.IPlugin as plugin
class TestPlugin(plugin.IPlugin):
'''
classdocs
'''
def __init__(self, loader):
self.name='Test Plugin'
super(TestPlugin, self).__init__(loader)
Now the IPlugin looks like this:
class IPlugin(object):
'''
classdocs
'''
name=''
def __init__(self, loader):
self.loader=loader
def activate(self):
pass
All the IPlugin classes works flawlessy by them selves, but when called by ThingLoader the program gets an exception:
File "./plugins\TestPlugin.py", line 13, in __init__
super(TestPlugin, self).__init__(loader) NameError:
global name 'super' is not defined
I looked all around and I simply don't know what is going on.

‘super’ is a builtin. Unless you went out of your way to delete builtins, you shouldn't ever see “global name 'super' is not defined”.
I'm looking at your user web link where there is a dump of IntrospectionHelper. It's very hard to read without the indentation, but it looks like you may be doing exactly that:
built_in_list = ['__builtins__', '__doc__', '__file__', '__name__']
for i in built_in_list:
if i in module.__dict__:
del module.__dict__[i]
That's the original module dict you're changing there, not an informational copy you are about to return! Delete these members from a live module and you can expect much more than ‘super’ to break.
It's very hard to keep track of what that module is doing, but my reaction is there is far too much magic in it. The average Python program should never need to be messing around with the import system, sys.path, and monkey-patching __magic__ module members. A little bit of magic can be a neat trick, but this is extremely fragile. Just off the top of my head from browsing it, the code could be broken by things like:
name clashes with top-level modules
any use of new-style classes
modules supplied only as compiled bytecode
zipimporter
From the incredibly round-about functions like getClassDefinitions, extractModuleNames and isFromBase, it looks to me like you still have quite a bit to learn about the basics of how Python works. (Clues: getattr, module.__name__ and issubclass, respectively.)
In this case now is not the time to be diving into import magic! It's hard. Instead, do things The Normal Python Way. It may be a little more typing to say at the bottom of a package's mypackage/__init__.py:
from mypackage import fooplugin, barplugin, bazplugin
plugins= [fooplugin.FooPlugin, barplugin.BarPlugin, bazplugin.BazPlugin]
but it'll work and be understood everywhere without relying on a nest of complex, fragile magic.
Incidentally, unless you are planning on some in-depth multiple inheritance work (and again, now may not be the time for that), you probably don't even need to use super(). The usual “IPlugin.__init__(self, ...)” method of calling a known superclass is the straightforward thing to do; super() is not always “the newer, better way of doing things” and there are things you should understand about it before you go charging into using it.

Unless you're running a version of Python earlier than 2.2 (pretty unlikely), super() is definitely a built-in function (available in every scope, and without importing anything).
May be worth checking your version of Python (just start up the interactive prompt by typing python at the command line).