We Keep Coding

marshmallow NewType can not be turned into a dataclass, python 3.10

I'm in the process of upgrading to python 3.10 and in that context I thought it would be nice to also upgrade packages used. Right now, the problem is with the marshmallow packages and at this point I can't even run their example code for NewType anymore.
This is my code (taken from the comment in the NewType definition):
from marshmallow_dataclass import NewType, dataclass, List
import marshmallow.validate
IPv4 = NewType('IPv4', str, validate=marshmallow.validate.Regexp(r'^([0-9]{1,3}\\.){3}[0-9]{1,3}$'))
#dataclass
class MyIps:
ips: List[IPv4]
MyIps.Schema().load({"ips": ["0.0.0.0", "grumble grumble"]})
I only added the first line to have the necessary commands available.
When I run this I receive the following error-message:
/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py:373: UserWarning: ****** WARNING ****** marshmallow_dataclass was called on the class <function NewType.<locals>.new_type at 0x7fb5a077fd90>, which is not a dataclass. It is going to try and convert the class into a dataclass, which may have undesirable side effects. To avoid this message, make sure all your classes and all the classes of their fields are either explicitly supported by marshmallow_dataclass, or define the schema explicitly using field(metadata=dict(marshmallow_field=...)). For more information, see https://github.com/lovasoa/marshmallow_dataclass/issues/51 ****** WARNING ******
warnings.warn(
Traceback (most recent call last):
File "/usr/lib/python3.10/dataclasses.py", line 1197, in fields
fields = getattr(class_or_instance, _FIELDS)
AttributeError: 'function' object has no attribute '__dataclass_fields__'. Did you mean: '__dataclass_params__'?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 370, in _internal_class_schema
fields: Tuple[dataclasses.Field, ...] = dataclasses.fields(clazz)
File "/usr/lib/python3.10/dataclasses.py", line 1199, in fields
raise TypeError('must be called with a dataclass type or instance')
TypeError: must be called with a dataclass type or instance
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 384, in _internal_class_schema
created_dataclass: type = dataclasses.dataclass(clazz)
File "/usr/lib/python3.10/dataclasses.py", line 1185, in dataclass
return wrap(cls)
File "/usr/lib/python3.10/dataclasses.py", line 1176, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
File "/usr/lib/python3.10/dataclasses.py", line 909, in _process_class
for b in cls.__mro__[-1:0:-1]:
AttributeError: 'function' object has no attribute '__mro__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/username/testscript.python", line 9, in <module>
MyIps.Schema().load({"ips": ["0.0.0.0", "grumble grumble"]})
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/lazy_class_attribute.py", line 33, in __get__
setattr(cls, self.name, self.func())
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 356, in class_schema
return _internal_class_schema(clazz, base_schema, clazz_frame)
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 402, in _internal_class_schema
attributes.update(
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 405, in <genexpr>
field_for_schema(
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 695, in field_for_schema
generic_field = _field_for_generic_type(typ, base_schema, typ_frame, **metadata)
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 503, in _field_for_generic_type
child_type = field_for_schema(
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 729, in field_for_schema
or _internal_class_schema(typ, base_schema, typ_frame)
File "/home/username/.local/share/virtualenvs/venv_3.10/lib/python3.10/site-packages/marshmallow_dataclass/__init__.py", line 387, in _internal_class_schema
raise TypeError(
TypeError: IPv4 is not a dataclass and cannot be turned into one.
My current install is python 3.10.6 and marshmallow_dataclasses 8.5.8. (The code was working with python 3.9 and marshmallow dataclasses 8.3.0)
Does anyone know why this is happening and how to solve this (other than downgrading, obviously)
Does anyone else have this issue? Is this new behaviour and I should adapt our code (if so, how?)? Is this a bug?

I've had a similar issue with a similar bit of code that is raising an error since trying to upgrade marshmallow_dataclasses.
import dataclasses
import marshmallow_dataclass
ARGON2_PASSWORD = marshmallow_dataclass.NewType(
"ARGON2_PASSWORD",
str,
validate=validate.Regexp(
r"[$]argon2(?:id|i)[$]v=\d{1,3}[$]m=\d{3,20},t=\d{1,4},"
r"p=\d{1,2}[$][^$]{1,100}[$][^$]{1,768}"))
#marshmallow_dataclass.add_schema
#dataclasses.dataclass
class Argon2Password:
password: ARGON2_PASSWORD
The detailed root cause is currently discussed in https://github.com/lovasoa/marshmallow_dataclass/issues/206 afaics.
For the time being, downgrading to typing-inspect==0.7.1 worked for me as a hotfix.

Couldn't Run the .py file | AttributeError: module 'collections' has no attribute 'MutableSequence'

I am working with odx files and I have a generate.py file to run. I am using pyXB. When I try to run I am getting this.
*Traceback (most recent call last):
File "C:\Users\rohitkr\Downloads\starter_kit_adas-master\starter_kit_adas-master\devops\scripts\generate_odxf\generate_odxf.py", line 15, in
from schema import odx
File "C:\Users\rohitkr\Downloads\starter_kit_adas-master\starter_kit_adas-master\devops\scripts\generate_odxf\schema\odx.py", line 9, in import pyxb.binding
File "C:\Users\rohitkr\AppData\Local\Programs\Python\Python310\lib\site-packages\pyxb\binding_init_.py", line 8, in
from . import datatypes
File "C:\Users\rohitkr\AppData\Local\Programs\Python\Python310\lib\site-packages\pyxb\binding\datatypes.py", line 1266, in
rom . import content
File "C:\Users\rohitkr\AppData\Local\Programs\Python\Python310\lib\site-packages\pyxb\binding\content.py", line 807, in
class _PluralBinding (collections.MutableSequence):
AttributeError: module 'collections' has no attribute 'MutableSequence'*
'''
What could be the problem?
Thanks in advance.

In python 3.10 MutableSequence was removed from collections in favor of collections.abc
Deprecated since version 3.3, will be removed in version 3.10: Moved Collections Abstract Base Classes to the collections.abc module. For backwards compatibility, they continue to be visible in this module through Python 3.9.
>>> from collections import MutableSequence
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
ImportError: cannot import name 'MutableSequence' from 'collections' (C:\Program Files\Python310\lib\collections\__init__.py)
>>> from collections.abc import MutableSequence

If you don't want to change the source code, there is an easier way. Just use this in your script after importing.
import collections
collections.MutableSequence = collections.abc.MutableSequence

Why do we get a NameError when trying to use the SharedMemoryManager (python 3.8) as a replacement for the BaseManager?

Python 3.8 introduces new shared memory features. We are trying to use the SharedMemoryManager and a NameError is thrown.
I thought that we might do something wrong in our complex scenario, so I broke it down using python documentation snippets.
try:
# python >= 3.8
from multiprocessing.managers import SharedMemoryManager as Manager
except:
# python < 3.8
from multiprocessing.managers import BaseManager as Manager
class MathsClass:
def add(self, x, y):
return x + y
def mul(self, x, y):
return x * y
class MyManager(Manager):
pass
MyManager.register('Maths', MathsClass)
if __name__ == '__main__':
with MyManager() as manager:
maths = manager.Maths()
print(maths.add(4, 3)) # prints 7
print(maths.mul(7, 8)) # prints 56
This is pretty much taken from the multiprocessing docs (except for the fallback import) and works fine in python 3.7 but throws the following error in python 3.8:
Traceback (most recent call last):
File "scripts/debug_shared_memory_issue.py", line 21, in <module>
maths = manager.Maths()
File "/usr/lib/python3.8/multiprocessing/managers.py", line 740, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 625, in _create
id, exposed = dispatch(conn, None, 'create', (typeid,)+args, kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 91, in dispatch
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/managers.py", line 210, in handle_request
result = func(c, *args, **kwds)
File "/usr/lib/python3.8/multiprocessing/managers.py", line 1312, in create
if hasattr(self.registry[typeid][-1], "_shared_memory_proxy"):
NameError: name 'self' is not defined
---------------------------------------------------------------------------
The release notes of python 3.8 and the documentation say that SharedMemoryManager is "A subclass of BaseManager" so we expected it to work as a drop-in replacement. But it doesn't seem so. What are we doing wrong? Looking at the current 3.8 branch of CPython there don't seem to be relevant changes to this. In Python 3.9 there is an explicit self arg in the create function though. But 3.9 is WIP, so we'd rather not use it in production software.
Thanks for your help!

It was a bug fixed in python/cpython#142566c (v3.9.0a1).
You can patch from python/cpython/blob/v3.9.0a1/Lib/multiprocessing/managers.py#L1269-L1277:
from multiprocessing.managers import SharedMemoryManager as Manager
import sys
if sys.version_info < (3, 9):
from multiprocessing.managers import Server, SharedMemoryServer
def create(self, c, typeid, /, *args, **kwargs):
if hasattr(self.registry[typeid][-1], "_shared_memory_proxy"):
kwargs['shared_memory_context'] = self.shared_memory_context
return Server.create(self, c, typeid, *args, **kwargs)
SharedMemoryServer.create = create

How can I get the definition of a class in a module?

Why can't I get the definition of Callable from module collections in the following code?
How can I get the definition of a class in a module? Thanks.
>>> from collections import Callable
>>> inspect.getsource(Callable)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/inspect.py", line 944, in getsource
lines, lnum = getsourcelines(object)
File "/usr/lib/python3.5/inspect.py", line 931, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python3.5/inspect.py", line 788, in findsource
raise OSError('could not find class definition')
OSError: could not find class definition
>>> inspect.getsourcelines(Callable)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/inspect.py", line 931, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python3.5/inspect.py", line 788, in findsource
raise OSError('could not find class definition')
OSError: could not find class definition
>>> inspect.getmodule(Callable)
<module 'collections.abc' from '/usr/lib/python3.5/collections/abc.py'>
>>> inspect.getfile(Callable)
'/usr/lib/python3.5/collections/abc.py'
>>> inspect.getsourcefile(Callable)
'/usr/lib/python3.5/collections/abc.py'

In general, this is easily done with inspect.getsource which accepts a module, a class, a method, a function, a traceback, a frame,
or a code object. The source they represent should of course be written in Python else an error is raised.
In this specific case, you just happen to be unlucky because while Callable is defined in _collections_abc the Callable.__module__ name is callections.abc:
>>> Callable.__module__
'collections.abc'
This throws getsource off because it won't look in _collections_abc that contains Callables definition but, instead, in collections.abc which merely imports all definitions from _collections_abc:
>>> print(getsource(collections.abc))
from _collections_abc import *
from _collections_abc import __all__
Normally, getsource doesn't have an issue finding the source, for example, on itself:
>>> print(getsource(getsource))
def getsource(object):
"""Return the text of the source code for an object.
The argument may be a module, class, method, function, traceback, frame,
or code object. The source code is returned as a single string. An
OSError is raised if the source code cannot be retrieved."""
lines, lnum = getsourcelines(object)
return ''.join(lines)
In this specific case, though, it does (due to Callable.__module__ returning collections.abc.) You could replace __module__ with '_collections_abc' for a tricky way to see the source:
>>> Callable.__module__ = '_collections_abc'
>>> src = getsource(Callable)
>>> print(src)
class Callable(metaclass=ABCMeta):
__slots__ = ()
#abstractmethod
def __call__(self, *args, **kwds):
return False
#classmethod
def __subclasshook__(cls, C):
if cls is Callable:
return _check_methods(C, "__call__")
return NotImplemented
but this doesn't make me feel very comfortable.

get_source(fullname)
Return the source code for the specified module.
So you should return the whole module, as Callable is defined in the module of _collections_abc, so your code should be this:
import _collections_abc
import inspect
print(inspect.getsource(_collections_abc))
and you can see the definition of Callable in the print result.

Pathos (python module) behaves different in IDE and shell

I am trying to understand how to use the Pathos package to run a function that calls a function. It was my understanding an the advantage of Pathos over the main multiprocessing package was that it allowed functions inside functions. However, I can't seem to make it work. Here is the simplest example I could come up with:
def testf(x):
print(x)
import dill
import pathos
from pathos.multiprocessing import ProcessingPool
pool = ProcessingPool(nodes=3)
out2 = pool.map(testf, [1,2,3,4,5,6,7,8,9])
pool.close()
Output:
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/james/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/home/james/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/james/.local/lib/python3.6/site-packages/pathos/helpers/mp_helper.py", line 14, in <lambda>
func = lambda args: f(*args)
File "<input>", line 2, in testf
NameError: name 'print' is not defined
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<input>", line 5, in <module>
File "/home/james/.local/lib/python3.6/site-packages/pathos/multiprocessing.py", line 136, in map
return _pool.map(star(f), zip(*args)) # chunksize
File "/home/james/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/james/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 608, in get
raise self._value
NameError: name 'print' is not defined
Edit: It seems that if I paste this code into a shell console, it works fine. No dice if I run it from my IDE of choice, PyCharm. So now my question is why the same code would work differently in the same version of the python interpreter (3.6.1) based on whether it is run from a shell or the in-app console/