We Keep Coding

Dynamically create class instance with complex Class input parameters in Python using reflection

I am trying to make design metadata driven data pipelines, so I define in external textual metadata (json, yaml) things like:
dataFunctionSequence = DataFunctionSequence(
functions=[
Function(
functionClass="Function1",
functionPackage="pipelines.experiments.meta_driven.function_1",
parameters=[
Parameter(name="param1", dataType="str", value="this is my function str value")
]
)
]
)
Now with help of importlib i can get class:
functionMainClass = getattr(
importlib.import_module(
functionMeta.functionPackage), functionMeta.functionClass)
But I want real instance, so I got it working with this piece of code which utilizes on top of importlib as well eval builtin function:
def create_instance(class_str:str):
"""
Create a class instance from a full path to a class constructor
:param class_str: module name plus '.' plus class name and optional parens with arguments for the class's
__init__() method. For example, "a.b.ClassB.ClassB('World')"
:return: an instance of the class specified.
"""
try:
if "(" in class_str:
full_class_name, args = class_name = class_str.rsplit('(', 1)
args = '(' + args
else:
full_class_name = class_str
args = ()
# Get the class object
module_path, _, class_name = full_class_name.rpartition('.')
mod = importlib.import_module(module_path)
klazz = getattr(mod, class_name)
# Alias the the class so its constructor can be called, see the following link.
# See https://www.programiz.com/python-programming/methods/built-in/eval
alias = class_name + "Alias"
instance = eval(alias + args, { alias: klazz})
return instance
except (ImportError, AttributeError) as e:
raise ImportError(class_str)
And I can construct a string that will construct the class and it works like charm.
Now my problem is that the class requires another parameter which is complex Spark DataFrame object which is not loaded from metadata but from some database or s3 bucket for example. Here I fail to be able to create dynamically instance with non-string variable.
I am failing here:
instance = eval(alias + args, { alias: klazz})
I tried to extend the create_instance() fnc with **kwargs, so I can dynamically search for parameter by name eg kwargs["dataFrame"], but how to assign it dynamically to init?
evel is not the right way probably or my expression is not correct?
NOTE: Another possible approach was to iterate somehoe over init object where I will get all params of constructor, but still I don't know what python reflection fuction to use to make a real instance.
Workaround: What I can do is that I can simply remove the dataFrame from class constructor, create instance only on simple params (eg. strings) and call method:
instance.setDataFrame(dataFrame)
And it will work. but I wanted some reflection base approach if possible.
Thank you very much.
Ladislav

How to solve class objecto has no atribute

beginner Python user here.
So, I´m trying to make a program that orders the files of my (many) Downloads folder.
I made a class object to work with the many folders:
class cContenedora:
def __int__(self, nCarp, dCarp): #nCarp Stands is the file name and dCarp Stands for file directory.
self.nCarp = nCarp
self.dCarp = dCarp
So, y wrote a instance like this:
Download = cContenedora()
Download.nCarp = "Downloads/"
#The side bar is for making a path to move my archives from with shutil.move(path, dest)
Download.dCarp = "/Users/MyName/Download/"
#This is for searching the folder with os.listdir(Something.dCarp)
Then, I wrote my function, and it goes something like this:
def ordenador(carpetaContenedora, formato, directorioFinal): #carpetaContenedora is a Download Folder
carpetaContenedora = cContenedora() #carpetaContenedora one of the class objects
dirCCont = os.listdir(carpetaContenedora.dCarp) #The to directory is carpetaContenedora.cCarp
for a in dirCCont:
if a.endswith(formato):
path = "/Users/Aurelio Induni/" + carpetaContenedora().nCarp + a
try:
shutil.move(path, directorioFinal)
print(Fore.GREEN + a + "fue movido exitosamente.")
except:
print(Fore.RED + "Error con el archivo" + a)
pass
for trys in range(len(listaCarpetasDestino)-1): #Is a list full of directories.
for container in listaCarpetasFuente: #A short list of all my Downloads Folder.
for formatx in listaFormatos: #listaFormatos is a list ful of format extensions like ".pdf"
#try: #I disabled this to see the error istead of "Error Total"
ordenador(container, formatx, listaCarpetasDestino[trys])
#except:
#print(Fore.RED + "Error Total") #I disabled this to see the error.
But every time I run it I get the following:
AttributeError: 'cContenedora' object has no attribute 'dCarp'
It says the error is in line 47 (the one with the os.listdir(carpetaContenedora.dCarp))
I´m sure is something small. Python is so amazing, but it also can be so frustrating not knowing what´s wrong.

There is a spelling mistake in the initialization of your instance. It should be "init" instead of "int".
In the class cContenedora, the function should be
class cContenedora:
def __init__(self, nCarp, dCarp):
self.nCarp = nCarp
self.dCarp = dCarp
Additionally, When you are passing in the parameter. Make sure to pass in both of the parameters in the line with Value.
CContenedora(nCarp="something",dCarp="something")

Your class initializer, i.e., __init__() function has 2 parameters nCarp and dCarp but when you are actually creating the object there are no parameters passed.
Your function ordenador takes the first parameter as carpetaContenedora, on the first line same variable is assigned a new object of cContenedora, at this line the original values you passed are lost forever.
This could be the reason it is giving for the error.
Refer this link for more details on how to create classes and instantiate the object.

NameError when defining a class variable

Obviously I'm doing something stupid. But what?
I get:
File "<path>", line 71, in args
filename = filename
NameError: name 'filename' is not defined
...on the next-to-last line below ("filename = filename"):
def parseLog(filename, explain=False, omitminor=False, omitexpected=False,
omitgdocs=False, args=None):
print(filename)
if not args:
class args:
filename = filename
explain = explain
Yet the 2nd line above ("print(filename)") works fine. So, why the error?
In case you're wondering why I'm doing this in the first place, it's because the function parseLog() can also be called by the command line, like so:
def parseLogCLI():
''' parse command line for arguments '''
parser = argparse.ArgumentParser()
parser.add_argument('filename')
parser.add_argument('-explain', action="store_true", help='Explain what program has done')
parser.add_argument('-omitminor', action="store_true", help='Omit minor errors ' + repr(minor_errors))
parser.add_argument('-omitexpected', action="store_true", help='Omit machines expected to be often offline')
parser.add_argument('-omitgdocs', action="store_true", help='Omit errors on Google Docs native files (not copyable) ' + repr(gdocs))
args = parser.parse_args()
parseLog(arg.filename, args=args)
...so I'm trying to construct an 'arg' class (as argparse does) to pass to my function. If there's a better way to do this, I'm interested.

With class args: you are starting the definition of a class.
In that context, the first occurrence of filename defines a class attribute, which you are trying to assign from its own value before it is fully defined.
First of all, I think you should investigate in more details the concepts of classes, scopes, and instances.
This will help you understand why your function argument filename is hidden by the new definition inside the class scope.

It seems to me like you have a scope issue, in that the inner class 'arg' doesn't have access to the scope of the outer class. One solution would be to use the 'global' keyword like so:
def parseLog(filename, explain=False, omitminor=False, omitexpected=False,
omitgdocs=False, args=None):
global fname, expl
fname = filename
expl = explain
print(filename)
if not args:
class args:
filename = fname
explain = expl
You can read more about Python variable scopes here.

Trying to understand Django source code and cause of missing argument TypeError

A screenshot (portrait view) of my IDE and Traceback showing all the code pasted here, may be easier to read if you have a vertical monitor.
Context: Trying to save image from a URL to a Django ImageField hosted on EC2 with files on S3 using S3BotoStorage. I'm confused because all of this suggests that Django is still treating it like local storage, while it should S3.
The lines in question that seem to be causing the error:
def get_filename(self, filename):
return os.path.normpath(self.storage.get_valid_name(os.path.basename(filename)))
def get_valid_name(self, name):
"""
Returns a filename, based on the provided filename, that's suitable for
use in the target storage system.
"""
return get_valid_filename(name)
TypeError Exception: get_valid_name() missing 1 required positional argument: 'name'
Last Local vars Tracback before error at get_valid_name:
filename 'testimagefilename'
self <django.db.models.fields.files.ImageField: image>
(Only the stuff inside these two horizontal dividers is from me, the rest is from Django 1.9)
image.image.save('testimagefilename', File(temp), save=True)
Local vars from Traceback at that point (not sure about the ValueError on image, I think it's because it hasn't been created yet):
File <class 'django.core.files.base.File'>
image Error in formatting: ValueError: The 'image' attribute has no file associated with it.
requests <module 'requests' from '/usr/local/lib/python3.4/site-packages/requests/__init__.py'>
Image <class 'mcmaster.models.Image'>
NamedTemporaryFile <function NamedTemporaryFile at 0x7fb0e1bb0510>
temp <tempfile._TemporaryFileWrapper object at 0x7fb0dd241ef0>
Relevant snippets of Django source code:
files.py
def save(self, name, content, save=True):
name = self.field.generate_filename(self.instance, name)
if func_supports_parameter(self.storage.save, 'max_length'):
self.name = self.storage.save(name, content, max_length=self.field.max_length)
else:
warnings.warn(
'Backwards compatibility for storage backends without '
'support for the `max_length` argument in '
'Storage.save() will be removed in Django 1.10.',
RemovedInDjango110Warning, stacklevel=2
)
self.name = self.storage.save(name, content)
setattr(self.instance, self.field.name, self.name)
# Update the filesize cache
self._size = content.size
self._committed = True
# Save the object because it has changed, unless save is False
if save:
self.instance.save()
save.alters_data = True
def get_directory_name(self):
return os.path.normpath(force_text(datetime.datetime.now().strftime(force_str(self.upload_to))))
def get_filename(self, filename):
return os.path.normpath(self.storage.get_valid_name(os.path.basename(filename)))
def generate_filename(self, instance, filename):
# If upload_to is a callable, make sure that the path it returns is
# passed through get_valid_name() of the underlying storage.
if callable(self.upload_to):
directory_name, filename = os.path.split(self.upload_to(instance, filename))
filename = self.storage.get_valid_name(filename)
return os.path.normpath(os.path.join(directory_name, filename))
return os.path.join(self.get_directory_name(), self.get_filename(filename))
storage.py
def get_valid_name(self, name):
"""
Returns a filename, based on the provided filename, that's suitable for
use in the target storage system.
"""
return get_valid_filename(name)
text.py
def get_valid_filename(s):
"""
Returns the given string converted to a string that can be used for a clean
filename. Specifically, leading and trailing spaces are removed; other
spaces are converted to underscores; and anything that is not a unicode
alphanumeric, dash, underscore, or dot, is removed.
>>> get_valid_filename("john's portrait in 2004.jpg")
'johns_portrait_in_2004.jpg'
"""
s = force_text(s).strip().replace(' ', '_')
return re.sub(r'(?u)[^-\w.]', '', s)
get_valid_filename = allow_lazy(get_valid_filename, six.text_type)

I'd make a guess you didn't instantiate the Storage class. How are you setting Django to use the custom storage? If you do this in models.py
image = models.ImageField(storage=MyStorage)
It will fail exactly as you describe. It should be
image = models.ImageField(storage=MyStorage())

Django Imagekit overwrite cachefile_name?

I'm trying to overwrite the cachefile_name property from the module django-imagekit.
Here is my code:
class Thumb150x150(ImageSpec):
processors = [ResizeToFill(150, 150)]
format = 'JPEG'
options = {'quality': 90}
#property
def cachefile_name(self):
# simplified for this example
return "bla/blub/test.jpg"
register.generator('blablub:thumb_150x150', Thumb150x150)
class Avatar(models.Model):
avatar= ProcessedImageField(upload_to=upload_to,
processors=[ConvertToRGBA()],
format='JPEG',
options={'quality': 60})
avatar_thumb = ImageSpecField(source='avatar',
id='blablub:thumb_150x150')
It doesn't work at all.When I debug (without my overwrite of cachefile_name), and look at the return value of cachefile_name, the result is a string like "CACHE/blablub/asdlkfjasd09fsaud0fj.jpg". Where is my mistake?
Any ideas?

Replicating the example as closely as I could, it worked fine. A couple of suggestions are:
1) Make sure you are using the avatar_thumb in a view. The file "bla/blub/test.jpg" won't be generated until then.
2) Check the configuration of your MEDIA_ROOT to make sure you know where "bla/blub/test.jpg" is expected to appear.
Let me give an example of something similar I was working on. I wanted to give my thumbnails unique names that can be predicted from the original filename. Imagekit's default scheme names the thumbnails based on a hash, which I can't guess. Instead of this:
media/12345.jpg
media/CACHE/12345/abcde.jpg
I wanted this:
media/photos/original/12345.jpg
media/photos/thumbs/12345.jpg
Overriding IMAGEKIT_SPEC_CACHEFILE_NAMER didn't work because I didn't want all of my cached files to end up in the "thumbs" directory, just those generated from a specific field in a specific model.
So I created this ImageSpec subclass and registered it:
class ThumbnailSpec(ImageSpec):
processors=[Thumbnail(200, 200, Anchor.CENTER, crop=True, upscale=False)]
format='JPEG'
options={'quality': 90}
# put thumbnails into the "photos/thumbs" folder and
# name them the same as the source file
#property
def cachefile_name(self):
source_filename = getattr(self.source, 'name', None)
s = "photos/thumbs/" + source_filename
return s
register.generator('myapp:thumbnail', ThumbnailSpec)
And then used it in my model like this:
# provide a unique name for each image file
def get_file_path(instance, filename):
ext = filename.split('.')[-1]
return "%s.%s" % (uuid.uuid4(), ext.lower())
# store the original images in the 'photos/original' directory
photoStorage = FileSystemStorage(
location=os.path.join(settings.MEDIA_ROOT, 'photos/original'),
base_url='/photos/original')
class Photo(models.Model):
image = models.ImageField(storage=photoStorage, upload_to=get_file_path)
thumb = ImageSpecField(
source='image',
id='myapp:thumbnail')

I think, the correct way is to set IMAGEKIT_SPEC_CACHEFILE_NAMER. Have a look at default namer names.py, it joins settings.IMAGEKIT_CACHEFILE_DIR with file path and hash, you should probably do the same.