We Keep Coding

Automatically generate custom migrations in Django

Django provides a really nice feature called makemigrations where it will create migration files based on the changes in models. We are developing a module where we will want to generate custom migrations.
I haven't found much info about creating custom migrations in the Django docs. There is documentation on the various Operation classes that can yield migrations but there's nothing about creating custom Operation classes that can yield custom migrations.
The autodetector module for generating new migrations also doesn't seem to leave much room for adding custom Operation classes: https://github.com/django/django/blob/master/django/db/migrations/autodetector.py#L160
It seems this is completely static. Are there other ways to generate custom migrations, perhaps by using existing classes with a custom management command?

You can create a custom class to hook into the makemigrations class and add your custom migrations stuff then execute using the "runscript" command. Below is a sample module where the file is named custom_migrations.py and located in a "scripts" folder off one of your apps:
from django.core.management.commands.makemigrations import Command
"""
To invoke this script use:
manage.py runscript custom_migrations --script-args [app_label [app_label ...]] name=my_special_migration verbosity=1
"""
class MyMigrationMaker(Command):
'''
Override the write method to add more stuff before finishing
'''
def write_migration_files(self, changes):
print("Do some stuff to \"changes\" object here...")
super().write_migration_files(changes)
def run(*args):
nargs = []
kwargs = {}
# Preload some options with defaults and then can be overridden in the args parsing
kwargs['empty'] = True
kwargs['verbosity'] = 1
kwargs['interactive'] = True
kwargs['dry_run'] = False
kwargs['merge'] = False
kwargs['name'] = 'custom_migration'
kwargs['check_changes'] = False
for arg in args:
kwarg = arg.split('=', 1)
if len(kwarg) > 1:
val = kwarg[1]
if val == "True":
arg_val = True
elif val == "False":
arg_val = False
elif val.isdigits():
arg_val = int(val)
else:
arg_val = val
the_kwargs[kwarg[0]] = arg_val
else:
nargs.append(arg)
MyMigrationMaker().handle(*nargs, **kwargs)

An alternative if you are not willing to Tango with Django internals is to use this script below to generate a migration file by invoking a random method that will produce the Python code you want to run in the migration and insert it into a valid migration file that will then be part of the standard Django migrations.
It you had an app named "xxx" and you had a method in a file xxx/scripts/test.py looking like this:
def run(*args, **kwargs):
return "print(\"BINGO!!!!!!!!! {} :: {}\".format(args[0], kwargs['name']))"
... and you invoked the script shown at the bottom of this post stored in xxx/scripts/custom_migrations.py with the following command
manage.py runscript custom_migrations --script-args xxx name=say_bingo callable=xxx.scripts.test.run
Then you would end up with a migration file in xxx/migrations with the appropriate number sequence (something like 0004_say_bingo.py) looking like this:
Generated by Django 2.1.2 on 2018-12-14 08:54
from django.db import migrations
def run(*args, **kwargs):
print("BINGO!!!!!!!!! {} :: {}".format(args[0], kwargs['name']))
class Migration(migrations.Migration):
dependencies = [
]
operations = [
migrations.RunPython(run,)
]
The script is as follows:
from django.core.management.base import no_translations
from django.core.management.commands.makemigrations import Command
from django.db.migrations import writer
from django import get_version
from django.utils.timezone import now
import os
import sys
"""
To invoke this script use:
manage.py runscript custom_migrations --script-args [app_label [app_label ...]] callable=my_migration_code_gen name=my_special_migration
-- the "name" argument will be set as part of the migration file generated
-- the "callable" argument will be the function that is invoked to generate the python code you want to execute in the migration
the callable will be passed all the args and kwargs passed in to this script from the command line as part of --script-args
Only app names are allowed in args so use kwargs for custom arguments
"""
class LazyCallable(object):
def __init__(self, name):
self.n, self.f = name, None
def __call__(self, *a, **k):
if self.f is None:
modn, funcn = self.n.rsplit('.', 1)
if modn not in sys.modules:
__import__(modn)
self.f = getattr(sys.modules[modn], funcn)
return self.f(*a, **k)
class MyMigrationMaker(Command):
'''
Override the write method to provide access to script arguments
'''
#no_translations
def handle(self, *app_labels, **options):
self.in_args = app_labels
self.in_kwargs = options
super().handle(*app_labels, **options)
'''
Override the write method to add more stuff before finishing
'''
def write_migration_files(self, changes):
code = LazyCallable(self.in_kwargs['callable'])(self.in_args, self.in_kwargs)
items = {
"replaces_str": "",
"initial_str": "",
}
items.update(
version=get_version(),
timestamp=now().strftime("%Y-%m-%d %H:%M"),
)
items["imports"] = "from django.db import migrations\n\ndef run(*args, **kwargs):\n "
items["imports"] += code.replace("\n", "\n ") + "\n\n"
items["operations"] = " migrations.RunPython(run,)\n"
directory_created = {}
for app_label, app_migrations in changes.items():
for migration in app_migrations:
# Describe the migration
my_writer = writer.MigrationWriter(migration)
dependencies = []
for dependency in my_writer.migration.dependencies:
dependencies.append(" %s," % my_writer.serialize(dependency)[0])
items["dependencies"] = "\n".join(dependencies) + "\n" if dependencies else ""
# Write the migrations file to the disk.
migrations_directory = os.path.dirname(my_writer.path)
if not directory_created.get(app_label):
if not os.path.isdir(migrations_directory):
os.mkdir(migrations_directory)
init_path = os.path.join(migrations_directory, "__init__.py")
if not os.path.isfile(init_path):
open(init_path, "w").close()
# We just do this once per app
directory_created[app_label] = True
migration_string = writer.MIGRATION_TEMPLATE % items
with open(my_writer.path, "w", encoding='utf-8') as fh:
fh.write(migration_string)
if self.verbosity >= 1:
self.stdout.write("Migration file: %s\n" % my_writer.filename)
def run(*args):
glob_args = []
glob_kwargs = {}
# Preload some options with defaults and then can be overridden in the args parsing
glob_kwargs['empty'] = True
glob_kwargs['verbosity'] = 1
glob_kwargs['interactive'] = True
glob_kwargs['dry_run'] = False
glob_kwargs['merge'] = False
glob_kwargs['name'] = 'custom_migration'
glob_kwargs['check_changes'] = False
for arg in args:
kwarg = arg.split('=', 1)
if len(kwarg) > 1:
glob_kwargs[kwarg[0]] = kwarg[1]
else:
glob_args.append(arg)
MyMigrationMaker().handle(*glob_args, **glob_kwargs)

Pyuno indexing issue that I would like an explanation for

The following python libreoffice Uno macro works but only with the try..except statement.
The macro allows you to select text in a writer document and send it to a search engine in your default browser.
The issue, is that if you select a single piece of text,oSelected.getByIndex(0) is populated but if you select multiple pieces of text oSelected.getByIndex(0) is not populated. In this case the data starts at oSelected.getByIndex(1) and oSelected.getByIndex(0) is left blank.
I have no idea why this should be and would love to know if anyone can explain this strange behaviour.
#!/usr/bin/python
import os
import webbrowser
from configobj import ConfigObj
from com.sun.star.awt.MessageBoxButtons import BUTTONS_OK, BUTTONS_OK_CANCEL, BUTTONS_YES_NO, BUTTONS_YES_NO_CANCEL, BUTTONS_RETRY_CANCEL, BUTTONS_ABORT_IGNORE_RETRY
from com.sun.star.awt.MessageBoxButtons import DEFAULT_BUTTON_OK, DEFAULT_BUTTON_CANCEL, DEFAULT_BUTTON_RETRY, DEFAULT_BUTTON_YES, DEFAULT_BUTTON_NO, DEFAULT_BUTTON_IGNORE
from com.sun.star.awt.MessageBoxType import MESSAGEBOX, INFOBOX, WARNINGBOX, ERRORBOX, QUERYBOX
def fs3Browser(*args):
#get the doc from the scripting context which is made available to all scripts
desktop = XSCRIPTCONTEXT.getDesktop()
model = desktop.getCurrentComponent()
doc = XSCRIPTCONTEXT.getDocument()
parentwindow = doc.CurrentController.Frame.ContainerWindow
oSelected = model.getCurrentSelection()
oText = ""
try:
for i in range(0,4,1):
print ("Index No ", str(i))
try:
oSel = oSelected.getByIndex(i)
print (str(i), oSel.getString())
oText += oSel.getString()+" "
except:
break
except AttributeError:
mess = "Do not select text from more than one table cell"
heading = "Processing error"
MessageBox(parentwindow, mess, heading, INFOBOX, BUTTONS_OK)
return
lookup = str(oText)
special_c =str.maketrans("","",'!|##"$~%&/()=?+*][}{-;:,.<>')
lookup = lookup.translate(special_c)
lookup = lookup.strip()
configuration_dir = os.environ["HOME"]+"/fs3"
config_filename = configuration_dir + "/fs3.cfg"
if os.access(config_filename, os.R_OK):
cfg = ConfigObj(config_filename)
#define search engine from the configuration file
try:
searchengine = cfg["control"]["ENGINE"]
except:
searchengine = "https://duckduckgo.com"
if 'duck' in searchengine:
webbrowser.open_new('https://www.duckduckgo.com//?q='+lookup+'&kj=%23FFD700 &k7=%23C9C4FF &ia=meanings')
else:
webbrowser.open_new('https://www.google.com/search?/&q='+lookup)
return None
def MessageBox(ParentWindow, MsgText, MsgTitle, MsgType, MsgButtons):
ctx = XSCRIPTCONTEXT.getComponentContext()
sm = ctx.ServiceManager
si = sm.createInstanceWithContext("com.sun.star.awt.Toolkit", ctx)
mBox = si.createMessageBox(ParentWindow, MsgType, MsgButtons, MsgTitle, MsgText)
mBox.execute()

Your code is missing something. This works without needing an extra try/except clause:
selected_strings = []
try:
for i in range(oSelected.getCount()):
oSel = oSelected.getByIndex(i)
if oSel.getString():
selected_strings.append(oSel.getString())
except AttributeError:
# handle exception...
return
result = " ".join(selected_strings)
To answer your question about the "strange behaviour," it seems pretty straightforward to me. If the 0th element is empty, then there are multiple selections which may need to be handled differently.

Plone: Change default schemata of a content type

I am using the Products.FacultyStaffDirectory product to manage contacts in my website.
One of the content types, i.e. FacultyStaffDirectory, has the description field in the 'categorization' tab when you are in edit mode. Now I need to remove the description field and put it in the default tab.
To do this, I'm using archetypes.schemaextender product to edit the field. In particular, I am using ISchemaModifier.
I have implemented the code but the field is still being shown in the categorization tab. May be something I missed. Below is the code:
This is the class that contains the description field that I want to modify:
# -*- coding: utf-8 -*-
__author__ = """WebLion <support#weblion.psu.edu>"""
__docformat__ = 'plaintext'
from AccessControl import ClassSecurityInfo
from Products.Archetypes.atapi import *
from zope.interface import implements
from Products.FacultyStaffDirectory.interfaces.facultystaffdirectory import IFacultyStaffDirectory
from Products.FacultyStaffDirectory.config import *
from Products.CMFCore.permissions import View, ManageUsers
from Products.CMFCore.utils import getToolByName
from Products.ATContentTypes.content.base import ATCTContent
from Products.ATContentTypes.content.schemata import ATContentTypeSchema, finalizeATCTSchema
from Products.membrane.at.interfaces import IPropertiesProvider
from Products.membrane.utils import getFilteredValidRolesForPortal
from Acquisition import aq_inner, aq_parent
from Products.FacultyStaffDirectory import FSDMessageFactory as _
schema = ATContentTypeSchema.copy() + Schema((
LinesField('roles_',
accessor='getRoles',
mutator='setRoles',
edit_accessor='getRawRoles',
vocabulary='getRoleSet',
default = ['Member'],
multiValued=1,
write_permission=ManageUsers,
widget=MultiSelectionWidget(
label=_(u"FacultyStaffDirectory_label_FacultyStaffDirectoryRoles", default=u"Roles"),
description=_(u"FacultyStaffDirectory_description_FacultyStaffDirectoryRoles", default=u"The roles all people in this directory will be granted site-wide"),
i18n_domain="FacultyStaffDirectory",
),
),
IntegerField('personClassificationViewThumbnailWidth',
accessor='getClassificationViewThumbnailWidth',
mutator='setClassificationViewThumbnailWidth',
schemata='Display',
default=100,
write_permission=ManageUsers,
widget=IntegerWidget(
label=_(u"FacultyStaffDirectory_label_personClassificationViewThumbnailWidth", default=u"Width for thumbnails in classification view"),
description=_(u"FacultyStaffDirectory_description_personClassificationViewThumbnailWidth", default=u"Show all person thumbnails with a fixed width (in pixels) within the classification view"),
i18n_domain="FacultyStaffDirectory",
),
),
))
FacultyStaffDirectory_schema = OrderedBaseFolderSchema.copy() + schema.copy() # + on Schemas does only a shallow copy
finalizeATCTSchema(FacultyStaffDirectory_schema, folderish=True)
class FacultyStaffDirectory(OrderedBaseFolder, ATCTContent):
"""
"""
security = ClassSecurityInfo()
implements(IFacultyStaffDirectory, IPropertiesProvider)
meta_type = portal_type = 'FSDFacultyStaffDirectory'
# Make this permission show up on every containery object in the Zope instance. This is a Good Thing, because it easy to factor up permissions. The Zope Developer's Guide says to put this here, not in the install procedure (http://www.zope.org/Documentation/Books/ZDG/current/Security.stx). This is because it isn't "sticky", in the sense of being persisted through the ZODB. Thus, it has to run every time Zope starts up. Thus, when you uninstall the product, the permission doesn't stop showing up, but when you actually remove it from the Products folder, it does.
security.setPermissionDefault('FacultyStaffDirectory: Add or Remove People', ['Manager', 'Owner'])
# moved schema setting after finalizeATCTSchema, so the order of the fieldsets
# is preserved. Also after updateActions is called since it seems to overwrite the schema changes.
# Move the description field, but not in Plone 2.5 since it's already in the metadata tab. Although,
# decription and relateditems are occasionally showing up in the "default" schemata. Move them
# to "metadata" just to be safe.
if 'categorization' in FacultyStaffDirectory_schema.getSchemataNames():
FacultyStaffDirectory_schema.changeSchemataForField('description', 'categorization')
else:
FacultyStaffDirectory_schema.changeSchemataForField('description', 'metadata')
FacultyStaffDirectory_schema.changeSchemataForField('relatedItems', 'metadata')
_at_rename_after_creation = True
schema = FacultyStaffDirectory_schema
# Methods
security.declarePrivate('at_post_create_script')
def at_post_create_script(self):
"""Actions to perform after a FacultyStaffDirectory is added to a Plone site"""
# Create some default contents
# Create some base classifications
self.invokeFactory('FSDClassification', id='faculty', title='Faculty')
self.invokeFactory('FSDClassification', id='staff', title='Staff')
self.invokeFactory('FSDClassification', id='grad-students', title='Graduate Students')
# Create a committees folder
self.invokeFactory('FSDCommitteesFolder', id='committees', title='Committees')
# Create a specialties folder
self.invokeFactory('FSDSpecialtiesFolder', id='specialties', title='Specialties')
security.declareProtected(View, 'getDirectoryRoot')
def getDirectoryRoot(self):
"""Return the current FSD object through acquisition."""
return self
security.declareProtected(View, 'getClassifications')
def getClassifications(self):
"""Return the classifications (in brains form) within this FacultyStaffDirectory."""
portal_catalog = getToolByName(self, 'portal_catalog')
return portal_catalog(path='/'.join(self.getPhysicalPath()), portal_type='FSDClassification', depth=1, sort_on='getObjPositionInParent')
security.declareProtected(View, 'getSpecialtiesFolder')
def getSpecialtiesFolder(self):
"""Return a random SpecialtiesFolder contained in this FacultyStaffDirectory.
If none exists, return None."""
specialtiesFolders = self.getFolderContents({'portal_type': 'FSDSpecialtiesFolder'})
if specialtiesFolders:
return specialtiesFolders[0].getObject()
else:
return None
security.declareProtected(View, 'getPeople')
def getPeople(self):
"""Return a list of people contained within this FacultyStaffDirectory."""
portal_catalog = getToolByName(self, 'portal_catalog')
results = portal_catalog(path='/'.join(self.getPhysicalPath()), portal_type='FSDPerson', depth=1)
return [brain.getObject() for brain in results]
security.declareProtected(View, 'getSortedPeople')
def getSortedPeople(self):
""" Return a list of people, sorted by SortableName
"""
people = self.getPeople()
return sorted(people, cmp=lambda x,y: cmp(x.getSortableName(), y.getSortableName()))
security.declareProtected(View, 'getDepartments')
def getDepartments(self):
"""Return a list of FSDDepartments contained within this site."""
portal_catalog = getToolByName(self, 'portal_catalog')
results = portal_catalog(portal_type='FSDDepartment')
return [brain.getObject() for brain in results]
security.declareProtected(View, 'getAddableInterfaceSubscribers')
def getAddableInterfaceSubscribers():
"""Return a list of (names of) content types marked as addable using the
IFacultyStaffDirectoryAddable interface."""
return [type['name'] for type in listTypes() if IFacultyStaffDirectoryAddable.implementedBy(type['klass'])]
security.declarePrivate('getRoleSet')
def getRoleSet(self):
"""Get the roles vocabulary to use."""
portal_roles = getFilteredValidRolesForPortal(self)
allowed_roles = [r for r in portal_roles if r not in INVALID_ROLES]
return allowed_roles
#
# Validators
#
security.declarePrivate('validate_id')
def validate_id(self, value):
"""Ensure the id is unique, also among groups globally."""
if value != self.getId():
parent = aq_parent(aq_inner(self))
if value in parent.objectIds():
return _(u"An object with id '%s' already exists in this folder") % value
groups = getToolByName(self, 'portal_groups')
if groups.getGroupById(value) is not None:
return _(u"A group with id '%s' already exists in the portal") % value
registerType(FacultyStaffDirectory, PROJECTNAME)
Below is the portion of code from the class that I have implemented to change the description field's schemata:
from Products.Archetypes.public import ImageField, ImageWidget, StringField, StringWidget, SelectionWidget, TextField, RichWidget
from Products.FacultyStaffDirectory.interfaces.facultystaffdirectory import IFacultyStaffDirectory
from archetypes.schemaextender.field import ExtensionField
from archetypes.schemaextender.interfaces import ISchemaModifier, ISchemaExtender, IBrowserLayerAwareExtender
from apkn.templates.interfaces import ITemplatesLayer
from zope.component import adapts
from zope.interface import implements
class _ExtensionImageField(ExtensionField, ImageField): pass
class _ExtensionStringField(ExtensionField, StringField): pass
class _ExtensionTextField(ExtensionField, TextField): pass
class FacultyStaffDirectoryExtender(object):
"""
Adapter to add description field to a FacultyStaffDirectory.
"""
adapts(IFacultyStaffDirectory)
implements(ISchemaModifier, IBrowserLayerAwareExtender)
layer = ITemplatesLayer
fields = [
]
def __init__(self, context):
self.context = context
def getFields(self):
return self.fields
def fiddle(self, schema):
desc_field = schema['description'].copy()
desc_field.schemata = "default"
schema['description'] = desc_field
And here is the code from my configure.zcml:
<adapter
name="apkn-FacultyStaffDirectoryExtender"
factory="apkn.templates.extender.FacultyStaffDirectoryExtender"
provides="archetypes.schemaextender.interfaces.ISchemaModifier"
/>
Is there something that I'm missing in this?

def fiddle(self, schema):
schema['description'].schemata = 'default'
should be sufficient. The copy() operation does not make any sense here.
In order to check if the fiddle() method is actually used: use pdb or add debug print statements.

It turns out that there was nothing wrong with the code.
I got it to work by doing the following:
Restarted the plone website
Re-installed the product
Clear and rebuilt my catalog
This got it working somehow.

jinja2 Custom filter "TemplateAssertionError: no filter named 'format_number'"

I'm getting the "TemplateAssertionError: no filter named 'format_number'" error, when trying to register a custom filter on the template environment by updating the filters dict on the environment.
In my module, I have imported environment module as follows:
from jinja2 import environment
In my class I defined the following method:
class DashboardHandler(SecurePageHandler):
def format_number(number):
s = '%d' % number
groups = []
while s and s[-1].isdigit():
groups.append(s[-3:])
s = s[:-3]
return s + ','.join(reversed(groups))
def do_get(self):
# ... snip ...
env = environment.Environment(self)
env.filters['format_number'] = self.format_number
# ... snip ...
Inside my html template file I tried to implement the filter as follows:
{{top_five_url .total|format_number}}
How is this happening?
Is there a default environment instance, we should use?

I could resolve my problem as follow,
Inside the module, I defined my method and updated the filters dictionary globally as follows:
import jinja2
def format_number(number):
s = '%d' % number
groups = []
while s and s[-1].isdigit():
groups.append(s[-3:])
s = s[:-3]
return s + ','.join(reversed(groups))
jinja2.filters.FILTERS['format_number'] = format_number

Why to copy a dictonairy from WSGI environment?

In the following example from wsgi.org is cur_named copied:
def __call__(self, environ, start_response):
script_name = environ.get('SCRIPT_NAME', '')
path_info = environ.get('PATH_INFO', '')
for regex, application in self.patterns:
match = regex.match(path_info)
if not match:
continue
extra_path_info = path_info[match.end():]
if extra_path_info and not extra_path_info.startswith('/'):
# Not a very good match
continue
pos_args = match.groups()
named_args = match.groupdict()
cur_pos, cur_named = environ.get('wsgiorg.routing_args', ((), {}))
new_pos = list(cur_pos) + list(pos_args)
new_named = cur_named.copy() # Why copy()?
new_named.update(named_args)
environ['wsgiorg.routing_args'] = (new_pos, new_named)
environ['SCRIPT_NAME'] = script_name + path_info[:match.end()]
environ['PATH_INFO'] = extra_path_info
return application(environ, start_response)
return self.not_found(environ, start_response)
Why not to call ur_named.update(named_args) directly?

Do you know where cur_named dict came from? Just imaging something like the following:
SOME_CONFIG = {
'some_key': ((..., ...), {...}),
...
}
environ['wsgiorg.routing_args'] = SOME_CONFIG['some_key']
Now when you update new_named in-place you are actually updating inner dictionary inside SOME_CONFIG which will bring your data to other requests. The safe way is to copy dictionary unless you are sure it's not needed.