We Keep Coding

How to transfer changes made in SQliteStudio to Flask models (SQlAlchemy) [duplicate]

I'm using Alembic as migration tool and I'm launching the following pseudo script on an already updated database (no revision entries for Alembic, the database schema is just up to date).
revision = '1067fd2d11c8'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
op.add_column('box', sa.Column('has_data', sa.Boolean, server_default='0'))
def downgrade():
pass
It gives me the following error only with PostgreSQL behind (it's all good with MySQL):
INFO [alembic.migration] Context impl PostgresqlImpl.
INFO [alembic.migration] Will assume transactional DDL.
INFO [root] (ProgrammingError) ERREUR: la colonne « has_data » de la relation « box » existe déjà
Last line means the column has_data already exists.
I want to check that the column exists before op.add_column.

We ran into the same issue: we had to accommodate an edge case when a column added in a revision might exist in the schema. Silencing the error is not an option, as that will rollback the current transaction (unless using sqlite), and the version table will not be updated. Checking for column existence seems optimal here. Here's our solution (same idea as in the accepted answer, but updated for 2022):
from alembic import op
from sqlalchemy import inspect
def column_exists(table_name, column_name):
bind = op.get_context().bind
insp = inspect(bind)
columns = insp.get_columns(table_name)
return any(c["name"] == column_name for c in columns)
This is called from a revision file, so the context accessed via op.get_context() has been configured (presumably in your env.py, and the bind exists.

The easiest answer is not to try to do this. Instead, make your Alembic migrations represent the full layout of the database. Then any migrations you make will be based off the changes to the existing database.
To make a starting migration if you already have a database, temporarily point at an empty database and run alembic revision --autogenerate -m "base". Then, point back at the actual database and run alembic stamp head to say that the current state of the database is represented by the latest migration, without actually running it.
If you don't want to do that for some reason, you can choose not to use --autogenerate and instead generate empty revisions that you fill in with the operations you want. Alembic won't stop you from doing this, it's just much less convenient.

I am, unfortunately, in a situation where we have multiple versions with different schemas that all need to migrate to a single codebase. There are no migrations anywhere yet and no versions tagged in any db. So the first migration will have these conditional checks. After the first migration, everything will be in a known state and I can avoid such hacks.
So I added this in my migration (credit belongs to http://www.derstappen-it.de/tech-blog/sqlalchemie-alembic-check-if-table-has-column):
from alembic import op
from sqlalchemy import engine_from_config
from sqlalchemy.engine import reflection
def _table_has_column(table, column):
config = op.get_context().config
engine = engine_from_config(
config.get_section(config.config_ini_section), prefix='sqlalchemy.')
insp = reflection.Inspector.from_engine(engine)
has_column = False
for col in insp.get_columns(table):
if column not in col['name']:
continue
has_column = True
return has_column
My upgrade function has the following checks (note that I have a batch flag set that adds the with op.batch_alter_table line, which probably isn't in most setups:
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('mytable', schema=None) as batch_op:
if not _table_has_column('mytable', 'mycol'):
batch_op.add_column(sa.Column('mycol', sa.Integer(), nullable=True))
if not _table_has_column('mytable', 'mycol2'):
batch_op.add_column(sa.Column('mycol2', sa.Integer(), nullable=True))

How to disable automatically generated fields in Python Eve?

How to disable fields _updated, _created, _etag, _links?
I want to limit bandwidth and those fields are bigger size than data which I actually need to get from my database (Mongodb)

With the exceptions of _links, which you can remove by disabling HATEOAS (HATEOAS = False), you can only rename the other meta fields.
While the framework itself won't remove them, you could hook up a custom callback and purge those fields yourself before the response is sent over the wire.
from eve import Eve
def on_fetched_resource(resource, response):
for document in response['_items']:
del(document['_etag'])
# etc.
app = Eve()
app.on_fetched_resource += on_fetched_resource
if __name__ == '__main__':
app.run()

Good question!
If you not too anxious about concurrency control which is for the user-side integrity, you can disable it, and so the _etag field. This can be done by adding a simple option in your settings.py:
IF_MATCH = False
This also may be handy if you want to edit database with an external tools or apps, as it lets you avoid doing additional jiggery-pokery with "_etag".
To understand whether you need ETag, check out: https://docs.python-eve.org/en/stable/features.html#data-integrity-and-concurrency-control
Also refer to Nicola Iarocci, who mentioned about disabling HATEOAS (HATEOAS = False) and a way to remove _etag field without disabling ETag checks. (Actually I wonder how you can check and post with latest _etag then?)

Unable to delete large data set in Django — OperationalError: (1040, 'Too many connections')

I am attempting to delete 300,000+ spam comments from a Django site that is using the Zinnia blogging app. Zinnia includes a command for deleting spam called, appropriately, spam_cleanup but running this command spews thousands of the following error before being terminated by the OS.
OperationalError: (1040, 'Too many connections')
The code for the spam_cleanup command is as follows:
class Command(NoArgsCommand):
"""
Command object for removing comments
marked as non-public and removed.
"""
help = "Delete the entries's comments marked as non-public and removed."
def handle_noargs(self, **options):
verbosity = int(options.get('verbosity', 1))
content_type = ContentType.objects.get_for_model(Entry)
spams = comments.get_model().objects.filter(
#is_public=False, is_removed=True,
content_type=content_type)
spams_count = spams.count()
spams.delete()
if verbosity:
print('%i spam comments deleted.' % spams_count)
My initial thought was just to break the query down to only delete say 80 items at at time using the limit property but Django tells me that I can't do that on delete:
AssertionError: Cannot use 'limit' or 'offset' with delete.
It's not reasonable to increase the max connections on MySQL to 300,000, right? I also read that Django emulates cascade on delete but does not set it at the DB level so a raw SQL query could orphan all the relations. I am lost as to how to perform this delete properly, please help!

How to Limit Ratings (or anything else in Django) by IP?

I am using DjangoRatings for a web app which allows anonymous ratings from registered as well as nonregistered users. After I set the IPLimit integer in the DjangoRatings settings.py file, everything works fine; however, when I exceed the number of votes that I have allowed per IP, the entire web page gets reloaded with an “RaiseIPLimit()” error and the entire site goes down which necessitates reloading the previous page via back button. My question is, what can I add to my views.py file to tell django that when DjangoRatings passes the RaiseIPLimit() error, simply print something like “You can only vote once!” message to the user and leave the loaded web page as it is instead of crashing the entire site.
If there’s an easier way to do this general IP checking besides DjangoRatings, I am open to implementing other ways, but DjangoRatings just seems much easier than anything else since the only thing I need IP limits on is rating stuff. To be more clear, here is the exact error that DjangoRatings gives me:
IPLimitReached at /myapp/rating /page1
And this is straight from the DjangoRatings source code:
num_votes = Vote.objects.filter(
content_type=kwargs['content_type'],
object_id=kwargs['object_id'],
key=kwargs['key'],
ip_address=ip_address,
).count()
if num_votes >= getattr(settings, 'RATINGS_VOTES_PER_IP', RATINGS_VOTES_PER_IP):
raise IPLimitReached() ...
kwargs.update(defaults)
if use_cookies:
# record with specified cookie was not found ...
cookie = defaults['cookie'] # ... thus we need to replace old cookie (if presented) with new one
kwargs.pop('cookie__isnull', '') # ... and remove 'cookie__isnull' (if presented) from .create()'s **kwargs
rating, created = Vote.objects.create(**kwargs), True

Python script to hide ploneformgen form after user has filled it out. (For Plone-4.3.2-64.)

After a user has filled out a (ploneformgen) form , I would like to use a custom script adapter to call a python script to change the user’s local role so that they can’t see the form anymore. In other words, I want to prevent the user from filling out (or viewing) the form twice.
I figured that one way to do this is to call the script permission_changer.py which is located in the form folder. The code I have in that script is this:
container.manage_delLocalRoles((‘bob',))
container.reindexObjectSecurity()
Where ‘bob’ is just an example user, who has only the global role FormFiller (which I created under the Security tab of the ZMI) and the local role “Reader” for the form folder.
When I fill out the form (which has a "private" state) as a system admin, the script is called successfully and bob loses his “Reader” local role (which is all he had to begin with), and he can’t see the form anymore. However, when bob fills out the form, a “You do not have sufficient privileges to view this page.” error is displayed, and bob’s local role is not removed. I can’t work out why –– and I’ve tried many different things:
I’ve changed the proxy for the permission_changer.py by clicking on “Proxy” tab for the script in ZMI. I changed it to “Manager”, "System Administrator”, and “Owner”, but that didn’t solve the problem (nor did any combination of those).
I tried changing the proxy by creating a file permission_changer.py.metdadata in the form folder and including this:
[default]
proxy = Manager
but that didn’t work either.
Strangely, when I change bob’s global role to Manager, or System Administrator, or even Viewer, or Editor, the problem goes away and the script runs just fine (I can also change the script so that it adds and removes arbitrary other local roles). (These options are not solutions for me because bob will still be able to see the form because of his global role.)
Also, I tried giving the role FormFiller role every possible permission under the Security tab, but didn’t work.
So, I’m guessing that the problem has to do with the proxy settings, but I can’t work out what I’m doing wrong. I've searched around a lot, and I can't find anyone discussing a similar problem.
Any help would be much appreciated!

Ugly ugly way to handle this may be to access to the data saver field's download method and parse its output to find data to check.
For example, if username is the second pfg field added into form, a custom script adapter that prevents furthers fillings by a user may be
alreadyInDB = False
savedData = ploneformgen.savefield.getSavedFormInputForEdit()
username = request.AUTHENTICATED_USER.getId()
usersInDB = [x.split(',')[1] for x in savedData.split('\r\n') if len(x)>0]
if username in usersInDB:
alreadyInDB = True
if alreadyInDB:
return {'username': 'No way man!'}

I worked out what was going on, but I'm not sure how to describe it precisely. Basically, I found that by calling the script as a Custom Success Action (form > edit > overrides), I don't get the problem. So I think that by calling the script as custom script adapter I was trying to change the user's permission while they were still engaged with the form and that is impossible, even with the Manager proxy role.
I hope that helps. And if anyone has a more precise description of the problem, that would be appreciated.

For granting and revoking the permissions to submit a form, you could:
Create a group (e.g. with the ID "Submitters") and assign the chosen users to it
Make sure the form-folder has the state 'private' and grant View-permissions via the sharing-tab of the form-folder to the group
Add a content-item of type 'Page' in the form-folder's parent (e.g. with the ID 'submitted') and set its state to 'public'
Add a content-item of type 'Custom Script Adapter', select 'Manager' in the field 'Proxy role', and insert the lines below into the field 'Script body':
# Remove current user of group and redirect to [FORM_PARENT_URL]/landing_page_id'.
# If user is not in group, fail silently and continue.
# Fail if landing_page_id does not exist in form-folder, or one of its parents.
#
# Assumes a page with the ID as declared in `landing_page_id` lives in the
# form-folder's parent (or one of its grand-parents, first found wins),
# and holds the state 'public', so users can view it regardless of their
# group-memberships.
#
# Ment to be used after submission of a PloneFormGen-form with private-state and
# a locally assigned Reader-role for the group, so only group-members can view and
# submit the form.
from Products.CMFCore.utils import getToolByName
group_id = 'Submitters' # change as needed
landing_page_id = 'submitted' # change as needed
portal_groups = getToolByName(ploneformgen, 'portal_groups')
user_id = ploneformgen.memberId()
parent_url = '/'.join(ploneformgen.absolute_url().split('/')[:-1])
redirect_to_url = parent_url + '/' + landing_page_id
# Revoke current user's group-membership:
portal_groups.removePrincipalFromGroup(user_id, group_id)
# Let user land in userland:
request.response.redirect(redirect_to_url)
Tested with Plone-4.3.11 and Products.PloneFormGen-1.7.25