We Keep Coding

Solved: Adding new Column to ORM SQLAlchemy table in a volatile setting

I am working on a open source persistance layer for a MQTT-Broker https://github.com/volkerjaenisch/amqtt_db
Incoming MQTT messages are irregular blobs of data so usually the DB-Backend is some kind of object storage.
I do it the hard way and deserialize the blobs into typed data colums and store them into a fast relational database. My finally target will be timescaleDB but first I go via SQLAlchemy to access a wide bunch of DBs with one API.
MQTT messages are volatile (think not always complete) so the DB scheme has to adjust dynamically e.g. adding new columns for new information.
First Message:
Time: 1234
Temperature : 23.4
Second Message:
Time: 1245
Temperature : 23.6
Rel Hum : 87 %
I have used SQLalchemy ORM for more than a decade but always for quite static databases. So I am new to work dynmically.
Utilizing the ORM to build DB tables dynamically from the structure of incoming MQTT-Messages was quite doable and worked out perfect.
But currently I am stuck with the case of additional information in the MQTT-Packages that extends the tables with new columns.
What I did so far:
Utilizing sqlalchemy-migration it was quite easy to dynamically add new columns to the existing table in the DB. In the code "topic_cls" is the declarative class and "column_def" a col_name - type mapping.
from migrate.versioning.schema import Table as MiTable, Column as MiColumn
def add_new_colums(self, topic_cls, column_def):
table_name = str(topic_cls.__table__.name)
table = MiTable(table_name, self.metadata)
for col_name, col_type in column_def.items():
col = MiColumn(col_name, col_type)
col.create(table)
Works like a charm. But how to get this changes to the DB reflected back into declarative classes? I tried to get a new inspection of the table:
new_table = Table(topic_cls.__table__.name, self.metadata, autoload_with=self.engine)
This also works but it gives me a new table but not a declarative base.
So my stupid questions are:
Is this the right way to achive my goal?
How can I get a declarative class by inspecting an already existing table in a DB?
"Drop the ORM and use SQL" is not the answer I am looking for.
Cheers,
Volker

Found a solution but it is a bit of a hack.
new_table = Table("test/topic_growth", Base.metadata, autoload_with=self.engine)
Base.metadata.remove(topic_cls.__table__)
new_dcl = type(str(table_name), (Base,), {'__table__': new_table})
Base.metadata._add_table(table_name, None, new_table)
After you obtained the new table via inspection, remove the old table entry from the metadata.
Then generate a new declarative base with the new table and same table name.
At last add the new table to the metadata.

How to get the data object of a newly inserted data row and flask-mysqldb?

I have work in Perl where I am able to get the newly created data object ID by passing the result back to a variable. For example:
my $data_obj = $schema->resultset('PersonTable')->create(\%psw_rec_hash);
Where the $data_obj contains the primary key's column value.
I want to be able to do the same thing using Python 3.7, Flask and flask-mysqldb,
but without having to do another query. I want to be able to use the specific
record's primary key column value for another method.
Python and flask-mysqldb inserts data like so:
query = "INSERT INTO PersonTable (fname, mname, lname) VALUES('Phil','','Vil')
cursor = db.connection.cursor()
cursor.execute(query)
db.connection.commit()
cursor.close()
The PersonTable has a primary key column called, id. So, the newly inserted data row would look
like:
23, 'Phil', 'Vil'
Because there are 22 rows of data before the last inserted data, I don't want to perform a search
for the data, because there could be more than one entry with the same data. However, all I want
the most recent data row.
Can I do something similar to Perl with python 3.7 and flask-mysqldb?

You may want to consider the Flask-SQLAlchemy package to help you with this.
Although the syntax is going to be slightly different from Perl, what you can do is, when you create the model object, you can set it to a variable. Then, when you either flush or commit on the Database session, you can pull up your primary key attribute on that model object you had created (whether it's "id" or something else), and use it as needed.
SQLAlchemy supports MySQL, as well as several other relational databases. In addition, it is able to help prevent SQL injection attacks so long as you use model objects and add/delete them to your database session, as opposed to straight SQL commands.

Simple SQLAlchemy update table based on value in another table

I am a newcomer to SQLAlchemy, so please forgive what must be an elementary question.
I have a database table properties (mapped in SQLALchemy as the object Property) which contains a field MEBID. I have another table mebs (mapped in SQLAlchemy as MEB). I want to set the properties.MEBID field to mebs.id where properties.PostCode == mebs.PostCode.
I can do this simply in SQL using the command
update properties, mebs set properties.mebid = mebs.id where mebs.PostCode = properties.PostCode
but am struggling with doing it in SQLAlchemy. If I try the command
session.query(Property, MEB).\
filter(Property.PostCode == MEB.PostCode).\
update({Property.MEBID : MEB.id})
I get
InvalidRequestError: This operation requires only one Table or entity be specified as the target.
I know that this must be elementary as it's such a fundamental operation, but can't work out how it's done.

To update:
for prop, meb in session.query(Property, MEB).filter(Property.PostCode == MEB.PostCode).all():
prop.MEBID=meb.id
session.add(prop)

mysql - How to determine the length of a field which stores a mutable string?

I'm working on a iOS application and I use Flask(a Python framework) to build my backend.
I store my data in mysql database.
Now I need to store a bunch of IDs in one attribute.
Firstly I convert the array which stores the IDs to a JSON format object.
Then I ran into a problem. How to store this object?
As the length of the object can be rather large(I cannot make sure how many IDs I store), and SQLAlchemy requires the attribute to have a exact length when I create the table, so how to determine the length of the attribute?

In case you use MySQL 5.7 or newer
you should look at the new JSON type.
You can use this MySQL feature through sqlalchemy's type.JSON. This will greatly simplify column data management.
data_table = Table('data_table', metadata,
Column('id', Integer, primary_key=True),
Column('loosely_related_ids', JSON)
)
with engine.connect() as conn:
conn.execute(
data_table.insert(),
loosely_related_ids = [1, 54, 56, 99, 104]
)
Later on accessing the loosely_related_ids field will return a python array that you access normally.
If you are using an older version of MySQL
you should use a TEXT field or a wrapper around a similar type.
SQLAlchemy provides the PickleType field which is implemented on top of a BLOB field and will handle pickling and unpickling the array for you. Keep in mind that all the caveats of pickling python objects and sharing them across interpreters still apply here.

I don't quiet know the situation you meet , But it's not recommended to store multi records in one column , It's more normalised to build a relation map between ID owner and ID .
For example , you can create a new table called 'IDs' with schema like that :
id int auto increment ,
idbla varchar(<Your ID Length>)
owner int not null
When you are trying to get all idbla of some user x you can use
SELECT * idbla from IDs where owner = x
Another choice :
You can use nosql (non relational database) to store your data , It's document like and fit your situation pretty well .

Values from models from Database (Django)

I have this model
class Type(models.Model):
type = models.CharField(max_length=50)
value = models.CharField(max_length=1)
And into it, I have some data from an sql file:
INSERT INTO quest_type (type, value) VALUES ('Noun', '1');
INSERT INTO quest_type (type, value) VALUES ('Adjective', '2');
INSERT INTO quest_type (type, value) VALUES ('Duration', '3');
How do I access these values in the python shell? For example, if I know the type, how do I get the value (and vice verse)? I'm not sure how the syntax works.

you should be able to get that with
Type.objects.filter(type=typeImInterestedIn)
A couple of things to be leary of:
-you probably want to avoid manually writing to a DB that you're using an ORM in. It just creates potential for mismatches.
-naming an object Type is little problematic since it's so close to the python native function type.

It's unclear from your question how much about databases you understand, so I apologize if this answer is too basic for you (if so, please edit your question to include information about what actual database engine you're using and show some sample code trying to read from the database).
The SQL file you have is not the same as an SQL database. It is a series of commands that will create records in an SQL database. First you must install and configure a database engine on your machine then "run" that .sql file so that the records are created in the database.
After you have an actual database, you will have to configure Django so that it knows what kind of SQL engine you're using and the name and location of the database.
Finally, once the database is created and Django configured to talk to the engine, you will write python code to instantiate an instance of the Type class, read a record from the database, and inspect the values.
Also, let me point out that Type is a really, really bad name for a class in any programming language, and type and value are both bad names for columns in SQL databases.

If you are using python shell from django (python manage.py shell) firstly You have to import to your namespace your model, so type from my_app.models import Type.
Now if You want to get only one object from db syntax is:
result = Type.objects.get(type='your_query')
If you want to fetch more then one object syntax goes like this:
result = Type.objects.filter(type='your_query')
second method returns list instead of single object
To loop through list after using filter write:
for item in result:
item.value #will print values from matched rows