We Keep Coding

SqlAlchemy 2.x with specific columns makes scalars() return non-orm objects

This question is probably me not understanding architecture of (new) sqlalchemy, typically I use code like this:
query = select(models.Organization).where(
models.Organization.organization_id == organization_id
)
result = await self.session.execute(query)
return result.scalars().all()
Works fine, I get a list of models (if any).
With a query with specific columns only:
query = (
select(
models.Payment.organization_id,
models.Payment.id,
models.Payment.payment_type,
)
.where(
models.Payment.is_cleared.is_(True),
)
.limit(10)
)
result = await self.session.execute(query)
return result.scalars().all()
I am getting first row, first column only. Same it seems to: https://docs.sqlalchemy.org/en/14/core/connections.html?highlight=scalar#sqlalchemy.engine.Result.scalar
My understanding so far was that in new sqlalchemy we should always call scalars() on the query, as described here: https://docs.sqlalchemy.org/en/14/changelog/migration_20.html#migration-orm-usage
But with specific columns, it seems we cannot use scalars() at all. What is even more confusing is that result.scalars() returns sqlalchemy.engine.result.ScalarResult that has fetchmany(), fechall() among other methods that I am unable to iterate in any meaningful way.
My question is, what do I not understand?

My understanding so far was that in new sqlalchemy we should always call scalars() on the query
That is mostly true, but only for queries that return whole ORM objects. Just a regular .execute()
query = select(Payment)
results = sess.execute(query).all()
print(results) # [(Payment(id=1),), (Payment(id=2),)]
print(type(results[0])) # <class 'sqlalchemy.engine.row.Row'>
returns a list of Row objects, each containing a single ORM object. Users found that awkward since they needed to unpack the ORM object from the Row object. So .scalars() is now recommended
results = sess.scalars(query).all()
print(results) # [Payment(id=1), Payment(id=2)]
print(type(results[0])) # <class '__main__.Payment'>
However, for queries that return individual attributes (columns) we don't want to use .scalars() because that will just give us one column from each row, normally the first column
query = select(
Payment.id,
Payment.organization_id,
Payment.payment_type,
)
results = sess.scalars(query).all()
print(results) # [1, 2]
Instead, we want to use a regular .execute() so we can see all the columns
results = sess.execute(query).all()
print(results) # [(1, 123, None), (2, 234, None)]
Notes:
.scalars() is doing the same thing in both cases: return a list containing a single (scalar) value from each row (default is index=0).
sess.scalars() is the preferred construct. It is simply shorthand for sess.execute().scalars().

How to query a table that has ENUM column and keep the ENUM type?

I'm using SQLAlchemy ORM.
I have a table in SQL DB with an id column, and a column called b, which is type enum, and can take values ('example_1', 'example_2').
In Python, I have an Enum class like this:
class BTypes(enum.Enum):
EXAMPLE_1 = 'example_1'
EXAMPLE_2 = 'example_2'
For querying the table, I have an ORM like this:
class Example(Base):
__tablename__ = "example"
id = Column(Integer, primary_key=True)
b = Column(Enum(BTypes).values_callable)
When I do session.query(Example).all(), the objects that I get back have str type for the b attribute. In other words:
data = session.query(Example).all()
print(data[0].b)
# Outputs
# example_1
I want that the Example object for the attribute b has an enum type, not str. What is the best way to achieve this?

Base.metadata.create_all(create_engine("sqlite://")) with:
b = Column(Enum(BTypes).values_callable)
gives me:
sqlalchemy.exc.CompileError: (in table 'example', column 'b'): Can't generate DDL for NullType(); did you forget to specify a type on this Column?
About NullType
Since Enum(BTypes).values_callable is None, SQLAlchemy defaults to NullType.
From https://docs.sqlalchemy.org/en/14/core/type_api.html#sqlalchemy.types.NullType:
The NullType can be used within SQL expression invocation without issue, it just has no behavior either at the expression construction level or at the bind-parameter/result processing level.
In other words, when we query, its value is simply assigned as-is from the database.
How to use the Enum.values_callable parameter
From https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.Enum:
In order to persist the values and not the names, the Enum.values_callable parameter may be used. The value of this parameter is a user-supplied callable, which is intended to be used with a PEP-435-compliant enumerated class and returns a list of string values to be persisted. For a simple enumeration that uses string values, a callable such as lambda x: [e.value for e in x] is sufficient.
That would be:
b = Column(Enum(BTypes, values_callable=lambda x: [e.value for e in x]))

Modify your code to query table as below to get as Enum:
class Example(Base):
__tablename__ = "example"
id = Column(Integer, primary_key=True)
b = Column(Enum(BTypes))
Note that values_callable typically return list of string values.
Do have a look into the documentation for more information
values_callable –
A callable which will be passed the PEP-435 compliant enumerated type, which should then return a list of string values to be persisted. This allows for alternate usages such as using the string value of an enum to be persisted to the database instead of its name.

How to Django ORM update() a value nested inside a JSONField with a numeric value?

I have a Django JSONField on PostgreSQL which contains a dictionary, and I would like to use the queryset.update() to bulk update one (eventually, several) keys with a numeric (eventually, computed) values. I see there is discussion about adding better support for this and an old extension, but for now it seems I need a DIY approach. Relying on those references, this is what I came up with:
from django.db.models import Func, Value, CharField, FloatField, F, IntegerField
class JSONBSet(Func):
"""
Update the value of a JSONField for a specific key.
"""
function = 'JSONB_SET'
arity = 4
output_field = CharField()
def __init__(self, field, path, value, create: bool = True):
path = Value('{{{0}}}'.format(','.join(path)))
create = Value(create)
super().__init__(field, path, value, create)
This seems to work fine for non-computed "numeric string" values like this:
# This example sets the 'nestedkey' to numeric 199.
queryset.update(inputs=JSONBSet('inputs', ['nestedkey'], Value("199"), False))
and for carefully quoted strings:
# This example sets the 'nestedkey' to 'some string'.
queryset.update(inputs=JSONBSet('inputs', ['nestedkey'], Value('"some string"'), False))
But it does not work for a number:
queryset.update(inputs=JSONBSet('inputs', ['nestedkey'], Value(1), False))
{ProgrammingError}function jsonb_set(jsonb, unknown, integer, boolean) does not exist
LINE 1: UPDATE "paiyroll_payitem" SET "inputs" = JSONB_SET("paiyroll...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
As per the HINT, I tried an explicit cast Value(1, IntegerField()). I'm not sure where I am going wrong.

sqlalchemy Stored Procedure with Output

I want to call a stored procedure and receive the output parameter in python. I am using sqlAlchemy and can use parameters but do not know how to have the output be read into a variable. I understand that there is a outParam() attribute in sqlAlchemy, but I have not found a useful example.
Here is a simple SQL code for testing:
CREATE PROCEDURE [dbo].[Test]
#numOne int,
#numTwo int,
#numOut int OUTPUT
AS
BEGIN
SET NOCOUNT ON;
SET #numOut = #numOne + #numTwo
END
And simple python:
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
outParam = 0
result = engine.execute('Test ? ,?, ? OUTPUT', [1, 2, outParam])
outParam is still 0. I have tried modifying it with:
outParam = sqlalchemy.sql.outparam("ret_%d", type_=int)
But this produces a "Programming Error." What am I missing?

SQLAlchemy returns a ResultProxy object. Try it like this:
engine = sqlalchemy.create_engine(...)
rproxy = engine.execute(...)
result = rproxy.fetchall()
result should be a list of RowProxy objects that you can treat like dictionaries, keyed on the column names from the query.
If you are looking for a true OUT param, your approach is almost correct. There is just a small error in the first parameter in the call to sqlalchemy.sql.outparam (but it is valid Python syntax). It should be like this:
outParam = sqlalchemy.sql.outparam("ret_%d" % i, type_=int)
Note the change to the first parameter: it just needed a value to substitute into the format string. The first parameter is the key value, and most likely %d is to be replaced with the column index number (which should be in i).

How to compare 2 columns of a table through dynamic querying

Have a requirement in dynamic querying where i would like to compare 2 columns of a table say "column_a" and "column_b" (all columns are strings). The actual columns to compare are decided at run-time.
I'm using kwargs to create a dictionary. But Django assumes that RHS is an absolute value & not a column of the same table. Using F() is an option, but i cant find any documentation of using F() in kwargs.
if i use kwargs = {'predicted_value':'actual_value'}
'actual_value' is used as a literal string instead of column name
How do i use something like:
kwargs = {'predicted_value':F('actual_value')} and pass it as Model.objects.filter(**kwargs)
Alternately, is there a way to use F('column') in LHS ?
e.g. Model.objects.filter(F(column1_name) = F(column2_name))

For a non-dynamic filter field, I would use:
from django.db.models import F
Model.objects.filter(some_col=F(kwargs.get('predicted_value')))
But if you need it all dynamically, you can try with:
kwargs = {'predicted_value':F('actual_value')}
Model.objects.filter(**kwargs)
You can even access related fields:
kwargs = {'fk_field__somefield':F('actual_value')}
Model.objects.filter(**kwargs)