We Keep Coding

SQLAlchemy insert on conflict (upsert) multiple rows at once

I know there are commands in SQLAlchemy called bulk_insert_mappings and bulk_update_mappings, which can insert/update multiple rows at once; and I know there is a way to upsert a single row like so:
from sqlalchemy.dialects.postgresql import insert
insert_stmt = insert(my_table).values(
id='some_existing_id',
data='inserted value'
)
do_update_stmt = insert_stmt.on_conflict_do_update(
index_elements=['id'],
set_=dict(data='updated value')
)
conn.execute(do_update_stmt)
What I would like to do is combine these two things into a bulk upsert without having to ping my database three times (once to see which rows I need to insert and which to update, once to do the actual insert and once for an update). Does anyone know if there is a way to do that? Thanks for any help in advance!

count subquery in sqlalchemy

I'm having some trouble translating a subquery into sqlalchemy. I have two tables that both have a store_id column that is a foreign key (but it isn't a direct many-to-many relationship) and I need to return the id, store_id and name from table 1 along with the number of records from table 2 that also have the same store_id. I know the SQL that I would use to return those records I'm just now sure how to do it using sqlalchemy.
SELECT
table_1.id
table_1.store_id,
table_1.name,
(
SELECT
count(table_2.id)
FROM
table_2
WHERE
table_1.store_id = table_2.store_id
) AS store_count FROM table_1;

This post actually answered my question. I must have missed it when I was searching initially. My solution below.
Generate sql with subquery as a column in select statement using SQLAlchemy
store_count = session.query(func.count(Table2.id)).filter(Table2.store_id == Table1.store_id)
session.query.add_columns(Table1.id, Table1.name, Table1.store_id, store_count.label("store_count"))

Query data from schema.table in Postgres

I'm very new to sqlalchemy and I encountered a problem regards to Postgres databases. I can successfully connect to the postgresql database, and I think I've directed the engine to my desired schema.
cstr = f"postgresql+psycopg2://{username}:{password}#{server}:{port}/{database}"
engine = create_engine(cstr,connect_args={'options': '-csearch_path={}'.format("schema_name")},echo=True)
con = engine.connect()
print(con.execute('SELECT * FROM table_name'))
This prints out the correct schema_name.
insp = inspect(con)
print(insp.default_schema_name)
However, I still get error messages saying that the table does not exist.
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "table_name" does not exist
I also tried without the ,connect_args={'options': '-csearch_path={}'.format("google")} clause and use schema_name.table_name in the sql query. Same error occurs. It's not a local database, so I can't do anything to the database except getting data from it. What should I do here？

It's interesting how I searched the answers for hours and decided to ask instead. And right after, I found the solution. Just in case anyone is interested in the answer. I got my solution from this answer
Selecting data from schema based table in Postgresql using psycopg2
print(con.execute("""SELECT DISTINCT "column_name" FROM schema_name."table_name";"""))
This is the way to do it, with a looot of quotation marks

I don't know about your framework alchemy but the correct query should be something like that:
SELECT table_name FROM information_schema.tables WHERE table_schema='public'
Reference docs

Rather than manually quoting the identifiers in queries you can let SQLAlchemy do the work for you.
Given this table and data:
test# create table "Horse-Bus" (
test(# id integer generated always as identity,
test(# name varchar,
test(# primary key(id)
test(# );
CREATE TABLE
test#
test# insert into "Horse-Bus" (name) values ('Alice'), ('Bob'), ('Carol');
INSERT 0 3
You can create a Table object and query it like this:
>>>import sqlalchemy as sa
>>> engine = sa.create_engine('postgresql:///test', echo=False, future=True)
>>> tbl = sa.Table('Horse-Bus', sa.MetaData(), autoload_with=engine)
>>> with engine.connect() as conn:
... rows = conn.execute(sa.select(tbl))
... for row in rows:
... print(row)
...
(1, 'Alice')
(2, 'Bob')
(3, 'Carol')
>>>

Sqlalchemy simple query filter doesn't work. It returns all rows

I am using a simple sql query like this
SELECT * FROM MyTable WHERE id="Some ID"
so in Python SQLAlchemy like this
results = session.query(MyClass).filter(MyClass.id=="Some ID").all()
But it returns all rows in the table. In fact if I replace "Some ID" value with something random which doesn't even exist in the column it also returns all the rows. Quite weird for me.
Anything I am doing wrong here? Thanks

How to do a group by/count(*) pr year, month and day in django based on a datetime database field?

I have a table describing files with a datetime field. I want to somehow create a report that gives me the number of files grouped by each year, number of files grouped by each year and month and number of files grouped by each year, month and day. I just want records where count(*) > 0. Preferably using the ORM in django or if that`s not possible, using some SQL that runs on both PostgreSQL and SQLite.
The number of records in this database can be huge so my attempts to do this in code, not in SQL ( or indirectly in SQL thru ORM ) don't work and if I get it to work I don`t think it will scale at all.
Grateful for any hints or solutions.

Normally I work on Oracle but a quick google search showed that this should also work for Postgres. For the minutes you could do like this
select to_char(yourtimestamp,'yyyymmdd hh24:mi'), count(*)
from yourtable
group by to_char(yourtimestamp,'yyyymmdd hh24:mi')
order by to_char(yourtimestamp,'yyyymmdd hh24:mi') DESC;
That works then all the way down to years:
select to_char(yourtimestamp,'yyyy'), count(*)
from yourtable
group by to_char(yourtimestamp,'yyyy')
order by to_char(yourtimestamp,'yyyy') DESC;
You are only getting the years where you got something. I think that is what you wanted.
Edit: You need to build an index on "yourtimestamp" otherwise the performance is ugly if you do have a lot of rows.

My mistake - the date() function only works for MySql:
Maybe try this (SQLite):
tbl = MyTable.objects.filter()
tbl = tbl.extra(select={'count':'count(strftime('%Y-%m-%d', timestamp))', 'my_date':'strftime('%Y-%m-%d', timestamp))'}
tbl = tbl.values('count', 'my_date')
tbl.query.group_by = ['strftime('%Y-%m-%d', timestamp)']
For day and month, you could replace '%Y-%m-%d' with variations of the date format strings.
This was for MySQL (just in case someone needs it)
tbl = MyTable.objects.filter()
tbl = tbl.extra(select={'count':'count(date(timestamp))', 'my_date':'date(timestamp)'}
tbl = tbl.values('count', 'my_date')
tbl.query.group_by = ['date(timestamp)']
That works for year.