I am looking for a way to get the size of a database with SQL Alchemy. Ideally, it will be agnostic to which underlying type of database is used. Is this possible?
Edit:
By size, I mean total number of bytes that the database uses.
The way I would do is to find out if you can run a SQL query to get the answer. Then, you can just run this query via SQLAlchemy and get the result.
Currently, SQLAlchemy does not provide any convenience function to determine the table size in bytes. But you can execute an SQL statement. Caveat is that you have to use a statement that is specific to your type of SQL (MySQL, Postgres, etc)
Checking this answer, for MySQL you can execute a statement manually like:
from sqlalchemy import create_engine
connection_string = 'mysql+pymysql://...'
engine = create_engine(connection_string)
statement = 'SELECT table_schema, table_name, data_length, index_length FROM information_schema.tables'
with engine.connect() as con:
res = con.execute(statement)
size_of_table = res.fetchall()
For SQLite you can just check the entire database size with the os module:
import os
os.path.getsize('sqlite.db')
For PostgreSQL you can do it like this:
from sqlalchemy.orm import Session
dbsession: Session
engine = dbsession.bind
database_name = engine.url.database
# https://www.a2hosting.com/kb/developer-corner/postgresql/determining-the-size-of-postgresql-databases-and-tables
# Return the database size in bytes
database_size = dbsession.execute(f"""SELECT pg_database_size('{database_name}')""").scalar()
Related
According to SQLAlchemy documentation you are supposed to use Session object when executing SQL statements. But using a Session with Pandas .read_sql, gives an error: AttributeError 'Session' object has no attribute 'cursor'.
However using the Connection object works even with the ORM Mapped Class:
with ENGINE.connect() as conn:
df = pd.read_sql_query(
sqlalchemy.select(MeterValue),
conn
)
Where MeterValue is a Mapped Class.
This doesn't feel like the correct solution, because SQLAlchemy documentation says you are not supposed to use engine connection with ORM. I just can't find out why.
Does anyone know if there is any issue using the connection instead of Session with ORM Mapped Class?
What is the correct way to read sql in to a DataFrame using SQLAlchemy ORM?
I found a couple of old answers on this where you use the engine directly as the second argument, or use session.bind and so on. Nothing works.
Just reading the documentation of pandas.read_sql_query:
pandas.read_sql_query(sql, con, index_col=None, coerce_float=True, params=None, parse_dates=None, chunksize=None, dtype=None)
Parameters:
sql: str SQL query or SQLAlchemy Selectable (select or text object)
SQL query to be executed.
con: SQLAlchemy connectable, str, or sqlite3 connection
Using SQLAlchemy makes it possible to use any DB supported by that library. If a DBAPI2 object, only sqlite3 is supported.
...
So pandas does allow a SQLAlchemy Selectable (e.g. select(MeterValue)) and a SQLAlchemy connectable (e.g. engine.connect()), so your code block is correct and pandas will handle the querying correctly.
with ENGINE.connect() as conn:
df = pd.read_sql_query(
sqlalchemy.select(MeterValue),
conn,
)
I need to create a SQLAlchemy engine for a database, that doesn't use the default schema. What I want to be able to do is something like this:
from sqlalchemy import create_engine
string = "oracle+cx_oracle://batman:batpassword#batcave.com:1525/some_database"
engine = create_engine(string, schema="WEIRD_SCHEMA")
tables = engine.table_names()
Is there a way to do this?
I'm working with some legacy code, that uses engine.table_names() method.
Here is the answer for getting raw sql from insert and I am wondering whether I can get raw sql if I use session.add() such as:
session.add(ormclass).compile()
If you want SQLAlchemy to log the actual raw SQL queries you can always set echo=True on the create_engine method. I know this seems rudimentary but it's the easiest way to see executed queries.
engine = create_engine("postgresql://localhost/dbname", echo=True)
My temporal solution to this is that when an error happens, there will be an exception and in this exception there are the parameters and statement.
except Exception as inst:
print(inst.statement % inst.params)
But the problem still exists because I cannot get the statement and parameters if there are no exceptions. In addition, there are no quotation marks for strings in the print so the string cannot be executed in mysql directly.
The way that I display the raw sql (which works most [but not all] of the time):
query = [my query that I created]
from sqlalchemy.dialects import mysql
str_query = query.compile(dialect=mysql.dialect(), compile_kwargs={"literal_binds": True})
Then I can just print the sql_query statement and it will show me the query with arguments. Some queries won't display, such as bulk inserts.
Can we specify a different cursor when executing queries via SQLAlchemy.
I want to be able to use Realdict cursor (as mentioned in this article: http://alexharvey.eu/database/postgresql/postgresql-query-result-as-python-dictionary/) with SQLAlchemy
This is a sample code of how the query is executed:
with connection:
result = connection.execute(select(columns_to_select).where(clause))
row = result.fetchone()
I am using sqlalchemy for connection pooling only (need to call existing procs) and want to return a REF CURSOR which is an out parameter.
There seems to be no cursor in sqlalchemy to do this.
Any advice greatly appreciated.
Gut feel - you may have to dive down a lower level than SQLAlchemy, perhaps to the underlying cx_oracle classes.
From an answer provided by Gerhard Häring on another forum :
import cx_Oracle
con = cx_Oracle.connect("me/secret#tns")
cur = con.cursor()
outcur = con.cursor()
cur.execute("""
BEGIN
MyPkg.MyProc(:cur);
END;""", cur=outcur)
for row in out_cur:
print row
I would presume that as SQLAlchemy uses cx_oracle under the hood there should be some way to use the same pooled connections.
Is there any option to wrap your function returning the REF CURSOR in a view on the Oracle side?? (Provided the shape of the REF CURSOR doesn't change, and you can somehow get the right parameters into your function - possibly by sticking them in as session variables, or in a temp table, this should work - I've used this approach to retrieve data from a REF CURSOR function to a language that only supports a limited subset of Oracle features).