We Keep Coding

SQLAlchemy engine keeps some idle PostgreSQL database connection opened even when using a context manager

When using an SQLAlchemy engine for GeoPandas read_postgis() method to read data from a PostgreSQL/PostGIS database, there are plenty of busy database slots which stay open for nothing, often showing simple COMMIT or ROLLBACK query statements.
Here's a test.py code snippet which represents this problem:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os
os.environ['USE_PYGEOS'] = '0'
import geopandas as gpd
from sqlalchemy import create_engine, text
def foo():
engine = create_engine("postgresql://postgres:password#db:5432/postgres")
with engine.begin() as connection:
gdf0 = gpd.read_postgis(
sql = text("WITH cte AS (SELECT * FROM public.cities) SELECT *, ST_Buffer(geom,0.0001) FROM public.cities WHERE name ILIKE 'bog';"),
con=connection,
geom_col='geom',
crs=4326,
)
gdf1 = gpd.read_postgis(
sql = text("WITH cte AS (SELECT * FROM public.cities) SELECT *, ST_Buffer(geom,0.0002) FROM public.cities WHERE name ILIKE 'bra';"),
con=connection,
geom_col='geom',
crs=4326,
)
gdf2 = gpd.read_postgis(
sql = text("WITH cte AS (SELECT * FROM public.cities) SELECT *, ST_Buffer(geom,0.0003) FROM public.cities WHERE country ILIKE 'ven';"),
con=connection,
geom_col='geom',
crs=4326,
)
i=-1
while i < 100:
i+=1
foo()
As you can see, I'm using engine.begin(), which is known as "Connect and Begin Once" in the SQLAlchemy documentation:
A convenient shorthand form for the above “begin once” block is to use the Engine.begin() method at the level of the originating Engine object, rather than performing the two separate steps of Engine.connect() and Connection.begin(); the Engine.begin() method returns a special context manager that internally maintains both the context manager for the Connection as well as the context manager for the Transaction normally returned by the Connection.begin() method:
For convenience, here is also a database initialization script:
psql -U postgres -d postgres -c "CREATE TABLE IF NOT EXISTS cities (
id int PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
name text,
country text,
geom geometry(Point,4326)
);
INSERT INTO public.cities (name, country, geom) VALUES
('Buenos Aires', 'Argentina', ST_SetSRID(ST_Point(-58.66, -34.58), 4326)),
('Brasilia', 'Brazil', ST_SetSRID(ST_Point(-47.91,-15.78), 4326)),
('Santiago', 'Chile', ST_SetSRID(ST_Point(-70.66, -33.45), 4326)),
('Bogota', 'Colombia', ST_SetSRID(ST_Point(-74.08, 4.60), 4326)),
('Caracas', 'Venezuela', ST_SetSRID(ST_Point(-66.86,10.48), 4326));"
I usually run this Python code in multiple parallel instances as a dockerized application.
When using the following SQL snippet while the app is running, e.g. from pgAdmin4, I can see plenty of idle connections which are not properly closed. They stay in an idle state as long as the app is running, showing simple COMMIT queries. (In the whole application, there may also be some idle ROLLBACK queries).
SELECT state, query, *
FROM pg_stat_activity
WHERE application_name NOT ILIKE '%pgAdmin 4%'
Here is the result of the previous query:
If by any chance there are too much of those pending idle connections, I can also face the following error, certainly because there are no more available "slots" in the database for extra transactions to take place:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
self._dbapi_connection = engine.raw_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3269, in raw_connection
return self.pool.connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1255, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
self.__connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 640, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 580, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: sorry, too many clients already
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/test.py", line 42, in <module>
foo()
File "/app/test.py", line 19, in foo
with engine.begin() as connection:
File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3209, in begin
with self.connect() as conn:
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3245, in connect
return self._connection_cls(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
Connection._handle_dbapi_exception_noconnection(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2410, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[4]) from e
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
self._dbapi_connection = engine.raw_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3269, in raw_connection
return self.pool.connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1255, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
self.__connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 640, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 580, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: sorry, too many clients already
(Background on this error at: https://sqlalche.me/e/20/e3q8)
This error regularly prevents my application from working correctly with many errors, e.g.: psycopg2.OperationalError: FATAL: sorry, too many clients already.
I conclude that the Python code above is not perfectly written. Certainly because I didn't fully understand how the SQLAlchemy engine actually works.
Hence my question: could you explain what's wrong with this code and, most importantly, how could I properly close these queries?
Version info:
Python 3.9.7
geopandas==0.12.2
psycopg2==2.9.5
SQLAlchemy==2.0.1

If I recall correctly, create_engine creates a connection pool by default.
You can adjust idle connections by adjusting the pool parameters you give to create_engine.
See Connection pooling in SQLAlchemy documentation.

What's the proper SSL parameter to pass into sqlalchemy's create_engine?

I'm having trouble figuring out the proper parameters for connecting to a Postgres database using Django on Heroku with Python 3.7.10 and sqlalchemy 1.3.23. I know that's not the latest release of sqlalchemy, but this started failing the other week when Python was updated from 3.7 so I'm puzzled as to what's going on. I'm finding sqlalchemy's documentation a little hard to navigate and it doesn't help that this stopped working when just Python was updated.
So basically in Python, the connection is being set in this line:
import records as records_client
DB_URL_FROM_ENV_VAR = 'postgresql+pg8000://user:password#host:5432/db?ssl=True'
db = records_client.Database(DB_URL_FROM_ENV_VAR)
And when I go to the implementation of Database, I get this:
class Database(object):
"""A Database connection."""
def __init__(self, db_url=None, **kwargs):
# If no db_url was provided, fallback to $DATABASE_URL.
self.db_url = db_url or DATABASE_URL
if not self.db_url:
raise ValueError('You must provide a db_url.')
self._engine = create_engine(self.db_url, **kwargs)
# Connect to the database.
self.db = self._engine.connect()
self.open = True
That appears to be a part of sqlalchemy.
The error I get when I run the script is: TypeError: connect() got an unexpected keyword argument 'ssl'.
I've tried changing the parameter to ssl_mode and also sslmode since I've seen SO posts and issues posted with similar questions, but neither of those changes have done anything. I've also tried adding another parameter to the Database call like so: db = records_client.Database(DB_URL_FROM_ENV_VAR, connect_args={'sslmode':'require'}) but all to no avail.
How do I configure the database URL and/or the connection attempt with SSL to connect to the database?
Here's a full stack trace with the following settings:
DB_URL_FROM_ENV_VAR = 'postgresql+pg8000://user:password#host:5432'
db = records_client.Database(DB_URL_FROM_ENV_VAR, connect_args={'sslmode':'require'})
Traceback (most recent call last):
File "appdir/manage.py", line 14, in <module>
execute_from_command_line(sys.argv)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\django\core\management\__init__.py", line 381, in execute_from_command_line
utility.execute()
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\django\core\management\__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\django\core\management\base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\django\core\management\base.py", line 364, in execute
output = self.handle(*args, **options)
File "C:\Users\user\Github\django-app\appdir\apps\etl\management\commands\load.py", line 118, in handle
step.load(items=ds_items, period=period)
File "C:\Users\user\Github\django-app\appdir\apps\etl\models\job.py", line 89, in load
match_status = self.domain_source.load(*args, **kwargs)
File "C:\Users\user\Github\django-app\appdir\apps\etl\models\domain.py", line 90, in load
return self.load_profile.load(items=items, data=data, **kwargs)
File "C:\Users\user\Github\django-app\appdir\apps\etl\models\load_source.py", line 99, in load
db = records_client.Database(DB_URL_FROM_ENV_VAR, connect_args={'sslmode':'require'})
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\records.py", line 245, in __init__
self.db = self._engine.connect()
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\engine\base.py", line 2263, in connect
return self._connection_cls(self, **kwargs)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\engine\base.py", line 104, in __init__
else engine.raw_connection()
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\engine\base.py", line 2370, in raw_connection
self.pool.unique_connection, _connection
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\engine\base.py", line 2336, in _wrap_pool_connect
return fn()
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\base.py", line 304, in unique_connection
return _ConnectionFairy._checkout(self)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\base.py", line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\base.py", line 495, in checkout
rec = pool._do_get()
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\impl.py", line 140, in _do_get
self._dec_overflow()
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\util\compat.py", line 182, in raise_
raise exception
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\impl.py", line 137, in _do_get
return self._create_connection()
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\base.py", line 309, in _create_connection
return _ConnectionRecord(self)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\base.py", line 440, in __init__
self.__connect(first_connect_check=True)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\base.py", line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\util\compat.py", line 182, in raise_
raise exception
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\pool\base.py", line 656, in __connect
connection = pool._invoke_creator(self)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\engine\strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "C:\Users\user\Github\django-app\venv-dev\lib\site-packages\sqlalchemy\engine\default.py", line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
TypeError: connect() got an unexpected keyword argument 'sslmode'

Sqlalchemy+pyodbc for mysql and pd.to_sql() giving strange and inconsistent error

I am trying to connect, read and write from a mysql db using odbc(ODBC is a requirement and I'll have to use it) in python. I am using sqlalchemy and pyodbc. I am able to connect to the db and read. But when I am trying to write, pyodbc is giving me strange errors. The code is given below.
params = urllib3.parse.quote_plus("DRIVER={MySQL ODBC 8.0 ANSI Driver};"
f"SERVER={host}:{port};"
f"DATABASE={db};"
f"UID={username};"
f"PWD={password};"
f"charset=utf8")
db_engine = create_engine(f"mysql+pyodbc:///?odbc_connect={params}")
connection = db_engine.connect()
# main_df is a pd.DataFrame()
maindf.to_sql("mytable", connection, if_exists="append", index=False, chunksize=1000)
The Error is given below:
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\engine\base.py", line 1685, in _execute_context
self.dialect.do_executemany(
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\engine\default.py", line 713, in do_executemany
cursor.executemany(statement, parameters)
pyodbc.ProgrammingError: ('42000', "[42000] [MySQL][ODBC 8.0(a) Driver][mysqld-5.7.31-log]You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '4' at line 1 (1064) (SQLParamData)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "E:/myusername/cx-index-score/nice_rpa/pipeline.py", line 17, in <module>
uploader.split_upload(os.path.abspath(Path('./datasets')))
File "E:\myusername\cx-index-score\nice_rpa\processandupload.py", line 163, in split_upload
self.writetosandbox()
File "E:\myusername\cx-index-score\nice_rpa\processandupload.py", line 216, in writetosandbox
self.maindf.to_sql("nice_daily_update", self.connection, if_exists="append",
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\pandas\core\generic.py", line 2779, in to_sql
sql.to_sql(
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\pandas\io\sql.py", line 590, in to_sql
pandas_sql.to_sql(
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\pandas\io\sql.py", line 1405, in to_sql
raise err
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\pandas\io\sql.py", line 1397, in to_sql
table.insert(chunksize, method=method)
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\pandas\io\sql.py", line 831, in insert
exec_insert(conn, keys, chunk_iter)
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\pandas\io\sql.py", line 748, in _execute_insert
conn.execute(self.table.insert(), data)
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\engine\base.py", line 1200, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\sql\elements.py", line 313, in _execute_on_connection
return connection._execute_clauseelement(
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\engine\base.py", line 1389, in _execute_clauseelement
ret = self._execute_context(
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\engine\base.py", line 1748, in _execute_context
self._handle_dbapi_exception(
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\engine\base.py", line 1929, in _handle_dbapi_exception
util.raise_(
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\util\compat.py", line 211, in raise_
raise exception
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\engine\base.py", line 1685, in _execute_context
self.dialect.do_executemany(
File "C:\ProgramData\Anaconda3\envs\nice_rpa\lib\site-packages\sqlalchemy\engine\default.py", line 713, in do_executemany
cursor.executemany(statement, parameters)
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [MySQL][ODBC 8.0(a) Driver][mysqld-5.7.31-log]You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '33\x03#333333\x03#333333\x03#' at line 1 (1064) (SQLParamData)")
If you see the syntax error part, it is actually supposed to be a simple string, which is utf-8. The Databse encoding is 'latin1'. Interestingly, even though each time the data and the error are same, but the "wrong syntax" part always changes. once, it was '4', then it was '33\x03#333333\x03#333333\x03#' and it changes on each run, the input data is always same though.
I have exhausted all the other stackoverflow questions related to this topic I can find. Please help.

Connect Cloud Run container to Cloud SQL

UPDATE:
I re-forked the GoogleCloudPlatform example project and tried again. Suddenly, it's working.
Half of my problem had to do with the fact that my target project uses Flask-SQLAlchemy. For that, I needed to use the MySQLdb dialect as shown in this answer:
https://stackoverflow.com/a/10900826/4455571
I'm still not sure why I couldn't get the GoogleCloudPlatform example to work the first time and why it suddenly started working after I re-forked.
Original post:
I am trying to connect a container running on Cloud Run to Cloud SQL using the following guide:
https://cloud.google.com/sql/docs/mysql/connect-run?hl=en_US
I have made sure to do the following:
Enable the API for my project
Add the "Cloud SQL Client" role to my
service account REDACTED-compute#developer.gserviceaccount.com
However, the connection fails with the following error:
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py",
line 630, in connect raise exc sqlalchemy.exc.OperationalError:
(pymysql.err.OperationalError) (2003, "Can't connect to MySQL server
on 'localhost' ([Errno 2] No such File or directory)")
I made a second attempt using the sample code from GitHub:
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/cloud-sql/mysql/sqlalchemy
These are the steps I followed:
Mirrored the repository and made it private
Edited app.yaml to add my credentials similar to the following (I didn't wrap the values in quotes - should I?):
env_variables:
CLOUD_SQL_CONNECTION_NAME: my-project-270323:us-central1:database
DB_USER: root
DB_PASS: areallygreatpassword
DB_NAME: database
Connected the repo to Cloud Build and triggered a new build
Deployed my container in a new service on Cloud Run making sure to select by database under Connections > Cloud SQL connections
I get the same error. What am I doing wrong?
EDIT
Here's the full error dump:
Traceback (most recent call last):()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app response = self.full_dispatch_request()()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1944, in full_dispatch_request self.try_trigger_before_first_request_functions()()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1992, in try_trigger_before_first_request_functions func()()
File "/app/main.py", line 81, in create_tables with db.connect() as conn:()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2209, in connect return self._connection_cls(self, **kwargs)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 103, in __init__ else engine.raw_connection()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2306, in raw_connection return self._wrap_pool_connect(()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect Connection._handle_dbapi_exception_noconnection(()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1547, in _handle_dbapi_exception_noconnection util.raise_from_cause(sqlalchemy_exception, exc_info)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 152, in reraise raise value.with_traceback(tb)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect return fn()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection return _ConnectionFairy._checkout(self)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout fairy = _ConnectionRecord.checkout(pool)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 492, in checkout rec = pool._do_get()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get self._dec_overflow()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ compat.reraise(exc_type, exc_value, exc_tb)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 153, in reraise raise value()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get return self._create_connection()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection return _ConnectionRecord(self)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 437, in __init__ self.__connect(first_connect_check=True)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 652, in __connect connection = pool._invoke_creator(self)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 489, in connect return self.dbapi.connect(*cargs, **cparams)()
File "/usr/local/lib/python3.8/site-packages/pymysql/__init__.py", line 94, in Connect return Connection(*args, **kwargs)()
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 325, in __init__ self.connect()()
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 630, in connect raise exc sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' ([Errno 2] No such()
File or directory)")
Also, I did this via the web console, not the CLI. Here are the settings I passed to the example project provided by Google:
drive.google.com/open?id=17nl_rQVTU2ZirCEu64bjfe90zGweg3a6
drive.google.com/open?id=1_Riy1HNSPvZZUGl4tJ0Z8puYuuMbUMPH

The question includes some wide-ranging research, nicely done. Addressing a couple points directly:
The app.yaml file is specific to App Engine, and is a method to pass Environment Variable configuration to your app. In Cloud Run, you use the --update-env-vars flag on deploy.
You may also be missing a flag in your service deploy operation: --add-cloudsql-instances INSTANCE-CONNECTION-NAME
These steps together:
gcloud run deploy SERVICE --image gcr.io/PROJECT/SERVICE \
--add-cloudsql-instances my-project-270323:us-central1:database
--update-env-vars CLOUD_SQL_CONNECTION_NAME=my-project-270323:us-central1:database \
--update-env-vars DB_USER=root \
--update-env-vars DB_PASS=areallygreatpassword \
--update-env-vars DB_NAME=database
Note that database in the INSTANCE_CONNECTION_NAME is the Cloud SQL instance, while database as the DB_NAME is a database created within that instance.
The configuration values need to be used as part of creating your database connection.
You can see more of the connection between these environment variables and creating the connection object in the complete sample code on Github.

LOAD DATA LOCAL INFILE sqlalchemy and python to mysql db

I'm trying to use LOAD LOCAL DATA INFILE with sqlalchemy to upload batch files to my server. I have edited my /etc/mysql/my.conf to contain local-infile == 1 under both [mysql] and [mysqld], yet I'm getting this traceback when I run the program.
The relevant line of code is:
Traceback (most recent call last):
File "main.py", line 48, in <module>
con.execute(sql)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 906, in execute
return self._execute_text(object, multiparams, params)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1054, in _execute_text
statement, parameters
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1332, in _handle_dbapi_exception
exc_info
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
cursor.execute(statement, parameters)
File "/Users/eoddata/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/Users/eoddata/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1148, 'The used command is not allowed with
this MySQL version') [SQL: "LOAD DATA LOCAL
INFILE 'stocks/AMEX_20150420.txt' INTO TABLE database.new_table\n
FIELDS TERMINATED BY ',' (Symbol,#Date,Open,High,Low,Close,Volume)\n
SET Date = STR_TO_DATE(#Date,'%%Y%%m%%d');"]
I've done some research from the docs and can't find a way to fix this error. With python MySQLdb (from pip install MySQL-Python) will allow me to pass in a local-infile paramter. I can't find an equivalent in SQLAlchemy. Any help is appreciated.
EDIT: I've worked on this all night and still haven't made any progress - a lot of dead ends. Any ideas?
Thanks, Jared

You should add local_infile=1 to the connection string.
SQLALCHEMY_DATABASE_URI = "mysql://user:pass#localhost/dbname?charset=utf8&local_infile=1"