UPDATE:
I re-forked the GoogleCloudPlatform example project and tried again. Suddenly, it's working.
Half of my problem had to do with the fact that my target project uses Flask-SQLAlchemy. For that, I needed to use the MySQLdb dialect as shown in this answer:
https://stackoverflow.com/a/10900826/4455571
I'm still not sure why I couldn't get the GoogleCloudPlatform example to work the first time and why it suddenly started working after I re-forked.
Original post:
I am trying to connect a container running on Cloud Run to Cloud SQL using the following guide:
https://cloud.google.com/sql/docs/mysql/connect-run?hl=en_US
I have made sure to do the following:
Enable the API for my project
Add the "Cloud SQL Client" role to my
service account REDACTED-compute#developer.gserviceaccount.com
However, the connection fails with the following error:
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py",
line 630, in connect raise exc sqlalchemy.exc.OperationalError:
(pymysql.err.OperationalError) (2003, "Can't connect to MySQL server
on 'localhost' ([Errno 2] No such File or directory)")
I made a second attempt using the sample code from GitHub:
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/cloud-sql/mysql/sqlalchemy
These are the steps I followed:
Mirrored the repository and made it private
Edited app.yaml to add my credentials similar to the following (I didn't wrap the values in quotes - should I?):
env_variables:
CLOUD_SQL_CONNECTION_NAME: my-project-270323:us-central1:database
DB_USER: root
DB_PASS: areallygreatpassword
DB_NAME: database
Connected the repo to Cloud Build and triggered a new build
Deployed my container in a new service on Cloud Run making sure to select by database under Connections > Cloud SQL connections
I get the same error. What am I doing wrong?
EDIT
Here's the full error dump:
Traceback (most recent call last):()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app response = self.full_dispatch_request()()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1944, in full_dispatch_request self.try_trigger_before_first_request_functions()()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1992, in try_trigger_before_first_request_functions func()()
File "/app/main.py", line 81, in create_tables with db.connect() as conn:()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2209, in connect return self._connection_cls(self, **kwargs)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 103, in __init__ else engine.raw_connection()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2306, in raw_connection return self._wrap_pool_connect(()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect Connection._handle_dbapi_exception_noconnection(()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1547, in _handle_dbapi_exception_noconnection util.raise_from_cause(sqlalchemy_exception, exc_info)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 152, in reraise raise value.with_traceback(tb)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect return fn()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection return _ConnectionFairy._checkout(self)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout fairy = _ConnectionRecord.checkout(pool)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 492, in checkout rec = pool._do_get()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get self._dec_overflow()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ compat.reraise(exc_type, exc_value, exc_tb)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 153, in reraise raise value()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get return self._create_connection()()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection return _ConnectionRecord(self)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 437, in __init__ self.__connect(first_connect_check=True)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 652, in __connect connection = pool._invoke_creator(self)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams)()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 489, in connect return self.dbapi.connect(*cargs, **cparams)()
File "/usr/local/lib/python3.8/site-packages/pymysql/__init__.py", line 94, in Connect return Connection(*args, **kwargs)()
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 325, in __init__ self.connect()()
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 630, in connect raise exc sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' ([Errno 2] No such()
File or directory)")
Also, I did this via the web console, not the CLI. Here are the settings I passed to the example project provided by Google:
drive.google.com/open?id=17nl_rQVTU2ZirCEu64bjfe90zGweg3a6
drive.google.com/open?id=1_Riy1HNSPvZZUGl4tJ0Z8puYuuMbUMPH
The question includes some wide-ranging research, nicely done. Addressing a couple points directly:
The app.yaml file is specific to App Engine, and is a method to pass Environment Variable configuration to your app. In Cloud Run, you use the --update-env-vars flag on deploy.
You may also be missing a flag in your service deploy operation: --add-cloudsql-instances INSTANCE-CONNECTION-NAME
These steps together:
gcloud run deploy SERVICE --image gcr.io/PROJECT/SERVICE \
--add-cloudsql-instances my-project-270323:us-central1:database
--update-env-vars CLOUD_SQL_CONNECTION_NAME=my-project-270323:us-central1:database \
--update-env-vars DB_USER=root \
--update-env-vars DB_PASS=areallygreatpassword \
--update-env-vars DB_NAME=database
Note that database in the INSTANCE_CONNECTION_NAME is the Cloud SQL instance, while database as the DB_NAME is a database created within that instance.
The configuration values need to be used as part of creating your database connection.
You can see more of the connection between these environment variables and creating the connection object in the complete sample code on Github.
Related
When using an SQLAlchemy engine for GeoPandas read_postgis() method to read data from a PostgreSQL/PostGIS database, there are plenty of busy database slots which stay open for nothing, often showing simple COMMIT or ROLLBACK query statements.
Here's a test.py code snippet which represents this problem:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os
os.environ['USE_PYGEOS'] = '0'
import geopandas as gpd
from sqlalchemy import create_engine, text
def foo():
engine = create_engine("postgresql://postgres:password#db:5432/postgres")
with engine.begin() as connection:
gdf0 = gpd.read_postgis(
sql = text("WITH cte AS (SELECT * FROM public.cities) SELECT *, ST_Buffer(geom,0.0001) FROM public.cities WHERE name ILIKE 'bog';"),
con=connection,
geom_col='geom',
crs=4326,
)
gdf1 = gpd.read_postgis(
sql = text("WITH cte AS (SELECT * FROM public.cities) SELECT *, ST_Buffer(geom,0.0002) FROM public.cities WHERE name ILIKE 'bra';"),
con=connection,
geom_col='geom',
crs=4326,
)
gdf2 = gpd.read_postgis(
sql = text("WITH cte AS (SELECT * FROM public.cities) SELECT *, ST_Buffer(geom,0.0003) FROM public.cities WHERE country ILIKE 'ven';"),
con=connection,
geom_col='geom',
crs=4326,
)
i=-1
while i < 100:
i+=1
foo()
As you can see, I'm using engine.begin(), which is known as "Connect and Begin Once" in the SQLAlchemy documentation:
A convenient shorthand form for the above “begin once” block is to use the Engine.begin() method at the level of the originating Engine object, rather than performing the two separate steps of Engine.connect() and Connection.begin(); the Engine.begin() method returns a special context manager that internally maintains both the context manager for the Connection as well as the context manager for the Transaction normally returned by the Connection.begin() method:
For convenience, here is also a database initialization script:
psql -U postgres -d postgres -c "CREATE TABLE IF NOT EXISTS cities (
id int PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
name text,
country text,
geom geometry(Point,4326)
);
INSERT INTO public.cities (name, country, geom) VALUES
('Buenos Aires', 'Argentina', ST_SetSRID(ST_Point(-58.66, -34.58), 4326)),
('Brasilia', 'Brazil', ST_SetSRID(ST_Point(-47.91,-15.78), 4326)),
('Santiago', 'Chile', ST_SetSRID(ST_Point(-70.66, -33.45), 4326)),
('Bogota', 'Colombia', ST_SetSRID(ST_Point(-74.08, 4.60), 4326)),
('Caracas', 'Venezuela', ST_SetSRID(ST_Point(-66.86,10.48), 4326));"
I usually run this Python code in multiple parallel instances as a dockerized application.
When using the following SQL snippet while the app is running, e.g. from pgAdmin4, I can see plenty of idle connections which are not properly closed. They stay in an idle state as long as the app is running, showing simple COMMIT queries. (In the whole application, there may also be some idle ROLLBACK queries).
SELECT state, query, *
FROM pg_stat_activity
WHERE application_name NOT ILIKE '%pgAdmin 4%'
Here is the result of the previous query:
If by any chance there are too much of those pending idle connections, I can also face the following error, certainly because there are no more available "slots" in the database for extra transactions to take place:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
self._dbapi_connection = engine.raw_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3269, in raw_connection
return self.pool.connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1255, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
self.__connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 640, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 580, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: sorry, too many clients already
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/test.py", line 42, in <module>
foo()
File "/app/test.py", line 19, in foo
with engine.begin() as connection:
File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3209, in begin
with self.connect() as conn:
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3245, in connect
return self._connection_cls(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
Connection._handle_dbapi_exception_noconnection(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2410, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[4]) from e
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
self._dbapi_connection = engine.raw_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3269, in raw_connection
return self.pool.connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1255, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
self.__connect()
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
with util.safe_reraise():
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 640, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 580, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: sorry, too many clients already
(Background on this error at: https://sqlalche.me/e/20/e3q8)
This error regularly prevents my application from working correctly with many errors, e.g.: psycopg2.OperationalError: FATAL: sorry, too many clients already.
I conclude that the Python code above is not perfectly written. Certainly because I didn't fully understand how the SQLAlchemy engine actually works.
Hence my question: could you explain what's wrong with this code and, most importantly, how could I properly close these queries?
Version info:
Python 3.9.7
geopandas==0.12.2
psycopg2==2.9.5
SQLAlchemy==2.0.1
If I recall correctly, create_engine creates a connection pool by default.
You can adjust idle connections by adjusting the pool parameters you give to create_engine.
See Connection pooling in SQLAlchemy documentation.
I tried to push my Flask app to heroku and my app uses Postgres as db. However, when I tried to migrate db on heroku using this command
heroku run python manage.py db init --app app
It gives this "too many clients" error for both psycopg2 and sqlalchchemy.
Running python manage.py db migrate on ⬢ petscom... up, run.5707 (Hobby)
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect
return fn()
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
return _ConnectionFairy._checkout(self)
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 238, in _do_get
return self._create_connection()
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
connection = pool._invoke_creator(self)
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 488, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: sorry, too many clients already
FATAL: sorry, too many clients already
After a bit googling, I tried:
heroku pg:killall
heroku restart
None of those works.....
And I checked heroku pg, it shows 0/20 connections......
Some said dont put db connection in loop, but I have to check if some data is in db. Here's my code:
try:
connection = psycopg2.connect(user="postgres",
password="pass",
host="127.0.0.1",
port="5432",
database="pets")
cursor = connection.cursor()
select_all = "select * from pets"
cursor.execute(select_all)
# Check if api data in db and save to db
for index in range(len(myData)):
# if not exist in db => save to db
key = myData[index]["animal_id"]
exists = check_exists(key, cursor)
if(exists):
pass
else:
doSomething()
I can connect to my local postgres db but not the db on heroku.
How can I deal with this problem? Any advise would help :) Thanks!
I had a similar problem. I was able to fix it by setting SQLALCHEMY_TRACK_MODIFICATIONS=False in the environment, destroying the database, and creating a new one.
You can destroy it by going to https://data.heroku.com/datastores, opening the database, going to Settings, and selecting 'Destroy Database'.
Then provision a new one as you normally would.
I started Kiwi TCMS in Docker (k8s) with Postgresql. It started ok, but when I wanted to run migrations it crashed. I used this docker-compose for inspiration.
Log shows problem with connection with MySQL. But I changed configuration for engine. See:
bash-4.2$ KIWI_DB_ENGINE=django.db.backends.postgresql_psycopg2 /Kiwi/manage.py migrate
django.db.backends.postgresql_psycopg2
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
self.connect()
File "/venv/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
self.connection = self.get_new_connection(conn_params)
File "/venv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
return Database.connect(**conn_params)
File "/venv/lib/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
return Connection(*args, **kwargs)
File "/venv/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Kiwi/manage.py", line 13, in <module>
execute_from_command_line(sys.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
self.execute(*args, **cmd_options)
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 350, in execute
self.check()
File "/venv/lib/python3.6/site-packages/django/core/management/base.py", line 379, in check
include_deployment_checks=include_deployment_checks,
File "/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 59, in _run_checks
issues = run_checks(tags=[Tags.database])
File "/venv/lib/python3.6/site-packages/django/core/checks/registry.py", line 71, in run_checks
new_errors = check(app_configs=app_configs)
File "/venv/lib/python3.6/site-packages/django/core/checks/database.py", line 10, in check_database_backends
issues.extend(conn.validation.check(**kwargs))
File "/venv/lib/python3.6/site-packages/django/db/backends/mysql/validation.py", line 9, in check
issues.extend(self._check_sql_mode(**kwargs))
File "/venv/lib/python3.6/site-packages/django/db/backends/mysql/validation.py", line 13, in _check_sql_mode
with self.connection.cursor() as cursor:
File "/venv/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor
return self._cursor()
File "/venv/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor
self.ensure_connection()
File "/venv/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
self.connect()
File "/venv/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/venv/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
self.connect()
File "/venv/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
self.connection = self.get_new_connection(conn_params)
File "/venv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
return Database.connect(**conn_params)
File "/venv/lib/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
return Connection(*args, **kwargs)
File "/venv/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0")
Any ideas where can be problem please?
bash-4.2$ KIWI_DB_ENGINE=django.db.backends.postgresql_psycopg2 /Kiwi/manage.py migrate
django.db.backends.postgresql_psycopg2
This is setting an environment variable inside the docker container and I'm not sure if that works or which one takes precendece.
If you want somebody to help you post the entire docker-compose.yml (or whatever kubernetes control file you use). Clearly the application thinks it is configured to talk to MySQL and fails when MySQL isn't available.
So problem was with version of used docker image.
I used docker image tag kiwitcms/kiwi:6.2 and I thought it's the latest release. But it's not. This version(docker image) doesn't have code for changing DB engine. I changed version of docker tag to latest and it works pretty well now.
I am trying to run following code snippet in python to connect to oracle, but constantly running into following error. I have tried a lot of combinations but it doesn't seem to work. I understand the error, but don't understand what is incompatible here.
Has anyone come across this issue? How do I fix it?
File "", line 1, in File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1613, in execute
connection = self.contextual_connect(close_with_result=True) File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
self.pool.connect(), File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/pool.py", line 326, in connect
return _ConnectionFairy(self).checkout() File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/pool.py", line 485, in __init__
rec = self._connection_record = pool._do_get() File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/pool.py", line 770, in _do_get
return self._create_connection() File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/pool.py", line 279, in _create_connection
return _ConnectionRecord(self) File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/pool.py", line 372, in __init__
self.connection = self.__connect() File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/pool.py", line 433, in __connect
connection = self.__pool._creator() File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
return dialect.connect(*cargs, **cparams) File "/workplace/applications/python2.7/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 283, in connect
return self.dbapi.connect(*cargs, **cparams)
TypeError: expecting string, unicode or buffer object
from sqlalchemy.ext.declarative import declarative_base;
from sqlalchemy import create_engine;
engine = create_engine(u'oracle+cx_oracle://localhost:1521/orcl', echo=True)
result = engine.execute(u"select 1 from dual");
Setup:
Python 2.7
SqlAlchemy 0.9.7 and 0.8.7
Cx Oracle (latest version)
Oracle Database 10g Release 10.2.0.2.0
If you are running into this problem, most likely the cause is that you are not passing in arguments required by the underlying dbapi call.
In my case I added additional arguments of user, password and dsn to the create_engine call along with existing ones, which got passed to cx_oracle call and it worked.
something like this should work
create_engine(u'oracle+cx_oracle://localhost:1521/orcl', echo=True, user='<>', password='<>', dsn='<>')
I'm trying to run a django project locally on a brand new Mac. It was working up until yesterday, and I have no clue what happened, but all of a sudden I started getting Redis errors.
I can load pages that don't query the database, but once I try to do a search, which relies partly on Redis, it doesn't work.
Any ideas? here is the traceback.
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/Library/Python/2.7/site-packages/django/contrib/staticfiles/handlers.py", line 67, in __call__
return self.application(environ, start_response)
File "/Library/Python/2.7/site-packages/django/core/handlers/wsgi.py", line 241, in __call__
response = self.get_response(request)
File "/Library/Python/2.7/site-packages/django/core/handlers/base.py", line 179, in get_response
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
File "/Library/Python/2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/Users/dlitwak/mozio/dotcloud/demo/search/views.py", line 391, in results
cache.setDistanceAndDuration(distance, time, request.user.username)
File "/Users/dlitwak/mozio/dotcloud/demo/cache.py", line 305, in setDistanceAndDuration
self.cache.set(key, value, 1800)
File "/Library/Python/2.7/site-packages/redis_cache/cache.py", line 218, in set
result = self._set(key, pickle.dumps(value), int(timeout), client, _add_only)
File "/Library/Python/2.7/site-packages/redis_cache/cache.py", line 199, in _set
return client.setex(key, value, timeout)
File "/Library/Python/2.7/site-packages/redis/client.py", line 1221, in setex
return self.execute_command('SETEX', name, time, value)
File "/Library/Python/2.7/site-packages/redis/client.py", line 338, in execute_command
connection.send_command(*args)
File "/Library/Python/2.7/site-packages/redis/connection.py", line 287, in send_command
self.send_packed_command(self.pack_command(*args))
File "/Library/Python/2.7/site-packages/redis/connection.py", line 269, in send_packed_command
self.connect()
File "/Library/Python/2.7/site-packages/redis/connection.py", line 217, in connect
raise ConnectionError(self._error_message(e))
ConnectionError: Error 2 connecting to unix socket: 127.0.0.1. No such file or directory.
We are running MYSQL. I can access the db through the terminal, so I don't think it's a DB access problem.
Sometimes this is because Django cannot connect to the database, and redis is throwing the error.
And since you can load pages without db, seems like it’s the case.
Ok, so we finally got it to work.
For some reason Redis is no longer being started automatically and is not running in the background.
I'm working on a Mac, and it previously did this automatically when I was running on Ubuntu. The solution was to run "redis-server" in a separate terminal window.
We are still trying to figure out how to make it start automatically, or why it stopped in the first place, but yeah.
If your using a Mac, using the Homebrew package manager is probably the best way to install and manage Redis. Once Homebrew is installed simply, brew install redis on the command line.
After it's installed you can set it up to run automatically on startup:
ln -sfv /usr/local/opts/redis/*.plist ~/Library/LaunchAgents
Then to run it now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis.plist