I use my Macbook Pro to develop DBT Models against Databricks. Everything was working well enough, but I couldn't leave well enough alone and recently ran "dbt update" and saw that a bunch of things were outdated. I had run into issues in the past where upgrading my Python (install via Brew) broke things so I avoided upgrading Python, but upgraded SQLite. Apparently, an upgrade Python was a dependency and upgraded Python. So, DBT no longer works.
After lots of searching, I ran
brew remove dbt
brew install dbt
since the brew install dbt told me that installing dbt-labs/dbt/dbt has been deprecated, I instead ran brew install dbt-postgre figuring I will get the core DBT files install this way.
then per https://github.com/databricks/dbt-databricks, I ran
pip install dbt-databricks
All seemed mostly well except for the standard warning about "distutils config files is deprecated", etc...
Now, I run dbt debug and I get
Running with dbt=1.2.2
dbt version: 1.2.2
python version: 3.9.14
python path: /opt/homebrew/Cellar/dbt-postgres/1.2.2/libexec/bin/python
os info: macOS-12.6-arm64-arm-64bit
Using profiles.yml file at /Users/andrewpark/.dbt/profiles.yml
Using dbt_project.yml file at /Users/andrewpark/iCloud Drive (Archive)/Documents/Work >Projects/Github Source/data-platform/transformation/databricks-dbt/dbt_project.yml
18:24:13 target not specified in profile 'databricks_sql', using 'default'
18:24:13 Error importing adapter: No module named 'dbt.adapters.databricks'
Configuration:
profiles.yml file [ERROR invalid]
dbt_project.yml file [OK found and valid]
Required dependencies:
- git [OK found]
1 check failed:
Profile loading failed for the following reason:
Runtime Error
Credentials in profile "databricks_sql", target "default" invalid: Runtime Error
Could not find adapter type databricks!
I haven't touched my profiles.yml file at all so nothing there has changed, but it says the file is invalid. It appears to me that the dbt-databricks adaptor is not being found. How can I resolve this?
OS: MacOS Monterey 12.6
Python Version: 3.10.7
DBT Version (Core): 1.2.2
Also, I had installed dbt-snowflake and dbt-postgres adaptors as well as dbt-databricks, but these adaptors don't appear when I run dbt --version, only the Postgres 1.2.2 plugin.
configure profile.yml
your_profile_name:
target: dev
outputs:
dev:
type: databricks
catalog:
schema:
host:
http_path:
token:
I am using Cx_oracle for a while in Python but I can not recently import cx_oracle in Python and got this error:
ImportError: DLL load failed: Insufficient system resources exist to complete the requested service.
In addition, I get McAfee security alert as shown below:
I reinstalled cx_oracle a few times and it has not worked. Any suggestions?
The easiest answer is to use an older version of cx_Oracle using the following commands:
pip uninstall cx_Oracle
pip install -Iv cx_Oracle=="enter the version number here"
The reason is that McAfee might not integrate with the latest version of cx_Oracle or any other Python modules.
There are several other solutions including:
Uninstalling McAfee or any antivirus software that you have
Updating the drivers
Scanning the disk drivers to solve the issue
Modify the registry
I am trying to connect to my postgres database using psycopg2 with sslmode='required' param; however, I get the following error
psycopg2.OperationalError: sslmode value "require" invalid when SSL support is not compiled in
Heres a couple details about my system
Mac OS X El Capitan
Python 2.7
Installed psycopg2 via pip
Installed python via homebrew
Here is what I tried to do to fix the problem
brew uninstall python
which python still shows python living in /usr/local/bin/python, tried to uninstall this but couldnt. And heard that this is the python that the OS uses and should not be uninstalled anyways
brew install python --with-brewed-openssl --build-from-source
pip uninstall psycopg2
pip install psycopg2
After doing all of this, the exception still happens. I am running this python script via #!/usr/bin/env python Not sure if it matters, but that is a different directory than the one that which python shows
Since you're installing via pip, you should be using the most recent version of psycopg2 (2.6.1).
After a little digging through the code, it seems that the exception is being thrown in connection_int.c, which directly calls the postgresql-c-libraries to set up the db-connection. The call happens like so:
self->pgconn = pgconn = PQconnectStart(self->dsn);
Dprintf("conn_connect: new postgresql connection at %p", pgconn);
if (pgconn == NULL)
{
Dprintf("conn_connect: PQconnectStart(%s) FAILED", self->dsn);
PyErr_SetString(OperationalError, "PQconnectStart() failed");
return -1;
}
else if (PQstatus(pgconn) == CONNECTION_BAD)
{
Dprintf("conn_connect: PQconnectdb(%s) returned BAD", self->dsn);
PyErr_SetString(OperationalError, PQerrorMessage(pgconn));
return -1;
}
The keywords which were specified in your connect statement to psycopg2.connect() are being handled to that function and errors are returned as OperationalError exception.
The error is actually being generated directly in the postgresql-lib - you may want to check which version you are using, how it was built and, if possible, upgrade it to a version with SSL support or rebuilt it from source with SSL enabled.
The postgresql-docs also state that missing SSL support will raise an error, if the sslmode is set to require, verify-ca or verify-full. See here under sslmode for reference.
The postgres-website lists several ways to install postgres from binary packages, so you might choose one which suits your needs. I'm not familiar with OSX, so I don't have a recommendation what's best.
This question may also be helpful.
You also need to reinstall the psycopg2-module, be sure to use the newly installed lib when rebuilding it. Refer to the linked question (in short, you will need to place the path to pg_config which is included in your new installation to $PATH when running pip install psycopg2).
I had this same error, which turned out to be because I was using the Anaconda version of psycopg2. To fix it, I had adapt VictorF's solution from here and run:
conda uninstall psycopg2
sudo ln -s /Users/YOURUSERNAME/anaconda/lib/libssl.1.0.0.dylib /usr/local/lib
sudo ln -s /Users/YOURUSERNAME/anaconda/lib/libcrypto.1.0.0.dylib /usr/local/lib
pip install psycopg2
Then when you run conda list you'll see psycopg2 installed with <pip> in the far right column. After that, I just restarted Python and everything worked.
The error you are receiving is caused by a problem with Postgres itself, and not psycopg2.
psycopg2.OperationalError: sslmode value "require" invalid when SSL support is not compiled in
The above indicates that the version of Postgres that psycopg2 is built against does not have SSL support compiled in. When you attempt to connect to the running Posgres server with ssl=require it throws this error.
You don't mention how you installed Postgres but since you are using Homebrew for other things, I recommend you also install Postgres the same way:
$ brew update
$ brew install postgresql
The formula for postgresql shows that it depends on openssl and compiles with the --with-openssl flag set. It will also install the neccessary libpq headers. You may need to reinstall psycopg2 after this step to ensure it picks up the correct libraries/version.
Interestingly, there is a bug listed against conda which lists the same error that you report occurring when the conda version of psycopg2 — linked on a system with Homebrew postgres — was installed on a system without, suggesting missing SSL libraries can also trigger this.
I would suggest uninstalling any existing Postgres versions (including any installed via Homebrew) before reinstalling to minimise the risk of the wrong one being used.
As others have said, the error message looks to be coming from Postgres. You can verify this by typing: psql sslmode=require if it gives you a pgsql terminal then ssl works with postgres, if it errors then it doesn't
Try and remove postgres and reinstall with openssl support:
brew uninstall postgres
brew update
brew install postgres --with-openssl
Alternatively, and this is the way I'd suggest, there is a one click installer at http://postgresapp.com that might also make it easier to get it installed. Follow the instructions on the site to get it installed correctly.
When I did it on Yosemite it installed at ~/Library/Application\ Support/Postgres93/var
You'll also want to create a certificate (this could also be where the error is coming from) either from a registrar if this is going to be public facing in the slightest or self signed if it's for a dev/test environment.
openssl req -new -text -out server.req
openssl rsa -in privkey.pem -out server.key
rm privkey.pem
openssl req -x509 -in server.req -text -key server.key -out server.crt
chmod og-rwx server.key
Navigate to your config directory, by default it is: ~/Library/Application\ Support/Postgres93/var
Enable ssl support:
vim postgresql.conf
# change this:
# ssl = on
# to this:
ssl = on
Restart the app and then check ssl with psql "sslmode=require"
If that works then try it through your Python code. If it works with the code above, but not Python then it's definitely a Python code problem that needs to be worked through.
As I can not comment:
Adding to Brideau's answer that this only worked for me after changing my version of Postgres.
brew uninstall postgres
brew update
brew install postgres --with-openssl
Then run the code provided by Brideau and it should work.
If you're using v2.6.1 or v2.6.2 of psycopg2 (like me), the answer was a simple upgrade to v2.7. Reading the release notes for psycopg2, there was a minor fix for SSL albeit it doesn't look particularly relevant.
My setup was as follows:
Mac OS X El Capitan 10.11.6
psycopg2 2.6.2 installed via pip
PostgreSQL installed via Enterprise DB Installer
Running pip uninstall psycopg2 followed by pip install psycopg2 resolved matters.
Try to install psycopg2 from MacPorts
sudo port install py27-psycopg2
I am trying to upload a new python project from my mac to PyPI via:
python setup.py sdist upload -r pypi
When I try to upload a package, I get the following error on OSX:
Submitting dist/PyTreasuryDirect-0.1.0.tar.gz to https://pypi.python.org/pypi
error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
This question is slightly related to these two questions, ssl with pip and when importing the ssl library. However I need setup.py to work in my case, how can I get past this ssl error?
I also get the same error on Python 2.7.9 and Python 3.5 using OSX 10.10.2 (14C109)
The solution for OS X 10.10 is to use its default Python version 2.7.6.
This problem is related to the fact that starting from Python version 2.7.9 certificate validation for stdlib http clients is enabled by default (PEP 476). This issue is described in Python bug tracker.
I am using the python stack on Heroku and am trying to use the requests library to access an https API that uses TLS 1.2
I have got thing working on my local environment by upgrading openssl to 1.0.1 and I need to do the same thing in my Heroku environment.
Here is the error in Heroku and I've confirmed that the openssl version is OpenSSL 0.9.8k 25 Mar 2009
requests.exceptions.SSLError: [Errno bad handshake] [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]
Is there a python buildpack that can support this? It looks like there is a ruby buildpack, but I haven't seen it verified to work OpenSSL 1.0.1 on Heroku
I tried to mimic that behavior, but don't really know what I'm doing with a fork of the standard python buildpack.
Thanks!
Cedar-14, has OpenSSL 1.0.1f. Heroku-16, the most recent Heroku stack, has OpenSSL 1.0.2g. Stack package details
The upgrade guide is here:
https://devcenter.heroku.com/articles/cedar-14-migration