We Keep Coding

Using database.ini versus hardcoding/using env with psycopg2

I have a problem that gave me a lot of grievance and would like to understand why it happened.
I want to access our Redshift instance using psycopg2. My initial attempt just hardcoded the information in the script and used an env variable for the password.
import psycopg2
import os
def connect_to_db():
return psycopg2.connect(
host="host url",
database="db name",
user="username",
password=os.getenv("db_pw"),
port=12345
)
Whenever I tried this, I would get this error:
psycopg2.OperationalError: FATAL: password authentication failed for user "username"
FATAL: no pg_hba.conf entry for host "::ffff:<host IP>", user "username", database "db name", SSL off
However if I use this following project structure I'm able to get the script to run as intended.
- project
- db.py
- database.ini
And these are the files:
db.py:
import psycopg2
from configparser import ConfigParser, Error
def config(filename='database.ini', section='postgresql'):
# create a parser
parser = ConfigParser()
# read config file
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception('Section {0} not found in the {1} file'.format(section, filename))
return db
def connect_to_db():
"""Connect to db and return the connection object"""
params = config()
return psycopg2.connect(**params)
database.ini
[postgresql]
host=hosturl
database=db
user=user
password=password
port=12345
Could someone explain why this is happening?
Mac Intel 2020 2.3 GHz Quad-Core Intel Core i7

How to Store safely Mysql database credentials in my code

try:
connection = mysql.connector.connect(host='localhost',database='USER',user='root',password='password')
sql_select_Query = "select * from AuthSys WHERE mac = '%s'"%mac
cursor = connection.cursor()
cursor.execute(sql_select_Query)
row_headers=[x[0] for x in cursor.description]
records = cursor.fetchall()
except mysql.connector.Error as e:
return [e]
finally:
if connection.is_connected():
connection.close()
cursor.close()
print("MySQL connection is closed")
I wanted to store the host='localhost',database='USER',user='root',password='password' securely in my python project.So that everyone whoever uses my script will not get access to my database
Note: I am new to stackoverflow.If i wrote something wrong please suggent me right.Thanks in Advance.

You should probably put the credentials in a separate config file that isn't deployed with the project. And pass the path of this file to the main entry of the application, something like this:
python main.py --config=/your-path/to/your-config-file.ini
You will also need to parse this --config argument and then read and parse the your-config-file.ini file.

If you dont have too many such settings one common option is to get them from system environment variables.
user= os.environ["myuser"]
password= os.environ["mypassword"]
connection = mysql.connector.connect(host='localhost',database='USER',user=user,password=password)
See https://12factor.net/ factor 3.
I’d prefix all app settings environment names with something common, giving bkapp_user, bkapp_password.

Lambda function cant connect to redshift: FATAL: no pg_hba.conf entry for host

I understand there are a few questions related to these issues. I have seen them but could not find the correct solution.
I am using AWS Secret Manager to store the credentials and right now trying to connect with AWS Redshift using AWS Lambda Function.
Here is my lambda.py
import json
import boto3
import base64
import psycopg2
from botocore.exceptions import ClientError
def get_secret(secret_name):
try:
session = boto3.session.Session()
client = session.client(service_name='secretsmanager', region_name='xx')
get_secret_value_response = client.get_secret_value( SecretId=secret_name )
secret = get_secret_value_response['SecretString']
secret = json.loads(secret)
return secret
def lambda_handler(event, context):
s = get_secret('secretname')
conn = psycopg2.connect(database = s['xx'],
user = s['username'],
password = secret['password'],
host = secret['host'],
port = secret['port']
)
print('connected')
cur = conn.cursor()
cur.execute("""Select * from table1;""")
rows = cur.fetchall()
for row in rows:
print(row[0])
ERROR: FATAL: no pg_hba.conf entry for host "::ffff:xx.xxx.xxx.xxx", user "xxxx", database "xxx", SSL off
I am using psycopg2.
I have tried inserting, " sslmode = 'require'", not working.
Any help will be appreciated

Python: Can't insert into MongoDB

I have been trying to insert data into my MongoDB collection but it's not working:
try:
client = MongoClient(uri,
connectTimeoutMS=30000,
socketTimeoutMS=None)
print("Connection successful")
print()
except:
print("Unsuccessful")
print(client)
print()
db = client["<database>"]
collection = db["<collection>"]
print(db)
print()
print(collection)
print()
doc = {"test": "success"}
collection.insert_one(doc)
print("success")
The URI variable is my connection string copied from MongoDB.
Everything works fine, even the the db and collection variables print out fine until I get to the line: collection.insert_one(doc)
When I run, it just stops at that line and then I get a timeout error after a while. I am using the latest versions of Python and Pymongo

So I resolved the issue:
1) I needed to configure the whitelist entries. (https://docs.atlas.mongodb.com/security-whitelist/)
2) I needed to get off of my Universities Wifi because they block certain things.

There might be few things at play
check if the URL is correct
check if you write rights on the DB
from pymongo import MongoClient
try:
client = MongoClient(uri,
connectTimeoutMS=30000,
socketTimeoutMS=None)
print("Connection successful")
except:
print("Unsuccessful")
db = client["<database>"]
doc = {"test": "success"}
db[collectionName].insert_one(doc)

Connecting to Postgres sql with Python

I am running postgressql on a docker container. I am trying to connect to postgres via python and display the tables below is the code that I am using to connect to postgres:
import psycopg2
conn_string = "host='192.168.99.100:15432' dbname='PREDICTIVE_DS_POSTGRESQL'
user='ds_user' password='ds_user'"
print("Connecting to database\n ->%s" % (conn_string))
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
print("Connected!\n")
Then I use the below Python code to display the existing tables within postgres:
def table_exists(con, table_str):
exists = False
try:
cur = con.cursor()
cur.execute("select exists(select relname from pg_class where relname='"
+ table_str + "')")
exists = cur.fetchone()[0]
print("exists")
cur.close()
except psycopg2.Error as e:
print(e)
return exists
def get_table_col_names(con, table_str):
col_names = []
try:
cur = con.cursor()
cur.execute("select * from " + table_str + " LIMIT 0")
for desc in cur.description:
col_names.append(desc[0])
cur.close()
except psycopg2.Error as e:
print(e)
However, it is not working at all. It says that it cannot connect translate host name "192.168.99.100:15432" to address: Unknown host. However, the container is up and running and that is the host name. Additionally, I don't know whether the rest of the code will work once it connects.

Have your database credentials defined in a separate file.
For example, have a file called database.ini and define it like this:
[creds]
host=192.168.99.100
port=15432
database=PREDICTIVE_DS_POSTGRESQL
user=ds_user
password=ds_user
Have another config parser file to parse this. Call it config.py
#!/usr/bin/python
try:
import configparser
except:
from six.moves import configparser
def config(section,filename='database.ini',):
parser = configparser.ConfigParser()
parser.read(filename)
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception('Section {0} not found in the {1}
file'.format(section, filename))
return db
Now, in your main file, import your config function like this:
from config import config
and connect like this:
dbParams = config("creds")
con = psycopg2.connect(**dbParams)