I'm trying to connect to mysql from my AWS Lambda script.I did pip install --allow-external mysql-connector-python mysql-connector-python -t <dir>
to install mysql-connector-python in local directory.
I zipped the file and uploaded it to AWS Lambda where my python files are being executed.
My scripts are executing correctly up to the point where I initialize a mysql connection.
I have this
log('about to set connection for db')
connection = mysql.connector.connect(user=DB_USER, password=DB_PASSWORD, host=DB_HOST, database=DB_DATABASE)
query = "SELECT * FROM customers WHERE customer_email LIKE '%s' LIMIT 1"
log('set connection for DB')
'about to set connection for db' is being logged but 'set connection for DB' is never logged and my program hits a timeout and stops executing.
What might I be doing wrong?
EDIT:
This is my class that I'm calling from lambda_function.py
import mysql.connector
import logging
from mysql.connector import errorcode
class MySql( object ):
USER =None
PASSWORD=None
HOST =None
DATABASE=None
logger = logging.getLogger()
logger.setLevel( logging.INFO )
def __init__(self, user, password, host, database):
global USER, PASSWORD, HOST, DATABASE
USER = user
PASSWORD = password
HOST = host
DATABASE = database
def getId( self, customer_email ):
email_exists = False
connection = mysql.connector.connect(user=USER, password=PASSWORD, host=HOST, database=DATABASE)
query = "SELECT * FROM customers WHERE customer_email LIKE '%s' LIMIT 1"
cursor = connection.cursor()
cursor.execute( query % customer_email )
data = cursor.fetchall()
id = None
for row in data :
id = row[1]
break
cursor.close()
connection.close()
return id
def insertCustomer( self, customer_email, id ):
log('about to set connection for db')
connection = mysql.connector.connect(user=USER, password=PASSWORD, host=HOST, database=DATABASE)
log('set connection for DB')
cursor = connection.cursor()
try:
cursor.execute("INSERT INTO customers VALUES (%s,%s)",( customer_email, id ))
connection.commit()
except:
connection.rollback()
connection.close()
def log( logStr):
logger.info( logStr )
def main():
user = 'xxx'
password = 'xxx'
host = ' xxx'
database = 'xxx'
mysql = MySql( user, password, host, database )
id = mysql.getId('testing')
if id == None:
mysql.insertCustomer('blah','blahblah')
print id
if __name__ == "__main__":
main()
When I execute the MySql.py locally my code works fine. My database gets updated but nothing happens when I run it from AWS.
Is it a MySQL instance on AWS (RDS) or on premise DB? If RDS, check the NACL inbound rules of the vpc associated with your DB instance. Inbound rules can allow/deny from specific IP sources
when you did a zip file create. Did you do a pip to target directory.
I have enclosed the syntax below.This copies the files to your target directory to zip.
That may be the reason you are able to execute it locally but not in a lambda.
This is the syntax
pip install module-name -t /path/to/PythonExampleDir
Related
I am new to Database operations and i am writing a python script to take backup and restore the backup of an db in postgres
Below is my python script
import subprocess
import psycopg2
user = "postgres"
password = "postgres"
host = "localhost"
port = "5432"
database_name = "test"
dest_file = "/home/admin/temp/db.sql"
#Taking db backup
process = subprocess.Popen(['pg_dump','--dbname=postgresql://{}:{}#{}:{}/{}'.format(user, password, host, port, database_name),'-Fc','-f', dest_file],stdout=subprocess.PIPE)
output = process.communicate()[0]
if process.returncode != 0:
print('Command failed. Return code : {}'.format(process.returncode))
exit(1)
print(str(process))
#Doing db changes
#Restoring db in a chance of error
conn = psycopg2.connect(user = user,password = password,host = host,port = port)
conn.autocommit = True
with conn.cursor() as cursor:
cursor.execute('DROP DATABASE test;')
cursor.execute('CREATE DATABASE test;')
process = subprocess.Popen(['pg_restore', '--no-owner','--dbname=postgresql://{}:{}#{}:{}/{}'.format(user, password, host, port, database_name), dest_file],stdout=subprocess.PIPE)
output = process.communicate()[0]
if process.returncode != 0:
print('Command failed. Return code : {}'.format(process.returncode))
exit(1)
print(output)
While executing code i am getting the follwing error..
psycopg2.errors.ObjectInUse: database "test" is being accessed by other users
Not sure whats wrong..
Please help
You have connections on your test database which must be closed before proceeding which can be achieved using the following:
-- Stop further connections
alter database test allow_connections = off;
-- Drop remaining (except this connection)
SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'test'
AND pid <> pg_backend_pid();
The database can now be dropped - either via your code or SQL i.e.
drop database test;
I've been given a postgres DB in my uni project in which I have to SSH into the network from which I can then access the DB. I've set up the connection in DBeaver using the SSH tab and it works perfectly fine. However, using Python, I can connect to the SSH just fine, but cannot connect to the DB itself. I've checked with another DB that doesn't require SSH and that works just fine. Here is my code. Note: I've already tried using SSHTunnel, too, to no avail. Also, ignore my quick hack to anonymize my SSH login data, as I didn't find how to use a proper config file with paramiko late at night yesterday...
import os
from psycopg2 import connect, Error
from paramiko import SSHClient
from config import config
with open("ssh_config.txt", "r") as f:
lines = f.readlines()
hostname = lines[0].strip()
username = lines[1].strip()
password = lines[2].strip()
ssh = SSHClient()
ssh.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
ssh.connect(hostname=hostname, username=username, password=password)
print("SSH connected.")
try:
params = config()
conn = connect(**params)
cursor = conn.cursor()
print("DB connected.")
# Print PostgreSQL connection properties.
print(conn.get_dsn_parameters(), "\n")
# Print PostgreSQL version.
cursor.execute("SELECT version();")
record = cursor.fetchone()
print("You are connected to - ", record, "\n")
except (Exception, Error) as error:
print("Error while connecting to PostgreSQL", error)
Any help would be appreciated. Thanks!
I've figured it out myself. Here is the updated code. Basically, I had to forward the remote address to localhost and then connect to localhost instead of the DB address.
from psycopg2 import connect, Error
from sshtunnel import SSHTunnelForwarder
from config import config
with open("ssh_config.txt", "r") as f:
lines = f.readlines()
hostname = lines[0].strip()
username = lines[1].strip()
password = lines[2].strip()
remote_bind_address = lines[3].strip()
try:
with SSHTunnelForwarder(
(hostname, 22),
ssh_username=username,
ssh_password=password,
remote_bind_address=(remote_bind_address, 5432),
local_bind_address=("localhost", 8080)) \
as tunnel:
tunnel.start()
print("SSH connected.")
params = config()
conn = connect(**params)
cursor = conn.cursor()
print("DB connected.")
# Print PostgreSQL connection properties.
print(conn.get_dsn_parameters(), "\n")
# Print PostgreSQL version.
cursor.execute("SELECT version();")
record = cursor.fetchone()
print("You are connected to - ", record, "\n")
cursor.close()
conn.close()
tunnel.close()
print("DB disconnected.")
except (Exception, Error) as error:
print("Error while connecting to DB", error)
I'm attempting to connect a database, located on a web server, to a robot but I do not know how to connect the database to the robot. I would like the robot to run SELECT and UPDATE queries from the robot. The other issue is that I do not intend on using C-languages or Java; I plan on using python in the main control system.
I do know:
PHP
VBScript
Batch
Python
If anyone knows how to connect the DB to a bot it would be a great help.
So basically how to connect to an SQL DB in python? I'm working on a virtual bot right now doing the same thing. Look into the module , SQL-connector! http://www.mysqltutorial.org/python-connecting-mysql-databases/
You would start with creating a config.ini with your credentials
[mysql]
host = localhost
database = python_mysql
user = root
password =
Read Config.ini and return a dictionary
from configparser import ConfigParser
def read_db_config(filename='config.ini', section='mysql'):
""" Read database configuration file and return a dictionary object
:param filename: name of the configuration file
:param section: section of database configuration
:return: a dictionary of database parameters
"""
# create parser and read ini configuration file
parser = ConfigParser()
parser.read(filename)
# get section, default to mysql
db = {}
if parser.has_section(section):
items = parser.items(section)
for item in items:
db[item[0]] = item[1]
else:
raise Exception('{0} not found in the {1} file'.format(section, filename))
return db
and connect to MYSQL database
from mysql.connector import MySQLConnection, Error
from python_mysql_dbconfig import read_db_config
def connect():
""" Connect to MySQL database """
db_config = read_db_config()
try:
print('Connecting to MySQL database...')
conn = MySQLConnection(**db_config)
if conn.is_connected():
print('connection established.')
else:
print('connection failed.')
except Error as error:
print(error)
finally:
conn.close()
print('Connection closed.')
if __name__ == '__main__':
connect()
and update statement would look like the following
def update_book(book_id, title):
# read database configuration
db_config = read_db_config()
# prepare query and data
query = """ UPDATE books
SET title = %s
WHERE id = %s """
data = (title, book_id)
try:
conn = MySQLConnection(**db_config)
# update book title
cursor = conn.cursor()
cursor.execute(query, data)
# accept the changes
conn.commit()
except Error as error:
print(error)
finally:
cursor.close()
conn.close()
if __name__ == '__main__':
update_book(37, 'The Giant on the Hill *** TEST ***')
I'm attempting to connect to a vertica database from OS/X 10.10.2 using the following small script:
#!/usr/bin/python
import sys
import pyodbc
def usage(retval):
if retval == 0:
file_ = sys.stdout
else:
file_ = sys.stderr
file_.write('Usage: {} --query sql_query --dsn data_source_name\n'.format(sys.argv[0]))
sys.exit(retval)
def main():
query = None
dsn = None
while sys.argv[1:]:
if sys.argv[1] == '--query':
query = sys.argv[2]
del sys.argv[1]
elif sys.argv[1] == '--dsn':
dsn = sys.argv[2]
del sys.argv[1]
elif sys.argv[1] in {'-h', '--help'}:
usage(0)
else:
sys.stderr.write('{}: Unrecognized option: {}\n'.format(sys.argv[0], sys.argv[1]))
usage(1)
del sys.argv[1]
if query is None:
sys.stderr.write('{}: --query is a required option\n'.format(sys.argv[0]))
usage(1)
if dsn is None:
sys.stderr.write('{}: --dsn is a required option\n'.format(sys.argv[0]))
usage(1)
connection_string = 'DSN={}'.format(dsn)
sys.stderr.write('connection_string is {}\n'.format(connection_string))
connection = pyodbc.connect(connection_string)
cursor = connection.cursor()
cursor.execute(query)
# Fetchall might be faster, but it also has memory limit issues
for row in cursor:
print(row)
main()
Invoking this script like so:
ODBCINI=/Library/ODBC/odbc.ini /simple-connect --query 'select * from GLITCH.METRIC' --dsn GLITCH_ADMIN
...fails with:
21L, 1L, u"FAILED: ('28000', '[28000] FATAL 3781: Invalid username or password\\n (3781) (SQLDriverConnect)')", u'unique_seconds', ...
...plus a little more text on the same line, and one additional line which I've deleted for protection.
I have another (much larger) program that connects fine using the same DSN, without prompting for a username or password.
The relevant DSN in /Library/ODBC/odbc.ini looks like (hostname, username and password changed for protection) :
Driver = Vertica
Description = Glitch Vertica Meta Data DSN
Database = Analytics
Servername = hostname
UID = username
PWD = password
PORT = 5433
SSLMode = require
I dtruss'd the script, and found that it was looking in /Library/ODBC/odbc.ini - or rather, it does with the ODBCINI environment variable set appropriately. Without that variable, it appeared to be ignoring my odbc.ini.
Does anyone know why it's failing to authenticate?
Thanks!
I am trying to write a simple program to connect MySQL and perform some operations
host = '10.0.106.40'
user = 'ddddd'
port = 3306
passwd = 'DDDDDD'
db = 'bbbbbbb'
''' Creates a MySQL connection and returns the cursor '''
def create_connection():
connection = mysql.connect(host, user, passwd, db, port)
cursor = connection.cursor()
return connection, cursor
''' Close the connection '''
def close_connection(cursor, connection):
cursor.close()
connection.commit()
connection.close()
The above functions are my skeletons. Now, when I try to do this
for user in users:
connection, cursor = create_connection()
...
close_connection(cursor, connection)
I get this error
TypeError: connect() argument 2 must be string, not long
However, when I do this
connection, cursor = create_connection()
for user in users:
...
close_connection(cursor, connection)
The code runs perfectly fine! I'm not sure but why should that be? I really want to run the earlier version of the code as the latter one is too slow for me.
My guess is that the user being passed to create_connection() is from the line
for user in users:
That is why it works outside the loop, because then it is accessing the correct user, namely the one with a String datatype.
An alternative would be to use a while loop instead or change the line to something like:
for u in users:
Or else you should do what warwaruk suggests.
users is a list of integers?
That's why it's bad to deal with globals. Better pass some parameters to create_connection instead of taking values from globals:
def create_connection(host, user, passwd, db, port):
connection = mysql.connect(host, user, passwd, db, port)
cursor = connection.cursor()
return connection, cursor