I'm trying to directly access Google Cloud SQL and create there table. I want to use as little services as possible (keep it simple), therefore I really don't want to use Cloud SDK whatever.
I want to use something similar, that I saw here. I tried to replicate it, but I ended up with error.
AttributeError: module 'socket' has no attribute 'AF_UNIX'
For all this I'm using Python with sqlalchemy & pymysql
I really don't know how to debug it since I'm using it first few hours, but I think that problem could be with URL or environmental variables (app.yamp file, which I created).
I think that I already have installed all dependencies which I need
db_user = os.environ.get("db_user")
db_pass = os.environ.get("db_pass")
db_name = os.environ.get("db_name ")
cloud_sql_connection_name = os.environ.get("cloud_sql_connection_name ")
db = sqlalchemy.create_engine(
# Equivalent URL:
# mysql+pymysql://<db_user>:<db_pass>#/<db_name>?unix_socket=/cloudsql/<cloud_sql_instance_name>
sqlalchemy.engine.url.URL(
drivername='mysql+pymysql',
username=db_user,
password=db_pass,
database=db_name,
query={
'unix_socket': '/cloudsql/{}'.format(cloud_sql_connection_name)
}
),
pool_size=5,
max_overflow=2,
pool_timeout=30,
pool_recycle=1800,
)
with db.connect() as conn:
conn.execute(
"CREATE TABLE IF NOT EXISTS votes "
"( vote_id SERIAL NOT NULL, time_cast timestamp NOT NULL, "
"candidate CHAR(6) NOT NULL, PRIMARY KEY (vote_id) );"
)
I do not use db_user etc. as real values. These are just examples.
It should pass successfully and create an table in Google SQL
Can I directly access database with SQLAlchemy - not locally
You are specifying a unix socket /cloudsql/{}. This requires that you set up the Cloud SQL Proxy on your local machine.
To access Cloud SQL directly, you will need to specify the Public IP address for Cloud SQL. In your call to the function sqlalchemy.engine.url.URL, specify the host and port parameters and remove the query parameter.
Related
I'm trying to query an RDS (Postgres) database through Python, more specifically a Jupyter Notebook. Overall, what I've been trying for now is:
import boto3
client = boto3.client('rds-data')
response = client.execute_sql(
awsSecretStoreArn='string',
database='string',
dbClusterOrInstanceArn='string',
schema='string',
sqlStatements='string'
)
The error I've been receiving is:
BadRequestException: An error occurred (BadRequestException) when calling the ExecuteSql operation: ERROR: invalid cluster id: arn:aws:rds:us-east-1:839600708595:db:zprime
In the end, it was much simpler than I thought, nothing fancy or specific. It was basically a solution I had used before when accessing one of my local DBs. Simply import a specific library for your database type (Postgres, MySQL, etc) and then connect to it in order to execute queries through python.
I don't know if it will be the best solution since making queries through python will probably be much slower than doing them directly, but it's what works for now.
import psycopg2
conn = psycopg2.connect(database = 'database_name',
user = 'user',
password = 'password',
host = 'host',
port = 'port')
cur = conn.cursor()
cur.execute('''
SELECT *
FROM table;
''')
cur.fetchall()
I'm trying to run a Python(version 2.7.1') script where I am using pymysql package to create a table into a database from a CSV file.
It runs correctly in my local system, however, the problem appears when running the same script as a part of a pipeline in Google Cloud Dataflow.
My Python function is the following one:
class charge_to_db(beam.DoFn):
def process(self, element):
import pymysql
with open(element, 'r') as f:
data = f.read().decode("UTF-8")
datalist = []
for line in data.split('\n'):
datalist.append(line.split(','))
db = pymysql.connect(host='IPaddress', user='root', password='mypassword', database='stack_model')
cursor = db.cursor()
cursor.execute("DROP TABLE IF EXISTS stack_convergence")
# create column names from the first line in fList
up = "upper_bnd"
primal = "primal"
d = "dualit"
gap = "gap_rel"
teta = "teta"
alpha = "alpha"
imba = "imba_avg"
price = "price_avg"
# create STUDENT table // place a comma after each new column except the last
queryCreateConvergenceTable = """CREATE TABLE stack_convergence(
{} float not null,
{} float not null,
{} float not null,
{} float not null,
{} float not null,
{} float not null,
{} float not null,
{} float not null )""".format(up, primal, d, gap, teta, alpha, imba, price)
cursor.execute(queryCreateConvergenceTable)
When running this function in the cloud I'm obtaining the following error:
RuntimeError: OperationalError: (2003, 'Can\'t connect to MySQL server on \'35.195.1.40\' (110 "Connection timed out")')
I don't know why this error is occurring because it runs correctly in local system, so from the local system I have access to my cloud SQL instance, but not from the dataflow in the cloud.
Why is this error occurring?
On Dataflow you cannot whitelist an IP to enable Dataflow to access a SQL instance. If you would be using Java, the easiest way would be to use JdbcIO / JDBC socket factory.
But since you're using Python, then mimicking the implementation of JdbcIO.read() using Python-specific database connectivity facilities would help. There's this related question with a workaround after changing some Cloud SQL settings and adding related python codes.
If this seems complex, alternatively you can export data from Cloud SQL to Cloud Storage and then load from Cloud Storage.
I am unable to connect to RDS using an Lambda Function via the test example they provide
This is the code:
import sys
import logging
import rds_config
import pymysql
#rds settings
rds_host = "connection_link"
name = rds_config.db_username
password = rds_config.db_password
db_name = rds_config.db_name
logger = logging.getLogger()
logger.setLevel(logging.INFO)
try:
conn = pymysql.connect(rds_host, user=name, passwd=password, db=db_name, connect_timeout=5)
except:
logger.error("ERROR: Unexpected error: Could not connect to MySql instance.")
sys.exit()
logger.info("SUCCESS: Connection to RDS mysql instance succeeded")
def handler(event, context):
"""
This function fetches content from mysql RDS instance
"""
item_count = 0
with conn.cursor() as cur:
cur.execute("create table Employee3 ( EmpID int NOT NULL, Name varchar(255) NOT NULL, PRIMARY KEY (EmpID))")
cur.execute('insert into Employee3 (EmpID, Name) values(1, "Joe")')
cur.execute('insert into Employee3 (EmpID, Name) values(2, "Bob")')
cur.execute('insert into Employee3 (EmpID, Name) values(3, "Mary")')
conn.commit()
cur.execute("select * from Employee3")
for row in cur:
item_count += 1
logger.info(row)
#print(row)
return "Added %d items from RDS MySQL table" %(item_count)
This is the structure of my deployment package
app/pymysql/...
app/app.py
app/rds_config.py
app/PyMySQL-0.7.11.dist-info/...
I have packaged all the files inside the app folder in a zip file.
This is the error is get
"errorMessage": "RequestId: 96fb4cd2-79c1-11e7-a2dc-f97407196dbb Process exited before completing request"
I have already checkedmy RDS connection on MYSQL Workbench its working fine
Update:
Let's assume that your actual Python code is actually indented correctly unlike the code you posted above.
For some reason, your function cannot connect to your database. And instead of returning an error to the user, you basically told it to sys.exit(1) so that's the reason why Lambda says "Process exited before completing the request".
-- Original Answer --
That does not look like an AWS lambda handler.
You're supposed to write a function handler that accepts the lambda event and context as arguments.
Please read more about it from the AWS Lambda documentation.
As #MarkB mentioned in the comments, For connectivity you need to set VPC, Subnets and Security Group in your Lambda Function same as your RDS instance:
and also you need to check Protocol, Port and Source for Inbound and Outbound in security group to make sure it is open for your IP and port range.
I just ran into the same problem, and it turns out it is because I did not specify a name for the database on creation. Easy Create doesn't give you this option, so you will have to go with Standard Create, where one can specify the name under Additional Configuration. This name is what you should specify for db_name in rds_config.py.
Alternatively, you can connect with your tool of choice without a database name, perform a CREATE DATABASE xxx; where xxx is the name of the database, and then use can use that database going forward.
Source: https://serverfault.com/a/996423
This one is also relevant: Why don't I have access to the database from aws lambda but have from a local computer with the same login data?
The problem with your zip file.
Im also using the same lambda function.
Please follow the below steps.
1. your app.py and rds_config.py files are good.
2. Then download the pymysql https://pypi.python.org/pypi/PyMySQL
3. Extract it.
4. Copy the pymysql to somewhere. (Note: Dont add all contents inside the PyMySQL-0.7.11 folder, we just need pymysql only.)
5. Then create a zip file with app.py, rds_config.py and pymysql folder.
I'm getting an error similar to other posts in this subject.
I tried switching from 1st gen to 2nd gen SQL server (both on us-central1), but it still doesn't work.
I copied my CLOUDSQL_PROJECT from the url on the top of my project.
I copied my CLOUDSQL_INSTANCE from the proprieties part in the SQL page.
In my main.py, I'm trying to run Google sample code, and it doesn't work (locally it does, of course):
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
db = MySQLdb.connect(
unix_socket='/cloudsql/{}:{}'.format(
CLOUDSQL_PROJECT,
CLOUDSQL_INSTANCE),
user=user,passwd=password)
# When running locally, you can either connect to a local running
# MySQL instance, or connect to your Cloud SQL instance over TCP.
else:
db = MySQLdb.connect(host=host,user=user,passwd=password)
cursor = db.cursor()
cursor.execute('SHOW VARIABLES')
for r in cursor.fetchall():
self.response.write('{}\n'.format(r))
The documentation is slightly outdated. You should be able to always use the "Instance connection name" property from the SQL properties page to construct the unix socket path; just append that value after the "/cloudsql/" prefix.
For second generation, the connection format is project:region:name. In your example, it maps to "hello-world-123:us-central1:sqlsomething3", and the unix socket path is "/cloudsql/hello-world-123:us-central1:sqlsomething3".
I have a connection string that looks like this
con_str = "myuser/mypass#oracle.sub.example.com:1521/ora1"
Where ora1 is the SID of my database. Using this information in SQL Developer works fine, meaning that I can connect and query without problems.
However, if I attempt to connect to Oracle using this string, it fails.
cx_Oracle.connect(con_str)
DatabaseError: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
This connection string format works if the ora1 is a service name, though.
I have seen other questions that seem to have the reverse of my problem (it works with SID, but not Service name)
Using Oracle Service Names with SQLAlachemy
Oracle SID and Service name; connection problems
cx_Oracle & Connecting to Oracle DB Remotely
What is the proper way to connect to Oracle, using cx_Oracle, using an SID and not a service name? How do I do this without the need to adjust the TNSNAMES.ORA file? My application is distributed to many users internally and making changes to the TNSNAMES file is less than ideal when dealing with users without administrator privileges on their Windows machines. Additionally, when I use service name, I don't need to touch this file at all and would like it keep it that way.
I a similar scenario, I was able to connect to the database by using cx_Oracle.makedsn() to create a dsn string with a given SID (instead of the service name):
dsnStr = cx_Oracle.makedsn("oracle.sub.example.com", "1521", "ora1")
This returns something like
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle.sub.example.com)(PORT=1521)))(CONNECT_DATA=(SID=ora1)))
which can then be used with cx_Oracle.connect() to connect to the database:
con = cx_Oracle.connect(user="myuser", password="mypass", dsn=dsnStr)
print con.version
con.close()
For those looking for how to specify service_name instead of SID.
From changelog for SQLAlchemy 1.0.0b1 (released on March 13, 2015):
[oracle] [feature] Added support for cx_oracle connections to a
specific service name, as opposed to a tns name, by passing
?service_name=<name> to the URL. Pull request courtesy Sławomir
Ehlert.
The change introduces new, Oracle dialect specific option service_name which can be used to build connect string like this:
from sqlalchemy import create_engine
from sqlalchemy.engine import url
connect_url = url.URL(
'oracle+cx_oracle',
username='some_username',
password='some_password',
host='some_host',
port='some_port',
query=dict(service_name='some_oracle_service_name'))
engine = create_engine(connect_url)
If you are using sqlalchemy and ORACLE 12, the following seems to work.
from sqlalchemy import create_engine
con='oracle://user:password#hostname:1521/?service_name=DDDD'
engine = create_engine(con)
Note, you have to use the service name and not the SID. I don't know why, but the simple connection string that uses SID does not work.
It still may not work. You need to take the output of dsnStr and modify the string by replacing SID with SERVICE_NAME and use that variable in the con string. This procedure worked for me.
SID's may not be easily accessible or you might not have it created for your database.
In my case, I'm working from the client side requesting access to a cloud database so creating an SID didn't really make sense.
Instead, you might have a string that looks similar to this:
"(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = something.cloud.company)
(PORT = 12345)) (ADDRESS = (PROTOCOL = TCP)(HOST = something.cloud.company)
(PORT = 12345)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME =
something.company)))"
You can use it in replacement of the SID.
connection = cx_Oracle.connect("username", "pw", "(DESCRIPTION = (ADDRESS =
(PROTOCOL = TCP)(HOST = something.cloud.company)(PORT = 12345)) (ADDRESS =
(PROTOCOL = TCP)(HOST = something.cloud.company)(PORT = 12345))
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = something.company)))")
I thought during a while that I would not be able to use Magic SQL (%sql, %%sql) because of service name issue in connection that would force to use the alternative way described above with cx_Oracle.connect(), cx_Oracle.makedsn()... I finally found a solution working for me: declare and set a variable for the service name first and then use it in the command (since not working if literal string for service name put in the command !)
import cx_Oracle
user='youruser'
pwd='youruserpwd'
dbhost='xx.xx.xx.xx'
service='yourservice'
%load_ext sql
%sql oracle+cx_oracle://$user:$pwd#$dbhost:1521/?service_name=$service
output (what you get in successful connection):
u'Connected: youruser#'
If someone is looking to set oracle.jdbc.proxyClientName property for cx_oracle, to connect using proxyClient, they can use -
cx_Oracle.init_oracle_client("../../oracle_local_client", config_dir= "../../oracle_local_client/network/admin")
connectDsn = cx_Oracle.makedsn('db.svr.net', 'portNumberHere',service_name="TEST_READWRITE")
#replace all prams above
pool = cx_Oracle.SessionPool(externalauth=True, homogeneous=False, dsn = connectDsn)
connection = pool.acquire(user="[PROXY_CLIENT_NAME]")
Notice the use of '[' braces to depict that the user is proxyClient.
I am using Kerberos authentication for this and my SQLNET.ora file contains the below properties.
NAMES.DIRECTORY_PATH=(TNSNAMES,HOSTNAME,EZCONNECT)
SQLNET.AUTHENTICATION_SERVICES = (BEQ,KERBEROS5PRE,KERBEROS5)
SQLNET.AUTHENTICATION_KERBEROS5_SERVICE=oracle
SQLNET.KERBEROS5_CC_NAME=OSMSFT:
SQLNET.KERBEROS5_CONF_MIT=TRUE
SQLNET.KERBEROS5_CONF=I:\projects\poc\resources\krb5.conf # krb5 config file complete path.
For more information, refer the video embedded in this article.
I also met this issue.
The solution is:
1: get the service name at tnsnames.ora
2: put the service name in
con_str = "myuser/mypass#oracle.sub.example.com:1521/ora1"