Python Pandas has both a to_sql and read_sql methods which can be used to communicate with a database.
I am connecting to a Postgres database.
Pandas seems to use SQL Alchemy and psycopg2 to perform the communication.
When using create_engine('mysql+mysqlconnector://user:password#amazon_endpoint:port/database_name',connect_args={'sslmode':'require'}, echo=True) I received an AttributeError: Unsupported argument 'sslmode' error.
I then saw the following posts: 1, 2 and amended the code to be create_engine('mysql+mysqlconnector://user:password#amazon_endpoint:port/database_name?ssl_ca=/home/user/Downloads/rds-combined-ca-bundle.pem'). The connection now works and I am able to connect to the database. Does this means that the connection is now secure? Is there a way I can confirm this?
I don't believe the connection is secure by default, unless your Postgres instance has been configured to require secure connections. First, your instance needs to have been built with SSL support, and then it's a simple matter of telling the client to connect securely. From the documentation:
PostgreSQL has native support for using SSL connections to encrypt client/server communications for increased security. This requires that OpenSSL is installed on both client and server systems and that support in PostgreSQL is enabled at build time (see Chapter 15).
With SSL support compiled in, the PostgreSQL server can be started with SSL enabled by setting the parameter ssl to on in postgresql.conf. The server will listen for both normal and SSL connections on the same TCP port, and will negotiate with any connecting client on whether to use SSL. By default, this is at the client's option; see Section 19.1 about how to set up the server to require use of SSL for some or all connections.
When you use to_sql or read_sql, pass it a SQLAlchemy connection that's been set up with sslmode=require. This question provides the full snippet for setting up your connection:
db = create_engine(
'postgresql+pg8000://user:pass#hostname/dbname',
connect_args={'sslmode':'require'},
echo=True
).connect()
Related
Can someone else connect to my SQL Server using my Windows Authentication or am I the only one? I saw that there is an option for "allow remote connections" in SSMS, so I'm wondering if someone has my connection credentials, such as server name and database, can they connect to it?
Is the server name "sensitive information" or does it not matter? I'm wondering because I always get hesitant typing out my server name which is DESKTOP-xxxxx (x in place of the actual numbers, which is the thing I'm not sure is sensitive or not)
example:
conn = pyodbc.connect('Driver={ODBC Driver 17 for SQL Server};'
'Server=DESKTOP-xxxxxx;' **<--------is this sensitive info?**
'Database=Test_Database;' **<--------is this sensitive info?**
'Trusted_Connection=yes;')
Ignore the SSMS allow remote connection option. Per the documentation:
This configuration option is an obscure SQL Server to SQL Server
communication feature that is deprecated
Use SQL Server Configuration Manager to view, enable, or disable protocols as desired. Remote TCP/IP and Named pipes are disabled by default.
If someone besides you knows your Windows credentials, you have bigger issues. Although they will not be able to connect to SQL Server remotely when the protocols are disabled, they could still get to your database via other means (e.g. RDP into your machine and access SQL Server locally).
The name of your machine could be considered sensitive but it's easily discoverable (e.g. DNS). You generally want multiple layers, which include firewalls and surface area reduction (e.g. disabled RDP), and perhaps obfuscation (non-standard SQL ports) as well for protecting particularly sensitive data.
I have used the following code in Python:
import mysql.connector as mysql
import sys
HOST = "34.87.95.90"
DATABASE = "CAO_db"
USER = "root"
PASSWORD = "*********"
db_connection = mysql.connect(user=USER, password=PASSWORD, host=HOST, database=DATABASE)
cur = db_connection.cursor()
When I run the above code, I get the following error messages:
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
InterfaceError: 2003: Can't connect to MySQL server on '34.87.95.90:3306' (10060 A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)
I am not sure of how to fix my code and/or resolve the given errors. Please ask me if you would like more details of the error messages to help with the issue. I would greatly appreciate all the help I can get towards resolving the issues.
One thing I'm not seeing here is whether or not you have configured your Cloud SQL instance to accept connections.
You can configure it to accept connections from within the GCP stratosphere using their "Private IP" internal networking magic, AND you can configure it to accept connections from other machines using a combination of Public IP and either an authorized external network (like if you were accessing your GCP Cloud SQL instance from, say, an Amazon EC2 instance), or their Cloud SQL Proxy tool (which is what I use to connect to my Cloud SQL instance from my laptop).
In the GCP Console, go to your project
From the hamburger menu, select SQL
Click on your Cloud SQL instance
In the left nav, click on Connections
If you have Private IP checked and you're running this code on a GCP Compute/GKE resource, confirm that the "Network" field is set to the network used by that resource.
If you're just trying to get a connection from your local machine and you don't have a static IP to whitelist, your best option is to use Public IP in combination with Cloud SQL Proxy.
Cloud SQL Proxy essentially creates a TCP tunnel that allows your laptop to connect to 'localhost' on a port you specify, and it then redirects your connection to the remote Cloud SQL instance.
Once you've established that your networking situation isn't the problem, you could use the same Python connection code that you wrote above, but change HOST to 127.0.0.1 and add an attribute for PORT=3308.
EDITED to add: I suggest using PORT=3308 for your cloud_sql_proxy connection so that it doesn't interfere with any existing port 3306 (MySQL default) connections that you may already be actually running on your local machine. If this isn't the case, you can either omit the PORT attribute or keep it explicit, but change it to 3306.
Could anyone share a functional connection method to Wonderware's Historian using python3 on OSX (or linux)?
Historian is apparently a Microsoft SQL Server OLE DB (see pg102 of http://www.logic-control.com/datasheets/1/Historian/HistorianConcepts.pdf).
Another SO post suggests that the only python library available capable of connecting to an OLE DB is 'adodbapi' (Connecting to Microsoft SQL server using Python)
an attempt in code (using default RO credentials):
import adodbapi
ServerName = "ServerName"
MSQLDatabase = "Runtime"
username = "aaUser"
password = "pwUser"
conn = adodbapi.connect("PROVIDER=INSQL;Data Source={0};Database={1};trusted_connection=yes;UID={2};PWD{3};".format(ServerName,MSQLDatabase,username,password))
That gives an error:
adodbapi.apibase.OperationalError: (InterfaceError("Windows COM Error: Dispatch('ADODB.Connection') failed.",)...
The error is probably due to the absence and unavailability of the pywin32 package, which is apparently Windows only (Pywin32 (com objects) on Mac)
Tips appreciated. I highly suspect that the Microsoft vs Mac/Linux worlds just can't be bridged in this situation.
You should be able to access an MS SQL Server database (e.g. Wonderware Historian database) with Python.
Things to check:
For ODBC, "Trusted Connection" setting should be "No". Trusted connection means that it tries to use Windows Authentication to log in. You want to use a username and password instead. I think for OLE DB you set "Integrated Security = SSPI" instead.
Connection String (Username, Server Hostname/IP, Database name, Correct Port, Syntax)
Port (make sure you use the correct port - may be a non-standard port)
Firewalls - make sure that any firewalls are set up to allow access
If you're using the hostname, make sure DNS is working (e.g. can you ping the server?)
You may need to install an ODBC driver for Linux and pyodbc. ODBC is a more open standard. As you've pointed out, OLEDB is COM-based (e.g. Windows-based) so I'm not sure if it would be compatible.
I am able to access a Wonderware server using Python3 both through sqlalchemy and pyodbc on Linux and Windows - I don't have a Mac so you're on your own there. I've read that there are other drivers available, but I don't have any experience with them. If you have any suggestions here I'd be glad to hear them.
This is how I modularize the functionality of pyodbc. Essentially, I've defined a function in our code that sets up the sql engine conncetion:
def get_conn():
conn_pyodbc = pyodbc.connect(DSN=<myDSN>, UID=<user>, PWD=<pass>)
return conn_pyodbc
And I use the connection as such:
def executeQuery(sql_query):
with get_conn() as conn:
df = pd.read_sql(sql_query, conn)
Using the context manager just seems like an easier way of handling opening and closing database connections.
As far as setting up the DSN, I needed to install the Microsoft ODBC driver, which was easy enough to do by following some links I found online. After a successful installation I edited the /etc/odbc.init and /etc/odbcinst.ini files manually such that they look like this now:
$ cat /etc/odbc.ini
[myDSN]
Driver=ODBC Driver 13 for SQL Server
Description=Awesome server
Trace=No
Server=<serverIP>
and
$ cat odbcinst.ini
[ODBC Driver 13 for SQL Server]
Description=Microsoft ODBC Driver 13 for SQL Server
Driver=/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.1.so.9.1
UsageCount=1
In my experience, you can use 'Trusted Connection', but that means that your computer will try to connect to the server using it's Windows credentials. These work if you are logged into a Windows machine that has access to the data in question. Running on Linux is a different story, so I keep to the user/password combo.
Anymore questions feel free to ask.
I am writing an app with wxPython that incorporates pyodbc to access SQL Server. A user must first establish a VPN connection before they can establish a connection with the SQL server. In cases where a user forgets to establish a VPN connection or is simply not authorized to access a particular server, the app will freeze for up to 60+ seconds before it produces an error message. Often, users will get impatient and force-close the app before the error message pops up.
I wonder if there is a way to test whether it's possible to connect to the server without freezing up. I thought about using timeout, but it seems that timeout can be used only after I establish a connection
A sample connection string I use is below:
connection = pyodbc.connect(r'DRIVER={SQL Server};SERVER=ServerName;database=DatabaseName;Trusted_Connection=True;unicode_results=True')
See https://code.google.com/archive/p/pyodbc/wikis/Connection.wiki under timeout
Note: This attribute only affects queries. To set the timeout for the
actual connection process, use the timeout keyword of the
pyodbc.connect function.
So change your connection string to:
connection = pyodbc.connect(r'DRIVER={SQL Server};SERVER=ServerName;database=DatabaseName;Trusted_Connection=True;unicode_results=True', timeout=3)
should work
took a while before it threw an error message about server not existing or access being denied
Your comment conflates two very different kinds of errors:
server not existing is a network error. Either the name has no address, or the address is unreachable. No connection can be made.
access being denied is a response from the server. For the server to respond, a connection must exist. This is not to be confused with connection refused (ECONNREFUSED), which means the remote is not accepting connections on the port.
SQL Server uses TCP/IP. You can use standard network functions to determine if the network hostname of the machine running SQL Server can be found, and if the IP address is reachable. One advantage to using them to "pre-test" the connection is that any error you'll get will be much more specific than the typical there was a problem connecting to the server.
Note that not all delay-inducing errors can be avoided. For example, if the DNS server is not responding, the resolver will typically wait 30 seconds before giving up. If an IP address is valid, but there's no machine with that address, attempting a connection will take a long time to fail. There's no way for the client to know there's no such machine; it could just be taking a long time to get a response.
I'm pretty new to python. I'm trying to connect to an informix server using python and ibm_db.connect(). However I can't seem to succeed and the error messages don't help.
Using java and jdbc I can connect successfully with the following connection url:
jdbc:informix-sqli://10.20.30.40:1234/mydb:INFORMIXSERVER=foo_bar;USER=user;PASSWORD=pass;
My attempt at using ibm_db is:
ibm_db.connect('HOSTNAME=10.20.30.40;PORT=1234;DATABASE=mydb;PROTOCOL=ONSOCTCP;UID=user;PASSWORD=pass;', '', '')
But it gives an error (Exception: [IBM][CLI Driver] SQL0902C A system error occurred. Subsequent SQL statements cannot be processed. IBM software support reason code: "". SQLSTATE=58005)
I would like if possible an equivalent string I can put in as the first argument to
ibm_db.connect('', '', '')
So I can connect with python.
Please check https://code.google.com/p/ibm-db/issues/detail?id=116&can=1&q=ONSOCTCP, This might be helpful for you. If you still facing issue then you can post your query to https://groups.google.com/forum/#!forum/ibm_db for quick response.
Protocol onsoctcp is not supported by ibm_db.
Please check https://www.ibm.com/support/knowledgecenter/SSGU8G_11.50.0/com.ibm.admin.doc/ids_admin_0207.htm, This can help you to configure a DDRA (tcpip) access to your database.
I have faced the same issue with ibm_db and now I use jayDeBeApi to connect to Informix through python. it requires java JDBC driver and application is work like a charm.
https://pypi.org/project/JayDeBeApi/#:~:text=The%20JayDeBeApi%20module%20allows%20you,of%20the%20Java%20JDBC%20driver.
Your JDBC connection string points to a SQLI Informix listener, but the 'ibm_db' python module uses DRDA (IBM Data Server Driver) to connect to the Informix engine.
Informix allows both SQLI and DRDA clients (and others like MongoDB). SQLI is the 'native' Informix protocol and supports all the Informix server features and data types. DRDA is what other IBM databases use (like DB2). It does have some limitations in terms of what types you can use.
You have two options:
Configure the Informix server to listen for DRDA connections in a another port (basically, create an DBALIASES using 'drsoctcp') as described here:
https://www.ibm.com/support/knowledgecenter/SSGU8G_11.50.0/com.ibm.admin.doc/ids_admin_0207.htm
Or leave the server as it is, and use a different Python module, one that uses SQLI like 'IfxPy'
https://github.com/OpenInformix/IfxPy