I have a postgresql database "Test2" hosted in my localhost. I am able to see the tables using pgAdmin. I want to fetch the data of the DB from Jupyter Notebook. I tried to connect to the DB by following the steps shown in "2) of Part 2" of https://towardsdatascience.com/python-and-postgresql-how-to-access-a-postgresql-database-like-a-data-scientist-b5a9c5a0ea43
Thus, my code is --
import config as creds
import pandas as pd
def connect():
# Set up a connection to the postgres server.
conn_string = "host="+ creds.PGHOST +" port="+ "5432" +" dbname="+ creds.PGDATABASE +" user=" + creds.PGUSER \
+" password="+ creds.PGPASSWORD
conn = psycopg2.connect(conn_string)
print("Connected!")
# Create a cursor object
cursor = conn.cursor()
return conn, cursor
#Connecting to DB
conn, cursor = connect()
#SQL command to create inventory table
abc = ("""SELECT * FROM clubs""")
#Execute SQL Command and commit to DB
cursor.execute(abc)
results = cursor.fetchall()
print(results)
conn.commit()
My config.py looks like this -->
PGHOST = 'localhost'
PGDATABASE = 'Test2'
PGUSER = '#####'
PGPASSWORD = '#####'
I able to get the output when the table name has all lowercase characters but for table names which has mixed character like "clubCategory", it throws an error stating "relation "clubcategory" does not exist"
I tried
abc = ("""SELECT * FROM 'clubCategory' """)
but its still throws error.
Any help please?
Try using double quotes:
abc = ('''SELECT * FROM "clubCategory" ''')
Also see this answer: https://stackoverflow.com/a/21798517/1453822
Related
In Jupyter Notebook, I was able to connect to my work database using pyobdc with 'dsn; userid; pw' connection string successfully. I tested out printing all the table names in the database. Then I tried to run the following code to test a simple query.
================================
query = "select * from TEST_TABLE"
cursor.execute(query)
================================
But I got the following error.
ProgrammingError: ('42S02', '[42S02] [IBM][CLI Driver][DB2/AIX64] SQL0204N "**myusername.TEST_TABLE**" is an **undefined name**. SQLSTATE=42704\r\n (-204) (SQLExecDirectW)')
TEST_TABLE does exist in the database I'm connected to. But for some reason, the code is adding myusername in front of the TEST_TABLE and tells me the table name is not defined.
Try setting up the PyODBC connection as follows:
import pyodbc
dbConn = pyodbc.connect("DSN=<dsn>;UID=<uid>;PWD=<pwd>;DATABASE=<db>")
dbCursor = dbConn.cursor()
query = "SELECT * FROM TEST_TABLE"
dbCursor.execute(query)
where:
dsn = the hostname (or IP address)
uid = the username
pwd = the password
db = the database name
I am trying to create tables out of json files containing the field names and types of each table of a database downloaded from Bigquery. The SQL request semt fine to me and but no table was created according to psql command-line interpreter typing \d
So, to begin I've just tried with a simpler sql request that doesn't work neither,
Here is the code :
import pandas as pd
import psycopg2
# information used to create a database connection
sqluser = 'postgres'
dbname = 'testdb'
pwd = 'postgres'
# Connect to postgres database
con = psycopg2.connect(dbname=dbname, user=sqluser, password=pwd )
curs=con.cursor()
q="""set search_path to public,public ;
CREATE TABLE tab1(
i INTEGER
);
"""
curs.execute(q)
q = """
SELECT table_name
FROM information_schema.tables
WHERE table_schema='public'
AND table_type='BASE TABLE';
"""
df = pd.read_sql_query(q, con)
print(df.head())
print("End of test")
The code written above displays this new table tab1, but actually this new table doesn't appear listed when typing \d within the psql command line interpreter. If I type in the psql interpreter :
SELECT table_name
FROM information_schema.tables
WHERE table_type='BASE TABLE';
it doesn't get listed neither , seems it's not actually created, Thanks in advance for your help
There was a commit() call missing, that must be written after the table creation sql request,
This code works:
import pandas as pd
import psycopg2
# information used to create a database connection
sqluser = 'postgres'
dbname = 'testdb'
pwd = 'postgres'
# Connect to postgres database
con = psycopg2.connect(dbname=dbname, user=sqluser, password=pwd )
curs=con.cursor()
q="""set search_path to public,public ;
CREATE TABLE tab1(
i INTEGER
);
"""
curs.execute(q)
con.commit()
q = """
SELECT table_name
FROM information_schema.tables
WHERE table_schema='public'
AND table_type='BASE TABLE';
"""
df = pd.read_sql_query(q, con)
print(df.head())
print("End of test")
I have a column called REQUIREDCOLUMNS in a SQL database which contains the columns which I need to select in my Python script below.
Excerpt of Current Code:
db = mongo_client.get_database(asqldb_row.SCHEMA_NAME)
coll = db.get_collection(asqldb_row.TABLE_NAME)
table = list(coll.find())
root = json_normalize(table)
The REQUIREDCOLUMNSin SQL contains values reportId, siteId, price, location
So instead of explicitly typing:
print(root[["reportId","siteId","price","location"]])
Is there a way to do print(root[REQUIREDCOLUMNS])?
Note: (I'm already connected to the SQL database in my python script)
You will have to use cursors if you are using mysql or pymysql , both the syntax are almost similar below i will mention for mysql
import mysql
import mysql.connector
db = mysql.connector.connect(
host = "localhost",
user = "root",
passwd = " ",
database = " "
)
cursor = db.cursor()
sql="select REQUIREDCOLUMNS from table_name"
cursor.execute(sql)
required_cols = cursor.fetchall()#this wll give ["reportId","siteId","price","location"]
cols_as_string=','.join(required_cols)
new_sql='select '+cols_as_string+' from table_name'
cursor.execute(new_sql)
result=cursor.fetchall()
This should probably work, i intentionally split many lines into several lines for understanding.
syntax could be slightly different for pymysql
I tried to connect snowflake(SSO Authentication) and get data from table.
But, When i run the code, i can login with my credentials in the pop-up browser window and it connect Snowflake, no response after that(the program neither terminate nor provide result).
Not sure, where am doing mistake, Pls help.
'''
import snowflake.connector
# Connecting to Snowflake using SAML 2.0-compliant IdP federated authentication
conn = snowflake.connector.connect(
user='G*****K',
account='abcdf',
authenticator='externalbrowser',
warehouse='abcdf',
database='abcdf',
schema='abcdf'
)
cur = conn.cursor()
sql = "select * from abcdf.ACCT limit 10"
x=cur.execute(sql)
cur.close()
print(x)
'''
I believe you are closing the cursor before the print;
try:
cur.execute("SELECT col1, col2 FROM test_table ORDER BY col1")
for (col1, col2) in cur:
print('{0}, {1}'.format(col1, col2))
finally:
cur.close()
Details: https://docs.snowflake.com/en/user-guide/python-connector-example.html#using-cursor-to-fetch-values
Results of the query are stored in cursor. The contents of cursor may then be stored in a local variable.
Also, best practice to close connection in the end.
https://www.psycopg.org/docs/cursor.html
import snowflake.connector
# Connecting to Snowflake using SAML 2.0-compliant IdP federated authentication
conn = snowflake.connector.connect(
user='G*****K',
account='abcdf',
authenticator='externalbrowser',
warehouse='abcdf',
database='abcdf',
schema='abcdf'
)
cur = conn.cursor()
sql = "select * from abcdf.ACCT limit 10"
cur.execute(sql)
print(cur.fetchall())
cur.close()
conn.close()
When I am trying to connect python with SQL Server, following error occurred.
"pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC SQL Server
Driver][DBNETLIB]SQL Server does not exist or access denied. (17)
(SQLDriverConnect)')"
Following is the my code.
import pyodbc
connection = pyodbc.connect("Driver={SQL Server}; Server=localhost;
Database=emotionDetection; uid=uname ;pwd=pw;Trusted_Connection=yes")
cursor = connection.cursor()
SQLCommand = ("INSERT INTO emotion" "(happy, sad, angry) "
"VALUES (?,?,?)")
Values = ['smile','cry','blame']
cursor.execute(SQLCommand,Values)
connection.commit()
connection.close()
This is my first attempt to connect Python with sql server. I don't have an idea what would be the driver name, server name, username and password.Do you have any idea of what should be my configuration. Please help me.
CONNECTION FROM WINDOWS TO MS SQL SERVER DATABASE:
Here you have an example I use myself to connect to MS SQL database table with a Python script:
import pyodbc
server = 'ip_database_server'
database = 'database_name'
username = 'user_name'
password = 'user_password'
driver = '{SQL Server}' # Driver you need to connect to the database
port = '1433'
cnn = pyodbc.connect('DRIVER='+driver+';PORT=port;SERVER='+server+';PORT=1443;DATABASE='+database+';UID='+username+
';PWD='+password)
cursor = cnn.cursor()
'User' and 'password' and 'table_name' are attibutes defined by the DB administrator, and he should give them to you. The port to connect to is also defined by the admin. If you are trying to connect from a Windows device to the DB, go to ODBC Data Source Administrator from Windows, and check if you have installed the driver:
Where is the ODBC data source administrator in a Windows machine.
The image is in spanish, but you only have to click on 'Drivers' tab, and check if the driver is there as in the image.
CONNECTION FROM LINUX/UNIX TO MS SQL SERVER DATABASE:
If you are working in Linux/Unix, then you shoud install a ODBC manager like 'FreeTDS' and 'unixODBC'. To configure them, you have some examples in the following links:
Example: Connecting to Microsoft SQL Server from Linux/Unix
Example: Installing and Configuring ODBC
I think you should check out this.
stackoverflow answer about odbc
Also, what sql server do you use?
The library pymssql doesnot require any drivers and works on both Windows as well as Ubunutu.
import pymssql
import pandas as pd
server = 'yourusername'
username = 'yourusername'
password = 'yourpassword'
database = 'yourdatabase'
table_name = 'yourtablename'
conn = pymssql.connect(host=server,user=username,password=password,database=database)
dat = pd.read_sql("select * from table_name,conn)
Try pyodbc with SQLalchemy
try this:
import sqlalchemy
import pyodbc
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://user:password#host:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
cnxn = engine.connect()
Use your corresponding driver
It works for me
Luck!
Working Examples Work Best For Me:
Need Mac ODBC Drivers?
If you need the mac driver I used homebrew and found the commands here
Detail
I personally learn best by reverse enginerring, with that said I am sharing one of my examples, it may be a bit crude but I'm growing my Python skills.
My script I created allows me to connect my Mac OS to a AWS RDS instance.
The whole script is a copy paste with a little modification for you about your server info, and you are off and running.
just modify these lines to connect.
server = 'yourusername'
username = 'yourusername'
password = 'yourforgottencomplicatedpassword'
database = 'yourdatabase'
Then Run the file: python3 ~/Your/path/pyodbc_mssqldbtest.py and you should be set.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# =============================================================================
# Created By : Jeromie Kirchoff
# Created Date: Mon July 31 22:32:00 PDT 2018
# FILENAME: pyodbc_mssqldbtest.py
# =============================================================================
"""The Module Has Been Build for Interaction with MSSQL DBs To Test the con."""
# =============================================================================
# Thanks to this post for headers https://stackoverflow.com/q/12704305/1896134
# Answer to an SO question: https://stackoverflow.com/q/42433408/1896134
# =============================================================================
import pyodbc
def runningwithqueries(query):
"""The Module Has Been Build to {Open, Run & Close} query connection."""
print("\nRunning Query: " + str(query) + "\nResult :\n")
crsr = cnxn.execute(query)
columns = [column[0] for column in crsr.description]
print(columns)
for row in crsr.fetchall():
print(row)
crsr.close()
# =============================================================================
# SET VARIABLES NEEDED FOR SERVER CONNECTION
# =============================================================================
server = 'yourusername'
username = 'yourusername'
password = 'yourforgottencomplicatedpassword'
database = 'yourdatabase'
connStr = (r'DRIVER={ODBC Driver 17 for SQL Server};' +
r"Integrated Security=True;" +
r'SERVER=' + server +
r';UID=' + username +
r';PWD=' + password +
r';DSN=MSSQL-PYTHON' +
r';DATABASE=' + database + ';'
)
print("Your Connection String:\n" + str(connStr) + "\n\n")
# =============================================================================
# CONNECT TO THE DB
# =============================================================================
cnxn = pyodbc.connect(connStr, autocommit=True)
# =============================================================================
# SET QUERIES TO VARIABLES
# =============================================================================
SQLQUERY1 = ("SELECT ##VERSION;")
SQLQUERY2 = ("SELECT * FROM sys.schemas;")
SQLQUERY3 = ("SELECT * FROM INFORMATION_SCHEMA.TABLES;")
SQLQUERY4 = ("SELECT * FROM INFORMATION_SCHEMA.COLUMNS;")
SQLQUERY5 = ("SELECT * FROM INFORMATION_SCHEMA.CHECK_CONSTRAINTS;")
SQLQUERY6 = ("EXEC sp_databases;")
SQLQUERY7 = ("EXEC sp_who2 'active';")
# =============================================================================
# RUN QUERIES
# YOU CAN RUN AS MANY QUERIES AS LONG AS THE CONNECTION IS OPEN TO THE DB
# =============================================================================
runningwithqueries(SQLQUERY1)
runningwithqueries(SQLQUERY2)
runningwithqueries(SQLQUERY3)
runningwithqueries(SQLQUERY4)
runningwithqueries(SQLQUERY5)
runningwithqueries(SQLQUERY6)
runningwithqueries(SQLQUERY7)
# =============================================================================
# CLOSE THE CONNECTION TO THE DB
# =============================================================================
cnxn.close()
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};' 'Server=**SERVER NAME**;' 'Database=**DATABASE NAME**;' 'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT * FROM Output3')
This works just check you specify the Respective Driver, Server and the Database names correctively!