I continue to get and error "ProgrammingError: 002003 (42502): SQL compilation error: Object 'Table' does not exist or not authorized. I am using the following code:
con = snowflake.connector.connect(
user = "user.name",
authenticator="externalbrowser",
warehouse = "ware house name",
database = "db name",
schema = "schema name"
)
cur.con.cursor()
sql = "select * from Table"
cur.execute(sql)
df = cur.fetch_pandas_all()
When I execute the code in Jupyter Notebook the browser window opens and authenticates my creds but when it gets to the sql execute line the error rises and tells me that the table does not exist. When I open up Snowflake in my browser I can see that the table does exist in the correct warehouse, database and schema I have in my code.
Has anyone else ever experienced this? Do I need to authorize my user to be able to access this table via Python and Jupyter Notebook?
It's likely your session doesn't have a role assigned to it (current role).
You can add the role in your list of connection session paramters,
e.g. add something like the following
role = 'RICH_ROLE',
You might want to consider setting a default role for your user.
ALTER USER userNameHere SET DEFAULT_ROLE = 'THE_BEST_ROLE';
docs link: https://docs.snowflake.com/en/sql-reference/sql/alter-user.html
Also, when all else fails, use the fully qualified table name, note this won't help much if the role isn't set:
sql = "select * from databaseName.schemaName.TableName"
Related
i'm beginner in sqlalchemy, i want to reflect my table in database to object, but always return invalid password, even though the password is correct. I dont understand why this happend. When i try to inspect they return my table name, so my password, connection string or on create_engine is correct.
when my database have no password is fine i can reflect it to Object, that's so weird.
but why when i reflect database with password it's error, always return "Not a valid password" ??,
My MS. Access Tbl 1
My MS. Access Tbl 2
Error in Reflect but My Table name is returned
This is my Code
because I was curious I also made a test select data, and it turned out to be successful in retrieving the data
it's returned my data and success created connection
when i add some code for testing
I think all it's correct but why cannot reflect??, Please Help.
My Reference connection_string
My Reference SqlAlchemy Automap Reflect
I have just released sqlalchemy-access version 1.1.1 to address this issue. Note that as described in Getting Connected if you want to use a pass-through ODBC connection string with an encrypted database you need to supply the password in two places:
driver = "{Microsoft Access Driver (*.mdb, *.accdb)}"
db_path = r"C:\Users\Public\test\sqlalchemy-access\gord_test.accdb"
pwd = "tiger"
connection_string = (
f"DRIVER={driver};"
f"DBQ={db_path};"
f"PWD={pwd};"
f"ExtendedAnsiSQL=1;"
)
connection_uri = (
f"access+pyodbc://admin:{pwd}#/"
f"?odbc_connect={urllib.parse.quote_plus(connection_string)}"
)
engine = sa.create_engine(connection_uri)
We have data in a Snowflake cloud database that we would like to move into an Oracle database. As we would like to work toward refreshing the Oracle database regularly, I am trying to use SQLAlchemy to automate this.
I would like to do this using Core because my team is all experienced with SQL, but I am the only one with Python experience. I think it would be easier to tweak the data pulls if we just pass SQL strings. Plus the Snowflake db has some columns with JSON that seems easier to parse using direct SQL since I do not see JSON in the SnowflakeDialect.
I have established connections to both databases and am able to do select queries from both. I have also manually created the tables in our Oracle db so that the keys and datatypes match what I am pulling from Snowflake. When I try to insert, though, my Jupyter notebook just continuously says "Executing Cell" and hangs. Any thoughts on how to proceed or how to get the notebook to tell me where the hangup is?
from sqlalchemy import create_engine,pool,MetaData,text
from snowflake.sqlalchemy import URL
import pandas as pd
eng_sf = create_engine(URL( #engine for snowflake
account = 'account'
user = 'user'
password = 'password'
database = 'database'
schema = 'schema'
warehouse = 'warehouse'
role = 'role'
timezone = 'timezone'
))
eng_o = create_engine("oracle+cx_oracle://{}[{}]:{}#{}".format('user','proxy','password','database'),poolclass=pool.NullPool) #engine for oracle
meta_o = MetaData()
meta_o.reflect(bind=eng_o)
person_o = meta_o['bb_lms_person'] # other oracle tables follow this example
meta_sf = MetaData()
meta_sf.reflect(bind=eng_sf,only=['person']) # other snowflake tables as well, but for simplicity, let's look at one
person_sf = meta_sf.tables['person']
person_query = """
SELECT ID
,EMAIL
,STAGE:student_id::STRING as STUDENT_ID
,ROW_INSERTED_TIME
,ROW_UPDATED_TIME
,ROW_DELETED_TIME
FROM cdm_lms.PERSON
"""
with eng_sf.begin() as connection:
result = connection.execute(text(person_query)).fetchall() # this snippet runs and returns result as expected
with eng_o.begin() as connection:
connection.execute(person_o.insert(),result) # this is a coinflip, sometimes it runs, sometimes it just hangs 5ever
eng_sf.dispose()
eng_o.dispose()
I've checked the typical offenders. The keys for both person_o and the result are all lowercase and match. Any guidance would be appreciated.
use the metadata for the table. the fTable_Stage update or inserted as fluent functions and assign values to lambda variables. This is very safe because only metadata field variables can be used in the lambda. I am updating three fields:LateProbabilityDNN, Sentiment_Polarity, Sentiment_Subjectivity
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
connection=engine.connect()
metadata=MetaData()
Session = sessionmaker(bind = engine)
session = Session()
fTable_Stage=Table('fTable_Stage', metadata,autoload=True,autoload_with=engine)
stmt=fTable_Stage.update().where(fTable_Stage.c.KeyID==keyID).values(\
LateProbabilityDNN=round(float(late_proba),2),\
Sentiment_Polarity=round(my_valance.sentiment.polarity,2),\
Sentiment_Subjectivity= round(my_valance.sentiment.subjectivity,2)\
)
connection.execute(stmt)
I have a sqlalchemy connection setup to snowflake which works, as I can run some queries and get results out. The attempts to query are also logged in my user_query history.
My connection:
engine = create_engine(URL(
user, password, account, database, warehouse, role
))
connection = engine.connect()
However, most of the time my queries fail returning Operational Error (i.e. its a snowflake error) https://docs.sqlalchemy.org/en/13/errors.html#error-e3q8. But these same queries will run fine in the snowflake web UI.
For example if I run
test_query = 'SELECT * FROM TABLE DB1.SCHEMA1.TABLE1'
test = pd.read_sql(test_query, connection)
When I look at my query_history it shows the sqlalchemy query failing, then a second later the base query itself being run successfully. However I'm not sure where this output goes in the snowflake setup, and why its not transferring through my sqlalchemy connection. What I'm seeing...
Query = 'DESC TABLE /* sqlalchemy:_has_object */ "SELECT * FROM DB1"."SCHEMA1"."TABLE1"
Error code = 2003 Error message = SQL compilation error: Database
'"SELECT * FROM DB1" does not exist.
Then 1 second later, the query itself will run successfully, but not clear where this goes as it doesn't get sent over the connection.
Query = SELECT * FROM TABLE DB1.SCHEMA1.TABLE1
Any help much appreciated!
Thanks
You can try adding schema also here
engine = create_engine(URL(
account = '',
user = '',
password = '',
database = '',
schema = '',
warehouse = '',
role='',
))
connection = engine.connect()
It is very unlikely that the query is running in WebUI and fails with syntax error when connected via CLI or other modes.
Suggest you print the query which is via CLI or via a connector, run the same to WebUI and also note that from which role you're running the query.
Please share what is your finding.
The mentioned query (SELECT * FROM TABLE DB1.SCHEMA1.TABLE1) is not a snowflake supported SQL syntax.
Link here will help you more with details.
Hope this helps!
I am trying to write a Python script to access Amazon Redshift to create a table in Redshift and copy data from S3 to the Redshift table.
My code is:
import psycopg2
import os
#import pandas as pd
import requests
requests.packages.urllib3.disable_warnings()
redshift_endpoint = os.getenv("END-point")
redshift_user = os.getenv("user")
redshift_pass = os.getenv("PASSWORD")
port = 5439
dbname = 'DBNAME'
conn = psycopg2.connect(
host="",
user='',
port=5439,
password='',
dbname='')
cur = conn.cursor()
aws_key = os.getenv("access_key") # needed to access S3 Sample Data
aws_secret = os.getenv("secret_key")
#aws_iam_role= os.getenv('iam_role') #tried using this too
base_copy_string= """copy %s from 's3://mypath/%s'.csv
credentials 'aws_access_key_id= %s aws_access_secrect_key= %s'
delimiter '%s';""" # the base COPY string that we'll be using
#easily generate each table that we'll need to COPY data from
tables = ["employee"]
data_files = ["test"]
delimiters = [","]
#the generated COPY statements we'll be using to load data;
copy_statements = []
for tab, f, delim in zip(tables, data_files, delimiters):
copy_statements.append(base_copy_string % (tab, f, aws_key, aws_secret, delim)%)
#create Table
cur.execute(""" create table employee(empname varchar(30),empno integer,phoneno integer,email varchar(30))""")
for copy_statement in copy_statements: # execute each COPY statement
cur.execute(copy_statement)
conn.commit()
for table in tables + ["employee"]:
cur.execute("select count(*) from %s;" % (table,))
print(cur.fetchone())
conn.commit() # make sure data went through and commit our statements permanently.
When I run this command I getting an Error at cur.execute(copy_statement)
**Error:** error: Invalid credentials. Must be of the format: credentials 'aws_iam_role=...' or 'aws_access_key_id=...;aws_secre
t_access_key=...[;token=...]'
code: 8001
context:
query: 582
location: aws_credentials_parser.cpp:114
process: padbmaster [pid=18692]
Is there a problem in my code? Or is it is an AWS access_key problem?
I even tried using an iam_role but I get an error:
IAM role cannot assume role even in Redshift
I have a managed IAM role permission by attaching S3FullAccess policy.
There are some errors in your script.
1) Change base_copy_string as below:
base_copy_string= """copy %s from 's3://mypath/%s.csv' credentials
'aws_access_key_id=%s;aws_secret_access_key=%s' delimiter '%s';""" #
the base COPY string that we'll be using
There must be a ; added in credentials and also other formatting issues with single-quotes. It is aws_secret_access_key and not aws_access_secrect_key.
check this link for detailed info: http://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-access-permissions.html#copy-usage_notes-iam-permissions
I suggest you use iam-roles instead of credentials.
http://docs.aws.amazon.com/redshift/latest/dg/loading-data-access-permissions.html
2) change copy_statements.append as below(remove extra % in the end):
copy_statements.append(base_copy_string % (tab, f, aws_key,
aws_secret, delim))
Correct these and try again.
To start with, NEVER, NEVER, NEVER hardcode access keys and secret keys in your code. So that rules out your first query. Now coming to right way of implementing things. You are right, IAM Role is the right way of doing it. Unfortunately, I can't get the exact error and use case from your description. As far as I understand, you are trying to run this python file from your computer(local machine). Hence, you need to attach permission with your IAM user to have access to RedShift(and all other services your code is touching). Please correct me if my assumption is wrong.
Just in case if you missed
Install AWS CLI
Run
aws configure
Put your credentials and region
Hope this helps.
I'm using pyodbc to access DB2 10.1.0
I have a login account named foobar and a schema with the same name. I have a table named users under the schema.
When I'm logged in as foobar, I can run the following query successfully from the command line:
select * from users
I have a small Python script that I'm using to connect to the database. The script is:
#!/usr/bin/python
import pyodbc
if __name__ == "__main__":
accessString ="DRIVER={DB2};DATABASE=MYDATABASE;SERVER=localhost;UID=foobar; PWD=foobarish1;CURRENTSCHEMA=FOOBAR"
print accessString
cnxn = pyodbc.connect(accessString , autocommit=True)
cursor = cnxn.cursor()
query = "SELECT * FROM USERS"
cursor.execute(query)
rows = cursor.fetchall()
for row in rows:
print 'Row data'
print row[0]
cursor.close()
cnxn.close()
When I run the script, I get the following error:
('42S02', '[42S02] [IBM][CLI Driver][DB2/LINUXX8664] SQL0204N "FOOBAR.USERS" is an undefined name. SQLSTATE=42704\n (-204) (SQLExecDirectW)')
This usually means that the schema isn't defined. However, if I change the query in the script to:
VALUES CURRENT SCHEMA
the script runs successfully and it returns
FOOBAR
Does anyone know how to fix this so I can query the user table? Your assistance and insight is appreciated.
EDIT: I've also tried adding the schema directly to the table name, making the query
SELECT * FROM FOOBAR.USERS
and I still get the same error.
I had the same issue and solved it by setting the SCHEMA explicit by query:
SET CURRENT SCHEMA foobar