I've been stuck for a few days trying to run some code in MySQL to fill a database that I have already created. Initially upon running I got the error 1251 :
"Client does not support authentication protocol requested by server; consider upgrading MySQL client". In the MySQL documentation and stackoverflow answers I found, I was led to change the default insecureAuth setting from the default false to true. Here is the code I am currently using...
import datetime
import MySQLdb as mdb
from math import ceil
def obtain_btc():
now = datetime.datetime.utcnow()
symbols = ['BTC', 'Crypto', 'Bitcoin', 'No Sector', 'USD', now, now]
return symbols
def insert_btc_symbols(symbols, insecureAuth):
db_host = 'localhost'
db_user = 'natrob'
db_pass = '**********'
db_name = 'securities_master'
con = mdb.connect(host=db_host,user=db_user,passwd=db_pass,db=db_name,{insecureAuth:true})
column_str = "ticker, instrument, name, sector, currency, created_date, last_updated_date"
insert_str = (("%s, ")*7)[:2]
final_str = ("INSERT INTO symbols (%s) VALUES (%s)" % (column_str,insert_str))
print (final_str,len(symbols))
with con:
cur = con.cursor()
for i in range(0,int(ceil(len(symbols)/100.0))):
cur.executemany(final_str,symbols[i*100:(i+1)*100-1])
if __name__ == "__main__":
symbols = obtain_btc()
insert_btc_symbols(symbols)
I recently have gotten the error: "non-keyword arg after keyword arg". I've tried to switch the order to no avail, which leads me to believe that I may not be changing the default setting correctly. Any help or advice is appreciated. Thank you.
The issue looks like is coming from {insecureAuth:true} where it is not a keyword argument. ie var=value. I'm not familiar with the library but if that is a keyword then you should be able to set it as a keyword or pass it with **
con = mdb.connect(host=db_host,user=db_user,passwd=db_pass,db=db_name,insecureAuth=True)
or
con = mdb.connect(host=db_host,user=db_user,passwd=db_pass,db=db_name,**{insecureAuth:true})
I managed to get the section of code working by getting the public key for the password and using that in place of the normal password. This was in lieu of using the insecureAuth parameters.
Related
I have the following python code
cred_dict = load_credentials()
user = cred_dict["user"]
pwd = cred_dict["pwd"]
host = cred_dict["host"]
port = cred_dict["port"]
db = cred_dict["db"]
schema = cred_dict["schema"]
con_string = f'Driver={{PostgreSQL Unicode}};Server={host};Database={db};Port={port};UID={user};PWD={pwd};'
params = urllib.parse.quote_plus(con_string)
con = sqlalchemy.create_engine("mssql:///?odbc_connect=%s" % params,fast_executemany=True)
con = con.connect()
but I keep getting ERROR: function schema_name() does not exist.
I have tested different drivers but im using the PostgreSQL Unicode for my DSN setting in ODBC, and it is working w/o any issues. I have also tried mssql+pyodbc but it does not do any difference.
Note, I cannot use pyodbc.connect("DSN="+DSN) since I have to use an sqlalchemy-connection for our production-environment
I managed to work around it by installing psycopg2 and then do
con = create_engine(f'postgresql://{self.user}:{self.pwd}#{self.host}:{self.port}/{self.db}')
which then is passed to
df = pd.read_sql(query=query, con=con)
I'm writing a simple script that checks if user account is about to expire. I'm having a problem with an UPDATE query - it doesn't update, basically. All examples I've found on the internet seem to use tuples to update rows, however my case requires parameters to be apart from each other.
I'm completely new to Python (I started literally yesterday). My database is MySQL (almost all examples on the web use SQLite) and I can't change that. I use Python 3 and the server is running on Ubuntu 18.04. I tried replacing %s with ? or :variable. I also tried insecure way of doing this (SQL Injection vulnerable) and it didn't work either.
This is my current code:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import mysql.connector
from mysql.connector import Error
import datetime
try:
mydb = mysql.connector.connect(
host="localhost",
user="rootery-root",
passwd="example",
database="playground"
)
sqlCursor = mydb.cursor()
sqlCursor.execute("SELECT id, email, last_email_date FROM users WHERE soon_expires = 1")
sqlResult = sqlCursor.fetchall()
setLastMailCmd = """UPDATE users SET last_email_date =%s WHERE id =%s"""
today = datetime.date.today()
for i in range(0, len(sqlResult)):
id = sqlResult[i][0]
email = sqlResult[i][1]
lastemaildate = sqlResult[i][2]
daydiff = lastemaildate - today
setLastMail = sqlCursor.execute(setLastMailCmd, (id, today))
if daydiff.days >= 30:
print "Sending mail and updating \"last_email_date\"."
setLastMail
else:
print "%s already received an e-mail this month - ignored." % email
setLastMail # debugging purposes
except mysql.connector.Error as error:
print("SQL connection error.".format(error))
finally:
if (mydb.is_connected()):
sqlCursor.close()
mydb.close()
print("Disconnected from database.")
print(today)
I expected it to update my table with data provided by the for loop, however it does nothing at all.
Try using functions more. You put everything in one place and it's not easy to debug.
Your problem is the way you use setLastMail # debugging purposes. It does nothing.
What would be better:
mydb = mysql.connector.connect(
host="localhost",
user="rootery-root",
passwd="example",
database="playground"
)
sqlCursor = mydb.cursor()
def set_last_email(id):
stmt = """UPDATE users SET last_email_date =%s WHERE id =%s"""
today = datetime.date.today()
sqlCursor.execute(stmt, (id, today))
And then just execute your set_last_email(id).
Remember to make cursor global, otherwise it won't be available in your function. Or acquire it directly in your function from global connection.
That's of course a dummy example, but you need to start somewhere :)
I'm trying to query an RDS (Postgres) database through Python, more specifically a Jupyter Notebook. Overall, what I've been trying for now is:
import boto3
client = boto3.client('rds-data')
response = client.execute_sql(
awsSecretStoreArn='string',
database='string',
dbClusterOrInstanceArn='string',
schema='string',
sqlStatements='string'
)
The error I've been receiving is:
BadRequestException: An error occurred (BadRequestException) when calling the ExecuteSql operation: ERROR: invalid cluster id: arn:aws:rds:us-east-1:839600708595:db:zprime
In the end, it was much simpler than I thought, nothing fancy or specific. It was basically a solution I had used before when accessing one of my local DBs. Simply import a specific library for your database type (Postgres, MySQL, etc) and then connect to it in order to execute queries through python.
I don't know if it will be the best solution since making queries through python will probably be much slower than doing them directly, but it's what works for now.
import psycopg2
conn = psycopg2.connect(database = 'database_name',
user = 'user',
password = 'password',
host = 'host',
port = 'port')
cur = conn.cursor()
cur.execute('''
SELECT *
FROM table;
''')
cur.fetchall()
I am trying to write a Python script to access Amazon Redshift to create a table in Redshift and copy data from S3 to the Redshift table.
My code is:
import psycopg2
import os
#import pandas as pd
import requests
requests.packages.urllib3.disable_warnings()
redshift_endpoint = os.getenv("END-point")
redshift_user = os.getenv("user")
redshift_pass = os.getenv("PASSWORD")
port = 5439
dbname = 'DBNAME'
conn = psycopg2.connect(
host="",
user='',
port=5439,
password='',
dbname='')
cur = conn.cursor()
aws_key = os.getenv("access_key") # needed to access S3 Sample Data
aws_secret = os.getenv("secret_key")
#aws_iam_role= os.getenv('iam_role') #tried using this too
base_copy_string= """copy %s from 's3://mypath/%s'.csv
credentials 'aws_access_key_id= %s aws_access_secrect_key= %s'
delimiter '%s';""" # the base COPY string that we'll be using
#easily generate each table that we'll need to COPY data from
tables = ["employee"]
data_files = ["test"]
delimiters = [","]
#the generated COPY statements we'll be using to load data;
copy_statements = []
for tab, f, delim in zip(tables, data_files, delimiters):
copy_statements.append(base_copy_string % (tab, f, aws_key, aws_secret, delim)%)
#create Table
cur.execute(""" create table employee(empname varchar(30),empno integer,phoneno integer,email varchar(30))""")
for copy_statement in copy_statements: # execute each COPY statement
cur.execute(copy_statement)
conn.commit()
for table in tables + ["employee"]:
cur.execute("select count(*) from %s;" % (table,))
print(cur.fetchone())
conn.commit() # make sure data went through and commit our statements permanently.
When I run this command I getting an Error at cur.execute(copy_statement)
**Error:** error: Invalid credentials. Must be of the format: credentials 'aws_iam_role=...' or 'aws_access_key_id=...;aws_secre
t_access_key=...[;token=...]'
code: 8001
context:
query: 582
location: aws_credentials_parser.cpp:114
process: padbmaster [pid=18692]
Is there a problem in my code? Or is it is an AWS access_key problem?
I even tried using an iam_role but I get an error:
IAM role cannot assume role even in Redshift
I have a managed IAM role permission by attaching S3FullAccess policy.
There are some errors in your script.
1) Change base_copy_string as below:
base_copy_string= """copy %s from 's3://mypath/%s.csv' credentials
'aws_access_key_id=%s;aws_secret_access_key=%s' delimiter '%s';""" #
the base COPY string that we'll be using
There must be a ; added in credentials and also other formatting issues with single-quotes. It is aws_secret_access_key and not aws_access_secrect_key.
check this link for detailed info: http://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-access-permissions.html#copy-usage_notes-iam-permissions
I suggest you use iam-roles instead of credentials.
http://docs.aws.amazon.com/redshift/latest/dg/loading-data-access-permissions.html
2) change copy_statements.append as below(remove extra % in the end):
copy_statements.append(base_copy_string % (tab, f, aws_key,
aws_secret, delim))
Correct these and try again.
To start with, NEVER, NEVER, NEVER hardcode access keys and secret keys in your code. So that rules out your first query. Now coming to right way of implementing things. You are right, IAM Role is the right way of doing it. Unfortunately, I can't get the exact error and use case from your description. As far as I understand, you are trying to run this python file from your computer(local machine). Hence, you need to attach permission with your IAM user to have access to RedShift(and all other services your code is touching). Please correct me if my assumption is wrong.
Just in case if you missed
Install AWS CLI
Run
aws configure
Put your credentials and region
Hope this helps.
I am trying to connect to a Postgres Database with variables like this:
cs = "dbname=%s user=%s password=%s host=%s port=%s",(dn,du,dp,dh,dbp)
con = None
con = psycopg2.connect(cs)
However I get the error message:
TypeError: argument 1 must be string, not tuple
I need to be able to use variables in the connection string. Anyone know how to accomplish this?
Your code currently creates a tuple with your string and the tuple you are trying to sub. You need:
cs = "dbname=%s user=%s password=%s host=%s port=%s" % (dn,du,dp,dh,dbp)
You could pass params directly without building connection string:
con = psycopg2.connect(
dbname=dn,
user=du,
password=dp,
host=dh,
port=dbp,
)
Docs for more details on psycopg2.connect