We Keep Coding

How to extract data from Oracle database with AWS Glue and other AWS services

I am new to AWS glue and other AWS stuff. I have a requirement to build an ETL framework for a project.
This is the high-level diagram. I want to understand, instead of creating 400 glue pipelines, can I create a template kind of a thing which is driven by reference data from a postgres aurora/mysql. I am familiar with Python.
Anyone has any ideas on this? Any references, code examples.

We had a config master table in our mysql db. The columns per convenience we had source_table_name as the identifier to fetch appropriate table column names/queries for CREATING STG TABLE, LOAD DATA INTO STG TABLE, INSERT/UPDATE INTO TARGET TABLEs etc.
We have also split the INSERT/UPDATE into two different columns in config master, since we were using ON DUPLICATE KEY to update existing records.
get the source table name, by processing the lambda events which will have landing file name.
Fetch all required data from the config master for the source table name. It would be something like following:
sql_query = "SELECT * FROM {0}.CONFIG_MASTER WHERE src_tbl_name = %s ".format(mydb)
cur.execute(sql_query, (source_fname))
result = cur.fetchall()
for row in result:
stg_table_name = row[1]
tgt_table_name = row[2]
create_stg_table_qry = row[3]
load_data_stg_table_qry = row[4]
insert_tgt_table_qry = row[5]
insert_tgt_table_qry_part_1 = row[6]
insert_tgt_table_qry_part_2 = row[7]
conn.commit()
cur.close()
Pass appropriate parameters to the generic functions as below:
create_stg_table(stg_table_name, create_stg_table_qry, load_data_stg_table_qry)
loaddata(tgt_table_name, insert_tgt_table_qry_part_1, insert_tgt_table_qry_part_2, stg_table_name)
The generic functions would be something like below, this is for aurora RDS, please make changes as needed.
def create_stg_table(stg_table_name, create_stg_table_qry, load_data_stg_table_qry):
cur, conn = connect()
createStgTable1 = "DROP TABLE IF EXISTS {0}.{1}".format(mydb, stg_table_name)
createStgTable2 = "CREATE TABLE {0}.{1} {2}".format(mydb, stg_table_name, create_stg_table_qry)
loadQry = "LOAD DATA FROM S3 PREFIX 's3://' REPLACE INTO TABLE ...".format()
cur.execute(createStgTable1)
cur.execute(createStgTable2)
cur.execute(loadQry)
conn.commit()
conn.close()
def loaddata(tgt_table_name, insert_tgt_table_qry_part_1, insert_tgt_table_qry_part_2, stg_table_name):
cur, conn = connect()
insertQry = "INSERT INTO target table, from the staging table query here"
print(insertQry)
cur.execute(insertQry)
conn.commit()
conn.close()
Hope this gives an idea.
Thanks

MySQL query result not reflecting values in database after query is executed once

I am working on a PyQT5 application with part of it being a table that displays movie ratings in a QTableWidget. When I add a movie to the database, I clear the old table and run a new select query to get the ten most recent movies logged. This query works correctly the first time, but fails every time afterwards. Each time after the first add, the select query just returns the 10 entries from the first time the query was executed instead of returning movies logged since then. What's confusing me is that if I run the query in MySQL workbench, it runs correctly every time. It's only when the entries are passed to my application that things go wrong. Here is the refresh method that is run after an entry is added:
def refreshLastTen(self):
#Refreshes the table of recently watched movies after a new entry is added or a change
print('Refreshing Last10 Table')
self.LastTenTable.clearContents()
#When adding one new movie, the refresh works fine. Anything more than that, the result from the first add is returned every time
sql = "SELECT * FROM log ORDER BY LOG_MOVIE_DATE desc LIMIT 0, 10" #Selects top 10 results from the table
cursor = dbConnection.cursor()
cursor.execute(sql)
myresult = cursor.fetchall()
cursor.close()
for row_number, row_data in enumerate(myresult): #Adds data from select statement to the table
for column_number, data in enumerate(row_data):
self.LastTenTable.setItem(row_number, column_number,QtWidgets.QTableWidgetItem(str(data)))
For example, I add the movie "Test 1" at the current date (theoretically the most recently watched) to the database and the refresh method returns the following:
After this, I add another movie "Test 2" at the current date and instead of returning "Test 2" and "Test 1" as the most recent 2 movies in the list, the same result from the first query is returned again. This continues no matter how many more movies you add. As stated before, if you were to then run the same query in MySQL workbench, it would correctly return "Test 2" and "Test 1" as the two most recent entries. Does anyone have any ideas of what could be happening?
*Edit 1: Here is the code used in my insert method to add a movie to the database:
if self.validateSubmission() == True:
sql = "INSERT INTO log (LOG_MOVIE_TITLE, LOG_MOVIE_DATE, LOG_MOVIE_RATING, LOG_MOVIE_GENRE, LOG_MOVIE_LOCATION, LOG_MOVIE_COMMENTS) VALUES (%s, %s, %s, %s, %s, %s)"
vals = [self.titleVal.text(), self.dateVal.date().toString('yyyy-MM-dd'), self.ratingVal.currentText(), self.genreVal.currentText(), self.locationVal, self.commentVal.toPlainText()]
cursor = dbConnection.cursor()
cursor.execute(sql, vals)
dbConnection.commit()
cursor.close()
dbConnection.close()
*Edit 2: Here is a screenshot from MySQL Workbench showing the select query working:
*Edit 3: Now I am wondering if it has to do with how I am handling the dbConnection. The refreshLastTen method is in the app's main page class whereas the add happens in an add form class. Each class has dbConnection explicitly defined like this:
dbConnection = mysql.connector.connect(
host="localhost",
user="root",
passwd="pwd",
database="moviesheet"
)
I am not sure if this causes a disconnect or collision of some sort. I added the following to close the connection in the add form class after the add is done just in case:
dbConnection.close()
One other oddity that I noticed is that if I run the query in the add form class after the add, it returns the correct result. It seems like it might just be related to the main window class.

python sqlite - deleting selected records

I'm trying to use sqlite3 in python to delete a selection of rows from a table. My attempt fails, but I can't work out why.
The sql query works ok, but I can't implement it within the python code.
I have a set of records that are moved from current_table to archive_table after a period of time.
I'd like to clean up the current_table by removing those rows that are in the archive_table (matched on id).
Intended SQL query:
DELETE FROM current_table WHERE id IN ( SELECT id FROM archive_table);
Attempted python code:
import sqlite3
def clean_up(current_table, archive_table):
db = sqlite3.connect(sqlite_db)
cursor = db.cursor()
sql_query_delete = '''DELETE FROM %s WHERE id IN ( SELECT id FROM %s);''' % (current_table, archive_table)
try:
cursor.execute(sql_query_delete)
db.commit()
db.close()
except:
print("error deleting")

Now working. The database file was locked by another process. Removing the pointless try/except led me to the detailed error message.

using python 2.7 to query sqlite3 database and getting "sqlite3 operational error no such table"

My simple test code is listed below. I created the table already and can query it using the SQLite Manager add-in on Firefox so I know the table and data exist. When I run the query in python (and using the python shell) I get the no such table error
def TroyTest(self, acctno):
conn = sqlite3.connect('TroyData.db')
curs = conn.cursor()
v1 = curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
print v1
conn.close()

When you pass SQLite a non-existing path, it'll happily open a new database for you, instead of telling you that the file did not exist before. When you do that, it'll be empty and you'll instead get a "No such table" error.
You are using a relative path to the database, meaning it'll try to open the database in the current directory, and that is probably not where you think it is..
The remedy is to use an absolute path instead:
conn = sqlite3.connect('/full/path/to/TroyData.db')
You need to loop over the cursor to see results:
curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
for row in curs:
print row[0]
or call fetchone():
print curs.fetchone() # prints whole row tuple

The problem is the SQL statment. you must specify the db name and after the table name...
'''SELECT * FROM db_name.table_name WHERE acctno = ? '''

Postgresql table update

I updated a database table using postgresql from python
My code was
import psycopg2
connection=psycopg2.connect("dbname=homedb user=ria")
cursor=connection.cursor()
l_dict= {'licence_id':1}
cursor.execute("SELECT * FROM im_entry.usr_table")
rows=cursor.fetchall()
for row in rows:
i=i+1
p = findmax(row)
#print p
idn="id"
idn=idn+str(i)
cursor.execute("UPDATE im_entry.pr_table SET (selected_entry) = ('"+p+"') WHERE image_1d ='"+idn+"'")
print 'DATABASE TO PRINT'
cursor.execute("SELECT * FROM im_entry.pr_table")
rows=cursor.fetchall()
for row in rows:
print row
I got the updated table displayed
But when i display updated table by psql as
homedb=# SELECT * FROM im_entry.pr_table;
i got an empty table displayed..what is wrong?? please help me

You're probably not committing the transaction, i.e. you need a connection.commit() after all your updates.
There are various different settings you can make to the isolation level, e.g. autocommit, so you don't need to issue commits yourself. See, for example, How do I do database transactions with psycopg2/python db api?