We Keep Coding

How to replace a column name header- Python/Airflow/Redshift/PostgresSQL

I am running a SQL statement and fetching the results(column/header) and I am trying to collect only the headers that have _check at the end of it and then do a .replace command to get a list of the newly named columns without the _check at the end.
Here is my code:
pg_hook = PostgresHook(postgre_conn_id="postgres_default", schema='schema1')
connection = pg_hook.get_conn()
col_query = "select * from schema.table"
cursor = connection.cursor()
cursor.execute(col_query)
#fetchall to dictonary
desc = cursor.description
column_names = [col[0] for col in desc]
data = [dict(zip(column_names, row)) for row in cursor.fetchall()]
for x in column_names:
if x='updated_check':
x.replace('_check','')
connection.commit()
connection.close()
Any ideas or suggestions on how to replace the name of the columns that have _check in them and them put them inside a list? Any help would be appreciated. I am using airflow, python, mysqldb, and psycopg2

Inserting csv into MySQL database with python library mysql.connector

I have trouble with insert of csv data into MySQL tabel with mysql.connector .
The code I use looks like this :
import mysql.connector
import csv
andreport = 'testowa.csv'
cnx = mysql.connector.connect(
user='xxxxx',
password='xxxxx',
host='xxxxxx',
database='xxxxx')
cursor = cnx.cursor()
with open(andreport, 'r') as csv_data:
for row in csv_data:
cursor.execute(
"INSERT INTO flex(date, Store, Vendor, Shelf)"
"VALUES({},{},{},{})", row)
cnx.commit()
cursor.close()
cnx.close()
print("Done")
The error I get :
C:\Users\Iw4n\PycharmProjects\Learning\venv\Scripts\python.exe C:/Users/Iw4n/PycharmProjects/Learning/Orange_android_MySQL_insertion.py
Traceback (most recent call last):
File "C:/Users/Iw4n/PycharmProjects/Learning/Orange_android_MySQL_insertion.py", line 15, in <module>
cursor.execute(
File "C:\Users\Iw4n\PycharmProjects\Learning\venv\lib\site-packages\mysql\connector\cursor.py", line 551, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\Iw4n\PycharmProjects\Learning\venv\lib\site-packages\mysql\connector\connection.py", line 490, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\Iw4n\PycharmProjects\Learning\venv\lib\site-packages\mysql\connector\connection.py", line 395, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '},{},{},{})' at line 1
When i wrapped {} into '' , as many rows as were in csv been inserted into datbase as {},{}
same story goes for %s if I use it , i got the same error as above, when it's wrapped in '' , %s is insetred into database.
I also found information to add f in fron of "INSERT~ but it did not help.
Can anyone give me some suggestion on how to overcome this and correctly insert data to MySQL ?
Final code that is working as intended :
import mysql.connector
import csv
andreport = 'testowa.csv'
cnx = mysql.connector.connect(
user='xxxxx',
password='xxxxx',
host='xxxxx',
database='xxxxx')
cursor = cnx.cursor()
with open(andreport, mode='r') as csv_data:
reader = csv.reader(csv_data, delimiter=';')
csv_data_list = list(reader)
for row in csv_data_list:
cursor.execute("""
INSERT INTO flex(
date, Agency, MediaSource, Campaign)
VALUES(%s,%s,%s,%s)""",
(row[0], row[1], row[2], row[3]))
cnx.commit()
cursor.close()
cnx.close()
print("Done")

I'm guessing that seems the problem is that you passed one argument (row) instead of four. So try this:
cursor.execute("""
INSERT INTO flex(date, Store, Vendor, Shelf)
VALUES(%s,%s,%s,%s)""",(row[0], row[1], row[2], row[3], ))

Looking at the documentation for MySQLCursor.excute() method, it seems like adding some %s as the parameters in your insert statement might fix this?
import mysql.connector
import csv
andreport = 'testowa.csv'
cnx = mysql.connector.connect(
user='xxxxx',
password='xxxxx',
host='xxxxxx',
database='xxxxx')
cursor = cnx.cursor()
insert_statement = (
"INSERT INTO flex(date, Store, Vendor, Shelf)"
"VALUES (%s, %s, %s, %s)"
)
with open(andreport, mode='r') as csv_data:
reader = csv.reader(csv_data, delimiter=';')
csv_data_list = list(reader)
for row in csv_data_list:
cursor.execute(insert_statement, row)
cnx.commit()
cursor.close()
cnx.close()
print("Done")
Let me know if this gets you anywhere, or if you see a new error!
Edit: updated CSV reading to convert to a list.

mariaDB: column count doesn't match value count at row 1

I don't get what's the problem here. I want to build a web scraper that scrapes amazon and takes the price and the name into a database. But for some reason, it tells me that the columns and values are not matching. I do have one additional column in my database called "timestamp" where I automatically put in the time, but that is handled by the database. I am using MariaDB. A friend said I can use the MySQL API for MariaDB as well.
P.S. preis = price, coming from Germany, switching between English and German sometimes, just in case anyone is wondering.
import requests, time, csv, pymysql
from bs4 import BeautifulSoup as bs
#URL = input("URL")
URL = "https://www.amazon.de/gp/product/B075FTXF15/ref=crt_ewc_img_bw_3?ie=UTF8&psc=1&smid=A24FLB4J0NZBNT"
def SOUPIT (tempURL):
URL = tempURL
page = requests.get(URL,headers={"User-Agent":"Defined"})
soup = bs(page.content, "html.parser")
raw_price = soup.find(id="priceblock_ourprice").get_text()
price = raw_price[:-2]
raw_name = soup.find(id="productTitle").get_text()
name = raw_name.strip()
for i in range(0,len(name)-1):
if name[i] == "(":
name = name[:i]
break
data = [name, price, time.strftime("%H:%M:%S"), time.strftime("%d.%m.%Y")]
return(data)
data = SOUPIT(URL)
while True:
data = SOUPIT(URL)
db = pymysql.connect("localhost", "root", "root", "test")
cursor = db.cursor()
if (data == None):
break
print("break")
else:
name = data[0]
preis = data[1]
sql = """INSERT INTO amazon_preise (Name, Preis) VALUES ('{}',{})""".format(name,preis)
cursor.execute(sql)
db.commit()
print("success")
print(data)
time.sleep(60)
error message:
Traceback (most recent call last):
File "amazonscraper_advanced.py", line 43, in <module>
cursor.execute(sql)
File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\pymysql\cursors.py", line 170, in execute
result = self._query(query)
File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\pymysql\cursors.py", line 328, in _query
conn.query(q)
File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\pymysql\connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\pymysql\connections.py", line 732, in _read_query_result
result.read()
File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\pymysql\connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\pymysql\connections.py", line 684, in _read_packet
packet.check_error()
File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\pymysql\protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "C:\Users\...\AppData\Local\Programs\Python\Python36\lib\site-packages\pymysql\err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1136, "Column count doesn't match value count at row 1")

The problem is caused, at least partially, by a using string formatting to insert values into an SQL statement.
Here is the scraped data:
>>> data = ['Sweatshirt Alien VS. Predator Z100088', '32,99', '14:08:43', '08.09.2019']
>>> name, preis, *_ = data
Let's create the SQL statement
>>> sql = """INSERT INTO amazon_preise (Name, Preis) VALUES ('{}',{})""".format(name,preis)
And display it:
>>> sql
"INSERT INTO amazon_preise (Name, Preis) VALUES ('Sweatshirt Alien VS. Predator Z100088',32,99)"
Observe that the VALUES clause contains three comma-separated values; this is because the web page displays currency in the German style, that is with commas separating the cents from the euros. When interpolated into the SQL statement
preis becomes two values instead of one.
The right way to fix this would be to convert preis from a string to a float or decimal, and use parameter substitution instead of string formatting to interpolate the values..
>>> fpreis = float(preis.replace(',', '.'))
>>> sql = """INSERT INTO amazon_preise (Name, Preis) VALUES (%s, %s)"""
>>> cursor.execute(sql, (name, fpreis))

Store Mysql coulmn names in array using Python mysql connector

I'm quite new to mysql as in manipulating the database itself. I succeeded to store new lines in a table but my next endeavor will be a little more complex.
I'd like to fetch the column names from an existing mysql database and save them to an array in python. I'm using the official mysql connector.
I'm thinking I can achieve this through the information_schema.columns command but I have no idea how to build the query and store the information in an array. It will be around 100-200 columns so performance might become an issue so I don't think its wise just to iterate my way through it for each column.
The base code to inject code into mysql using the connector is:
def insert(data):
query = "INSERT INTO templog(data) " \
"VALUES(%s,%s,%s,%s,%s)"
args = (data)
try:
db_config = read_db_config()
conn = MySQLConnection(db_config)
cursor = conn.cursor()
cursor.execute(query, args)
#if cursor.lastrowid:
# print('last insert id', cursor.lastrowid)
#else:
# print('last insert id not found')
conn.commit()
cursor.close()
conn.close()
except Error as error:
print(error)
As said this above code needs to be modified in order to get data from the sql server. Thanks in advance!

Thanks for the help!
Got this as working code:
def GetNames(web_data, counter):
#get all names from the database
connection = create_engine('mysql+pymysql://user:pwd#server:3306/db').connect()
result = connection.execute('select * from price_usd')
a = 0
sql_matrix = [0 for x in range(counter + 1)]
for v in result:
while a == 0:
for column, value in v.items():
a = a + 1
if a > 1:
sql_matrix[a] = str(('{0}'.format(column)))
This will get all column names from the existing sql database

MySQLdb Results

I want to be able to get all the information from MySQLdb using the SELECT * FROM query. I have the following code:
database = MySQLdb.connect("127.0.0.1", "root", "pswd", "Kazzah")
cursor = database.cursor()
cursor.execute("SELECT * FROM Accounts WHERE Email=%s AND Password=%s", (_Email, _Password))
database.commit()
numrows = cursor.rowcount
results = cursor.fetchall()
print numrows
for result in results:
print result
How can I make variables that hold each piece of info from result. If it result returns:
(28L, 'Name', 'Last', 'email#email.com', 'pswd', '10000')
I want to make a variable called ID and get the first part of the result which is 28L, and so forth with each other pieces of information.
Thank you!

Edit
To set the full set of data use this :
id, fname, lname, email, pswd, whatever = result
And for particular values, try indexing.