We Keep Coding

How to gzip query results-Python/Airflow

I am trying to GZIP my query results and write it to a location in Airflow. However I get the error of
TypeError: memoryview: a bytes-like object is required, not 'str'
whenever I run my code.
Check out the fp variable in my code:
def create_tunnel_postgres():
try:
tunnel = SSHTunnelForwarder((ssh_host, 22),
ssh_username=ssh_username,
ssh_private_key=pkf,
remote_bind_address=(psql_host,
5432))
# local_bind_address=('localhost',6543) # could be any available port
# Start the tunnel
tunnel.start()
except:
print 'connection'
else:
conn = psycopg2.connect(database='my_db', user='user',
password='my_pwd',
host=tunnel.local_bind_host,
port=tunnel.local_bind_port)
cur = conn.cursor()
cur.execute("""
select * from pricing.public.seller_tiers ;
""")
result = cur.fetchall()
# Getting Field Header names
column_names = [i[0] for i in cur.description]
fp = gzip.open(path, 'wb')
myFile = csv.writer(fp, delimiter=',')
myFile.writerow(column_names)
myFile.writerows(result)
fp.close()
conn.close
tunnel.stop
Any ideas or suggestions? I am new to python/airflow so anything would help.

I think the error would be with respect to the way you are using gzip writer file content.
You are trying to open the gzip file in byte mode and writing string to it, in here fp = gzip.open(path, 'wb').
As per python documentation in here it's stated:
'rt', 'at', 'wt', or 'xt' for text mode.
Change your code to use wt which is write text or use encode feature to encode to bytes:
import gzip
import csv
with gzip.open("sample.gz", "wt") as gz_fp:
fieldnames = ['first_name', 'last_name']
writer = csv.writer(gz_fp, delimiter=",")
writer.writerow({'first_name': 'Baked', 'last_name': 'Beans'})
writer.writerow({'first_name': 'Lovely', 'last_name': 'Spam'})
writer.writerow({'first_name': 'Wonderful', 'last_name': 'Spam'})
If you want to write bytes only then:
with gzip.open('file.gz', 'wb') as f:
f.write('Hello world!'.encode())

Python MySQLdb - Export query to csv without line terminators

So basically im using MySQLdb query dialy images of my tables and i want to save them in .csv but one of the fields has line terminators (\n) and i cant figure out how to get rid of them so my csv doesnt break.
Here is the python im using:
results = cur.execute(sql)
db = MySQLdb.connect(host="",
user="",
passwd="",
db="" )
cur = db.cursor()
sql = """" big query here """
results = cur.execute(sql)
with open("out.csv", "wb") as csv_file:
csv_writer = csv.writer(csv_file)
csv_writer.writerow([i[0] for i in cur.description])
csv_writer.writerow(cur)
Is there a easy way to replace \n chars for just spaces?

Try this:
import csv
import sys
csv_writer = csv.writer(sys.stdout, lineterminator='\n')
Or:
with open("out.csv","wb",newline='') as csv_file:

If the newline is in appearing in the text of your column maybe something like this wouldwork.
csv_writer.writerow([i[0].replace('\n',' ') for i in cur.description])

SQL query output to .csv

I am running SQL query from python API and want to collect data in Structured(column-wise data under their header).CSV format.
This is the code so far I have.
sql = "SELECT id,author From researches WHERE id < 20 "
cursor.execute(sql)
data = cursor.fetchall()
print (data)
with open('metadata.csv', 'w', newline='') as f_handle:
writer = csv.writer(f_handle)
header = ['id', 'author']
writer.writerow(header)
for row in data:
writer.writerow(row)
Now the data is being printed on the console but not getting in .CSV file this is what I am getting as output:
What is that I am missing?

Here is a simple example of what you are trying to do:
import sqlite3 as db
import csv
# Run your query, the result is stored as `data`
with db.connect('vehicles.db') as conn:
cur = conn.cursor()
sql = "SELECT make, style, color, plate FROM vehicle_vehicle"
cur.execute(sql)
data = cur.fetchall()
# Create the csv file
with open('vehicle.csv', 'w', newline='') as f_handle:
writer = csv.writer(f_handle)
# Add the header/column names
header = ['make', 'style', 'color', 'plate']
writer.writerow(header)
# Iterate over `data` and write to the csv file
for row in data:
writer.writerow(row)

import pandas as pd
import numpy as np
from sqlalchemy import create_engine
from urllib.parse import quote_plus
params = quote_plus(r'Driver={SQL Server};Server=server_name; Database=DB_name;Trusted_Connection=yes;')
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
sql_string = '''SELECT id,author From researches WHERE id < 20 '''
final_data_fetch = pd.read_sql_query(sql_string, engine)
final_data_fetch.to_csv('file_name.csv')
Hope this helps!

with mysql - export csv with mysqlclient library - utf8
import csv
import MySQLdb as mariadb;
import sys
tablelue="extracted_table"
try:
conn = mariadb.connect(
host="127.0.0.1",
port=3306,
user="me",
password="mypasswd",
database="mydb")
cur = conn.cursor()
instruction="show columns from " + tablelue
cur.execute(instruction)
myresult = cur.fetchall()
work=list()
i=0
for x in myresult:
work.append(x[0])
i=i+1
wsql = "SELECT * FROM " + tablelue
cur.execute(wsql)
wdata = cur.fetchall()
# Create the csv file
fichecrit=tablelue+".csv"
with open(fichecrit, 'w', newline='', encoding="utf8") as f_handle:
writer = csv.writer(f_handle,delimiter=";")
# Add the header/column names
header = work
writer.writerow(header)
# Iterate over `data` and write to the csv file
for row in wdata:
writer.writerow(row)
conn.close()
except Exception as e:
print(f"Error: {e}")
sys.exit(0)

You can dump all results to the csv file without looping:
data = cursor.fetchall()
...
writer.writerows(data)

Write sql data to csv from python API

I am running SQL query from python API and want to collect data in Structured(column-wise data under their own header).CSV format.
This is the code so far I have.
import pymysql.cursors
import csv
conn = pymysql.connect(host='159.XXX.XXX.XXX',user='proXXX',password='PXX',db='pXX',charset='utf8mb4',cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
print (type(conn))
sql = "SELECT id,author From researches WHERE id < 20 "
cursor.execute(sql)
data = cursor.fetchall()
print (data)
with open('metadata.csv', 'w', newline='') as f_handle:
writer = csv.writer(f_handle,delimiter=',')
header = ['id', 'author']
writer.writerow(header)
for row in data:
writer.writerow(row)
Now the data is being printed on the console but not getting in.CSV file this is what I am getting asnoutput. What is that I am missing? Please help.

with open('metadata.csv', 'w', newline='') as f_handle:
fieldnames = ['id', 'author']
writer = csv.DictWriter(f_handle, fieldnames=fieldnames)
writer.writeheader()
for row in data:
writer.writerow(row)
So the thing is, your data is in the form of dictionaries, while the Writer object expects tuples. You should be using the DictWriter object instead.

Python csv from database query adding a custom column to csv file

here is what I try to achieve my current code is working fine I get the query to run on my sql server but I will need to gather information from several servers. How would I add a column with the dbserver listed in that column?
import pyodbc
import csv
f = open("dblist.ini")
dbserver,UID,PWD = [ variable[variable.find("=")+1 :] for variable in f.readline().split("~")]
connectstring = "DRIVER={SQL server};SERVER=" + dbserver + ";DATABASE=master;UID="+UID+";PWD="+PWD
cnxn = pyodbc.connect(connectstring)
cursor = cnxn.cursor()
fd = open('mssql1.txt', 'r')
sqlFile = fd.read()
fd.close()
cursor.execute(sqlFile)
with open("out.csv", "wb") as csv_file:
csv_writer = csv.writer(csv_file, delimiter = '!')
csv_writer.writerow([i[0] for i in cursor.description]) # write headers
csv_writer.writerows(cursor)

You could add the extra information in your sql query. For example:
select "dbServerName", * from table;
Your cursor will return with an extra column in front of your real data that has the db Server name. The downside to this method is you're transferring a little more extra data.