I am running SQL query from python API and want to collect data in Structured(column-wise data under their own header).CSV format.
This is the code so far I have.
import pymysql.cursors
import csv
conn = pymysql.connect(host='159.XXX.XXX.XXX',user='proXXX',password='PXX',db='pXX',charset='utf8mb4',cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
print (type(conn))
sql = "SELECT id,author From researches WHERE id < 20 "
cursor.execute(sql)
data = cursor.fetchall()
print (data)
with open('metadata.csv', 'w', newline='') as f_handle:
writer = csv.writer(f_handle,delimiter=',')
header = ['id', 'author']
writer.writerow(header)
for row in data:
writer.writerow(row)
Now the data is being printed on the console but not getting in.CSV file this is what I am getting asnoutput. What is that I am missing? Please help.
with open('metadata.csv', 'w', newline='') as f_handle:
fieldnames = ['id', 'author']
writer = csv.DictWriter(f_handle, fieldnames=fieldnames)
writer.writeheader()
for row in data:
writer.writerow(row)
So the thing is, your data is in the form of dictionaries, while the Writer object expects tuples. You should be using the DictWriter object instead.
Related
I am making a program that fetches column names and dumps the data into csv format.
Now everything is working just fine and data is being dumped into csv, the problem is,
I am not able to fetch headers into csv. If I open the exported csv file into excel, only data shows up not the column headers. How do I do that?
Here's my code:
import cx_Oracle
import csv
dsn_tns = cx_Oracle.makedsn(--Details--)
conn = cx_Oracle.connect(--Details--)
d = conn.cursor()
csv_file = open("profile.csv", "w")
writer = csv.writer(csv_file, delimiter=',', lineterminator="\n", quoting=csv.QUOTE_NONNUMERIC)
d.execute("""
select * from all_tab_columns where OWNER = 'ABBAS'
""")
tables_tu = d.fetchall()
for row in tables_tu:
writer.writerow(row)
conn.close()
csv_file.close()
What code do I use to export headers too in csv?
Place this just above your for loop:
writer.writerow(i[0] for i in d.description)
Because d.description is a read-only attribute containing 7-tuples that look like:
(name,
type_code,
display_size,
internal_size,
precision,
scale,
null_ok)
How can I export my scraped data into a CSV file. My code below prints out all of the data correctly, but I would like to export it the same way to a CSV file, line by line.
How can I write the data to a csv file?
import requests
import json
import csv
with open('Links.csv', 'r') as csvfile:
readCSV = csv.reader(csvfile, delimiter=',', quotechar='"')
for row in readCSV:
data = row[0]
for b in row:
r = requests.get(b)
json_object = json.loads('{"data":%s}}' % (r.content.decode("utf-8").replace("jQuery111002521088376353553_1491736907010(", "")[:-2].replace("\'", "")))
for game in json_object["data"]["docs"]:
print ("Name: %s, Price: %s, CatalogId: %s, slug: %s" % (game["name"], game["minPrice"], game["catalogId"], game["slug"]))
You can write the data to a row in a csv file like:
writeCSV.writerow([game["name"], game["minPrice"], game["catalogId"], game["slug"]])
Here is this added to your code, with the init code needed:
import csv
import json
import requests
with open('Links.csv', 'r') as r_csvfile, open('outp.csv', 'w') as w_csvfile:
readCSV = csv.reader(r_csvfile, delimiter=',', quotechar='"')
writeCSV = csv.writer(w_csvfile, delimiter=',', quotechar='"')
writeCSV.writerow("Name Price CatalogId slug".split())
for row in readCSV:
data = row[0]
for b in row:
r = requests.get(b)
json_object = json.loads('{"data":%s}}' % (
r.content.decode("utf-8").replace(
"jQuery111002521088376353553_1491736907010(", "")[:-2]
.replace("\'", "")))
for game in json_object["data"]["docs"]:
writeCSV.writerow([game["name"], game["minPrice"],
game["catalogId"], game["slug"]])
You did not give any actual data, so I could not test this, but it will be close.
I think pandas is the package you‘re looking for.
Use pandas.dataframe.from_dict or pandas.dataframe.from_json. Once you have your pandas dataframe writing a csv file is as easy as it gets.
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html
I am running SQL query from python API and want to collect data in Structured(column-wise data under their header).CSV format.
This is the code so far I have.
sql = "SELECT id,author From researches WHERE id < 20 "
cursor.execute(sql)
data = cursor.fetchall()
print (data)
with open('metadata.csv', 'w', newline='') as f_handle:
writer = csv.writer(f_handle)
header = ['id', 'author']
writer.writerow(header)
for row in data:
writer.writerow(row)
Now the data is being printed on the console but not getting in .CSV file this is what I am getting as output:
What is that I am missing?
Here is a simple example of what you are trying to do:
import sqlite3 as db
import csv
# Run your query, the result is stored as `data`
with db.connect('vehicles.db') as conn:
cur = conn.cursor()
sql = "SELECT make, style, color, plate FROM vehicle_vehicle"
cur.execute(sql)
data = cur.fetchall()
# Create the csv file
with open('vehicle.csv', 'w', newline='') as f_handle:
writer = csv.writer(f_handle)
# Add the header/column names
header = ['make', 'style', 'color', 'plate']
writer.writerow(header)
# Iterate over `data` and write to the csv file
for row in data:
writer.writerow(row)
import pandas as pd
import numpy as np
from sqlalchemy import create_engine
from urllib.parse import quote_plus
params = quote_plus(r'Driver={SQL Server};Server=server_name; Database=DB_name;Trusted_Connection=yes;')
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
sql_string = '''SELECT id,author From researches WHERE id < 20 '''
final_data_fetch = pd.read_sql_query(sql_string, engine)
final_data_fetch.to_csv('file_name.csv')
Hope this helps!
with mysql - export csv with mysqlclient library - utf8
import csv
import MySQLdb as mariadb;
import sys
tablelue="extracted_table"
try:
conn = mariadb.connect(
host="127.0.0.1",
port=3306,
user="me",
password="mypasswd",
database="mydb")
cur = conn.cursor()
instruction="show columns from " + tablelue
cur.execute(instruction)
myresult = cur.fetchall()
work=list()
i=0
for x in myresult:
work.append(x[0])
i=i+1
wsql = "SELECT * FROM " + tablelue
cur.execute(wsql)
wdata = cur.fetchall()
# Create the csv file
fichecrit=tablelue+".csv"
with open(fichecrit, 'w', newline='', encoding="utf8") as f_handle:
writer = csv.writer(f_handle,delimiter=";")
# Add the header/column names
header = work
writer.writerow(header)
# Iterate over `data` and write to the csv file
for row in wdata:
writer.writerow(row)
conn.close()
except Exception as e:
print(f"Error: {e}")
sys.exit(0)
You can dump all results to the csv file without looping:
data = cursor.fetchall()
...
writer.writerows(data)
I am new to Python and request your kind assistance. I have five tables in the MS Access database and I need to compile a CSV file for each of the tables. One of the tables is Perm_Reviews, which is part of the snippet. Fortunately, I am able to query the MS Access data and it returns rows and the columns associated from the database. Can someone please provide assistance on how to store the tables as CSV files.
import pyodbc
import csv
conn_string = ("DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=T:\\DataDump\\7.18.2016 PCR etrakit.accdb")
save_csv = 'C:\Desktop\CSVFiles'
conn = pyodbc.connect(conn_string)
cursor = conn.cursor()
SQL = 'select * from Perm_Reviews;'
for row in cursor.execute(SQL):
print row
cursor.close()
conn.close()
print 'All done for now'
I think this is what you are looking for.
import pyodbc
import csv
conn_string = ("DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=T:\\DataDump\\7.18.2016 PCR etrakit.accdb")
conn = pyodbc.connect(conn_string)
cursor = conn.cursor()
cursor.execute('select * from Perm_Reviews;')
with open('Perms_Review.csv','w') as f:
writer = csv.writer(f)
writer.writerows([i[0] for i in cursor.description])
writer.writerows(cursor)
cursor.close()
conn.close()
Python has a built-in csv module that you can use readily, below a simple example on csv with headers:
import csv
with open('names.csv', 'w') as csvfile:
fieldnames = ['first_name', 'last_name']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
writer.writerow({'first_name': 'Baked', 'last_name': 'Beans'})
writer.writerow({'first_name': 'Lovely', 'last_name': 'Spam'})
writer.writerow({'first_name': 'Wonderful', 'last_name': 'Spam'})
I am trying to connect to oracle table and execute a sql. I need to export result set to a csv file. My code is below:
import pyodbc
import csv
cnxn = pyodbc.connect("DSN=11g;UID=test101;PWD=passwd")
cursor = cnxn.cursor()
cursor.execute(sql)
row = cursor.fetchall()
with open('data.csv', 'w', newline='') as fp:
a = csv.writer(fp, delimiter=',')
for line in row:
a.writerows(line)
cursor.close()
when I do print to line within for loop, I get something like this:
('Production', 'farm1', 'dc1prb01', 'web')
('Production', 'farv2', 'dc2pr2db01', 'app.3')
('Production', 'farm5', 'dc2pr2db02', 'db.3')
this is not working. Any ideas what I might be missing?
It would be writerow for a single row:
a.writerow(line)
writerows expects an iterable of iterables, so it will iterate over the substrings writing each char individually.
If you want to use writerows call it on row:
row = cursor.fetchall()
with open('data.csv', 'w', newline='') as fp:
a = csv.writer(fp, delimiter=',')
a.writerows(row)
If you are using python2 remove newline='', newline is a *python*3 keyword:
row = cursor.fetchall()
with open('data.csv', 'w') as fp:
a = csv.writer(fp, delimiter=',')
a.writerows(row)