I am trying to get a Oracle SQL database into python so I can aggregate/analyze the data. Pandas would be really useful for this task. But anytime I try to use my code, it just hangs and does not output anything. I am not sure its because I am using the cx oracle package and then using the pandas package?
import cx_Oracle as cxo
import pandas as pd
dsn=cxo.makedsn(
'host.net',
'1111',
service_name='servicename'
)
conn=cxo.connect(
user='Username',
password='password',
dsn=dsn)
c=conn.cursor()
a=c.execute("SELECT * FROM data WHERE date like '%20%'")
conn.close
df=pd.DataFrame(a)
head(df)
However when I use the code below, it prints out the data I am looking for. I need to convert this data into a panda data frame,
for row in c: print(row)
conn.close()
I am very new to python so any help will be really appreciated!!
To convert a cx_Oracle cursor to dataframe you can use de following code.
with conn.cursor() as cursor:
cursor.execute("SELECT * FROM data WHERE date like '%20%'")
from pandas import DataFrame
df = DataFrame(cursor.fetchall())
df.columns = [x[0] for x in cursor.description]
print("I got %d lines " % len(df))
Note I'm using the cursor as context manager. So it will be closed automatically on the end of the block.
Related
I am trying to create a dataframe using the data in redshift table.
But I am getting "Memory error" because the data I am fetching is huge in volume.
how to sove this issue, (I found chunking is one option. How to implement chucking) Is there any other library useful for such situations ?
The following is an example code
import pandas as pd
import psycopg2
conn = psycopg2.connect(host=host_name,user=usr,port=pt,password=pass,db_name=DB)
sql_query = "SELECT * FROM Table_Name"
df = pd.read_sql_query(conn,sql_query)
Hopefully someone can help me with this !! I am using cx_Oracle for oracle DB connection, I want to store a few SQL queries in excel. by running script in python, the sql can be imported from excel can be executed.
The sql1 has successfully import the sql1 but the value cannot pass to c.execute. How can I make it right? Adding """ will not help.
excel_data_df = pandas.read_excel('C:\\Python\Excel\sql.xlsx', sheet_name='SQL1')
caseno = excel_data_df['Case no']
sql1 = excel_data_df['SQL']
c = conn.cursor()
c.execute(sql1)*
Many Thanks for your help
I am converting a csv file into a Pandas dataframe and then converting it to Postgres table essentially.
The problem is that I am able to create a table in Postgres but I am unable to select column names from the table while querying it.
This is the sample code I have:
import pandas as pd
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql://postgres:pwd#localhost:5432/test')
def convertcsvtopostgres(csvfileloc, table_name, delimiter):
data = pd.read_csv(csvfileloc, sep=delimiter, encoding='latin-1')
data.head()
data1 = data.rename(columns=lambda x: x.strip())
data1.to_sql(table_name, engine, index=False)
convertcsvtopostgres("Product.csv","t_product","~")
I can do a select * from test.t_product; but I am unable to do a select product_id from test.t_product;
I am not sure if that is happening because of the encoding of the file and the conversion because of that. Is there any way around this, since I do not want to specify the table structure each time.
I have used Python to parse a txt file for specific information (dates, $ amounts, lbs, etc) and now I want to export that data to an Oracle table that I made in SQL Developer.
I have successfully connected Python to Oracle with the cx_Oracle module, but I am struggling to export or even print any data to my database from Python.
I am not proficient at using SQL, I know of simple queries and that's about it. I have explored the Oracle docs and haven't found straightforward export commands. When exporting data to an Oracle table via Python is it Python code I am going to be using or SQL code? Is it the same as importing a CSV file, for example?
I would like to understand how to write to an Oracle table from Python; I need to parse and export a very large amount of data so this won't be a one time export/import. I would also ideally like to have a way to preview my import to ensure it aligns correctly with my already created Oracle table, or if a simple undo action exists that would suffice.
If my problem is unclear I am more than happy to clarify it. Thanks for all help.
My code so far:
import cx_Oracle
dsnStr = cx_Oracle.makedsn("sole.wh.whoi.edu", "1526", "sole")
con = cx_Oracle.connect(user="myusername", password="mypassword", dsn=dsnStr)
print (con.version)
#imp 'Book1.csv' [this didn't work]
cursor = con.cursor()
print (cursor)
con.close()
From Import a CSV file into Oracle using CX_Oracle & Python 2.7 you can see overall plan.
So if you already parsed data into csv you can easily do it like:
import cx_Oracle
import csv
dsnStr = cx_Oracle.makedsn("sole.wh.whoi.edu", "1526", "sole")
con = cx_Oracle.connect(user="myusername", password="mypassword", dsn=dsnStr)
print (con.version)
#imp 'Book1.csv' [this didn't work]
cursor = con.cursor()
print (cursor)
text_sql = '''
INSERT INTO tablename (firstfield, secondfield) VALUES(:1,:2)
'''
my_file = 'C:\CSVData\Book1.csv'
cr = csv.reader(open(my_file,"rb"))
for row in cr:
print row
cursor.execute(text_sql, row)
print 'Imported'
con.close()
I am noob in python but I need to export MySQL table into .xls file using xlwt in python. I succeeded in exporting the table using example from here
http://ryrobes.com/featured-articles/using-xlwt-and-python-to-export-an-oracle-dataset-to-excel-python-simple-etl-part-2/
but the order of table column in excel and MySQL does not match if there are more than two columns in MySQL table.
Here's a part of the code:
from xlwt import *
import sys
import MySQLdb
table_name='student'
sql_select="SELECT * FROM %s"%table_name
conn1 =MySQLdb.connect(host='localhost',user='root',passwd='',db='test')
cu_select=conn1.cursor(MySQLdb.cursors.DictCursor)
try:
cu_select.execute(sql_select)
except MySQLdb.Error, e:
errInsertSql = "Insert Sql ERROR!! sql is==>%s" %(sql_select)
sys.exit(errInsertSql)
result_set = cu_select.fetchall()'
I tried printing result_set and found that mismatch starts from here. Can anyone help me.
Tables are organized alphabetically or by by ascending order ,if you want organised order use rows instead .