Hi I am new to python (programming and stackoverflow). First let me start by saying a little bit about what I am doing and trying to do.
I am using an internal XML API to pull data from an internal database
I parse/format xml result into a txt document (automated this to occur at a
set interval)
I want to write or import the contents of this document to an oracle database
How would i go about importing or writing this document into an existing oracle database? I can't seem to find much in the way of documentation with regards to the cx_Oracle module that i am using to establish a connection with my database. Could any of you kind folk point me in a direction / resource to accomplish this?
SHORT ANSWER:
query = """
insert into TABLE(FIELD1, FIELD2, ...) values (VAL1, VAL2, ...)
"""
cur = con.cursor()
cur.execute(query)
cur.commit()
I strongly suggest you to use prepared statements.
LONG ANSWER:
This is for Linux, in particular for Red Hat. Only the Python code at the end can be used on every OS. Try to adapt these steps to your OS.
0: Install the packages libaio and python-dev (or python-devel, check your distro)
1: If you don't have pip, install it
2: Install oracle instantclient-basic and instantclient-sdk (or instantclient-devel) from Oracle site
3: Launch these commands using bash. If you don't have /etc/profile.d/, check your distro.
echo 'ORACLE_HOME="/usr/lib/oracle/12.1/client64"' | \
sudo tee /etc/profile.d/cx_oracle.sh
pip install cx_Oracle
4: Logout and login again
5: Before using cx_Oracle, you have to set LD_LIBRARY_PATH I recommend you to NOT set it globally:
export LD_LIBRARY_PATH="$ORACLE_HOME/lib"
6: Finally, the Python code:
import cx_Oracle
os.environ["NLS_LANG"] = "AMERICAN_AMERICA.UTF8"
con_str = "USERNAME/PASSWORD#HOST:PORT/DBNAME"
con = cx_Oracle.connect(con_str)
query = """
select 1 from dual
"""
cur = con.cursor()
cur.execute(query)
rows = cur.fetchall()
for row in rows:
print(row) # it should print "1"
con.close()
You have to change the con_str with your username, password etc. The line that sets utf-8 encoding is optional and adaptable to your needs, but recommended.
7: If you want to insert rows:
query = """
insert into TABLE(FIELD1, FIELD2, ...) values (VAL1, VAL2, ...)
"""
cur = con.cursor()
cur.execute(query)
cur.commit()
I strongly suggest you to use prepared statements if you can't trust the source of the data.
Sources:
http://chilipuppy.blogspot.it/2008/10/purpose-im-working-on-building-python.html
http://agiletesting.blogspot.it/2005/05/installing-and-using-cxoracle-on-unix.html
http://cx-oracle.readthedocs.org/en/latest/index.html
Personal hassle
Your question is basically "how do I get started with cx_Oracle?"
There's some snippets here:
http://markharrison.net/cx-oracle-demos
and your simplest cx_Oracle program is something like this:
import cx_Oracle
conn = cx_Oracle.connect('scott/tiger')
curs = conn.cursor()
curs.execute('select 2+2 from dual')
print curs.fetchall()
curs.execute('insert into mytable(x) values(3)')
conn.commit()
curs.execute('select * from mytable')
for row in curs:
print row
conn.close()
Related
Hopefully someone can help me with this !! I am using cx_Oracle for oracle DB connection, I want to store a few SQL queries in excel. by running script in python, the sql can be imported from excel can be executed.
The sql1 has successfully import the sql1 but the value cannot pass to c.execute. How can I make it right? Adding """ will not help.
excel_data_df = pandas.read_excel('C:\\Python\Excel\sql.xlsx', sheet_name='SQL1')
caseno = excel_data_df['Case no']
sql1 = excel_data_df['SQL']
c = conn.cursor()
c.execute(sql1)*
Many Thanks for your help
I am trying to learn how to get Microsoft SQL query results using python and pyodbc module and have run into an issue in returning the same results using the same query that I use in Microsoft SQL Management Studio.
I've looked at the pyodbc documentation and set up my connection correctly... at least I'm not getting any connection errors at execution. The only issue seems to be returning the table data
import pyodbc
import sys
import csv
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=<server>;DATABASE=<db>;UID=<uid>;PWD=<PWD>')
cursor = cnxn.cursor()
cursor.execute("""
SELECT request_id
From audit_request request
where request.reception_datetime between '2019-08-18' and '2019-08-19' """)
rows = cursor.fetchall()
for row in cursor:
print(row.request_id)
When I run the above code i get this in the python terminal window:
Process returned 0 (0x0) execution time : 0.331 s
Press any key to continue . . .
I tried this same query in SQL Management Studio and it returns the results I am looking for. There must be something I'm missing as far as displaying the results using python.
You're not actually setting your cursor up to be used. You should have something like this before executing:
cursor = cnxn.cursor()
Learn more here: https://github.com/mkleehammer/pyodbc/wiki/Connection#cursor
I have used Python to parse a txt file for specific information (dates, $ amounts, lbs, etc) and now I want to export that data to an Oracle table that I made in SQL Developer.
I have successfully connected Python to Oracle with the cx_Oracle module, but I am struggling to export or even print any data to my database from Python.
I am not proficient at using SQL, I know of simple queries and that's about it. I have explored the Oracle docs and haven't found straightforward export commands. When exporting data to an Oracle table via Python is it Python code I am going to be using or SQL code? Is it the same as importing a CSV file, for example?
I would like to understand how to write to an Oracle table from Python; I need to parse and export a very large amount of data so this won't be a one time export/import. I would also ideally like to have a way to preview my import to ensure it aligns correctly with my already created Oracle table, or if a simple undo action exists that would suffice.
If my problem is unclear I am more than happy to clarify it. Thanks for all help.
My code so far:
import cx_Oracle
dsnStr = cx_Oracle.makedsn("sole.wh.whoi.edu", "1526", "sole")
con = cx_Oracle.connect(user="myusername", password="mypassword", dsn=dsnStr)
print (con.version)
#imp 'Book1.csv' [this didn't work]
cursor = con.cursor()
print (cursor)
con.close()
From Import a CSV file into Oracle using CX_Oracle & Python 2.7 you can see overall plan.
So if you already parsed data into csv you can easily do it like:
import cx_Oracle
import csv
dsnStr = cx_Oracle.makedsn("sole.wh.whoi.edu", "1526", "sole")
con = cx_Oracle.connect(user="myusername", password="mypassword", dsn=dsnStr)
print (con.version)
#imp 'Book1.csv' [this didn't work]
cursor = con.cursor()
print (cursor)
text_sql = '''
INSERT INTO tablename (firstfield, secondfield) VALUES(:1,:2)
'''
my_file = 'C:\CSVData\Book1.csv'
cr = csv.reader(open(my_file,"rb"))
for row in cr:
print row
cursor.execute(text_sql, row)
print 'Imported'
con.close()
I have a query that returns over 125K rows.
The goal is to write a script the iterates through the rows, and for each, populate a second table with data processed from the result of the query.
To develop the script, I created a duplicate database with a small subset of the data (4126 rows)
On the small database, the following code works:
import os
import sys
import random
import mysql.connector
cnx = mysql.connector.connect(user='dbuser', password='thePassword',
host='127.0.0.1',
database='db')
cnx_out = mysql.connector.connect(user='dbuser', password='thePassword',
host='127.0.0.1',
database='db')
ins_curs = cnx_out.cursor()
curs = cnx.cursor(dictionary=True)
#curs = cnx.cursor(dictionary=True,buffered=True) #fail
with open('sql\\getRawData.sql') as fh:
sql = fh.read()
curs.execute(sql, params=None, multi=False)
result = curs.fetchall() #<=== script stops at this point
print len(result) #<=== this line never executes
print curs.column_names
curs.close()
cnx.close()
cnx_out.close()
sys.exit()
The line curs.execute(sql, params=None, multi=False) succeeds on both the large and small databases.
If I use curs.fetchone() in a loop, I can read all records.
If I alter the line:
curs = cnx.cursor(dictionary=True)
to read:
curs = cnx.cursor(dictionary=True,buffered=True)
The script hangs at curs.execute(sql, params=None, multi=False).
I can find no documentation on any limits to fetchall(), nor can I find any way to increase the buffer size, and no way to tell how large a buffer I even need.
There are no exceptions raised.
How can I resolve this?
I was having this same issue, first on a query that returned ~70k rows and then on one that only returned around 2k rows (and for me RAM was also not the limiting factor). I switched from using mysql.connector (i.e. the mysql-connector-python package) to MySQLdb (i.e. the mysql-python package) and then was able to fetchall() on large queries with no problem. Both packages seem to follow the python DB API, so for me MySQLdb was a drop-in replacement for mysql.connector, with no code changes necessary beyond the line that sets up the connection. YMMV if you're leveraging something specific about mysql.connector.
Pragmatically speaking, if you don't have a specific reason to be using mysql.connector the solution to this is just to switch to a package that works better!
I'm new to python. I'm trying to query a MSSQL database.
import pymssql
conn = pymssql.connect(host='hostname', user='username', password='password', database='dbname')
cursor = conn.cursor()
sql = "select count(*) from T_Email with (nolock) where Transmit is null"
cursor.execute(sql)
results = cursor.fetchall()
for row in results:
print (row)
The query successfully runs is Microsoft SQL Server Management Studio, but my python script always returns nothing.
I verified I have network connectivity. I verified the username, password and database name. If I change the password, then the script will give an error.
I have tried results = cursor.fetchone(), but that didn't help.
Any suggestions?
If you're using Ubuntu you may have used (like me) apt-get to install pymssql package.
This is a known bug of the bundled version: https://bugs.launchpad.net/ubuntu/+source/pymssql/+bug/918896
Try to install it manually using easy_install.
I had the same issue on Ubuntu 12.04, indeed the fix is doing the sequence:
$ apt-get purge python-pymssql
$ apt-get install freetds-dev
$ pip install Cython
$ pip install pymssql
Try adding a conn.commit() to your query
Without sufficient information to reproduce the example, it's hard to say the specific problem you're having. However, here are a few guesses I have as for possible problems:
Maybe your actual column name (presuming your example above was just a mock up) is too long. See: http://code.google.com/p/pymssql/wiki/FAQ (look for Column names get silently truncated to 30 characters. (1.x only) ) This is a common one to trip folks up because it SILENTLY fails!!
If you are creating tables before querying into them, or other things that require commits, it can mess things up, even with autocommit turned on ( autocommit(1) ). See my answer to myself :) at pymssql ( python module ) unable to use temporary tables
Good luck!
Mike
import pymssql
conn = pymssql.connect(
server="server",
port=port,
user="user",
password=password,
database="database")
conn
cursor = conn.cursor()
cursor.execute("select count(*) from T_Email with (nolock) where Transmit is null")
for row in cursor.fetchall():
print ("%s\n" % str(row))
conn.close()
Change the following lines in your code snippet
results = cursor.fetchall()
for row in results:
print (row)
into
# results = cursor.fetchall()
for row in cursor:
print (row)
pymssql has bug in cursor.fetchall()
For reference https://github.com/pymssql/pymssql/issues/141