I am new to python. What I am trying to achieve is to insert values from my list/tuple into my redshift table without iteration.I have around 1 million rows and 1 column. Below is the code I am using to create my list/tuple.
cursor1.execute("select domain from url limit 5;")
for record, in cursor1:
ext = tldextract.extract(record)
mylist.append(ext.domain + '.' + ext.suffix)
mytuple = tuple(mylist)
I am not sure what is best to use, tuple or list. output of print(mylist) and print(mytuple) are as follows.
List output
['friv.com', 'steep.tv', 'wordpress.com', 'fineartblogger.net',
'v56.org'] Tuple Output('friv.com', 'steep.tv', 'wordpress.com',
'fineartblogger.net', 'v56.org')
Now, below is the code I am using to insert the values into my redshift table but I am getting an error:
cursor2.execute("INSERT INTO sample(domain) VALUES (%s)", mylist) or
cursor2.execute("INSERT INTO sample(domain) VALUES (%s)", mytuple)
Error - not all arguments converted during string formatting
Any help is appreciated. If any other detail is required please let me know, I will edit my question.
UPDATE 1:
Tried using below code and getting different error.
args_str = ','.join(cur.mogrify("(%s)", x) for x in mylist)
cur.execute("INSERT INTO table VALUES " + args_str)
ERROR - INSERT has more expressions than target columns
I think you're looking for Fast Execution helpers:
mylist=[('t1',), ('t2',)]
execute_values(cursor2, "INSERT INTO sample(domain) %s", mylist, page_size=100)
what this does is it replaces the %s with 100 VALUES. I'm not sure how high you can set page_size, but that should be far more performant.
Finally found a solution. For some reason cur.mogrify was not giving me proper sql string for insert. Created my own SQl string and it works alot faster than cur.executeall()
list_size = len(mylist)
for len in range(0,list_size):
if ( len != list_size-1 ):
sql = sql + ' ('+ "'"+ mylist[len] + "'"+ ') ,'
else:
sql = sql + '('+ "'"+ mylist[len] + "'"+ ')'
cursor1.execute("INSERT into sample(domain) values " + sql)
Thanks for your help guys!
The problem I have is to identify the type of data entering the database, I think everyone as IntegerField model in Django and Python code only inserts it all in one list and then insert it into the base data.
On the other hand I have no very clear writing Python code length according to the rules of the line, what I do is just see that in the end the code is very long as separated with spaces to align the next line below do not know if this good in this way and that the code will not fail.
The data that has to enter ip_unidad is ('186 .99.41.000 ', 3333) found in' self.addr [0] 'and the data date is '091211' which is in 'self.Cadenapura [17] '
and try "self.Cadenapura [17] = int (self.Cadenapura [17])" but nothing
It records the input data in the database but the two spaces are 0.
any ideas would be grateful.
The console error is:
Warning: Incorrect integer value: 'self.addr[0]' for column 'ip_unidad' at row 1
('self.addr[0]','self.Cadenapura[17]')
Warning:Incorrect integer value: 'self.Cadenapura[17]' for column 'fecha' at row 1
('self.addr[0]','self.Cadenapura[17]')
The code. Py used is:
sql = """INSERT INTO carro ( ip_unidad , hora ) VALUES (%s,%s)"""
db = MySQLdb.Connect(host="localhost", user="root",passwd="--------",db="gprslmgs")
cursor = db.cursor()
try :
cursor.execute(sql,('self.addr[0]','self.Cadenapura[17]'))
db.commit()
except:
db.rollback()
Django model used to create the database is:
class Carro(models.Model):
ip_unidad = models.IntegerField(max_length=15)
fecha = models.IntegerField(max_length=6)
Thank you.
I am not very familiar with Django but what i see is that you specify object within ' - think you don't need to do this. Have you tried something like:
cursor.execute(sql % (self.addr[0], self.Cadenapura[17]))
Or:
cursor.execute(sql, (self.addr[0], self.Cadenapura[17],))
While browsing i found the following MySQLdb sample code:
import MySQLdb
db = MySQLdb.connect(passwd="moonpie",db="thangs")
c = db.cursor()
c.executemany(
"""INSERT INTO breakfast (name, spam, eggs, sausage, price)
VALUES (%s, %s, %s, %s, %s)""",
[
("Spam and Sausage Lover's Plate", 5, 1, 8, 7.95 ),
("Not So Much Spam Plate", 3, 2, 0, 3.95 ),
("Don't Wany ANY SPAM! Plate", 0, 4, 3, 5.95 )
] )
So i think it should work the second way i mentioned.
I see two main problems - first is that '186.99.41.000' is not an integer, this is a string. Django provides an ip address field which is designed for this.
Your second problem is similar to the first, in that '09876' is a string, and your column type is IntegerField, however when you convert '09876' to an integer, you'll get 9876 because in Python, a number starting from 0 is an octal literal:
>>> print 127
127
>>> print 0177
127
>>> print 0x7f
127
So you need to store '09876' as a string in your database, and to do that you need to change your column type to CharField.
Like I said on your newer question, you should use Django's model api to handle the SQL for you.
Carro.objects.create(ip_unidad=self.addr[0], fecha=self.Cadenapura[17])
Furthermore, you should also revise your model. Instead of using IntegerFields, you should instead use
class Carro(models.Model):
ip_unidad = models.IPAddressField()
fecha = models.DateField()
Then when saving your data(with the model's objects.create), you need to make sure that your date is a python datetime.date object. Using the IPAddressField also means that you don't need to bother trying to convert the IP address to an int.
Hey,
I'm trying to run the following query:
self.cursor.execute('SELECT courses.courseid, days, starttime, bldg, roomnum, '
'area, title, descrip, prereqs, endtime FROM '
'classes, courses, crosslistings, coursesprofs, profs WHERE '
'classes.courseid = courses.courseid AND '
'courses.courseid = crosslistings.courseid AND '
'courses.courseid = coursesprofs.courseid AND '
'coursesprofs.profid = profs.profid AND '
'classes.classid LIKE %s'
';',
(self.classid))
classid is an int(11) field in the db. When I set self.classid = %, it returns all the results, but as soon as I set it to say, '3454' or some other amount it returns nothing even when there is a class with that classid. Am I querying incorrectly against int fields?
Even a simpler query like
select * from classes where classes.classid = '3454'; does not work
Try:
select * from classes where classes.classid = 3454;
I resolved this on my own. Based on my db structure, I was querying the wrong fields. I was looking for values that weren't there so that's why I was always returning an empty result set. Thanks for the help on the = operator though, that was utilized.
I've googled around a bit, but maybe I didn't put the correct magik incantation into the search box.
Does anyone know how to get output parameters from a stored procedure in Python? I'm using pymssql to call a stored procedure, and I'm not sure of the correct syntax to get the output parameter back. I don't think I can use any other db modules since I'm running this from a Linux box to connect to a mssql database on a MS Server.
import pymssql
con = pymssql.connect(host='xxxxx',user='xxxx',password='xxxxx',database='xxxxx')
cur = con.cursor()
query = "EXECUTE blah blah blah"
cur.execute(query)
con.commit()
con.close()
I'm not a python expert but after a brief perusing of the DB-API 2.0 I believe you should use the "callproc" method of the cursor like this:
cur.callproc('my_stored_proc', (first_param, second_param, an_out_param))
Then you'll have the result in the returned value (of the out param) in the "an_out_param" variable.
If you cannot or don't want to modify the original procedure and have access to the database you can write a simple wrapper procedure that is callable from python.
For example, if you have a stored procedure like:
CREATE PROC GetNextNumber
#NextNumber int OUTPUT
AS
...
You could write a wrapper like so which is easily callable from python:
CREATE PROC GetNextNumberWrap
AS
DECLARE #RNextNumber int
EXEC GetNextNumber #RNextNumber
SELECT #RNextNumber
GO
Then you could call it from python like so:
import pymssql
con = pymssql.connect(...)
cur = con.cursor()
cur.execute("EXEC GetNextNumberWrap")
next_num = cur.fetchone()[0]
If you make your procedure produce a table, you can use that result as a substitute for out params.
So instead of:
CREATE PROCEDURE Foo (#Bar INT OUT, #Baz INT OUT) AS
BEGIN
/* Stuff happens here */
RETURN 0
END
do
CREATE PROCEDURE Foo (#Bar INT, #Baz INT) AS
BEGIN
/* Stuff happens here */
SELECT #Bar Bar, #Baz Baz
RETURN 0
END
It looks like every python dbapi library implemented on top of freetds (pymssql, pyodbc, etc) will not be able to access output parameters when connecting to Microsoft SQL Server 7 SP3 and higher.
http://www.freetds.org/faq.html#ms.output.parameters
I was able to get an output value from a SQL stored procedure using Python. I could not find good help getting the output values in Python. I figured out the Python syntax myself, so I suspect this is worth posting here:
import sys, string, os, shutil, arcgisscripting
from win32com.client import Dispatch
from adoconstants import *
#skip ahead to the important stuff
conn = Dispatch('ADODB.Connection')
conn.ConnectionString = "Provider=sqloledb.1; Data Source=NT38; Integrated Security = SSPI;database=UtilityTicket"
conn.Open()
#Target Procedure Example: EXEC TicketNumExists #ticketNum = 8386998, #exists output
Cmd = Dispatch('ADODB.Command')
Cmd.ActiveConnection = conn
Cmd.CommandType = adCmdStoredProc
Cmd.CommandText = "TicketNumExists"
Param1 = Cmd.CreateParameter('#ticketNum', adInteger, adParamInput)
Param1.Value = str(TicketNumber)
Param2 = Cmd.CreateParameter('#exists', adInteger, adParamOutput)
Cmd.Parameters.Append(Param1)
Cmd.Parameters.Append(Param2)
Cmd.Execute()
Answer = Cmd.Parameters('#exists').Value
2016 update (callproc support in pymssql 2.x)
pymssql v2.x offers limited support for callproc. It supports OUTPUT parameters using the pymssql.output() parameter syntax. Note, however, that OUTPUT parameters can only be retrieved with callproc if the stored procedure does not also return a result set. That issue is discussed on GitHub here.
For stored procedures that do not return a result set
Given the T-SQL stored procedure
CREATE PROCEDURE [dbo].[myDoubler]
#in int = 0,
#out int OUTPUT
AS
BEGIN
SET NOCOUNT ON;
SELECT #out = #in * 2;
END
the Python code
import pymssql
conn = pymssql.connect(
host=r'localhost:49242',
database='myDb',
autocommit=True
)
crsr = conn.cursor()
sql = "dbo.myDoubler"
params = (3, pymssql.output(int, 0))
foo = crsr.callproc(sql, params)
print(foo)
conn.close()
produces the following output
(3, 6)
Notice that callproc returns the parameter tuple with the OUTPUT parameter value assigned by the stored procedure (foo[1] in this case).
For stored procedures that return a result set
If the stored procedure returns one or more result sets and also returns output parameters, we need to use an anonymous code block to retrieve the output parameter value(s):
Stored Procedure:
ALTER PROCEDURE [dbo].[myDoubler]
#in int = 0,
#out int OUTPUT
AS
BEGIN
SET NOCOUNT ON;
SELECT #out = #in * 2;
-- now let's return a result set, too
SELECT 'foo' AS thing UNION ALL SELECT 'bar' AS thing;
END
Python code:
sql = """\
DECLARE #out_value INT;
EXEC dbo.myDoubler #in = %s, #out = #out_value OUTPUT;
SELECT #out_value AS out_value;
"""
params = (3,)
crsr.execute(sql, params)
rows = crsr.fetchall()
while rows:
print(rows)
if crsr.nextset():
rows = crsr.fetchall()
else:
rows = None
Result:
[('foo',), ('bar',)]
[(6,)]
You might also look at using SELECT rather than EXECUTE. EXECUTE is (iirc) basically a SELECT that doesn't actually fetch anything (, just makes side-effects happen).
You can try to reformat query:
import pypyodc
connstring = "DRIVER=SQL Server;"\
"SERVER=servername;"\
"PORT=1043;"\
"DATABASE=dbname;"\
"UID=user;"\
"PWD=pwd"
conn = pypyodbc.connect(connString)
cursor = conn.cursor()
query="DECLARE #ivar INT \r\n" \
"DECLARE #svar VARCHAR(MAX) \r\n" \
"EXEC [procedure]" \
"#par1=?," \
"#par2=?," \
"#param1=#ivar OUTPUT," \
"#param2=#svar OUTPUT \r\n" \
"SELECT #ivar, #svar \r\n"
par1=0
par2=0
params=[par1, par2]
result = cursor.execute(query, params)
print result.fetchall()
[1]https://amybughunter.wordpress.com/tag/pypyodbc/
Here's how I did it, the key is to declare output parameter first:
import cx_Oracle as Oracle
conn = Oracle.connect('xxxxxxxx')
cur = conn.cursor()
idd = cur.var(Oracle.NUMBER)
cur.execute('begin :idd := seq_inv_turnover_id.nextval; end;', (idd,))
print(idd.getvalue())
I use pyodbc and then convert the pyodbc rows object to a list. Most of the answers show a query declaring variables as part of the query. But I would think you declare your variables as part of the sp, thus eliminating an unnecessary step in python. Then, in python, all you have to do is pass the parameters to fill in those variables.
Here is the function I use to convert the pyodbc rows object to a usable list (of lists) (note that I have noticed pyodbc sometimes adds trailing spaces, so I account for that which works well for me):
def convert_pyodbc(pyodbc_lst):
'''Converts pyodbc rows into usable list of lists (each sql row is a list),
then examines each list for list elements that are strings,
removes trailing spaces, and returns a usable list.'''
usable_lst = []
for row in pyodbc_lst:
e = [elem for elem in row]
usable_lst.append(e)
for i in range(0,len(usable_lst[0])):
for lst_elem in usable_lst:
if isinstance(lst_elem[i],str):
lst_elem[i] = lst_elem[i].rstrip()
return usable_lst
Now if I need to run a stored procedure from python that returns a results set, I simply use:
strtdate = '2022-02-21'
stpdate = '2022-02-22'
conn = mssql_conn('MYDB')
cursor = conn.cursor()
qry = cursor.execute(f"EXEC mystoredprocedure_using_dates
'{strtdate}','{stpdate}' ")
results = convert_pyodbc(qry.fetchall())
cursor.close()
conn.close()
And sample results which I then take and write to a spreadsheet or w/e:
[[datetime.date(2022, 2, 21), '723521', 'A Team Line 1', 40, 9],
[datetime.date(2022, 2, 21), '723522', 'A Team Line 2', 15, 10],
[datetime.date(2022, 2, 21), '723523', 'A Team Line 3', 1, 5],
[datetime.date(2022, 2, 21), '723686', 'B Team Line 1', 39, 27],
[datetime.date(2022, 2, 21), '723687', 'B Team Line 2', 12, 14]]