I am trying to run a system SP (which is a pre-built) using python pyodbc. In fact I am trying to see the dependencies on a table using sp_depends '<Object name>';
I am using the below code snippet.
df_f=[]
l_table = ['table_1','table_2','table_3']
try:
for l in l_table:
sql = """EXEC sp_depends '{0}';""".format(l)
while cur.nextset():
cur.execute(sql)
c = cur.fetchall()
df_l= pd.DataFrame.from_records(c, columns = [desc[0] for desc in cur.description])
df_l['Referenced_Object'] = l
df_f.append(df_l)
break
except pyodbc.Error as err:
s = str(err)
print(s)
finally:
cur.close()
cnxn.close()
The above code is not running. It is not throwing error but not appending anything in df_f.
If I run the above SP separately, I am getting the below error:
ProgrammingError: No results. Previous SQL was not a query.
I have taken help from this SO thread.
I am not able to SET NOCOUNT ON in this SP as this is a built-in and therefore I am not able to get the desired information in dataframe.
Any clue on this?
As I mentioned in the comments, sp_depends has been deprecated; it should not be used. As per the documentation you should be using sys.dm_sql_referencing_entities and sys.dm_sql_referenced_entities instead.
You can get similar results to the sp_depends with the following queries. You would need to replace the variables with parameters from your programming language (not inject them):
SELECT CONCAT(re.referenced_schema_name,'.', re.referenced_entity_name) AS [name],
o.type_desc AS [Type],
CASE re.is_updated WHEN 0 THEN 'no' WHEN 1 THEN 'yes' END AS updated,
CASE re.is_selected WHEN 0 THEN 'no' WHEN 1 THEN 'yes' END AS selected,
re.referenced_minor_name AS [column]
FROM sys.dm_sql_referenced_entities(QUOTENAME(#SchemaName) + N'.' + QUOTENAME(#ObjectName) ,'OBJECT') re
JOIN sys.objects o ON re.referenced_id = o.object_id;
SELECT DISTINCT
CONCAT(re.referencing_schema_name,'.', re.referencing_entity_name) AS [name],
o.type_desc AS [Type]
FROM sys.dm_sql_referencing_entities(QUOTENAME(#SchemaName) + N'.' + QUOTENAME(#ObjectName) ,'OBJECT') re
JOIN sys.objects o ON re.referencing_id = o.object_id;
I would like get a list of values from a DB table in python using cx_oracle. I am unable to write a query with two where conditions one of single value and another of a list.
I am able to achieve it when I filter it two strings separately or only filter it by a list of string. But could not achieve it together!!
output_list=[]
catlist = ','.join(":x%d" % i for i, _ in enumerate(category_list))
db_cursor = connection.cursor()
db_cursor.execute("""
SELECT LWEX_WORD_EXCLUDE
FROM WCG_SRC_WORD_EXCLUDE
WHERE LWEX_CATEGORY IN (%s) and LWIN_USER_UPDATED = :did""" % catlist, category_list, did =argUser)
for word in db_cursor :
output_list.append(word[0])
The current code throws an error. But if I have either of the conditions separately then it works fine. The python version that I am using is 3.5
You cannot mix and match "bind by position" and "bind by name", which is what you are doing in the above code. My suggestion would be to do something like this instead:
output_list=[]
catlist = ','.join(":x%d" % i for i, _ in enumerate(category_list))
bindvals = category_list + [arguser]
db_cursor = connection.cursor()
db_cursor.execute("""
SELECT LWEX_WORD_EXCLUDE
FROM WCG_SRC_WORD_EXCLUDE
WHERE LWEX_CATEGORY IN (%s) and LWIN_USER_UPDATED = :did""" % catlist, bindvals)
for word in db_cursor :
output_list.append(word[0])
Let's suppose I have the following table :
Id (int, Primary Key) | Value (varchar)
----------------------+----------------
1 | toto
2 | foo
3 | bar
I would like to know if giving two request, the result of the first must be contained in the result of the second without executing them.
Some examples :
# Obvious example
query_1 = "SELECT * FROM example;"
query_2 = "SELECT * FROM example WHERE id = 1;"
is_sub_part_of(query_2, query_1) # True
# An example we can't know before executing the two requests
query_1 = "SELECT * FROM example WHERE id < 2;"
query_2 = "SELECT * FROM example WHERE value = 'toto' or value = 'foo';"
is_sub_part_of(query_2, query_1) # False
# An example we can know before executing the two requests
query_1 = "SELECT * FROM example WHERE id < 2 OR value = 'bar';"
query_2 = "SELECT * FROM example WHERE id < 2 AND value = 'bar';"
is_sub_part_of(query_2, query_1) # True
# An example about columns
query_1 = "SELECT * FROM example;"
query_2 = "SELECT id FROM example;"
is_sub_part_of(query_2, query_1) # True
Do you know if there's a module in Python that is able to do that, or if it's even possible to do ?
Interesting problem. I don't know of any library that will do this for you. My thoughts:
Parse the SQL, see this for example.
Define which filtering operations can be added to a query that can only result in the same or a narrower result set. "AND x" can always be added, I think, without losing the property of being a subset. "OR x" can not. Anything else you can do to the query? For example "SELECT *", vs "SELECT x", vs "SELECT x, y".
Except for that, I can only say it's an interesting idea. You might get some more input on DBA. Is this an idea you're researching or is it related to a real-world problem you are solving, like optimizing a DB query? Maybe your question could be updated with information about this, since this is not a common way to optimize queries (unless you're working on the DB engine itself, I guess).
I have this sqlalchemy query:
query = session.query(Store).options(joinedload('salesmen').
joinedload('comissions').
joinedload('orders')).\
filter(Store.store_code.in_(selected_stores))
stores = query.all()
for store in stores:
for salesman in store.salesmen:
for comission in salesman.comissions:
#generate html for comissions for each salesman in each store
#print html document using PySide
This was working perfectly, however I added two new filter queries:
filter(Comissions.payment_status == 0).\
filter(Order.order_date <= self.dateEdit.date().toPython())
If I add just the first filter the application hangs for a couple of seconds, if I add both the application hangs indefinitely
What am I doing wrong here? How do I make this query fast?
Thank you for your help
EDIT: This is the sql generated, unfortunately the class and variable names are in Portuguese, I just translated them to English so it would be easier to undertand,
so Loja = Store, Vendedores = Salesmen, Pedido = Order, Comission = Comissao
Query generated:
SELECT "Loja"."CodLoja", "Vendedores_1"."CodVendedor", "Vendedores_1"."NomeVendedor", "Vendedores_1"."CodLoja", "Vendedores_1"."PercentualComissao",
"Vendedores_1"."Ativo", "Comissao_1"."CodComissao", "Comissao_1"."CodVendedor", "Comissao_1"."CodPedido",
"Pedidos_1"."CodPedido", "Pedidos_1"."CodLoja", "Pedidos_1"."CodCliente", "Pedidos_1"."NomeCliente", "Pedidos_1"."EnderecoCliente", "Pedidos_1"."BairroCliente",
"Pedidos_1"."CidadeCliente", "Pedidos_1"."UFCliente", "Pedidos_1"."CEPCliente", "Pedidos_1"."FoneCliente", "Pedidos_1"."Fone2Cliente", "Pedidos_1"."PontoReferenciaCliente",
"Pedidos_1"."DataPedido", "Pedidos_1"."ValorProdutos", "Pedidos_1"."ValorCreditoTroca",
"Pedidos_1"."ValorTotalDoPedido", "Pedidos_1"."Situacao", "Pedidos_1"."Vendeu_Teflon", "Pedidos_1"."ValorTotalTeflon",
"Pedidos_1"."DataVenda", "Pedidos_1"."CodVendedor", "Pedidos_1"."TipoVenda", "Comissao_1"."Valor", "Comissao_1"."DataPagamento", "Comissao_1"."StatusPagamento"
FROM "Comissao", "Pedidos", "Loja" LEFT OUTER JOIN "Vendedores" AS "Vendedores_1" ON "Loja"."CodLoja" = "Vendedores_1"."CodLoja"
LEFT OUTER JOIN "Comissao" AS "Comissao_1" ON "Vendedores_1"."CodVendedor" = "Comissao_1"."CodVendedor" LEFT OUTER JOIN "Pedidos" AS "Pedidos_1" ON "Pedidos_1"."CodPedido" = "Comissao_1"."CodPedido"
WHERE "Loja"."CodLoja" IN (:CodLoja_1) AND "Comissao"."StatusPagamento" = :StatusPagamento_1 AND "Pedidos"."DataPedido" <= :DataPedido_1
Your FROM clause is producing a Cartesian product and includes each table twice, once for filtering the result and once for eagerly loading the relationship.
To stop this use contains_eager instead of joinedload in your options. This will look for the related attributes in the query's columns instead of constructing an extra join. You will also need to explicitly join to the other tables in your query, e.g.:
query = session.query(Store)\
.join(Store.salesmen)\
.join(Store.commissions)\
.join(Store.orders)\
.options(contains_eager('salesmen'),
contains_eager('comissions'),
contains_eager('orders'))\
.filter(Store.store_code.in_(selected_stores))\
.filter(Comissions.payment_status == 0)\
.filter(Order.order_date <= self.dateEdit.date().toPython())
I've googled around a bit, but maybe I didn't put the correct magik incantation into the search box.
Does anyone know how to get output parameters from a stored procedure in Python? I'm using pymssql to call a stored procedure, and I'm not sure of the correct syntax to get the output parameter back. I don't think I can use any other db modules since I'm running this from a Linux box to connect to a mssql database on a MS Server.
import pymssql
con = pymssql.connect(host='xxxxx',user='xxxx',password='xxxxx',database='xxxxx')
cur = con.cursor()
query = "EXECUTE blah blah blah"
cur.execute(query)
con.commit()
con.close()
I'm not a python expert but after a brief perusing of the DB-API 2.0 I believe you should use the "callproc" method of the cursor like this:
cur.callproc('my_stored_proc', (first_param, second_param, an_out_param))
Then you'll have the result in the returned value (of the out param) in the "an_out_param" variable.
If you cannot or don't want to modify the original procedure and have access to the database you can write a simple wrapper procedure that is callable from python.
For example, if you have a stored procedure like:
CREATE PROC GetNextNumber
#NextNumber int OUTPUT
AS
...
You could write a wrapper like so which is easily callable from python:
CREATE PROC GetNextNumberWrap
AS
DECLARE #RNextNumber int
EXEC GetNextNumber #RNextNumber
SELECT #RNextNumber
GO
Then you could call it from python like so:
import pymssql
con = pymssql.connect(...)
cur = con.cursor()
cur.execute("EXEC GetNextNumberWrap")
next_num = cur.fetchone()[0]
If you make your procedure produce a table, you can use that result as a substitute for out params.
So instead of:
CREATE PROCEDURE Foo (#Bar INT OUT, #Baz INT OUT) AS
BEGIN
/* Stuff happens here */
RETURN 0
END
do
CREATE PROCEDURE Foo (#Bar INT, #Baz INT) AS
BEGIN
/* Stuff happens here */
SELECT #Bar Bar, #Baz Baz
RETURN 0
END
It looks like every python dbapi library implemented on top of freetds (pymssql, pyodbc, etc) will not be able to access output parameters when connecting to Microsoft SQL Server 7 SP3 and higher.
http://www.freetds.org/faq.html#ms.output.parameters
I was able to get an output value from a SQL stored procedure using Python. I could not find good help getting the output values in Python. I figured out the Python syntax myself, so I suspect this is worth posting here:
import sys, string, os, shutil, arcgisscripting
from win32com.client import Dispatch
from adoconstants import *
#skip ahead to the important stuff
conn = Dispatch('ADODB.Connection')
conn.ConnectionString = "Provider=sqloledb.1; Data Source=NT38; Integrated Security = SSPI;database=UtilityTicket"
conn.Open()
#Target Procedure Example: EXEC TicketNumExists #ticketNum = 8386998, #exists output
Cmd = Dispatch('ADODB.Command')
Cmd.ActiveConnection = conn
Cmd.CommandType = adCmdStoredProc
Cmd.CommandText = "TicketNumExists"
Param1 = Cmd.CreateParameter('#ticketNum', adInteger, adParamInput)
Param1.Value = str(TicketNumber)
Param2 = Cmd.CreateParameter('#exists', adInteger, adParamOutput)
Cmd.Parameters.Append(Param1)
Cmd.Parameters.Append(Param2)
Cmd.Execute()
Answer = Cmd.Parameters('#exists').Value
2016 update (callproc support in pymssql 2.x)
pymssql v2.x offers limited support for callproc. It supports OUTPUT parameters using the pymssql.output() parameter syntax. Note, however, that OUTPUT parameters can only be retrieved with callproc if the stored procedure does not also return a result set. That issue is discussed on GitHub here.
For stored procedures that do not return a result set
Given the T-SQL stored procedure
CREATE PROCEDURE [dbo].[myDoubler]
#in int = 0,
#out int OUTPUT
AS
BEGIN
SET NOCOUNT ON;
SELECT #out = #in * 2;
END
the Python code
import pymssql
conn = pymssql.connect(
host=r'localhost:49242',
database='myDb',
autocommit=True
)
crsr = conn.cursor()
sql = "dbo.myDoubler"
params = (3, pymssql.output(int, 0))
foo = crsr.callproc(sql, params)
print(foo)
conn.close()
produces the following output
(3, 6)
Notice that callproc returns the parameter tuple with the OUTPUT parameter value assigned by the stored procedure (foo[1] in this case).
For stored procedures that return a result set
If the stored procedure returns one or more result sets and also returns output parameters, we need to use an anonymous code block to retrieve the output parameter value(s):
Stored Procedure:
ALTER PROCEDURE [dbo].[myDoubler]
#in int = 0,
#out int OUTPUT
AS
BEGIN
SET NOCOUNT ON;
SELECT #out = #in * 2;
-- now let's return a result set, too
SELECT 'foo' AS thing UNION ALL SELECT 'bar' AS thing;
END
Python code:
sql = """\
DECLARE #out_value INT;
EXEC dbo.myDoubler #in = %s, #out = #out_value OUTPUT;
SELECT #out_value AS out_value;
"""
params = (3,)
crsr.execute(sql, params)
rows = crsr.fetchall()
while rows:
print(rows)
if crsr.nextset():
rows = crsr.fetchall()
else:
rows = None
Result:
[('foo',), ('bar',)]
[(6,)]
You might also look at using SELECT rather than EXECUTE. EXECUTE is (iirc) basically a SELECT that doesn't actually fetch anything (, just makes side-effects happen).
You can try to reformat query:
import pypyodc
connstring = "DRIVER=SQL Server;"\
"SERVER=servername;"\
"PORT=1043;"\
"DATABASE=dbname;"\
"UID=user;"\
"PWD=pwd"
conn = pypyodbc.connect(connString)
cursor = conn.cursor()
query="DECLARE #ivar INT \r\n" \
"DECLARE #svar VARCHAR(MAX) \r\n" \
"EXEC [procedure]" \
"#par1=?," \
"#par2=?," \
"#param1=#ivar OUTPUT," \
"#param2=#svar OUTPUT \r\n" \
"SELECT #ivar, #svar \r\n"
par1=0
par2=0
params=[par1, par2]
result = cursor.execute(query, params)
print result.fetchall()
[1]https://amybughunter.wordpress.com/tag/pypyodbc/
Here's how I did it, the key is to declare output parameter first:
import cx_Oracle as Oracle
conn = Oracle.connect('xxxxxxxx')
cur = conn.cursor()
idd = cur.var(Oracle.NUMBER)
cur.execute('begin :idd := seq_inv_turnover_id.nextval; end;', (idd,))
print(idd.getvalue())
I use pyodbc and then convert the pyodbc rows object to a list. Most of the answers show a query declaring variables as part of the query. But I would think you declare your variables as part of the sp, thus eliminating an unnecessary step in python. Then, in python, all you have to do is pass the parameters to fill in those variables.
Here is the function I use to convert the pyodbc rows object to a usable list (of lists) (note that I have noticed pyodbc sometimes adds trailing spaces, so I account for that which works well for me):
def convert_pyodbc(pyodbc_lst):
'''Converts pyodbc rows into usable list of lists (each sql row is a list),
then examines each list for list elements that are strings,
removes trailing spaces, and returns a usable list.'''
usable_lst = []
for row in pyodbc_lst:
e = [elem for elem in row]
usable_lst.append(e)
for i in range(0,len(usable_lst[0])):
for lst_elem in usable_lst:
if isinstance(lst_elem[i],str):
lst_elem[i] = lst_elem[i].rstrip()
return usable_lst
Now if I need to run a stored procedure from python that returns a results set, I simply use:
strtdate = '2022-02-21'
stpdate = '2022-02-22'
conn = mssql_conn('MYDB')
cursor = conn.cursor()
qry = cursor.execute(f"EXEC mystoredprocedure_using_dates
'{strtdate}','{stpdate}' ")
results = convert_pyodbc(qry.fetchall())
cursor.close()
conn.close()
And sample results which I then take and write to a spreadsheet or w/e:
[[datetime.date(2022, 2, 21), '723521', 'A Team Line 1', 40, 9],
[datetime.date(2022, 2, 21), '723522', 'A Team Line 2', 15, 10],
[datetime.date(2022, 2, 21), '723523', 'A Team Line 3', 1, 5],
[datetime.date(2022, 2, 21), '723686', 'B Team Line 1', 39, 27],
[datetime.date(2022, 2, 21), '723687', 'B Team Line 2', 12, 14]]