This python code should run statements on the database, but the sql statements are not executed:
from sqlalchemy import *
sql_file = open("test.sql","r")
sql_query = sql_file.read()
sql_file.close()
engine = create_engine(
'postgresql+psycopg2://user:password#localhost/test', echo=False)
conn = engine.connect()
print sql_query
result = conn.execute(sql_query)
conn.close()
The test.sql file contains SQL statements which create 89 tables.
The tables are not created if I specify 89 tables, but if I reduce the number of tables to 2 to it works.
Is there a limit on the number of queries that can be executed within the conn.execute? How do a run any number of raw queries like this?
Perhaps, forcing the autocommit:
conn.execute(RAW_SQL).execution_options(autocommit=True))
Other approach is using transactions and doing the commit:
t = conn.begin()
try:
conn.execute(RAW_SQL)
t.commit()
except:
t.rollback()
PD: You can put the execution_options in the create_engine parameters too.
Why do you use raw SQL with SQLAlchemy? If you have no good reason for that, you should use other methods:
http://docs.sqlalchemy.org/en/rel_0_7/orm/tutorial.html
http://docs.sqlalchemy.org/en/rel_0_7/core/schema.html#metadata-describing
Related
I have parameterized queries with f strings such that the queries will select some data from a series of tables and joins, and I want to insert the resulting set of data into another pre-created table (tables been designed to house these results).
Python executes the code but the query results never show up in my table.
Assuming target_table is already created in singlestore database:
qry_load = 'insert into target_table select * from some_tables'
conn = engine.connect()
trans = conn.begin()
try:
conn.execute(qry_load)
trans.commit()
except:
trans.rollback()
raise
The code executes and acts as if all is ok, but the data never shows up in the target table.
How do I see what singlestore is passing back to better debug what is happening within the database?
Just replace begin() with cursor() function:
conn = engine.connect()
trans = conn.cursor()
If not resolved
1- Verify structure of source and destination tables if they are same or not.
2- remove try ,except and rollback() block so you can know the actual error.
Ex.
qry_load = 'insert into target_table select * from some_tables'
conn = engine.connect()
trans = conn.cursor()
conn.execute(qry_load)
trans.commit()
Can the cursor.execute call below execute multiple SQL queries in one go?
cursor.execute("use testdb;CREATE USER MyLogin")
I don't have python setup yet but want to know if above form is supported by cursor.execute?
import pyodbc
# Some other example server values are
# server = 'localhost\sqlexpress' # for a named instance
# server = 'myserver,port' # to specify an alternate port
server = 'tcp:myserver.database.windows.net'
database = 'mydb'
username = 'myusername'
password = 'mypassword'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
#Sample select query
cursor.execute("SELECT ##version;")
row = cursor.fetchone()
while row:
print(row[0])
row = cursor.fetchone()
Multiple SQL statements in a single string is often referred to as an "anonymous code block".
There is nothing in pyodbc (or pypyodbc) to prevent you from passing a string containing an anonymous code block to the Cursor.execute() method. They simply pass the string to the ODBC Driver Manager (DM) which in turn passes it to the ODBC Driver.
However, not all ODBC drivers accept anonymous code blocks by default. Some databases default to allowing only a single SQL statement per .execute() to protect us from SQL injection issues.
For example, MySQL/Connector ODBC defaults MULTI_STATEMENTS to 0 (off) so if you want to run an anonymous code block you will have to include MULTI_STATEMENTS=1 in your connection string.
Note also that changing the current database by including a USE … statement in an anonymous code block can sometimes cause problems because the database context changes in the middle of a transaction. It is often better to execute a USE … statement by itself and then continue executing other SQL statements.
Yes, it is possible.
operation = 'SELECT 1; INSERT INTO t1 VALUES (); SELECT 2'
for result in cursor.execute(operation, multi=True):
But it is not a comprehensive solution. For example, in queries with two selections, you have problems.
Consider that two types of answers must be fetch all in the cursor!
So the best solution is to break the query to sub queries and do your work step by step.
for example :
s = "USE some_db; SELECT * FROM some_table;"
s = filter(None, s.split(';'))
for i in s:
cur.execute(i.strip() + ';')
in the pyodbc documentation should give you the example your looking for. more over in the GitHub wiki: https://github.com/mkleehammer/pyodbc/wiki/Objects#cursors
you can see an example here:
cnxn = pyodbc.connect(...)
cursor = cnxn.cursor()
cursor.execute("""
select user_id, last_logon
from users
where last_logon > ?
and user_type <> 'admin'
""", twoweeks)
rows = cursor.fetchall()
for row in rows:
print('user %s logged on at %s' % (row.user_id, row.last_logon))
from this example and exploring the code, I would say your next step is testing a multi cursor.execute("<your_sql_Querie>").
if this test works, maybe try and create a CLASS then create instances of that class for each query you want to run.
This would be the basic evolution of a developers effort of reproducing documentation...hope this helps you :)
Yes, you can results for multiple queries by using the nextset() method...
query = "select * from Table1; select * from Table2"
cursor = connection.cursor()
cursor.execute(query)
table1 = cursor.fetchall()
cursor.nextset()
table2 = cursor.fetchall()
The code explains it... cursors return result "sets", which you can move between using the nextset() method.
i've been trying to get some data from my db by using below code, but the code is not working. is there any mistake that i made in the code, if so how can i fix it.
NOTE: i took the below code from just a script not a django or flesk web app.
def db():
conn = psycopg2.connect(
"dbname=mydb user=postgres password=****** host=*.*.*.*")
cur = conn.cursor()
cur.execute("""SELECT * FROM MddPublisher""")
query_results = cur.fetchall()
print(query_results)
db()
ERROR: psycopg2.errors.UndefinedTable: relation "mddpublisher" does not exist LINE 1: SELECT * FROM MddPublisher
additionally,i want to show below code to prove that connection is ok. the problem is that i can't receive data from my db whenever i try to execute select command through python.
def print_tables():
conn = psycopg2.connect(
"dbname=mydb user=postgres password=***** host=*.*.*.*.*")
cur = conn.cursor()
cur.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""")
for table in cur.fetchall():
print(table)
print_tables()
OUTPUT:
('MddPublisher',)
This is probably an issue with case sensitivity. Postgresql names are usually normalized to lower case. However, when used inside double quotes, they keep their case. So, to access a table named MddPublisher you must write it like "MddPublisher".
All the gory details are in Section 4.1.1, Identifiers and Key Words in the Postgresql 14 docs.
I'm trying to create a database with the name a user will provide. As far as I know the correct way is to use the second argument of execute().
So I did as follows:
import psycopg2
conn = psycopg2.connect(host="...", dbname="...",
user="...", password="...", port='...')
cursor = conn.cursor()
query = ''' CREATE DATABASE %s ;'''
name = 'stackoverflow_example_db'
conn.autocommit = True
cursor.execute(query, (name,))
cursor.close()
conn.close()
And I got this error:
psycopg2.errors.SyntaxError: syntax error at or near "'stackoverflow_example_db'"
LINE 1: CREATE DATABASE 'stackoverflow_example_db' ;
I need to do this statement avoiding SQL injection, so using the second argument is a must.
You can't pass values as second argument of execute(), if the statement is a CREATE DATABASE one.
As pointed out by unutbu one way to approach this is using the psycopg2.sql submodule and use identifiers to build the statement avoiding SQL injection.
The code:
import psycopg2
from psycopg2 import sql
conn = psycopg2.connect(host="...", dbname="...",
user="...", password="...", port='...')
cursor = conn.cursor()
query = ''' CREATE DATABASE {} ;'''
name = 'stackoverflow_example_db'
conn.autocommit = True
cursor.execute(sql.SQL(query).format(
sql.Identifier(name)))
cursor.close()
conn.close()
Other aditional observations:
format() do not work with %s, use {} instead
Autocommit mode is a must for this statement to work
The specified connection user needs creation privileges
It does not seem like SqlAlchemy supports calling stored procedures. Did anybody find a workaround for this that works with SQL Server?
Sample procedure:
CREATE PROCEDURE list_lock_set #name varchar (5), #requester varchar(30)
AS
BEGIN
SET NOCOUNT ON;
INSERT INTO list_lock (name, requester, acquired) values (#name, #requester, GETDATE())
RETURN 0
END
GO
This works:
import pyodbc
dbh = pyodbc.connect(driver=''{SQL Server}'', server=srv, database=db, uid=uid, pwd=pwd)
dbc = dbh.cursor()
dbc.execute("list_lock_set ?, ?", ['bbc', 'pyodbc'])
dbc.commit()
This does not produce an error but also but does not work:
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://usr:passw#srv/db?driver=SQL Server', echo=True)
engine.execute("list_lock_set ?, ?", ['bbc', 'sqlalchemy'])
Thank you.
EDIT: It appears the best solution is to fish out pyodbc cursor from the engine:
cursor = engine.raw_connection().cursor()
cursor.execute("list_lock_set ?, ?", ['bbc', 'using cursor'])
cursor.commit()
I can also obtain pyodbc Connection:
engine.raw_connection().connection
and set autocommit=True, but that might interfere with engine logic. Many thanks to #Batman.
To have it working in sqlalchemy I managed to do it this way:
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://usr:passw#srv/db?driver=SQL Server', echo=True)
with engine.begin() as conn:
conn.execute("exec dbo.your_proc")
I remember this giving me grief before too. From memory either session.execute() or connection.execute() worked for me. There's also a callproc() method, which is probably the right way to go. http://docs.sqlalchemy.org/en/latest/core/connections.html
Also, I've had issues in the past with MSSQL which seemed to be due to something asynchronous happening where the method was returning before the procedure was finished, which resulted in unpredictable results on the database. I found that putting a time.sleep(1) (or whatever the appropriate number is) right after the call fixed this.
Yes, Even I was facing the same issue. SQLAlchemy was executing the procedure but no actions were happening.
So I tweaked it by adding connection.execute("exec <Procedure_name>")
Following on from #MATEITA LUCIAN:
Add input parameters to the stored procedure
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://usr:passw#srv/db?driver=SQLServer', echo=True)
with engine.begin() as conn:
conn.execute('EXEC dbo.your_proc #textinput_param = "sometext", #intinput_param = 4')
For those that aren't sure about how or what to refer to with the '....//usr:passw#srv/db?driver=SQL server', echo=True... part of creating the engine I would suggest reading this and choosing Option 1 - Provide a DSN.
Retrieve output from the stored procedure
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc://usr:passw#srv/db?driver=SQLServer', echo=True)
with engine.begin() as conn:
result = conn.execute('EXEC dbo.your_proc #textinput_param = "sometext", #intinput_param = 4')
for row in result:
print('text_output: %s \t int_output: %d' % (row['text_field'], row['int_field']))