I have Python code that connects with SQL Server database using PYODBC and Streamlit to create a web app.
The problem is when I try to perform a select query with multiple conditions the result is empty where as the result it must return records.
If I try the SQL query direct on the database it return the below result:
SELECT TOP (200) ID, first, last
FROM t1
WHERE (first LIKE '%tes%') AND (last LIKE '%tesn%')
where as the query from the python it return empty
sql="select * from testDB.dbo.t1 where ID = ? and first LIKE '%' + ? + '%' and last LIKE '%' + ? + '%' "
param0 = vals[0]
param1=f'{vals[1]}'
param2=f'{vals[2]}'
rows = cursor.execute(sql, param0,param1,param2).fetchall()
Code:
import pandas as pd
import streamlit as st
vals = []
expander_advanced_search = st.beta_expander('Advanced Search')
with expander_advanced_search:
for i, col in enumerate(df.columns):
val = st_input_update("search for {}".format(col))
expander_advanced_search.markdown(val, unsafe_allow_html=True)
vals.append(val)
if st.form_submit_button("search"):
if len(vals)>0:
sql='select * from testDB.dbo.t1 where ID = ? and first LIKE ? and last LIKE ? '
param0 = vals[0]
param1=f'%{vals[1]}%'
param2=f'%{vals[2]}%'
rows = cursor.execute(sql, param0,param1,param2).fetchall()
df = pd.DataFrame.from_records(rows, columns = [column[0] for column in cursor.description])
st.dataframe(df)
Based on suggestion of Dale k I use the OR operator in the select query:
sql="select * from testDB.dbo.t1 where ID = ? OR first LIKE ? or last LIKE ? "
param0 = vals[0] # empty
param1=f'%{vals[1]}%' # nabi
param2=f'%{vals[2]}%' # empty
rows = cursor.execute(sql, param0,param1,param2).fetchall()
The displayed result:
all the records in the database
The expected result:
id first last
7 nabil jider
I think this is probably in your parameters - your form is only submitting first/last values, but your query says ID=?
You're not providing an ID from the form so there are no results. Or it's putting the value from the 'first' input into vals[0] and the resulting query is looking for an ID = 'tes'.
Also, look into pd.read_sql() to pipe query results directly into a DataFrame/
OR statement might be what you're after if you want each clause treated separately:
where ID = ? or first LIKE ? or last LIKE ?'
Related
I am using HANA Python Connector's Cursor.execute(sql,hanaparams) method, the parameters to this method are sql statement and hanaparams.My query is like this.
"SELECT * FROM TABLE WHRE COLUMN1 IN(?)" and My PARAMETES ARE VALUE1 ,VALUE2; LIKE LIST/TUPLE;
I am unable to retrieve resultset, Whereas when i run this in HANA with Query and Input Parameters Hard Coded in ,it runs perfectly fine
I am following this tutorial https://developers.sap.com/tutorials/hana-clients-python.html
Any pointers how should i Pass multiple values in Params
Something simple like this seems to work just fine. Count of ? must be equal to count of parameters you have. In your case it takes only VALUE1.
from hdbcli import dbapi
conn = dbapi.connect(
key='HDBKEY'
)
cursor = conn.cursor()
parameters = [11, "2020-12-24"]
params = '?,'*len(parameters)
params2 = params[0:-1]
sql_command2 = f"SELECT {params2} FROM DUMMY;"
cursor.execute(sql_command2, parameters)
rows = cursor.fetchall()
for row in rows:
for col in row:
print ("%s" % col, end=" ")
print (" ")
cursor.close()
conn.close()
So instead of SELECT * FROM TABLE WHERE COLUMN1 IN(?) it should be SELECT * FROM TABLE WHERE COLUMN1 IN(?, ?)
I am performing an ETL task where I am querying tables in a Data Warehouse to see if it contains IDs in a DataFrame (df) which was created by joining tables from the operational database.
The DataFrame only has ID columns from each joined table in the operational database. I have created a variable for each of these columns, e.g. 'billing_profiles_id' as below:
billing_profiles_dim_id = df['billing_profiles_dim_id']
I am attempting to iterated row by row to see if the ID here is in the 'billing_profiles_dim' table of the Data Warehouse. Where the ID is not present, I want to populate the DWH tables row by row using the matching ID rows in the ODB:
for key in billing_profiles_dim_id:
sql = "SELECT * FROM billing_profiles_dim WHERE id = '"+str(key)+"'"
dwh_cursor.execute(sql)
result = dwh_cursor.fetchone()
if result == None:
sqlQuery = "SELECT * from billing_profile where id = '"+str(key)+"'"
sqlInsert = "INSERT INTO billing_profile_dim VALUES ('"+str(key)+"','"+billing_profile.name"')
op_cursor = op_connector.execute(sqlInsert)
billing_profile = op_cursor.fetchone()
So far at least, I am receiving the following error:
SyntaxError: EOL while scanning string literal
This error message points at the close of barcket at
sqlInsert = "INSERT INTO billing_profile_dim VALUES ('"+str(key)+"','"+billing_profile.name"')
Which I am currently unable to solve. I'm also aware that this code may run into another problem or two. Could someone please see how I can solve the current issue and please ensure that I head down the correct path?
You are missing a double tick and a +
sqlInsert = "INSERT INTO billing_profile_dim VALUES ('"+str(key)+"','"+billing_profile.name+"')"
But you should really switch to prepared statements like
sql = "SELECT * FROM billing_profiles_dim WHERE id = '%s'"
dwh_cursor.execute(sql,(str(key),))
...
sqlInsert = ('INSERT INTO billing_profile_dim VALUES '
'(%s, %s )')
dwh_cursor.execute(sqlInsert , (str(key), billing_profile.name))
I'm trying to create an SQL queries for a large list of records (>42 million) to insert into a remote database. Right now I'm building queries in the format INSERT INTO tablename (columnnames) VALUES (values)
tablename, columnnames, and values are all of varying length so I'm generating a number of placeholders equal to the number of values required.
The result is I have a string called sqcommand that looks like INSERT INTO ColName (?,?,?) VALUES (?,?,?); and a list of parameters that looks like ([Name1, Name2, Name3, Val1, Val2, Val3]).
When try to execute the query as db.execute(sqlcommand, params) I get errors indicating I'm trying to insert into columns "#P1", "#P2", "#P3" et cetera. Why aren't the values from my list properly translating? Where is it getting "#P1" from? I know I don't have a column of that name and as far as I can tell I'm not referencing a column of that name yet the execute method is still trying to use it.
UPDATE: As per request, the full code is below, modified to avoid anything that might be private. The end result of this is to move data, row by row, from an sqlite3 db file to an AWS SQL server.
newDB = pyodbc.connect(newDataBase)
oldDB = sqlite3.connect(oldDatabase)
tables = oldDB.execute("SELECT * FROM sqlite_master WHERE type='table';").fetchall()
t0 = datetime.now()
for table in tables:
print('Parsing:', str(table[1]))
t1 = datetime.now()
colInfo = oldDB.execute('PRAGMA table_info('+table[1]+');').fetchall()
cols = list()
cph = ""
i = 0
for col in colInfo:
cph += "?,"
cols.append(str(col[1]))
rowCount = oldDB.execute("SELECT COUNT(*) FROM "+table[1]+" ;").fetchall()
count = 0
while count <= int(rowCount[0][0]):
params = list()
params.append(cols)
count += 1
row = oldDB.execute("SELECT * FROM "+table[1]+" LIMIT 1;").fetchone()
ph = ""
for val in row:
ph += "?,"
params = params.append(str(val))
ph = ph[:-1]
cph = cph[:-1]
print(str(table[1]))
sqlcommand = "INSERT INTO "+str(table[1])+" ("+cph+") VALUES ("+ph+");"
print(sqlcommand)
print(params)
newDB.execute(sqlcommand, params)
sqlcommand = "DELETE FROM ? WHERE ? = ?;"
oldDB.execute(sqlcommand, (str(table[1]), cols[0], vals[0],))
newDB.commit()
Unbeknownst to me, column names can't be passed as parameters. Panagiotis Kanavos answered this in a comment. I guess I'll have to figure out a different way to generate the queries. Thank you all very much, I appreciate it.
I am trying to get the last record from my tinyDB, so I want to make a query that looks like that:
"SELECT * FROM table ORDER BY id DESC LIMIT 1"
which should give me the last row. though I can't figure out how to do it with TinyDB.
If you want to order db by time descending for example:
od = sorted(db.all(), key=lambda k: k['time'])
print(od[-1])
According to the documentation, the following would return the doc id of the final element in the db in TinyDB 4.7.0:
el = db.all()[-1]
record = db.get(doc_id=el.doc_id)
Using Document IDs
How about:
table = db.table('table_name')
table.get(doc_id=len(table))
See the way of doc_id here
Using a Query and doing an update :
with TinyDB('db.json') as db:
my_table = db.table('a_table_name')
my_query= Query()
first_of_table_for_this_query = my_table.search(my_query.some_field == some_value)[1]
last_of_table_for_this_query = my_table.search(my_query.some_field == some_value)[-1]
# exemple of use for updating first inserted and last inserted only
my_table.update({'some_field': some_value+42}, doc_ids=[
first_of_table_for_this_query,
last_of_table_for_this_query
])
I have a list of tuples of which i'm inserting into a Table.
Each tuple has 50 values. How do i insert without having to specify the column names and how many ? there is?
col1 is an auto increment column so my insert stmt starts in col2 and ends in col51.
current code:
l = [(1,2,3,.....),(2,4,6,.....),(4,6,7,.....)...]
for tup in l:
cur.execute(
"""insert into TABLENAME(col2,col3,col4.........col50,col51)) VALUES(?,?,?,.............)
""")
want:
insert into TABLENAME(col*) VALUES(*)
MySQL's syntax for INSERT is documented here: http://dev.mysql.com/doc/refman/5.7/en/insert.html
There is no wildcard syntax like you show. The closest thing is to omit the column names:
INSERT INTO MyTable VALUES (...);
But I don't recommend doing that. It works only if you are certain you're going to specify a value for every column in the table (even the auto-increment column), and your values are guaranteed to be in the same order as the columns of the table.
You should learn to use code to build the SQL query based on arrays of values in your application. Here's a Python example the way I do it. Suppose you have a dict of column: value pairs called data_values.
placeholders = ['%s'] * len(data_values)
sql_template = """
INSERT INTO MyTable ({columns}) VALUES ({placeholders})
"""
sql = sql_template.format(
columns=','.join(keys(data_values)),
placeholders=','.join(placeholders)
)
cur = db.cursor()
cur.execute(sql, data_values)
example code to put before your code:
cols = "("
for x in xrange(2, 52):
cols = cols + "col" + str(x) + ","
test = test[:-1]+")"
Inside your loop
for tup in l:
cur.execute(
"""insert into TABLENAME " + cols " VALUES {0}".format(tup)
""")
This is off the top of my head with no error checking