I am trying to get the last record from my tinyDB, so I want to make a query that looks like that:
"SELECT * FROM table ORDER BY id DESC LIMIT 1"
which should give me the last row. though I can't figure out how to do it with TinyDB.
If you want to order db by time descending for example:
od = sorted(db.all(), key=lambda k: k['time'])
print(od[-1])
According to the documentation, the following would return the doc id of the final element in the db in TinyDB 4.7.0:
el = db.all()[-1]
record = db.get(doc_id=el.doc_id)
Using Document IDs
How about:
table = db.table('table_name')
table.get(doc_id=len(table))
See the way of doc_id here
Using a Query and doing an update :
with TinyDB('db.json') as db:
my_table = db.table('a_table_name')
my_query= Query()
first_of_table_for_this_query = my_table.search(my_query.some_field == some_value)[1]
last_of_table_for_this_query = my_table.search(my_query.some_field == some_value)[-1]
# exemple of use for updating first inserted and last inserted only
my_table.update({'some_field': some_value+42}, doc_ids=[
first_of_table_for_this_query,
last_of_table_for_this_query
])
Related
I have created a dictionary as follows:
dict = { 'NASDAQ': {'AMZN%', 'AAPL%', 'ABNB%'}, 'TSX': {'SHOP%', 'L%', 'RY%', 'XIF%'}}
I want to write a for loop within a query to fetch data from a table TICKER_TABLE with a column called TICKER which contains the dict values.
The query I am writing is part of a broader Python loop that looks like this:
for key in dict.keys():
query = """SELECT * FROM "TICKER_TABLE"
WHERE "TICKER" LIKE (FOR ITEM IN dict.VALUES)""""
Is there a way to do this?
If TICKER_TABLE and TICKER it's varable use f before string and {} to add varable. I'm not very advanced with sql, so I will use an extra for loop in python, although it can probably be done better but I don't know what u actually need:
for key in dict:
for value in dict[key]:
query = f"SELECT * FROM {TICKER_TABLE}
WHERE {TICKER} LIKE {value}"
Every iteretion of second for loop to AMZN%, AAPL%, SHOP% etc
Maby something for this will be helpful for u.
If you really wanted to do this using SQLAlchemy you could do it like this, combining the "likes" for each key into a single query:
import sqlalchemy as sa
engine = sa.create_engine(<connection_url>, echo=True, future=True)
# Reflect the database table into an object.
tbl = sa.Table('ticker_table', sa.MetaData(), autoload_with=engine)
with engine.connect() as conn:
for k, v in dict_.items():
q = sa.select(tbl).where(sa.or_(tbl.c.ticker.like(t) for t in v))
rows = conn.execute(q)
for row in rows:
print(row)
Generates SQL like this:
SELECT ticker_table.id, ticker_table.ticker
FROM ticker_table
WHERE ticker_table.ticker LIKE ?
OR ticker_table.ticker LIKE ?
OR ticker_table.ticker LIKE ?
OR ticker_table.ticker LIKE ?
If you don't need to process the results by key you can get them all in a single query:
import functools
import sqlalchemy as sa
# Merge the dictionary values into a single set.
vals = functools.reduce(set.union, dict_.values(), set())
with engine.connect() as conn:
q = sa.select(tbl).where(sa.or_(tbl.c.ticker.like(t) for t in vals))
rows = conn.execute(q)
for row in rows:
print(row)
print()
I have Python code that connects with SQL Server database using PYODBC and Streamlit to create a web app.
The problem is when I try to perform a select query with multiple conditions the result is empty where as the result it must return records.
If I try the SQL query direct on the database it return the below result:
SELECT TOP (200) ID, first, last
FROM t1
WHERE (first LIKE '%tes%') AND (last LIKE '%tesn%')
where as the query from the python it return empty
sql="select * from testDB.dbo.t1 where ID = ? and first LIKE '%' + ? + '%' and last LIKE '%' + ? + '%' "
param0 = vals[0]
param1=f'{vals[1]}'
param2=f'{vals[2]}'
rows = cursor.execute(sql, param0,param1,param2).fetchall()
Code:
import pandas as pd
import streamlit as st
vals = []
expander_advanced_search = st.beta_expander('Advanced Search')
with expander_advanced_search:
for i, col in enumerate(df.columns):
val = st_input_update("search for {}".format(col))
expander_advanced_search.markdown(val, unsafe_allow_html=True)
vals.append(val)
if st.form_submit_button("search"):
if len(vals)>0:
sql='select * from testDB.dbo.t1 where ID = ? and first LIKE ? and last LIKE ? '
param0 = vals[0]
param1=f'%{vals[1]}%'
param2=f'%{vals[2]}%'
rows = cursor.execute(sql, param0,param1,param2).fetchall()
df = pd.DataFrame.from_records(rows, columns = [column[0] for column in cursor.description])
st.dataframe(df)
Based on suggestion of Dale k I use the OR operator in the select query:
sql="select * from testDB.dbo.t1 where ID = ? OR first LIKE ? or last LIKE ? "
param0 = vals[0] # empty
param1=f'%{vals[1]}%' # nabi
param2=f'%{vals[2]}%' # empty
rows = cursor.execute(sql, param0,param1,param2).fetchall()
The displayed result:
all the records in the database
The expected result:
id first last
7 nabil jider
I think this is probably in your parameters - your form is only submitting first/last values, but your query says ID=?
You're not providing an ID from the form so there are no results. Or it's putting the value from the 'first' input into vals[0] and the resulting query is looking for an ID = 'tes'.
Also, look into pd.read_sql() to pipe query results directly into a DataFrame/
OR statement might be what you're after if you want each clause treated separately:
where ID = ? or first LIKE ? or last LIKE ?'
I have a sql code that will print out all events that user with id=3 did not join particular yet:
SELECT * from Event where id not in (select event_id from Participant where user_id =3);
I want to write it in SQLAlchemy and so far I've got this
Event.query.filter(not_(Participant.user_id==3))
but produced query is not what I initially wrote:
SELECT "Event".id AS "Event_id",
"Event".name AS "Event_name",
"Event".published AS "Event_published",
"Event".published_when AS "Event_published_when",
"Event".published_by AS "Event_published_by"
FROM "Event", "Participant"
WHERE "Participant".user_id != ?
The above is not giving any results. I guess I wrote this SQLAlchemy query incorrectly. What is wrong with it?
Correct syntax:
Event.query.filter(Event.id.notin_(db.session.query(Participant.event_id).filter(Participant.user_id==session['uid'])))
Subquery had to be limited to event_id only. Thanks to this it will now return correct data from DB.
Try something like this:
sub = Participant.query(Participant.event_id).filter(Participant.user_id==3)
res = Event.query.filter(Event.id.notin_(sub))
or maybe this way:
res = Session.query(Event, Participant).filter(Event.id == Participant.user_id).filter(Participant.user_id == 3)
I have a flask app:
db = SQLAlchemy(app)
#app.route('/')
def home():
query = "SELECT door_id FROM table WHERE id = 2422628557;"
result = db.session.execute(query)
return json.dumps([dict(r) for r in result])
When I execute: curl http://127.0.0.1:5000/
I got result very quickly [{"door_id": 2063805}]
But when I reverse my query: query = "SELECT id FROM table WHERE door_id = 2063805;"
everything work very-very slow.
Probably I have index on id attribute and don't have such on door_id.
How can I improve performance? How to add index on door_id?
If you want index on that column, just create it:
create index i1 on table (door_id)
Then depending on your settings you might have to analyze it to introduce to query planner, eg:
analyze table;
keep in mind - all indexes require additional IO on data manipulation
Look at the explain plan for your query.
EXPLAIN
"SELECT door_id FROM table WHERE id = 2422628557;"
It is likely you are seeing something like this:
QUERY PLAN
------------------------------------------------------------
Seq Scan on table (cost=0.00..483.00 rows=99999 width=244)
Filter: (id = 2422628557)
The Seq Scan is checking every single row in this table and this is then being filtered by the id you are restricting by.
What you should do in this occasion is to add an index to the ID column.
The plan will change to something like:
QUERY PLAN
-----------------------------------------------------------------------------
Index Scan using [INDEX_NAME] on table (cost=0.00..8.27 rows=1 width=244)
Index Cond: (id = 2422628557)
The optimiser will be using the index to reduce the row look ups for your query.
This will speed up your query.
I have one database with two tables, both have a column called barcode, the aim is to retrieve barcode from one table and search for the entries in the other where extra information of that certain barcode is stored. I would like to have bothe retrieved data to be saved in a DataFrame. The problem is when I want to insert the retrieved data into DataFrame from the second query, it stores only the last entry:
import mysql.connector
import pandas as pd
cnx = mysql.connector(user,password,host,database)
query_barcode = ("SELECT barcode FROM barcode_store")
cursor = cnx.cursor()
cursor.execute(query_barcode)
data_barcode = cursor.fetchall()
Up to this point everything works smoothly, and here is the part with problem:
query_info = ("SELECT product_code FROM product_info WHERE barcode=%s")
for each_barcode in data_barcode:
cursor.execute(query_info % each_barcode)
pro_info = pd.DataFrame(cursor.fetchall())
pro_info contains only the last matching barcode information! While I want to retrieve all the information for each data_barcode match.
That's because you are consistently overriding existing pro_info with new data in each loop iteration. You should rather do something like:
query_info = ("SELECT product_code FROM product_info")
cursor.execute(query_info)
pro_info = pd.DataFrame(cursor.fetchall())
Making so many SELECTs is redundant since you can get all records in one SELECT and instantly insert them to your DataFrame.
#edit: However if you need to use the WHERE statement to fetch only specific products, you need to store records in a list until you insert them to DataFrame. So your code will eventually look like:
pro_list = []
query_info = ("SELECT product_code FROM product_info WHERE barcode=%s")
for each_barcode in data_barcode:
cursor.execute(query_info % each_barcode)
pro_list.append(cursor.fetchone())
pro_info = pd.DataFrame(pro_list)
Cheers!