I'm using python 3.6.4 and sqlite3 2.6.0 to query the nearest consecutive dates in my table in a sqlite 3.27.2 file.
I've tried to get the actual sql string with vscode debugger and test it with DB Browser for SQLite. It works as I expect.
Here's the code:
sql = 'WITH \
dates(cast_date) AS (\
SELECT DISTINCT play_date\
FROM TimeTable\
),\
groups AS (\
SELECT\
date(cast_date, \'-\'||(ROW_NUMBER() OVER (ORDER BY cast_date))||\' days\') AS grp,\
cast_date\
FROM dates\
)\
SELECT\
MIN(cast_date) AS date_start,\
MAX(cast_date) AS date_end\
FROM groups GROUP BY grp ORDER BY 2 DESC LIMIT 1'
cursor = conn.cursor()
result = []
try:
cursor.execute(sql)
result = cursor.fetchone()
except sqlite3.OperationalError:
FileLogger.exception('Exception at '+__file__+' '+__name__)
An exception occurs:
cursor.execute(sql)
sqlite3.OperationalError: near "OVER": syntax error
Window functions support was first added to SQLite with release version 3.25.0 (2018-09-15), according to official documentation.
When using Python, you are using Python SQLite3 client library (which is distributed with Python) instead of your system SQLite3 installation. For Python 2.7, the version is 3.11.0, which is below your required version.
You may try using a newer SQLite3 client library, as suggested by these answers.
Related
I am using flask-sqlalchemy to perform a ranking query on a PlayerKillData table, which has 4 columns: ref_id (Primary Key), kill, all_kill and lvl. Specifically, what I want to achieve is to Rank by kill for player on the same lvl and add a Rank column
#Columns define in Create Table SQL statement and resolved by reflection at runtime
class PlayerKillData(db.Model):
__tablename__ = "player_kill_data"
The following python snippet is that I have tried but no matter how I have tried, it always return sqlite3.OperationalError:
subquery = excDB.db.session.query(
PlayerKillData,
excDB.db.func.rank().over(
order_by = PlayerKillData.kill.desc(),
partition_by = PlayerKillData.lvl).label('RANK')
)
subquery.all()
The SQL statement printed by the error message is:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "(": syntax error
[SQL: SELECT player_kill_data.ref_id AS player_kill_data_ref_id, player_kill_data.kill AS player_kill_data_kill, player_kill_data.all_kill AS player_kill_data_all_kill, player_kill_data.lvl AS player_kill_data_lvl, rank() OVER (PARTITION BY player_kill_data.lvl ORDER BY player_kill_data.kill DESC) AS "RANK"
FROM player_kill_data]
I have also tried running the SQL statement directly using Db Browser for SQLite and it returned the table successfully without any error.
I am really puzzled by this and would like any help I can get to solve this.
Your version of SQLite might be smaller than 3.25.0.
As mentionned here and confirmed in the Release History of SQLite, SQLite supports window function (RANK() OVER ...) since version 3.25.0, which was released on 2018-09-15.
To check your version of SQLite in python:
import sqlite3
print(sqlite3.sqlite_version)
# returns '3.22.0' for me
If many people work on the same codebase and you'd like to have a more explicit error message for other developers, you can assert:
assert sqlite3.sqlite_version_info >= (3,25,0), """sqlite3 version must be >= 3.25.0 because that's when it started to support window functions"""
I've created a database and am able to create a simple view using the following command:
CREATE VIEW IF NOT EXISTS monthly_terminals (year, month) AS
SELECT reportYear, reportMonth FROM osMonthlyTerminals
This command works just fine if I paste into the SQLite Manager plugin on FireFox:
I then drop this view and try to use the exact same command in the following Python 3 code to create this view...
import sqlite3
db_name = "../data/OsReportMerchants.sqlite"
conn = sqlite3.connect(db_name)
cur = conn.cursor()
cur.execute("CREATE VIEW IF NOT EXISTS monthly_terminals (year, month) AS SELECT reportYear, reportMonth FROM osMonthlyTerminals")
... but I get the following error:
sqlite3.OperationalError: near "(": syntax error
More specifically, it looks like this in ipython:
Yes, I know I should be parameterizing this and I have tried to do so, but keep running into this same snag.
I look at this: https://www.sqlite.org/lang_createview.html and it says that column lists are supported in 3.9 and later, so I updated to version 3.13 of sqlite3 and still ran into this issue.
When I remove the column list and use this:
import sqlite3
db_name = "../data/OsReportMerchants.sqlite"
conn = sqlite3.connect(db_name)
cur = conn.cursor()
cur.execute("CREATE VIEW IF NOT EXISTS monthly_terminals AS SELECT reportYear, reportMonth FROM osMonthlyTerminals")
it works fine. Not sure why it works in one context and not in the other.
Python has its own copy of the SQLite library (see sqlite3.sqlite_version).
If the latest Python version does not have a recent enough version of SQLite, the only way to update it would be to recompile Python, or use some other DB driver like APSW.
I have query which works in SQLite Spy (a tool for viewing SQLite Databases) but I can't get the query to work in python with peewee as a raw query.
The (simplified) query is
WITH tags(name) AS ( VALUES('foo'),('bar') ) SELECT * FROM tags
The (simplified) python code is:
from peewee import *
db = SqliteDatabase(":memory:")
db.execute_sql("WITH tags(name) AS ( VALUES('foo'),('bar') ) SELECT * FROM tags")
I get an error
peewee.OperationalError: near "WITH": syntax error
I also tried the RawQuery function or using SqliteExtDatabase from the PlayHouse extension.
Is this error caused by me or is it a problem of peewee?
CTE was introduced in SQLite as of version 3.8.3
http://www.sqlite.org/releaselog/3_8_3.html
it is very likely that your Python is using a previous version that does not support WITH
Check sqlite3.sqlite_version_info
https://docs.python.org/2/library/sqlite3.html
I have a very strange issue. I have a SQL statement that works locally, but the same statement does not work on a remote ubuntu machine (same mysql version). I think it has to do with how the SQL string is being encoded by the driver. Here is the statement I have:
group_ids = ('43ede7a1e1f048872c025867602dc54d', '43ede7a1e1f048872c025867602dc54d', '7a8ec12901c43606aee041f1e6d5b2d4', '0f57f4ad
cursor = connection.cursor()
cursor.execute( '''
SELECT
DISTINCT c.group_id
FROM
main_cue c
LEFT OUTER JOIN
main_passongroup p
ON
(c.group_id=p.group_id AND p.user_id=%s)
WHERE
c.group_id in %s
ORDER BY
p.timestamp ASC, c.id DESC''', (user.pk, group_ids))
results = cursor.fetchall()
print '>>> 1', results
On my local machine, it seems to be properly SQL-encoding the statement, but on the remote server, it is only working if I hard code the exact sql statement.
Is there a better way to encode the SQL statement?
This seemed to be an issue with an outdated version of MySQLdb, which was not properly encoding the string.
I was able to solve this by uninstalling MySQLdb and then reinstalling the newer version, (MySQL-python==1.2.5).
Is there a module available for connection of MSSQL and python 2.7?
I downloaded pymssql but it is for python 2.6. Is there any equivalent module for python 2.7?
I am not aware of it if anyone can provide links.
Important note: in the meantime there is a pymssql module available. Don't miss to read the answer at the end of this page: https://stackoverflow.com/a/25749269/362951
You can also use pyodbc to connect to MSSQL from Python.
An example from the documentation:
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=localhost;DATABASE=testdb;UID=me;PWD=pass')
cursor = cnxn.cursor()
cursor.execute("select user_id, user_name from users")
rows = cursor.fetchall()
for row in rows:
print row.user_id, row.user_name
The SQLAlchemy library (mentioned in another answer), uses pyodbc to connect to MSSQL databases (it tries various libraries, but pyodbc is the preferred one). Example code using sqlalchemy:
from sqlalchemy import create_engine
engine = create_engine("mssql://me:pass#localhost/testdb")
for row in engine.execute("select user_id, user_name from users"):
print row.user_id, row.user_name
If you're coming across this question through a web search, note that pymssql nowadays does support Python 2.7 (and 3.3) or newer. No need to use ODBC.
From the pymssql requirements:
Python 2.x: 2.6 or newer. Python 3.x: 3.3 or newer.
See http://pymssql.org/.
Install pyodbc using pip as follows: pip install pyodbc
import pyodbc
cnxn = pyodbc.connect("DRIVER={SQL Server};SERVER=SOME-PC;DATABASE=my_db")
cursor = cnxn.cursor()
cursor.execute("insert into test_tb values(6, 'name')")
cursor.execute("select id, name from my_tb")
rows = cursor.fetchall()
for row in rows:
print row.id, row.name
For details, see
https://github.com/mkleehammer/pyodbc/wiki
You can try out SQLAlchemy:
The SQLAlchemy Object Relational Mapper presents a method of associating user-defined Python classes with database tables, and instances of those classes (objects) with rows in their corresponding tables.
You can refer following links:
1> http://www.sqlalchemy.org/docs/
2> http://www.rmunn.com/sqlalchemy-tutorial/tutorial.html