We Keep Coding

Convert json query to insert a variable and re-convert it to json query

I am kinda frustrated. I copied the following Metabase query string from the network tab in the browser:
query = "{\"database\":17,\"query\":{\"source-table\":963,\"filter\":[\"and\",[\"=\",[\"field\",17580,null],\"XXXXXX_XXXXXX\"],[\"=\",[\"field\",17599,null],\"**chl-43d813dd-05a7-45b8-a5b0-8eb960289aa5**\"]],\"fields\":[[\"field\",17579,null],[\"field\",17569,null],[\"field\",17572,null],[\"field\",17586,null],[\"field\",17592,{\"temporal-unit\":\"default\"}],[\"field\",17611,null],[\"field\",17582,null],[\"field\",17597,null],[\"field\",17603,null],[\"field\",17607,null],[\"field\",17576,null],[\"field\",17588,null],[\"field\",17596,null],[\"field\",17608,null],[\"field\",17587,{\"temporal-unit\":\"default\"}],[\"field\",17578,{\"temporal-unit\":\"default\"}],[\"field\",17602,null],[\"field\",17606,null],[\"field\",17605,{\"temporal-unit\":\"default\"}],[\"field\",17601,null],[\"field\",17590,null],[\"field\",17580,null],[\"field\",17598,{\"temporal-unit\":\"default\"}],[\"field\",17577,null],[\"field\",164910,null],[\"field\",46951,null],[\"field\",46952,{\"temporal-unit\":\"default\"}]]},\"type\":\"query\",\"middleware\":{\"js-int-to-string?\":true,\"add-default-userland-constraints?\":true}}"
As the next step I wanted to convert it to a String to replace the bold reference with a variable.
The String looks like this:
query = '{"database\":17,\"query\":{\"source-table\":963,\"filter\":[\"and\",[\"=\",[\"field\",17580,null],\"XXXXXXXX-XXXXXXXX\"],[\"=\",[\"field\",17599,null],\"'+channelRef+'\"]],\"fields\":[[\"field\",17579,null],[\"field\",17569,null],[\"field\",17572,null],[\"field\",17586,null],[\"field\",17592,{\"temporal-unit\":\"default\"}],[\"field\",17611,null],[\"field\",17582,null],[\"field\",17597,null],[\"field\",17603,null],[\"field\",17607,null],[\"field\",17576,null],[\"field\",17588,null],[\"field\",17596,null],[\"field\",17608,null],[\"field\",17587,{\"temporal-unit\":\"default\"}],[\"field\",17578,{\"temporal-unit\":\"default\"}],[\"field\",17602,null],[\"field\",17606,null],[\"field\",17605,{\"temporal-unit\":\"default\"}],[\"field\",17601,null],[\"field\",17590,null],[\"field\",17580,null],[\"field\",17598,{\"temporal-unit\":\"default\"}],[\"field\",17577,null],[\"field\",164910,null],[\"field\",46951,null],[\"field\",46952,{\"temporal-unit\":\"default\"}]]},\"type\":\"query\",\"middleware\":{\"js-int-to-string?\":true,\"add-default-userland-constraints?\":true}}'
With
q = json.dumps(query)
the result looks exactly as I want to:
q = "{\"database\":17,\"query\":{\"source-table\":963,\"filter\":[\"and\",[\"=\",[\"field\",17580,null],\"XXXXXXXX-XXXXXXXX\"],[\"=\",[\"field\",17599,null],\"**chl-caabef81-f081-4532-9b6e-ac20b3d4c6cf**\"]],\"fields\":[[\"field\",17579,null],[\"field\",17569,null],[\"field\",17572,null],[\"field\",17586,null],[\"field\",17592,{\"temporal-unit\":\"default\"}],[\"field\",17611,null],[\"field\",17582,null],[\"field\",17597,null],[\"field\",17603,null],[\"field\",17607,null],[\"field\",17576,null],[\"field\",17588,null],[\"field\",17596,null],[\"field\",17608,null],[\"field\",17587,{\"temporal-unit\":\"default\"}],[\"field\",17578,{\"temporal-unit\":\"default\"}],[\"field\",17602,null],[\"field\",17606,null],[\"field\",17605,{\"temporal-unit\":\"default\"}],[\"field\",17601,null],[\"field\",17590,null],[\"field\",17580,null],[\"field\",17598,{\"temporal-unit\":\"default\"}],[\"field\",17577,null],[\"field\",164910,null],[\"field\",46951,null],[\"field\",46952,{\"temporal-unit\":\"default\"}]]},\"type\":\"query\",\"middleware\":{\"js-int-to-string?\":true,\"add-default-userland-constraints?\":true}}"
But when I use this query string to send an API request, I get the following error message(s):
{"via":[{"type":"java.lang.ClassCastException"}],"trace":[],"message":null}
Traceback (most recent call last):
File "c:\Users\XXXX\Documents\XXXXXXXX\Test.py", line 308, in
main()
File "c:\Users\XXXX\Documents\XXXXXXXX\Test.py", line 114, in main some_function(XXXX, window, selected_path)
File "c:\Users\XXXX\Documents\XXXXXXXX\Test.py", line 290, in some_function
dataframe = DataFrame(result)
File "C:\Users\XXXX\AppData\Roaming\Python\Python310\site-packages\pandas\core\frame.py", line 756, in init
raise ValueError("DataFrame constructor not properly called!")
ValueError: DataFrame constructor not properly called!
Does have anyone have an idea?
Thank you very much in advance!

You can use the built-in json module:
import json
query = "{\"database\":17,\"query\":{\"source-table\":963,\"filter\":[\"and\",[\"=\",[\"field\",17580,null],\"XXXXXX_XXXXXX\"],[\"=\",[\"field\",17599,null],\"**chl-43d813dd-05a7-45b8-a5b0-8eb960289aa5**\"]],\"fields\":[[\"field\",17579,null],[\"field\",17569,null],[\"field\",17572,null],[\"field\",17586,null],[\"field\",17592,{\"temporal-unit\":\"default\"}],[\"field\",17611,null],[\"field\",17582,null],[\"field\",17597,null],[\"field\",17603,null],[\"field\",17607,null],[\"field\",17576,null],[\"field\",17588,null],[\"field\",17596,null],[\"field\",17608,null],[\"field\",17587,{\"temporal-unit\":\"default\"}],[\"field\",17578,{\"temporal-unit\":\"default\"}],[\"field\",17602,null],[\"field\",17606,null],[\"field\",17605,{\"temporal-unit\":\"default\"}],[\"field\",17601,null],[\"field\",17590,null],[\"field\",17580,null],[\"field\",17598,{\"temporal-unit\":\"default\"}],[\"field\",17577,null],[\"field\",164910,null],[\"field\",46951,null],[\"field\",46952,{\"temporal-unit\":\"default\"}]]},\"type\":\"query\",\"middleware\":{\"js-int-to-string?\":true,\"add-default-userland-constraints?\":true}}"
my_json = json.loads(query)
# make edit's (works like a dict)
query = json.dumps(my_json)

I don't see a bold reference in your JSON string, but this is all handled with the json library:
import json
query = "YOUR QUERY STRING"
object = json.dumps(query)
# Make your changes to your dict object here
new_query = json.loads(object)

Django - Postgresql - JSON Field - #>> operator - Index of out range

In a django project which is using postgresql DB, there is a collection called 'table1' which has a JSON field called 'data'. In this JSON field, we will be storing an email against a dynamic key. For ex:
ID | DATA
1 | '{"0_email": "user1#mail.com"}'
2 | '{"3_email": "user2#mail.com"}'
3 | '{"1_email": "user3#mail.com"}'
Problem Statement:
Filter out the rows in which "user2#mail.com" exists in the "data" field.
My Approach:
from django.db import connection
#transaction.atomic
def run(given_email):
with connection.cursor() as crsr:
crsr.execute(
"""
DECLARE mycursor CURSOR FOR
SELECT id, data
FROM table1
WHERE
data #>> '{}' like '%\"%s\"%'
""",
[given_email]
)
while True:
crsr.execute("FETCH 10 FROM mycursor")
chunk = crsr.fetchall()
# DO SOME OPERATIONS...
Explanation for data #>> '{}' like '%\"%s\"%':
I am using the #>> operator to get object at specific path of JSON as text.
I am providing '{}' empty path so that I will get the complete JSON as a text.
From this stringified JSON, I am checking if the given_email (user2#mail.com from the above example) is present
Then, I have pointed this function to a API in django in which I will get the given_email in payload. I am facing the below error which triggering this function:
Traceback (most recent call last): File
"project/lib/python3.9/site-packages/django_extensions/management/debug_cursor.py",
line 49, in execute
return utils.CursorWrapper.execute(self, sql, params) File "project/lib/python3.9/site-packages/django/db/backends/utils.py",
line 67, in execute File
"project/lib/python3.9/site-packages/django/db/backends/utils.py",
line 76, in _execute_with_wrappers File
"project/lib/python3.9/site-packages/django/db/backends/utils.py",
line 87, in _execute
self.db.validate_no_broken_transaction() IndexError: tuple index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "", line 1, in
File
"project/lib/python3.9/site-packages/django_extensions/management/debug_cursor.py",
line 54, in execute
raw_sql = raw_sql[:truncate] TypeError: 'NoneType' object is not subscriptable
Observations:
I don't think there is anything wrong with the query. In fact, I have tried this query in DBeaver and I am getting the expected response.
I am suspecting that the '{}' part in the query is causing some issue. So, I have tried to replace it with '\\{\\}', '{{}}' but it didn't work.

First, '{}' is interpreted as a text whereas a json path is of type text[].
So you should cast it as text[] in your query : WHERE data #>> '{}' :: text[] like '%\"%s\"%'
Then data #>> '{}' :: text[] could be simplified as data :: text which will provide the same result.
Finally, you convert your json data into a text and then you use the pattern matching like operator. This solution may provide some unexpected results with some values which contains the expected email as a substring but which are not equal to that expected email.
In order to have the exact result for the expected email, you should use a jsonb function, for example :
WHERE jsonb_path_exists(data :: jsonb, ('$.* ? (# == "' || expected_email || '")') :: jsonpath)

How Can My sqllite3 interaction be fixed?

I'm trying to get an admin account to edit a 'rank' (basically access level) for one of the profiles in my data-base. The error is:
Traceback (most recent call last):
File "U:/A-level Computor Science/Y12-13/SQL/sqlite/Databases/ork task/Python for SQL V_2.py", line 154, in <module>
main()
File "U:/A-level Computor Science/Y12-13/SQL/sqlite/Databases/ork task/Python for SQL V_2.py", line 9, in main
start_menu()
File "U:/A-level Computor Science/Y12-13/SQL/sqlite/Databases/ork task/Python for SQL V_2.py", line 22, in start_menu
login()
File "U:/A-level Computor Science/Y12-13/SQL/sqlite/Databases/ork task/Python for SQL V_2.py", line 72, in login
Mek_menu()
File "U:/A-level Computor Science/Y12-13/SQL/sqlite/Databases/ork task/Python for SQL V_2.py", line 108, in Mek_menu
where Uzaname = %s""" % (NewRank, Findaname))
sqlite3.OperationalError: unrecognized token: "0rk_D4T4B453"`
The code that seems to be the problem is:
cursor.execute(""" update 0rk_D4T4B453.Da_Boyz
set Rank = %s
where Uzaname = %s""" % (NewRank, Findaname))
Originally, it was all on one line and it didn't work, and now I've tried it on multiple lines and it still doesn't work. So I checked here to see if anyone could help.
EDIT1: Thanks for the suggestions. None of them have fixed the code, but I've narrowed the problem code to: where Uzaname = %s""" % (NewRank, Findaname))

Unless you use ATTACH, SQLite (a file-level database) does not recognize other databases. Usually server-level databases (Oracle, Postgres, SQL Server, etc.) use the database.schema.table reference. However, in SQLite the very database file you connect to is the main database in scope. But ATTACH allows you to connect to other SQLite databases and then recognizes database.table referencing.
Additionally, for best practices:
In sqlite3 and any other Python DB-APIs, use parameterization for literal values and do not format values to SQL statement.
In general Python, stop using the de-emphasized (not deprecated yet) string modulo operator, %. Use str.format or more recent F-string for string formatting. But neither is needed here.
Altogether, if you connect to the 0rk_D4T4B453 database, simply query without database reference:
conn = sqlite3.connect('/path/to/0rk_D4T4B453.db')
cursor = conn.cursor()
# PREPARED STATEMENT WITH QMARK PLACEHOLDERS
sql = """UPDATE Da_Boyz
SET Rank = ?
WHERE Uzaname = ?"""
# BIND WITH TUPLE OF PARAMS IN SECOND ARG
cursor.execute(sql, (NewRank, Findaname))
conn.commit()
If you do connect to a different database, call ATTACH. Here also, you can alias other database with better naming instead of number leading identifier.
cursor.execute("ATTACH '/path/to/0rk_D4T4B453.db' AS other_db")
sql = """UPDATE other_db.Da_Boyz
SET Rank = ?
WHERE Uzaname = ?"""
cursor.execute(sql, (NewRank, Findaname))
conn.commit()
cur.execute("DETACH other_db")

I'm trying to use sql command on Python 3.51 and dbf 0.96.005 (for VFP). it have error

import dbf
table = dbf.Table('C:/test/MAS.DBF')
table.open()
records = table.sql("select * where SUPCOD = 1.YATHAI")
which gives:
Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
records = table.sql("select * where SUPCOD = 1.YATHAI")
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\dbf\ver_33.py", line 4637, in __getattr__
return object.__getattribute__(self, name)
AttributeError: 'VfpTable' object has no attribute '(supcod = column, 1.YATHAI = value in condition )
I need to make the program read data from vfp and insert into PostgreSQL.

For some examples of dbf and sql (called pql in the dbf module as it is not full sql syntax) check out this answer.
I suspect your immediate problem will by fixed by adding quotes around the value you want to match and doubling the equal sign (it's really Python syntax):
records = table.sql("select * where SUPCOD == '1.YATHAI'")

mysqldb interfaceError

I have a very weird problem with mysqldb (mysql module for python).
I have a file with queries for inserting records in tables. If I call the functions from the file, it works just fine; but when trying to call one of the functions from another file it throws me a
_mysql_exception.InterfaceError: (0, '')
I really don't get what I'm doing wrong here..
I call the function from buildDB.py :
import create
create.newFormat("HD", 0,0,0)
The function newFormat(..) is in create.py (imported) :
from Database import Database
db = Database()
def newFormat(name, width=0, height=0, fps=0):
format_query = "INSERT INTO Format (form_name, form_width, form_height, form_fps) VALUES ('"+name+"',"+str(width)+","+str(height)+","+str(fps)+");"
db.execute(format_query)
And the class Database is the following :
import MySQLdb
from MySQLdb.constants import FIELD_TYPE
class Database():
def __init__(self):
server = "localhost"
login = "seq"
password = "seqmanager"
database = "Sequence"
my_conv = { FIELD_TYPE.LONG: int }
self.conn = MySQLdb.connection(host=server, user=login, passwd=password, db=database, conv=my_conv)
# self.cursor = self.conn.cursor()
def close(self):
self.conn.close()
def execute(self, query):
self.conn.query(query)
(I put only relevant code)
Traceback :
Z:\sequenceManager\mysql>python buildDB.py
D:\ProgramFiles\Python26\lib\site-packages\MySQLdb\__init__.py:34: DeprecationWa
rning: the sets module is deprecated
from sets import ImmutableSet
INSERT INTO Format (form_name, form_width, form_height, form_fps) VALUES ('HD',0
,0,0);
Traceback (most recent call last):
File "buildDB.py", line 182, in <module>
create.newFormat("HD")
File "Z:\sequenceManager\mysql\create.py", line 52, in newFormat
db.execute(format_query)
File "Z:\sequenceManager\mysql\Database.py", line 19, in execute
self.conn.query(query)
_mysql_exceptions.InterfaceError: (0, '')
The warning has never been a problem before so I don't think it's related.

I got this error when I was trying to use a closed connection.

Problem resolved.. I was initializing the database twice.. Sorry if you lost your time reading this !

I couldn't get your setup to work. I gives me the same error all the time. However the way you connect to and make queries to the db with the query seems to be "non-standard".
I had better luck with this setup:
conn = MySQLdb.Connection(user="user", passwd="******",
db="somedb", host="localhost")
cur = conn.cursor()
cur.execute("insert into Format values (%s,%s,%s,%s);", ("hd",0,0,0))
This way you can take advantage of the db modules input escaping which is a must to mitigate sql injection attacks.