We Keep Coding

Django - Postgresql - JSON Field - #>> operator - Index of out range

In a django project which is using postgresql DB, there is a collection called 'table1' which has a JSON field called 'data'. In this JSON field, we will be storing an email against a dynamic key. For ex:
ID | DATA
1 | '{"0_email": "user1#mail.com"}'
2 | '{"3_email": "user2#mail.com"}'
3 | '{"1_email": "user3#mail.com"}'
Problem Statement:
Filter out the rows in which "user2#mail.com" exists in the "data" field.
My Approach:
from django.db import connection
#transaction.atomic
def run(given_email):
with connection.cursor() as crsr:
crsr.execute(
"""
DECLARE mycursor CURSOR FOR
SELECT id, data
FROM table1
WHERE
data #>> '{}' like '%\"%s\"%'
""",
[given_email]
)
while True:
crsr.execute("FETCH 10 FROM mycursor")
chunk = crsr.fetchall()
# DO SOME OPERATIONS...
Explanation for data #>> '{}' like '%\"%s\"%':
I am using the #>> operator to get object at specific path of JSON as text.
I am providing '{}' empty path so that I will get the complete JSON as a text.
From this stringified JSON, I am checking if the given_email (user2#mail.com from the above example) is present
Then, I have pointed this function to a API in django in which I will get the given_email in payload. I am facing the below error which triggering this function:
Traceback (most recent call last): File
"project/lib/python3.9/site-packages/django_extensions/management/debug_cursor.py",
line 49, in execute
return utils.CursorWrapper.execute(self, sql, params) File "project/lib/python3.9/site-packages/django/db/backends/utils.py",
line 67, in execute File
"project/lib/python3.9/site-packages/django/db/backends/utils.py",
line 76, in _execute_with_wrappers File
"project/lib/python3.9/site-packages/django/db/backends/utils.py",
line 87, in _execute
self.db.validate_no_broken_transaction() IndexError: tuple index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "", line 1, in
File
"project/lib/python3.9/site-packages/django_extensions/management/debug_cursor.py",
line 54, in execute
raw_sql = raw_sql[:truncate] TypeError: 'NoneType' object is not subscriptable
Observations:
I don't think there is anything wrong with the query. In fact, I have tried this query in DBeaver and I am getting the expected response.
I am suspecting that the '{}' part in the query is causing some issue. So, I have tried to replace it with '\\{\\}', '{{}}' but it didn't work.

First, '{}' is interpreted as a text whereas a json path is of type text[].
So you should cast it as text[] in your query : WHERE data #>> '{}' :: text[] like '%\"%s\"%'
Then data #>> '{}' :: text[] could be simplified as data :: text which will provide the same result.
Finally, you convert your json data into a text and then you use the pattern matching like operator. This solution may provide some unexpected results with some values which contains the expected email as a substring but which are not equal to that expected email.
In order to have the exact result for the expected email, you should use a jsonb function, for example :
WHERE jsonb_path_exists(data :: jsonb, ('$.* ? (# == "' || expected_email || '")') :: jsonpath)

psycopg2 fetchone() method returning a single-element tuple containing a string representing the «result» tuple

I am trying to write chess-oriented functionalities for a discord bot.
One of these is to being able to play correspondence games.
I am using a postgresql database to store every games.
The problem i have is that when i call cursor.fetchone() after the execution of a SELECT sql request, the returned object is a single-element tuple containing a string that represents the wanted tuple.
For instance :
('(351817698172207105,"",1)',) instead of (351817698172207105,"",1)
I installed psycopg2 with pip3 (and using it with python 3.6.7).
I got round that problem by using ast.literal_eval in the first SELECT request (creating and accepting a challenge)
But the other request contains the PGN of the game, which contains many quotation marks, and that makes the literal_eval function fail.
I could mess with the returned str, but i'm not sure it is the best option (and i'd like to understand why it is like this).
The first «get-around» :
from ast import literal_eval as make_tuple
cdesc = psycopg2.connect(**params_db)
curs = cdesc.cursor()
modele_req = "SELECT (id_j1, id_j2, id_challenge) FROM challenges WHERE (id_j2='{0}' OR id_j2 = '') AND id_challenge = {1};"
# id_acceptant and id_partie_acceptee are given
req = modele_req.format(id_acceptant, str(id_partie_acceptee))
res_tuple = curs.fetchone()
# print(res_tuple) produces the following output :
# ('(351817698172207105,"",1)',)
(idj1, idj2, idpartie) = make_tuple(res_tuple[0])
Also, the following is an example of the returned tuple in the "second" request :
( '("[Event ""?""]\n[Site ""?""]\n[Date ""????.??.??""]\n[Round ""?""]\n[White ""?""]\n[Black ""?""]\n[Result ""*""]\n\n*",351817698172207105,351817698172207105,t)' ,)
When using the make_tuple/literal_eval function, i (obviously) have the following error:
(...)
File "/home/synophride/projets/discord_bot/bot/commandes_echecs.py", line 568, in move_bd
(game_pgn, id_blanc, id_noir, joueur_jouant) = make_tuple(str_tuple)
File "/usr/lib/python3.6/ast.py", line 85, in literal_eval
return _convert(node_or_string)
File "/usr/lib/python3.6/ast.py", line 59, in _convert
return tuple(map(_convert, node.elts))
File "/usr/lib/python3.6/ast.py", line 84, in _convert
raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Name object at 0x7fe1956f5978>
In short :
Is it normal that the cursor.fetchone() method returns that kind of tuple ?
If no, what did i do wrong and what can i do to rectify that ?
Thanks for reading, and sorry for my eventually dubious english.

When you are using brackets in your query, postgres doesnt return the columns (they are stacked into one record).
Your query should look like this, to get all columns:
modele_req = "SELECT id_j1, id_j2, id_challenge FROM challenges WHERE ..."
Then psycopg2 will return a tuple of 3 items, that is not represented as a string:
((351817698172207105,"",1),)

not enough arguments for format string python mysql

I have a trouble with my program. I want input database from file txt. This is my source code
import MySQLdb
import csv
db=MySQLdb.connect(user='root',passwd='toor',
host='127.0.0.1',db='data')
cursor=db.cursor()
csv_data=csv.reader(file('test.txt'))
for row in csv_data:
sql = "insert into `name` (`id`,`Name`,`PoB`,`DoB`) values(%s,%s,%s,%s);"
cursor.execute(sql,row)
db.commit()
cursor.close()
After run that program, here the error
Traceback (most recent call last):
File "zzz.py", line 9, in <module>
cursor.execute(sql,row)
File "/home/tux/.local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 187, in execute
query = query % tuple([db.literal(item) for item in args])
TypeError: not enough arguments for format string
and this is my test.txt
4
zzzz
sby
2017-10-10
Please help, and thanks in advance.

Now that you have posted the CSV file, the error should now be obvious to you - each line contains only one field, not the four that the SQL statement requires.
If that is the real format of your data file, it is not CSV data. Instead you need to read each group of four lines as one record, something like this might work:
LINES_PER_RECORD = 4
SQL = 'insert into `name` (`id`,`Name`,`PoB`,`DoB`) values (%s,%s,%s,%s)'
with open('test.txt') as f:
while True:
try:
record = [next(f).strip() for i in range(LINES_PER_RECORD)]
cursor.execute(SQL, record)
except StopIteration:
# insufficient lines available for record, treat as end of file
break
db.commit()

python sqlite3 cursor.execute() with parameters leads to syntax error near ? (paramstyle qmark)

after searching untill madness, i decided to post a question here.
I try to create a sqlite3 database where i'd like to make use of the secure variable substituation function of the cursor.execute(SQL, param) function. My function goes like this:
#!/usr/bin/env python3
#-*- coding: utf-8 -*-
import sqlite3
def create():
values = ("data")
sql = "CREATE TABLE IF NOT EXISTS ? ( name TEXT, street TEXT, time REAL, age INTEGER )"
con = sqlite3.connect("database.db")
c = con.cursor()
c.execute(sql, values)
con.commit()
c.close()
con.close()
if __name__ = "__main__":
create()
I know that the first argument should be the sql command in form of a string and the second argument must be a tuple of the values which are supposed to be substituted where the ? is in the sql string.
However, when i run the file it returns the following error:
$ ./test.py
Traceback (most recent call last):
File "./test.py", line 21, in <module>
create()
File "./test.py", line 14, in create
c.execute(sql, values)
sqlite3.OperationalError: near "?": syntax error
This also happens when paramstyle is set to named (e.g. the :table form).
I can't spot a syntax error here, so i think that the problem must be caused somewhere in the system. I tested it on an Archlinux and Debian install, both post me the same error.
Now it is up yo you, as I have no idea anymore where to look for the cause.

SQL parameters can only apply to insert data, not table names. That means parameters are not even parsed for DDL statements.
For that you'll have to use string formatting:
sql = "CREATE TABLE IF NOT EXISTS {} ( name TEXT, street TEXT, time REAL, age INTEGER )".format(*values)

As I understand, your parameter is the table name?
so your command would be
tbl = 'my_table'
sql = "CREATE TABLE IF NOT EXISTS '%s' ( name TEXT, street TEXT, time REAL, age INTEGER )" % tbl

could not determine data type of parameter $1 in python-pgsql

I have a simple table (named test) as:
id | integer
name | character varying(100)
intval | integer
When I try to use prepare statement to update the name like this in python. (I am using python-pgsql http://pypi.python.org/pypi/python-pgsql/)
>>> for i in db.execute("select * from test"): print i
...
(1, 'FOO', None)
>>> query = "UPDATE test set name = '$1' where name = '$2'"
>>> cu.execute(query, "myname", "FOO")
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/pgsql.py", line 119, in execute
ret = self._source.execute(operation, params)
ProgrammingError: ERROR: could not determine data type of parameter $1
The demo file for the module can be seen at http://sprunge.us/VgLY?python.

I'm guessing that it could be your single quotes around $1 and $2 inside your string. In the main changes from PYGresql it says that:
support for bind parameters, alleviating the need for extensive,
expensive and vulnerable quoting of user-supplied data
So I'm assuming the single quotes are overloading the string with too many quotes, or just breaking the parser in python-pgsql.