We Keep Coding

Python - how to convert a json.loads variable to list and append

Basically I need to get the json content of a file, replace a variable and then append it with a new dictionary. I've tried in every way and searched but I can't find a solution. *The first line exemplifies the data that comes from the .json file
liststr = [{"type": "divider"},{"type": "section","text": {"type": "mrkdwn","text": "Olá {user} :wave:\n\n Segue abaixo a listagem de colaboradores terceiro sob sua gestão e a data de expiração da conta. \n\nA data de expiração máxima para a conta é de 30 dias. Caso o período de renovação exceda esse limite, a data será fixada em D+30. "}}]
liststr = str(liststr)
liststr = liststr.replace("{user}", "Bruno")
listlist = json.loads(json.dumps(liststr))
print(listlist)
Even that part works fine. The issue is that as I said, I need to append with one more information, for example:
liststr = [{"type": "divider"},{"type": "section","text": {"type": "mrkdwn","text": "Olá {user} :wave:\n\n Segue abaixo a listagem de colaboradores terceiro sob sua gestão e a data de expiração da conta. \n\nA data de expiração máxima para a conta é de 30 dias. Caso o período de renovação exceda esse limite, a data será fixada em D+30. "}}]
liststr = str(liststr)
liststr = liststr.replace("{user}", "Bruno")
listlist = json.loads(json.dumps(liststr))
listlist.append({
"user": "bruno"
})
print(listlist)
I get the error message
"AttributeError: 'str' object has no attribute 'append'"
because it considers the json.loads variable to be str instead of list. I've tried converting it to a list but it doesn't work either.

def replace(v, needle, replacement):
if isinstance(v, str):
return v.replace(needle, replacement)
if isinstance(v, list):
return [replace(i, needle, replacement) for i in v]
if isinstance(v, dict):
return {k: replace(i, needle, replacement) for k, i in v.items()}
return v
l = [{"type": "divider"},{"type": "section","text": {"type": "mrkdwn","text": "Olá {user} :wave:\n\n Segue abaixo a listagem de colaboradores terceiro sob sua gestão e a data de expiração da conta. \n\nA data de expiração máxima para a conta é de 30 dias. Caso o período de renovação exceda esse limite, a data será fixada em D+30. "}}]
l = replace(l, '{user}', 'buno')
This is what you're trying to do. Recursively iterate over all values in the list of dicts, and replace the string "{user}" in any of them, all the while keeping everything as a list of dicts and not turning everything into a string.

Not certain what the logic is here so making an assumption that any dictionary with a key of 'text' whose value is a string may need modification. If that is the case then:
list_ = [{"type": "divider"},{"type": "section","text": {"type": "mrkdwn","text": "Olá {user} :wave:\n\n Segue abaixo a listagem de colaboradores terceiro sob sua gestão e a data de expiração da conta. \n\nA data de expiração máxima para a conta é de 30 dias. Caso o período de renovação exceda esse limite, a data será fixada em D+30. "}}]
def process(d):
for k, v in d.items():
if isinstance(v, dict):
process(v)
else:
if isinstance(v, str) and k == 'text':
d['text'] = v.replace('{user}', 'Bruno')
break
for d in list_:
process(d)
print(list_)

JSON string Replace single quote with two single quotes in Python

I'm using Python to grab a JSON response. I have a json string that has fields that contain a single quote inside of the data. I'm trying to replace a single escaped quote ' with two single quotes so that I can insert it into SQL server. The problem is, when I use:
json_str = json_string.replace("\'","''")
it is changing it to ''.
How can I massage this data so that I can get it into SQL server?
Code snippet:
import pyodbc
import json
import requests
import csv
import datetime as DT
from datetime import datetime
import pytz
from pytz import timezone
# Dates / Times
today = DT.date.today()
week_ago = today - DT.timedelta(days=2)
# Settings
auth = ''
view_tickets = []
# List tickets from a View
print(f'Getting tickets...')
url = f'https:... created>{week_ago}'
headers = {'Content-Type': 'application/json'}
while url:
response = requests.get(url, auth=auth, headers=headers)
page_data = response.json()
# convert json to string
json_string = json.dumps(page_data)
# remove backslashes that screw up SQL import
json_string = json_string.encode().decode('unicode_escape')
#json_str = json_string.replace("\'","''")
# SQL EXPRESS INSERT Process
conn=pyodbc.connect('Driver={SQL Server};Server=NY-.\SQLEXPRESS;Database=Metrics;Trusted_Connection=yes;')
conn.timeout = 60
conn.autocommit = True
# Call SQL and trap Error if raised
try:
cursor = conn.cursor()
# SQL string
sql_declare = "DECLARE #json NVARCHAR(MAX) = N'" + json_str + "'"
sql_stat = "INSERT INTO Metrics.dbo.Tickets SELECT * FROM OPENJSON(#json, '$.results') WITH (id INTEGER '$.id', subject VARCHAR(255) '$.subject')"
sql = sql_declare + " " + sql_stat
cursor.execute(sql)
print('Connected to database')
except pyodbc.Error as err:
print('Error !!!!! %s' % err)
except:
print('something else failed')
conn.close()
print('closed db connection')
I have tried string.replace but that is not working.
Sample String:
'{"results": [{"subject": "Golf International de Longwy & Golf de Thionville Rive Droite - Add course request", "raw_subject": "Golf International de Longwy & Golf de Thionville Rive Droite - Add course request", "description": "Name GILBERT RAVAGLI\nEmail xxxxx\nPhone 00352781695321 france\n\nMessage\n\nBonjour J'ai acheter une montre Les deux golf qui sont dans mon secteur n'apparaissent pas sur la montre Le golf internationnal de Longwy et le golf de Thionville rive droite France Pouvez vous les cartographier que la montre puisse me servir? Cordialement"}], "facets": null, "next_page": null, "previous_page": null, "count": 72}'
SQL Code:
DECLARE #json NVARCHAR(MAX)
SET #json = N'{"results": [{"subject": "Golf International de Longwy & Golf de Thionville Rive Droite - Add course request", "raw_subject": "Golf International de Longwy & Golf de Thionville Rive Droite - Add course request", "description": "Name GILBERT RAVAGLI\nEmail xxxxx\nPhone 00352781695321 france\n\nMessage\n\nBonjour J'ai acheter une montre IZZO SWAMI Les deux golf qui sont dans mon secteur n'apparaissent pas sur la montre Le golf internationnal de Longwy et le golf de Thionville rive droite France Pouvez vous les cartographier que la montre puisse me servir? Cordialement"}], "facets": null, "next_page": null, "previous_page": null, "count": 72}'
SELECT * FROM OPENJSON (#json)
WITH (
subject VARCHAR(250) '$.subject'
)

Shouldn't all of this:
sql_declare = "DECLARE #json NVARCHAR(MAX) = N'" + json_str + "'"
sql_stat = "INSERT ... FROM OPENJSON(#json, '$.results') ..."
sql = sql_declare + " " + sql_stat
cursor.execute(sql)
Just be:
sql = "INSERT ... FROM OPENJSON(?, '$.results') ...);"
cursor.execute(sql, json_string)
Now you're passing in a parameter and don't have to deal with local variables in T-SQL, string delimiters, single quotes, etc.

Concat fields using apache beam

I have following function. So I would like to take 2 fields and concatenate them but when my pipeline finishes it doesn´t work
the pipeline finishes as correct but when I see in bigquery the fields have not been concatenated
It will be great if u can help me.
it´s the code used in the function:
import apache_beam as beam
from ..tools import ProcessLogger
_logger = ProcessLogger()
class ConcatFieldsFn(beam.DoFn):
"""Concatena los valores de los campos especificados en una pCollection por un valor especificado"""
def __init__(self, process: str, data_name: str, parameters: dict):
# Configuracion del logger
self.logger_data_name = data_name
self.logger_process = process
self.logger_subprocess = "Concatenar valores"
_logger.data_name = self.logger_data_name
_logger.process = self.logger_process
_logger.subprocess = self.logger_subprocess
# Parametros del proceso
self._fields = [field.get("name") for field in parameters.get("fields", None)]
_logger.info(
f"Se aplica regla: {_logger.subprocess} con los parametros: {parameters}"
)
def process(self, element):
# Configuracion del logger
_logger.data_name = self.logger_data_name
_logger.process = self.logger_process
_logger.subprocess = self.logger_subprocess
for field in self._fields:
if element[field] != None:
try:
element[field] = "|".join(element[field])
except Exception as ex:
_logger.error(
f"No se pueden eliminar las letras de los campos seleccionados: {ex}"
)
return [element]

A query in SQLite3 with python

I'm testing a query with python and sqlite3. First method works fine, but
second is not working. It is about the defined type of variable containing the resgisters in DB:
import sqlite3
def insertar():
db1=sqlite3.connect('tabla.db')
print("Estas en la funcion insertar")
nombre1=raw_input("Escribe el titulo de la novela: ")
autor1=raw_input("Escribe el autor de la novela: ")
year1=str(input("Digita el any de la novela: "))
consulta=db1.cursor()
strConsulta = "insert into tabla(nombre, autor, year) values\
('"+nombre1+"','"+autor1+"','"+year1+"')"
print(strConsulta)
consulta.execute(strConsulta)
consulta.close()
db1.commit()
db1.close()
def consultar():
db2 = sqlite3.connect("tabla.db")
print("Estas en la funcion insertar")
db2row_factory = sqlite3.Row
consulta = db2.cursor()
consulta.execute("select * from tabla")
filas = consulta.fetchall()
lista = []
for fila in filas:
s = {}
s['nombre'] = fila['nombre']
s['autor'] = fila['autor']
s['year'] = str(fila['year'])
lista.append(s)
consulta.close()
db2.close()
return(lista)
#consultar()
def menu():
Opcion= input("\nIngresa la opcion deseada\n1.Inserta un valor en la tabla\n2.Consultar los valores de la tabla\n")
if Opcion==1:
insertar()
menu()
elif Opcion==2:
ListaNovelas = consultar()
for novela in ListaNovelas:
print(novela['nombre'],novela['autor'],novela['year'])
menu()
menu()
I get this error while testing the second method consultar().
$ python file.py
Ingresa la opcion deseada
1.Inserta un valor en la tabla
2.Consultar los valores de la tabla
2
Estas en la funcion insertar
Traceback (most recent call last):
File "insertar.py", line 56, in <module>
menu()
File "insertar.py", line 51, in menu
ListaNovelas = consultar()
File "insertar.py", line 33, in consultar
s['nombre'] = fila['nombre']
TypeError: tuple indices must be integers, not str

db2row_factory = sqlite3.Row
This is the problematic line. Instead you meant to set the row_factory factory on the db2 connection instance:
db2.row_factory = sqlite3.Row
Then, all the fetched rows would be now sqlite3.Row instances having dictionary-like access to field values.

Python, database results are saved with ('result')

I select some urls from my database using a simple code
def crearArchivo(self):
#Se conecta a la base de datos
db = MySQLdb.connect("localhost","root","","proyectoacademias" )
cursor = db.cursor()
#Selecciona la lista de valores que no hayan sido procesados
sql = "SELECT url_paper_web FROM acw_papers_web WHERE finalizado = 0"
cursor.execute(sql)
#Recibe todos los datos
datos = cursor.fetchall()
#Crea el archivo
archivo = open('urls.txt','w')
archivo.close()
#Lo abre
archivo = open('urls.txt','w')
#Establece un contador para determinar el numero de filas
contador=0;
#Para cada fila la guarda en el archivo y si no es la ultima agrega un salto de linea
for columna in datos:
contador+=1;
archivo.write(str(columna))
if(contador!=len(datos)):
archivo.write("\n")
#Se cierra todo
archivo.close()
db.close()
This method places those links in a file... the problem is that those links instead of being saved like http:// google.com they are saved like ('http://google.com')

Each columna variable is actualy a row, not a single column. Simply select the first value in that row:
archivo.write(columna[0])

That is because columna is a tuple. It looks like you want to write the first (and in this case, the only element):
archivo.write(columna[0])