We Keep Coding

Delete rows that contains no information on Tweet text on pandas

I´m trying to remove rows containing blank texts or in tweet texts column. But I have tried in different ways counting the rows that only contain whitespace or counting the leading spaces and trailing spaces but to get a criterion to eliminate it.
ID tweet WhiteSpaceCount HaveWhiteSpace
0 this is a text 0 False
1 0 False
2 Hello im fine 0 False
I want to delete all the rows that don´t have any information on the tweet column.
Code here:
def extractAndSave(api, name):
# Creamos una lista de tweets:
previous_date = date.today() - timedelta(days=1)
query_date = date.today()
name = name
tweets = API_EXTRACTOR.search(q=name + "-filter:retweets", result_type='recent', timeout=999999, count=200,
end_time=previous_date, tweet_mode='extended')
# Podemos crear un dataframe como sigue:
tweet_list = []
for tweet in tweets:
tweet_list.append(tweet.full_text)
datos = pd.DataFrame(data=tweet_list, columns=['TWEETS'])
# CREANDO COLUMNA DE ID
id_list = []
for id in tweets:
id_list.append(id.id)
id = pd.DataFrame(data=id_list, columns=['ID'])
# CREANDO COLUMNA DE ID
creado_list = []
for creado in tweets:
creado_list.append(creado.created_at)
creado = pd.DataFrame(data=creado_list, columns=['FECHA_CREACION'])
# CREANDO COLUMNA DE nombre de usuario
user_list = []
for usuario in tweets:
user_list.append(usuario.user.screen_name)
usuario = pd.DataFrame(data=user_list, columns=['USUARIO'])
# CREANDO COLUMNA DE FUENTE
fuente_list = []
for fuente in tweets:
fuente_list.append(fuente.source)
fuente = pd.DataFrame(data=fuente_list, columns=['FUENTE'])
# CREANDO COLUMNA DE ME GUSTA
like_list = []
for like in tweets:
like_list.append(like.favorite_count)
like = pd.DataFrame(data=like_list, columns=['ME_GUSTA'])
# CREANDO COLUMNA DE RT
rt_list = []
for rt in tweets:
rt_list.append(rt.retweet_count)
retweet = pd.DataFrame(data=rt_list, columns=['ME_GUSTA'])
# CREANDO COLUMNA DE IDIOMA
idioma_list = []
for idioma in tweets:
idioma_list.append(idioma.lang)
idioma = pd.DataFrame(data=idioma_list, columns=['IDIOMA'])
# CREANDO COLUMNA DE IDIOMA
quote_list = []
for quote in tweets:
quote_list.append(quote.is_quote_status)
quote = pd.DataFrame(data=quote_list, columns=['CITADO'])
# CREANDO COLUMNA DE IDIOMA
location_list = []
for location in tweets:
location_list.append(location.user.location)
location = pd.DataFrame(data=location_list, columns=['LOCACION'])
# CONCATENANDO DATAFRAMES
datos = pd.concat([datos, id, creado, usuario, fuente, like, retweet, quote, idioma, location], axis=1)
# Dropear toda la fila si la columna tweets viene vacia.
datos['pass/fail'] = np.where(datos['TWEETS'].astype(str).str.fullmatch(r"\s*"),'FAIL','PASS')
datos['CONTEO_ESPACIOS']= (datos['TWEETS'].str.startswith(" ") | datos['TWEETS'].str.endswith(" ")).sum()
# Hora de publicación
datos['HORA_PUBLICACION'] = datos['FECHA_CREACION'].dt.hour
datos['DIA_SEMANA'] = datos['FECHA_CREACION'].dt.day_name()
# Extrayendo solo los tweets del día anterior
datos['FECHA_CREACION'] = pd.to_datetime(datos['FECHA_CREACION']).dt.date
datos = datos[datos['FECHA_CREACION'] == previous_date]
print(datos)
# Guardando en dataframe.
return datos

Instead of removing rows that you don't need, keep only the ones you do need:
df = df[df["tweet"].str.strip().str.len()>0]
>>> df
ID tweet WhiteSpaceCount HaveWhiteSpace
0 0 this is a text 0 False
2 2 Hello im fine 0 False

Issue with SQLite query between 2 dates

I'm working on a Python project which use SQlite3 database.
I created database with only one table called "Message" with this kind of data:
connexion = sqlite3.connect(BDD)
c = connexion.cursor()
c.execute(f""" CREATE TABLE IF NOT EXISTS {arg}(
Id INTEGER PRIMARY KEY AUTOINCREMENT,
Jour_Heure_Reception text,
Jour_Heure_Reponse text,
Theme text,
Motif text,
Risque_incident_client text,
Transfert_sans_action text,
Matricule text,
Origine text)""")
connexion.commit()
connexion.close();
My goal is to search between 2 dates in the Jour_Heure_Reponse column, and return the number of entries by matricule.
To do that, I use this SQlite query:
def nbr(arg):
"requette dans la bdd de statistiques mails retournant le nombre de messages par utilisateur sur la période arg"
#Création de la liste des utilisateurs ayant saisies des entrées dans la BDD sur la période
connexion = sqlite3.connect(BDD)
c = connexion.cursor()
date_selection = str((datetime.now() - timedelta(arg)).strftime('%d/%m/%Y'))
yesterday = str(datetime.now().strftime('%d/%m/%Y'))
c.execute(f"""
select Matricule from Message Where
Jour_Heure_Reception >= "{date_selection}"
and Jour_Heure_Reponse < "{yesterday}" """ )
agents = c.fetchall()
liste_agents = []
for i in agents:
if not i[0] in liste_agents:
liste_agents.append(i[0])
c.close()
# calcul du nombre d'entrées pour chaque matricules présent dans la liste crée précédemment
connexion = sqlite3.connect(BDD)
c = connexion.cursor()
liste_affichage = []
for i in liste_agents:
c.execute(f"""SELECT * FROM Message where
Matricule = "{i}" and
Jour_Heure_Reception >= "{date_selection}" and
Jour_Heure_Reponse < "{yesterday}" """)
test = c.fetchall()
print(i)
for i in test:
print(i[1])
data_list = [str(i),str(len(test))]
liste_affichage.append(data_list)
c.close()
The problème is that one:
When i call mi nbr func, it return nothing if mi arg is not 1, and even with 1 arg, the result is not logical.
for exemple, calling nrb(1), return this ( i only print dates ):
02/06/2020
02/06/2020
02/06/2020
02/06/2020
02/06/2020
02/06/2020
02/06/2020
03/06/2020
03/06/2020
03/06/2020
03/06/2020
02/07/2020
02/07/2020
02/07/2020
03/07/2020
03/08/2020
02/09/2020
03/09/2020
03/09/2020
09/08/2020
09/08/2020
09/08/2020
09/08/2020
09/08/2020
01/10/2020
02/10/2020
02/10/2020
As you can see, the timedelta is not respected.
As the datatype is stored as Text, I send dates as str after a time.strftime() conversion.
Where am I going wrong?

Sqlite does not have a dedicated date type. It supports storing date using a text field in ISO-8601 format. You will have to do date format conversion, so that you can perform your comparison. For the format details and documentation see this:
https://www.sqlite.org/lang_datefunc.html
You should use %Y-%m-%d as format string.

UPDATE mysql error with Python

Here's my situation:
I got a table on a database, as following:
nome_imagem estado type
57260-tracker-_tracker_face awake 0
57261-tracker-_tracker_face drowsiness 1
57268-tracker-_tracker_face noface 2
57289-tracker-_tracker_face distracted 3
57290-tracker-_tracker_face awake 1
57291-tracker-_tracker_face drowsiness 2
57293-tracker-_tracker_face noface 3
And I want to update de type column according to serveral if conditions, but I'm getting a error on line 29:
mysql.connector.errors.InternalError: Unread result found
I'm pretty sure its the way im making the query, and I already searched another questions about this before opening this one, but I couldnt solve it anyway.
EDIT 1: Changed the query and applied the solution gave by gzc to get rid of the error, but now it updates all the type column instead of just the cases where the if is true
import mysql.connector
from mysql.connector import errorcode
import os
cnx = mysql.connector.connect(user='root', database='empresa')
cursor = cnx.cursor()
fileDir = os.path.dirname(os.path.realpath(__file__))
textDir = os.path.join(fileDir, "test_img")
query = ("SELECT nome_imagem, estado, type FROM alertas ")
cursor.execute(query)
results = list(cursor)
for (nome_imagem, estado, type) in results:
print nome_imagem, estado
my_file_name = nome_imagem+'.txt'
my_file = open("test_img/"+my_file_name, 'r')
content = my_file.readline()
status = content.strip().split()[-1].split("=")[1]
face = content.strip().split()[0].split("=")[1]
print status, face #1 tem face, 0 nao tem
if (face == '1' and estado == status): #se tem cara e o estado que tem na bd for igual ao estado que o programa classificou = correto
print "correto"
cursor.execute("UPDATE alertas SET type='1' WHERE nome_imagem=nome_imagem")
if (face == '1' and estado == 'drowsiness' and status == 'awake') or (face == 1 and estado == 'awake' and status == 'drowsiness'): #verificar isto
print "trocado"
if (estado != '' and face == '0'): # se tiver estado mas nao tiver cara classifico logo como errado 3
print "errado"
if (estado == 'distracted' and face == '1'): # se tem cara mas for distracted deixo normal pois nao consigo classificar
print "normal"
cursor.close()
cnx.close()
What am I doing wrong?
Thanks
EDIT 2: gzc solver it again :)

You must read all results before doing another operation on the same cursor.
...
query = ("SELECT nome_imagem, estado, type FROM alertas ")
cursor.execute(query)
results = list(cursor)
for (nome_imagem, estado, type) in results:
...

A query in SQLite3 with python

I'm testing a query with python and sqlite3. First method works fine, but
second is not working. It is about the defined type of variable containing the resgisters in DB:
import sqlite3
def insertar():
db1=sqlite3.connect('tabla.db')
print("Estas en la funcion insertar")
nombre1=raw_input("Escribe el titulo de la novela: ")
autor1=raw_input("Escribe el autor de la novela: ")
year1=str(input("Digita el any de la novela: "))
consulta=db1.cursor()
strConsulta = "insert into tabla(nombre, autor, year) values\
('"+nombre1+"','"+autor1+"','"+year1+"')"
print(strConsulta)
consulta.execute(strConsulta)
consulta.close()
db1.commit()
db1.close()
def consultar():
db2 = sqlite3.connect("tabla.db")
print("Estas en la funcion insertar")
db2row_factory = sqlite3.Row
consulta = db2.cursor()
consulta.execute("select * from tabla")
filas = consulta.fetchall()
lista = []
for fila in filas:
s = {}
s['nombre'] = fila['nombre']
s['autor'] = fila['autor']
s['year'] = str(fila['year'])
lista.append(s)
consulta.close()
db2.close()
return(lista)
#consultar()
def menu():
Opcion= input("\nIngresa la opcion deseada\n1.Inserta un valor en la tabla\n2.Consultar los valores de la tabla\n")
if Opcion==1:
insertar()
menu()
elif Opcion==2:
ListaNovelas = consultar()
for novela in ListaNovelas:
print(novela['nombre'],novela['autor'],novela['year'])
menu()
menu()
I get this error while testing the second method consultar().
$ python file.py
Ingresa la opcion deseada
1.Inserta un valor en la tabla
2.Consultar los valores de la tabla
2
Estas en la funcion insertar
Traceback (most recent call last):
File "insertar.py", line 56, in <module>
menu()
File "insertar.py", line 51, in menu
ListaNovelas = consultar()
File "insertar.py", line 33, in consultar
s['nombre'] = fila['nombre']
TypeError: tuple indices must be integers, not str

db2row_factory = sqlite3.Row
This is the problematic line. Instead you meant to set the row_factory factory on the db2 connection instance:
db2.row_factory = sqlite3.Row
Then, all the fetched rows would be now sqlite3.Row instances having dictionary-like access to field values.

Python, database results are saved with ('result')

I select some urls from my database using a simple code
def crearArchivo(self):
#Se conecta a la base de datos
db = MySQLdb.connect("localhost","root","","proyectoacademias" )
cursor = db.cursor()
#Selecciona la lista de valores que no hayan sido procesados
sql = "SELECT url_paper_web FROM acw_papers_web WHERE finalizado = 0"
cursor.execute(sql)
#Recibe todos los datos
datos = cursor.fetchall()
#Crea el archivo
archivo = open('urls.txt','w')
archivo.close()
#Lo abre
archivo = open('urls.txt','w')
#Establece un contador para determinar el numero de filas
contador=0;
#Para cada fila la guarda en el archivo y si no es la ultima agrega un salto de linea
for columna in datos:
contador+=1;
archivo.write(str(columna))
if(contador!=len(datos)):
archivo.write("\n")
#Se cierra todo
archivo.close()
db.close()
This method places those links in a file... the problem is that those links instead of being saved like http:// google.com they are saved like ('http://google.com')

Each columna variable is actualy a row, not a single column. Simply select the first value in that row:
archivo.write(columna[0])

That is because columna is a tuple. It looks like you want to write the first (and in this case, the only element):
archivo.write(columna[0])