We Keep Coding

Python slicing output

I have a school project on analyzing logs & roughly, I need to be able to retrieve the day & time from a line in the format "MM DD HH:MM:SS" and it always shows as "['MM DD HH:MM:SS']"
def get_complete_date(line):
"""
Pre : line est une ligne de log bien formée (str)
Post : Retourne la date et l'heure sous forme de chaine de caractère sans changer le format.
"""
# splt = line.split(sep=" ", maxsplit=5)
# dt = splt[2]
complete_date = line.split(" ")
# print(complete_date)
return complete_date[0:3]

Very naive approach, but if the length of datetime part is the same (same pattern on each line), you can simply slice it from 0 to pattern length:
def get_complete_date(l):
tpl = "MM DD HH:MM:SS"
return l[0: len(tpl)]
line = "MM DD HH:MM:SS some text"
print(get_complete_date(line))
>>>'MM DD HH:MM:SS'

You almost there. It's an issue with slicing.
def get_complete_date(line):
"""
Pre : line est une ligne de log bien formée (str)
Post : Retourne la date et l'heure sous forme de chaine de caractère sans changer le format.
"""
# splt = line.split(sep=" ", maxsplit=5)
# dt = splt[2]
complete_date = line.split(" ")
# print(complete_date)
return complete_date[1:3:1]
line = "MM DD HH:MM:SS"
print(get_complete_date(line))
Gives #
['DD', 'HH:MM:SS']
Explanation: Your slicing should be
[1:3:1] = [start:stop:step]
The 0th element is month which you don't need to start from 1.

convert a tuple of tuples into bytes in python

I'm making a video game in upbge (which for this aspect is basically python) but to send the info between sockets it needs to be in bytes but i only found a way to do it turning it into a string, (because the list get's created with the game (i have a script that once a new entity appears it get's added to the list in the form of (Player-entity|data) but i want the data to variate between entities and i ran out of symbols to use to split it into lists so i'm trying to send the data as a list to be able to simply check it's components, but as it's a list inside of a list (and sometimes more) the bytearray won't work (at least i can't figure out how to make it work)
I'm unsure of how to summarize the code, but this is the code that I have now to generate that info
import socket
mi_socket = socket.socket()
mi_socket.bind(('127.0.0.1',55555))
mi_socket.listen()
TP = 0 #Total players
AP = 0 #Actual player
PL = ["host"] #Player List
AE = 0 #Actual entity
TE = 0 #Total entities
EL = ["|PlayerEntity$[Data]|"] #Entity list
PI = [] #Player Intel (Player name, Total Players, Player Code) (apartado uno en las comunicaciones)
order = None #Variable reservada para guardar comandos u identificaciones (ej: nombre de la entidad) en las comunicaciones
content = None #Variable reservada para almacenar la información recibida
incoming = 0 #Variable reservada para analizar el mensaje recibido de entrada
def login():
global AP
global TP
global PL
PJ = str(incoming).split("$")[1] #incoming Player
if AP <= TP:
if PJ!=str(PL[int(AP)]): #Nombre jugador que se conecta
AP+=1
login()
pass
else:
identity()
pass
if AP > TP:
PL.append(str("Player" + str(AP))) #Agregar jugador a la lista
TP = int(AP)
def identity(): #identifica de qué se trata la orden para responder en consecuencia, si se trata de entidad anota tmb la entidad en la lista
global TE
global AE
global EL
global AP
global PL
PJ = str(incoming).split("$")[1] # incoming Player
PE = str(incoming).split("$")[2] # incoming entity
if AE <= TE:
# Si nombre de jugador|nombre de objeto no és EL[AE]([1]|[2])
if str(str(PL[AP])+"-"+str(PE)) != str(str(EL[AE]).split("|")[1]).split("$")[0]:
AE+=1
identity()
pass
else:
EL[AE] = "|"+PL[AP]+"-"+PE+"$"+str(incoming).split("$")[3]+"|"
if AE > TE:
EL.append("|"+PL[AP]+"-"+PE+"$"+str(incoming).split("$")[3]+"|")
TE = int(AE)
def main():
global AP
global AE
global TE
global incoming
conexion, ip = mi_socket.accept()
incoming = conexion.recv(1024)
login()
conexion.send(str("#"+str(PL[AP])+"#"+str(EL[1:TE+1])+"#").encode())
AP=0
AE=0
conexion.close()
while True:
main()

Selenium/Python - "If not" statement followed by navigator.find_element

Good evening everyone,
I am currently trying to extract data from this website :
https://classic.sportsbookreview.com/betting-odds/nba-basketball/
The program logic is based on the followed loop.
Once the page from the website is open, the first step is to open the calendar and to seek if there were matches played this month.
If this is the case, for each of these day, the data will be written in an xls file. Then, when there aren't anymore matches to extract, the program click to the previous month, and perform the same statements.
On the contrary, if there isn't a single day where a match was played during the month, the treatment will click on the previous month and will perform the same statements as before.
Here is my code (sorry for the use of the french, if this is not clear, I could translate it) :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
import xlsxwriter
## Création du fichier excel
tableur = xlsxwriter.Workbook("PrédictionMachine.xlsx") ;
feuille1 = tableur.add_worksheet("10-05-2018") ;
## Ouverture du navigateur (pour Chrome, remplacer Firefox par Chrome)
navigateur = webdriver.Firefox() ;
navigateur.get("https://classic.sportsbookreview.com/betting-odds/nba- basketball/") ;
time.sleep(3) ;
## Initialisation des variables nécessaires pour écriture dans tableur
row = 0 ;
col = 0 ;
moistraite = 0 ;
matchtreeltrouve = 0 ;
## PHASE TEST À SUPPRIMER EN VF
passage = 0
try :
## Boucle : le programme s'arrête lorsque qu'une touche quelqconque du clavier est pressée
while True :
## Ouverture du calendrier
print("Passage : ", passage)
Calendrier = navigateur.find_element_by_xpath("//a[#class='dd-go-button']").click()
time.sleep(3)
## Récupération du mois/année en cours de traitement
Mois = navigateur.find_element_by_xpath("//h2[#class='tbl-cal-top-middle']")
## Si aucun match n'est joué ce mois-ci, le programme clique sur le mois précédent
if not (navigateur.find_elements_by_xpath("//a[contains(#onclick, '20')]")) :
print("------------------------------------------------")
print("Aucun match trouvé pour le mois de :", Mois.text)
print("------------------------------------------------")
Moisprecedent = navigateur.find_element_by_xpath("//img[#alt='Left Arrow']").click()
moistraite += 1
print("------------------------------------------------")
print("Nombre de mois traité :", moistraite) ;
print("------------------------------------------------")
time.sleep(3) ;
else :
## Récupération de tous les jours du mois où un match a supposément été joué
JMtheorique = navigateur.find_elements_by_xpath("//a[contains(#onclick, '20')]")
print("------------------------------------------------")
print("Matchs supposés trouvés pour le mois de :", Mois.text)
print("------------------------------------------------")
for jmtheorique in JMtheorique :
print("ON Y CROIT", jmtheorique.text)
navigateur.find_element_by_xpath("//a[contains(#onclick, 'OddsEvent.GetLinkDate')]").click()
time.sleep(3)
passage += 1
Calendrier = navigateur.find_element_by_xpath("//a[#class='dd-go-button']").click()
time.sleep(2)
Moisprecedent = navigateur.find_element_by_xpath("//img[#alt='Left Arrow']").click()
moistraite += 1
## Arrêt Manuel du programme
except KeyboardInterrupt :
print("!/_\/_\/_\! Interruption manuelle du programme !/_\/_\/_\! ")
print("Le programme a été stoppé au mois de :", Mois.text)
print("Nombre de mois traité :", moistraite)
print("Nombre de match écrit :", matchtreeltrouve) ;
##navigateur.quit()
For now, I just want to navigate through the different months/days.
For August, the code source works fine.
But when I arrive at July, the program is looping. It seems that the 'if not' statement isnt' correctly written, since the treatment find a match whereas there isn't a single one played during this month.

Okay, I think I understood my problem. When I open the calendar, some days of the previous/next month seem to be present. When the treatment reaches the month of July, the 4 August (a day where a game was played) is also present on the calendar, which explains why the program is looping between July and August. I need to extract only the a href contained in the td and not the ones contained in a td class as showed below:
The calendar
The kind of a href I need
I will investigate further to learn how to do that. Thanks for the help.

Histogram representing number of substitutions, insertions and deleting in sequences

l have two columns that represent : right sequence and predicted sequence. l want to make statistics on the number of deletion, substitution and insertion by comparing each right sequence with its predicted sequence.
l did the levenstein distance to get the number of characters which are different (see the function below) and error_dist function to get the most common errors (in terms of substitution) :
here is a sample of my data :
de de
date date
pour pour
etoblissemenls etablissements
avec avec
code code
communications communications
r r
seiche seiche
titre titre
publiques publiques
ht ht
bain bain
du du
ets ets
premier premier
dans dans
snupape soupape
minimum minimum
blanc blanc
fr fr
nos nos
au au
bl bl
consommations consommations
somme somme
euro euro
votre votre
offre offre
forestier forestier
cs cs
de de
pour pour
de de
paye r
cette cette
votre votre
valeurs valeurs
des des
gfda gfda
tva tva
pouvoirs pouvoirs
de de
revenus revenus
offre offre
ht ht
card card
noe noe
montant montant
r r
comprises comprises
quantite quantite
nature nature
ticket ticket
ou ou
rapide rapide
de de
sous sous
identification identification
du du
document document
suicide suicide
bretagne bretagne
tribunal tribunal
services services
cif cif
moyen moyen
gaec gaec
total total
lorsque lorsque
contact contact
fermeture fermeture
la la
route route
tva tva
ia ia
noyal noyal
brie brie
de de
nanterre nanterre
charcutier charcutier
semestre semestre
de de
rue rue
le le
bancaire bancaire
martigne martigne
recouvrement recouvrement
la la
sainteny sainteny
de de
franc franc
rm rm
vro vro
here is my code
import pandas as pd
import collections
import numpy as np
import matplotlib.pyplot as plt
import distance
def error_dist():
df = pd.read_csv('data.csv', sep=',')
df = df.astype(str)
df = df.replace(['é', 'è', 'È', 'É'], 'e', regex=True)
df = df.replace(['à', 'â', 'Â'], 'a', regex=True)
dictionnary = []
for i in range(len(df)):
if df.manual_raw_value[i] != df.raw_value[i]:
text = df.manual_raw_value[i]
text2 = df.raw_value[i]
x = len(df.manual_raw_value[i])
y = len(df.raw_value[i])
z = min(x, y)
for t in range(z):
if text[t] != text2[t]:
d = (text[t], text2[t])
dictionnary.append(d)
#print(dictionnary)
dictionnary_new = dict(collections.Counter(dictionnary).most_common(25))
pos = np.arange(len(dictionnary_new.keys()))
width = 1.0
ax = plt.axes()
ax.set_xticks(pos + (width / 2))
ax.set_xticklabels(dictionnary_new.keys())
plt.bar(range(len(dictionnary_new)), dictionnary_new.values(), width, color='g')
plt.show()
enter image description here
and the levenstein distance :
def levenstein_dist():
df = pd.read_csv('data.csv', sep=',')
df=df.astype(str)
df['string diff'] = df.apply(lambda x: distance.levenshtein(x['raw_value'], x['manual_raw_value']), axis=1)
plt.hist(df['string diff'])
plt.show()
enter image description here
Now l want to make a histograms showing three bins : number of substitution, number of insertion and number of deletion . How can l proceed ?
Thank you

Thanks to the suggestions of #YohanesGultom the answer for the problem can be found here :
http://www.nltk.org/_modules/nltk/metrics/distance.html
or
https://gist.github.com/kylebgorman/1081951

Pandas + Python: More efficient code

This is my code:
import pandas as pd
import os
import glob as g
archivos = g.glob('C:\Users\Desktop\*.csv')
for archiv in archivos:
nombre = os.path.splitext(archiv)[0]
df = pd.read_csv(archiv, sep=",")
d = pd.to_datetime(df['DATA_LEITURA'], format="%Y%m%d")
df['FECHA_LECTURA'] = d.dt.date
del df['DATA_LEITURA']
df['CONSUMO']=""
df['DIAS']=""
df["SUMDIAS"]=""
df["SUMCONS"]=""
df["CONSANUAL"] = ""
ordenado = df.sort_values(['NR_CPE','FECHA_LECTURA', 'HORA_LEITURA'], ascending=True)
##Agrupamos por el CPE
agrupado = ordenado.groupby('NR_CPE')
for name, group in agrupado: #Recorremos el grupo
indice = group.index.values
inicio = indice[0]
fin = indice[-1]
#Llenamos la primeras lectura de cada CPE, con esa lectura (porque no hay una lectura anterior)
ordenado.CONSUMO.loc[inicio] = 0
ordenado.DIAS.loc[inicio] = 0
cont=0
for i in indice: #Recorremos lo que hay dentro de los grupos, dentro de los CPES(lecturas)
if i > inicio and i <= fin :
cont=cont+1
consumo = ordenado.VALOR_LEITURA[indice[cont]] - ordenado.VALOR_LEITURA[indice[cont-1]]
dias = (ordenado.FECHA_LECTURA[indice[cont]] - ordenado.FECHA_LECTURA[indice[cont-1]]).days
ordenado.CONSUMO.loc[i] = consumo
ordenado.DIAS.loc[i] = dias
# Hago las sumatorias, el resultado es un objeto DataFrame
dias = agrupado['DIAS'].sum()
consu = agrupado['CONSUMO'].sum()
canu = (consu/dias) * 365
#Contador con el numero de courrencias de los campos A,B y C
conta=0
contb=0
contc=0
#Como es un DF, para recorrerlo tengo que iterar sobre ellos para hacer la comparacion
print "Grupos:"
for ind, sumdias in dias.iteritems():
if sumdias <= 180:
grupo = "A"
conta=conta+1
elif sumdias > 180 and sumdias <= 365:
grupo = "B"
contb=contb+1
elif sumdias > 365:
grupo = "C"
contc=contc+1
print "grupo A: " , conta
print "grupo B: " , contb
print "grupo C: " , contc
#Formateamos los campos para no mostrar todos los decimales
Fdias = dias.map('{:.0f}'.format)
Fcanu = canu.map('{:.2f}'.format)
frames = [Fdias, consu, Fcanu]
concat = pd.concat(frames,axis=1).replace(['inf','nan'],[0,0])
with open('C:\Users\Documents\RPE_PORTUGAL\Datos.csv','a') as f:
concat.to_csv(f,header=False,columns=['CPE','DIAS','CONSUMO','CONSUMO_ANUAL'])
try:
ordenado.to_excel(nombre+'.xls', columns=["NOME_DISTRITO",
"NR_CPE","MARCA_EQUIPAMENTO","NR_EQUIPAMENTO","VALOR_LEITURA","REGISTADOR","TIPO_REGISTADOR",
"TIPO_DADOS_RECOLHIDOS","FACTOR_MULTIPLICATIVO_FINAL","NR_DIGITOS_INTEIRO","UNIDADE_MEDIDA",
"TIPO_LEITURA","MOTIVO_LEITURA","ESTADO_LEITURA","HORA_LEITURA","FECHA_LECTURA","CONSUMO","DIAS"],
index=False)
print (archiv)
print ("===============================================")
print ("*****Se ha creado el archivo correctamente*****")
print ("===============================================")
except IOError:
print ("===================================================")
print ("¡¡¡¡¡Hubo un error en la escritura del archivo!!!!!")
print ("===================================================")
This takes a file where I have lectures of energy consumption from different dates for every light meter('NR_CPE') and do some calculations:
Calculate the energy consumption for every 'NR_CPE' by substracting the previous reading with the next one and the result put in a new column named 'CONSUMO'.
Calculate the number of days where I'v got a reading and sum up the number of days
Add the consumption for every 'NR_CPE' and calculate the anual consumption.
Finally I want to classify by number of days that every light meter('NR_CPE') has a lecture. A if it has less than 180 days, B between 180 and 1 year and C more than a year.
And finally write this result in two differents files.
Any idea of how should I re-code this to have the same output and be faster?
Thank you all.
BTW this is my dataset:
,NOME_DISTRITO,NR_CPE,MARCA_EQUIPAMENTO,NR_EQUIPAMENTO,VALOR_LEITURA,REGISTADOR,TIPO_REGISTADOR,TIPO_DADOS_RECOLHIDOS,FACTOR_MULTIPLICATIVO_FINAL,NR_DIGITOS_INTEIRO,UNIDADE_MEDIDA,TIPO_LEITURA,MOTIVO_LEITURA,ESTADO_LEITURA,DATA_LEITURA,HORA_LEITURA
0,GUARDA,A002000642VW,101,1865411,4834,001,S,1,1,4,kWh,1,1,A,20150629,205600
1,GUARDA,A002000642VW,101,1865411,4834,001,S,1,1,4,kWh,2,2,A,20160218,123300
2,GUARDA,A002000642VJ,122,204534,25083,001,S,1,1,5,kWh,1,1,A,20150629,205700
3,GUARDA,A002000642VJ,122,204534,27536,001,S,1,1,5,kWh,2,2,A,20160218,123200
4,GUARDA,A002000642HR,101,1383899,11734,001,S,1,1,5,kWh,1,1,A,20150629,205600
5,GUARDA,A002000642HR,101,1383899,11800,001,S,1,1,5,kWh,2,2,A,20160218,123000
6,GUARDA,A002000995VM,101,97706436,12158,001,S,1,1,5,kWh,1,3,A,20150713,155300
7,GUARDA,A002000995VM,101,97706436,12163,001,S,1,1,5,kWh,2,2,A,20160129,162300
8,GUARDA,A002000995VM,101,97706436,12163,001,S,1,1,5,kWh,2,2,A,20160202,195800
9,GUARDA,A2000995VM,101,97706436,12163,001,S,1,1,5,kWh,1,3,A,20160404,145200
10,GUARDA,A002000996LV,168,5011703276,3567,001,V,1,1,6,kWh,1,1,A,20150528,205900
11,GUARDA,A02000996LV,168,5011703276,3697,001,V,1,1,6,kWh,2,2,A,20150929,163500
12,GUARDA,A02000996LV,168,5011703276,1287,002,P,1,1,6,kWh,1,1,A,20150528,205900

Generally you want to avoid for loops in pandas.
For example, the first loop where you calculate total consumption and days could be rewritten as a groupby apply something like:
def last_minus_first(df):
columns_of_interest = df[['VALOR_LEITURA', 'days']]
diff = columns_of_interest.iloc[-1] - columns_of_interest.iloc[0]
return diff
df['date'] = pd.to_datetime(df['DATA_LEITURA'], format="%Y%m%d")
df['days'] = (df['date'] - pd.datetime(1970,1,1)).dt.days # create days column
df.groupby('NR_CPE').apply(last_minus_first)
(btw I don't understand why you are subtracting each entry from the previous, surely for meter readings this is the same as last-first?)
Then given the result of the above as consumption, you can replace your second for loop (for ind, sumdias in dias.iteritems()) with something like:
pd.cut(consumption.days, [-1, 180, 365, np.inf], labels=['a', 'b', 'c']).value_counts()