We Keep Coding

Streamlit, Python, and Pandas: Duplicate keys and writing

With Python and Streamlit I'm build apps to assist teachers grading essays. In this segment of the app the user is provided with all the student submissions as .txt files. These files are displayed on the main screen, and users scroll down to display additional texts. In a sidebar there are input fields for entering in comments and grades for each individual txt. The layout for the app is below:
So far so good. However, I'm at a loss to figure out how to capture the output from the sidebar radio buttons and number inputs into a csv with pandas. Here's my working code below:
import os
import streamlit as st
import pandas as pd
# for establishing length of loops below
path, dirs, files = next(os.walk("TXTs"))
float_file_count = ((len(files)) / 3)
file_count = int(float_file_count)
txts = [f"TXTs\\feedback_{n}.txt" for n in range(file_count)]
#streamlit layout
st.set_page_config(layout="wide")
st.sidebar.header("Grading Interface")
button1 = st.sidebar.button("Record comments and grades.")
st.header("Demo - Level 3.6.2 (Prototype)")
st.subheader("In this level...")
# loop to display txt files in main screen
def display_txts():
for txt in txts:
with open(txt, 'r', encoding="utf8") as infile:
contents = infile.read()
st.markdown(":point_right: " + "**" + txt + "**" + " :point_left:" + "\n\n" + contents)
# loop to display `engagement_comment` and `engagement_grade` fields in sidebar
def display_engagement():
for txt in txts:
engagement_comment = st.sidebar.radio("Engagement Comment: " + txt, ["Substantial engagement with text - good work.", "Satisfactory engagement with text.", "Little engagement with text - needs improvement."], key = txt)
engagement_grade = st.sidebar.number_input("Engagement Grade: " + txt)
display_txts()
display_engagement()
if button1:
df = pd.read_csv("Control2.csv")
df['engagement_grade'] = df.apply(display_engagement()) # <<<< Here's the crux of my difficulty. How do I write the outputs to be written to csv?
df
df.to_csv("Control2.csv", index=False)
st.write("Grades and comments recorded. This process can be repeated as needed.")
However, when I click on button1 to write the output I receive the following error:
DuplicateWidgetID: There are multiple identical st.radio widgets with key='TXTs\feedback_0.txt'. To fix this, please make sure that the key argument is unique for each st.radio you create.
Traceback:
File "C:\Users\danie\Desktop\Python\Streamlit\Experiment_3_6_2.py", line 50, in <module>
df['engagement_grade'] = df.apply(display_engagement())
File "C:\Users\danie\Desktop\Python\Streamlit\Experiment_3_6_2.py", line 37, in display_engagement
engagement_comment = st.sidebar.radio("Engagement Comment: " + txt, ["Substantial engagement with text - good work.", "Satisfactory engagement with text
I thought I had set up individual keys within the display_engagement function, but it appears something is amiss. Any ideas or suggestion to resolve this, or to structure this differently? All advice and assistance appreciated.

I modified your code a bit, but here's the general solution. There were a few problems.
Problem 1: This made it so that only one item in your list was being passed to the unique key (print off the txts in streamlit and you will see what I mean)
float_file_count = ((len(files)) / 3)
Problem 2: You need to allow the user to generate the data dynamically as a form so that it is all passed at once.
Problem 3: You need to access the unique key in streamlit's session state.
Problem 4: You need to append to the existing DataFrame
The code below works and it should give you the framework to finish up your app. By the way, super cool idea! I tried to keep a lot of your syntax the sam
import os
import streamlit as st
import pandas as pd
import glob
st.set_page_config(layout="wide")
st.sidebar.header("Grading Interface")
st.header("Demo - Level 3.6.2 (Prototype)")
st.subheader("In this level...")
txts = glob.glob("TXTs/*.txt")
def display_txts():
with open(txt, 'r', encoding="utf8") as infile:
contents = infile.read()
st.markdown(":point_right: " + "**" + txt + "**" + " :point_left:" + "\n\n" + contents)
form = st.sidebar.form(key='grading')
submit_button = form.form_submit_button(label='Submit')
for txt in txts:
display_txts()
txt = txt.replace("TXTs\\", "").replace(".txt", "").replace("_", "")
rb = form.radio("Engagement Comment: " + txt, ["Substantial engagement with text - good work.", "Satisfactory engagement with text.", "Little engagement with text - needs improvement."], key = txt)
if submit_button:
# df = pd.read_csv("Control2.csv")
df = pd.DataFrame(columns=['data'])
st.write(st.session_state.feedback0)
#Find out which items in the session state have feedback in the name
for item in st.session_state:
if "feedback" in item:
df = df.append({'data': st.session_state[item]}, ignore_index=True)
df.to_csv("Control2.csv", index=False)

Tkinter inserting text "live" into a textbox

Bit of a beginner here :)
So for the past couple of days I've been working on a relatively simple web scraper, but with a special goal in mind. Namely, I wanted to get better with tkinter.
So, I've made a devlog window, which is simply going to be a separate console kind of thing. But the problem is that I want to insert text "live" in to the devlog window, and the text comes from a different script in a different thread. How could I implement this feature? This devlog window is ALSO run in a thread, as well as 3 other scripts along side it. This is the code I have so far:
import Manager as AM
import tkinter as tk
from time import sleep # For some simple timing algorithms
Width = 700
Height = 500
FrameGeometry = str(Width) + "x" + str(Height)
# COLORS
LightGrey = "#D9D9D9"
DarkGrey = "#AAAAAA"
# Global variables.
ThreadsList = list()
def CreateDevLogWindow():
global Width
global Height
ThreadTemp_ = str(AM.FileManager(ReadData=2))
devlog = tk.Tk() # Secondary Devlog Interface.
FrameInfo = tk.Frame(master=devlog)
FrameText = tk.Frame(master=devlog)
ThreadText = tk.Label(master=FrameInfo, text="Total Threads: " + ThreadTemp_, bg="white")
ThreadTextActive = tk.Label(master=FrameInfo, text="Active Threads: " + ThreadTemp_, bg="white")
InfoText = tk.Text(FrameText, border=1, highlightthickness=1)
devlog.title("Devlog: YourOptimalScrapper")
devlog.geometry(str(Width) + "x" + str(Height))
devlog.config(bg="white")
FrameInfo.config(bg="white")
FrameText.config(bg="white", padx=5, pady=5)
FrameInfo.pack(side=tk.TOP, fill=tk.BOTH)
FrameText.pack(side=tk.BOTTOM, expand=True, fill=tk.BOTH)
ThreadText.grid(row=0)
ThreadTextActive.grid(row=1)
InfoText.pack(expand=True, fill=tk.BOTH)
while True:
if ThreadTemp_ != AM.FileManager(ReadData=2):
ThreadTemp_ = str(AM.FileManager(ReadData=2))
ThreadText.config(text="Total Threads: " + ThreadTemp_)
ThreadTextActive.config(text="Active Threads: " + ThreadTemp_)
devlog.update()
Any and all help is appreciated, and NOTE: There is more code under this, but I don't think its particularly necessary :)

First of all you can insert text in your TextBox with just a few lines.
def update_textbox(textbox, data):
textbox.config(state=tk.NORMAL) # unlock the textbox
textbox.insert(tk.END, "\n"+str(data)) # add newline and append the data
textbox.config(state=tk.DISABLED) # lock back the textbox to readonly
textbox.see(tk.END) # scroll to the bottom to see the last line
With that if your script is in a different thread, then you can create a file that will be the "carrier" of your data.
Whenever you need to write data to the textbox, simply write data to the file with your text in any thread and update the while loop in the above code to read this file and update the textbox if the file isn't empty.
EDIT: You don't need to call this function outside of this thread. Just check in your while loop whether new data has to be added to the log
...
while True:
with open("carrier.txt", 'r') as f:
content = f.read() # read carrier data
if content: # if file isn't empty
update_textbox(InfoText, content) # update the textbox with the data in the file
open("carrier.txt", 'w').close() # clear the file to prevent reading the data twice
if ThreadTemp_ != AM.FileManager(ReadData=2):
ThreadTemp_ = str(AM.FileManager(ReadData=2))
ThreadText.config(text="Total Threads: " + ThreadTemp_)
ThreadTextActive.config(text="Active Threads: " + ThreadTemp_)
devlog.update()
And in your other thread
def update_log(data):
with open("carrier.txt", 'a') as f: # append to the file
f.write(str(data))

How to correctly store user input (keyboard,mouse) in file using keyboard & mouse Python modules?

I am using the keyboard and mouse modules to record user interaction with our software so that we can have some high level GUI tests.
Currently I am trying to store the recorded events in a text file and later play this recording again.
However when I load the recorded events from said file I only see played mouse events and no keyboard events.
One cause for this problem may be the implementation of KeyboardEvents.
KeyboardEvents does not contain a correct implementation of __repr__. This prevents us from calling print(keyboard_events, file=f) and reading the lines with eval(line). (This works with mouse and ButtonEvent and MoveEvent)
So we have decided to work with the json format of KeyboardEvents. Basically what we're doing is we retrieve the json format of each KeyboardEvent and write the json in the file. Then we load the json file and parse the json as KeyboardEvents.
Currently we're storing both mouse & keyboard input in a single file. However, since mouse supports a correct implementation of __repr__ we can directly print and the mouse events and use eval() on it to retrieve the stored events.
This is the file used for recording and playing:
import threading
import mouse
import keyboard
from mouse import ButtonEvent
from mouse import MoveEvent
from mouse import WheelEvent
from keyboard import KeyboardEvent
import time
import json
import sys
def record(file='record.txt'):
f = open(file, 'w+')
mouse_events = []
keyboard_events = []
keyboard.start_recording()
starttime = time.time()
mouse.hook(mouse_events.append)
keyboard.wait('esc')
keyboard_events = keyboard.stop_recording()
mouse.unhook(mouse_events.append)
#first line = start of recording
#mouse events = second line
#keyboard events = every remaining line = 1 event
print(starttime, file=f)
print(mouse_events, file=f)
for kevent in range(0, len(keyboard_events)):
print(keyboard_events[kevent].to_json(), file = f)
f.close()
def play(file, speed = 0.5):
f = open(file, 'r')
#per definition the first line is mouse events and the rest is keyboard events
lines = f.readlines()
f.close()
mouse_events = eval(lines[1])
keyboard_events = []
for index in range(2,len(lines)):
keyboard_events.append(keyboard.KeyboardEvent(**json.loads(lines[index])))
starttime = float(lines[0])
keyboard_time_interval = keyboard_events[0].time - starttime
keyboard_time_interval /= speed
mouse_time_interval = mouse_events[0].time - starttime
mouse_time_interval /= speed
print(keyboard_time_interval)
print(mouse_time_interval)
#Keyboard threadings:
k_thread = threading.Thread(target = lambda : time.sleep(keyboard_time_interval) == keyboard.play(keyboard_events, speed_factor=speed) )
#Mouse threadings:
m_thread = threading.Thread(target = lambda : time.sleep(mouse_time_interval) == mouse.play(mouse_events, speed_factor=speed))
#start threads
m_thread.start()
k_thread.start()
#waiting for both threadings to be completed
k_thread.join()
m_thread.join()
if __name__ == '__main__':
if len(sys.argv) > 2 and sys.argv[1] == 'play':
play(sys.argv[2])
elif len(sys.argv) >= 2 and sys.argv[1] == 'record':
if(len(sys.argv)) == 3:
record(sys.argv[2])
else:
record()
else:
print("missing either 'play' or 'record' or filename")
I expect the same behavior with this code like when it is run in a single function (see edit in https://stackoverflow.com/a/57670484/7345513).
Meaning: I expect the playback in the threads to be synced and the keys to be pressed. What i actually get is that the mouse events are played back as desired but no KeyboardEvents are being processed. When I use the function from the linked SO it works.
Can someone please point me to the right direction?

I forgot to answer my own question but here it goes:
Thanks to the guidance from blubberdiblub I have tracked the start time for the tracking and can then add an offset to when the first input occurred. This allows playbacks to be timed somewhat accurately.

EOFError: Ran out of input

When I run the code below I get this error message "EOFError: Ran out of input"
what does it mean?? How it can be corrected?? and how to output the records details on the screen.
import pickle # this library is required to create binary files
class CarRecord:
def __init__(self):
self.VehicleID = " "
self.Registration = " "
self.DateOfRegistration = " "
self.EngineSize = 0
self.PurchasePrice = 0.00
ThisCar = CarRecord()
Car = [ThisCar for i in range(2)] # list of 2 car records
Car[0].VehicleID = "CD333"
Car[0].Registration = "17888"
Car[0].DateOfRegistration = "18/2/2017"
Car[0].EngineSize = 2500
Car[0].PurchasePrice = 22000.00
Car[1].VehicleID = "AB123"
Car[1].Registration = "16988"
Car[1].DateOfRegistration = "19/2/2017"
Car[1].EngineSize = 2500
Car[1].PurchasePrice = 20000.00
CarFile = open ('Cars.TXT', 'wb' ) # open file for binary write
for j in range (2): # loop for each array element
pickle.dump (Car[j], CarFile) # write a whole record to the binary file
CarFile.close() # close file
CarFile = open ('Cars.TXT','rb') # open file for binary read
Car = [] # start with empty list
while True: #check for end of file
Car.append(pickle.load(CarFile)) # append record from file to end of list
CarFile.close()

Short answer: The simplest solution is to write the complete list to file using pickle.dump(). There's no need to write all objects one by one in a loop. Pickle is designed to do this for you.
Example code and alternative solutions:
Below is a fully working example. Some notes:
I've updated your __init__ function a bit to make the initialization code a lot easier and shorter.
I've also added a __repr__ function. This could be used to print the record details to screen, which you also asked. (Note that you could also implement a __str__ function, but I chose to implement __repr__ for this example).
This code example uses standard Python coding styles (PEP-8).
This code uses a context manager to open the file. This is safer and avoid the need to manually close the file.
If you really want to write the objects manually, for whatever reason, there are a few alternatives to do that safely. I'll explain them after this code example:
import pickle
class CarRecord:
def __init__(self, vehicle_id, registration, registration_date, engine_size, purchase_price):
self.vehicle_id = vehicle_id
self.registration = registration
self.registration_date = registration_date
self.engine_size = engine_size
self.purchase_price = purchase_price
def __repr__(self):
return "CarRecord(%r, %r, %r, %r, %r)" % (self.vehicle_id, self.registration,
self.registration_date, self.engine_size,
self.purchase_price)
def main():
cars = [
CarRecord("CD333", "17888", "18/2/2017", 2500, 22000.00),
CarRecord("AB123", "16988", "19/2/2017", 2500, 20000.00),
]
# Write cars to file.
with open('Cars.TXT', 'wb') as car_file:
pickle.dump(cars, car_file)
# Read cars from file.
with open('Cars.TXT', 'rb') as car_file:
cars = pickle.load(car_file)
# Print cars.
for car in cars:
print(car)
if __name__ == '__main__':
main()
Output:
CarRecord('CD333', '17888', '18/2/2017', 2500, 22000.0)
CarRecord('AB123', '16988', '19/2/2017', 2500, 20000.0)
Instead of dumping the list at once, you could also do it in a loop. The following code snippets are alternative implementations to "Write cars to file" and "Read cars from file".
Alternative 1: write number of objects to file
At the start of the file, write the number of cars. This can be used to read the same amount of cars from the file.
# Write cars to file.
with open('Cars.TXT', 'wb') as car_file:
pickle.dump(len(cars), car_file)
for car in cars:
pickle.dump(car, car_file)
# Read cars from file.
with open('Cars.TXT', 'rb') as car_file:
num_cars = pickle.load(car_file)
cars = [pickle.load(car_file) for _ in range(num_cars)]
Alternative 2: use an "end" marker
At the end of the file, write some recognizable value, for example None. When reading this object can be used to detect the end of file.
# Write cars to file.
with open('Cars.TXT', 'wb') as car_file:
for car in cars:
pickle.dump(car, car_file)
pickle.dump(None, car_file)
# Read cars from file.
with open('Cars.TXT', 'rb') as car_file:
cars = []
while True:
car = pickle.load(car_file)
if car is None:
break
cars.append(car)

You can change you while loop to this:
this will break out of your while loop at the end of the input when it recieves the EOFError
while True: #check for end of file
try:
Car.append(pickle.load(CarFile)) # append record from file to end of list
except EOFError:
break
CarFile.close()

You get that error when the file you are trying to load with pickle is empty.So make sure that there's things written into '.pkl file'

Python reload textfile into string every x seconds in a loop

I am trying to read a text file into a string, do something with the string, then ever X seconds,
re-read the text file (in case it has changed) to update the string and do the same thing again, over and over in a loop, without spawning an infinite number of processes.
so something like :
in an infinite loop
open 'MyTextFile' and read it into a String
do stuff with the string of text it reads from the file
close the file (if I need to...to allow another script to write to it)
wait x seconds before clearing the string and re-reading the
same 'MyTextFile' file to update the string and do it all again
(in an infinite loop until I tell it to stop)
What would be a good sturdy (reliable to run for a long time) way to do that?
Thanks!
<code>
import os
import time
myDataFile = "MyData.txt"
data = ""
def ReadFile(data):
# only need to read it if data has changed
# detect 'modified' date of myDataFile and only PrintToPrinter IF it has change
# and if it has NOT changed, wait 10 seconds and check again
# IF CHANGED
with open (myDataFile, "r") as myFile:
try:
data=myFile.read().replace('\n', ' ')
finally:
myFile.close()
ShowOnScreen(data)
PrintToPrinter(data)
# ELSE
# data has not changed..
# no need to print it so I shouldn't send back to PrintToPrinter
# but I still want to ShowOnScreen(data) on screen
# and keep the program running to check for changes
ShowOnScreen(data)
sleep(10)
ReadFile(data)
def ShowOnScreen(data):
print(time+' '+data)
def PrintToPrinter(data):
# send DateTimeStamp+data to printer here
# go back to checking for changes
ReadFile(data)
# kick off the program to start looking for changes
# data starts out as "" so it should always read the first time
# and send to printer and screen
ReadFile(data)
</code>

This can be easily done using the time.sleep from the time module. A sample code is pasted below:
import time
def file_handler(filename):
with open(filename) as fh:
line = fh.read()
print line
print "Length of string:%s"% len(line)
fh.close()
while True:
file_handler("test.txt")
time.sleep(10)

Ended up doing something like this
#!/usr/bin/env python
import sys
import time from sleep
import datetime
import time
# probably should not import both time and datetime
MyFile = "MyFile.txt"
MyText = ""
MyDate = ""
def GetDateTime():
MyDate = time.strftime("%x %I:%M %p")
return MyDate+" "
def ReadFile(MyText):
with open (MyFile, "r" as myfile:
try:
MyText2=myfile.read().replace('\n',' ')
finally:
myfile.close()
CurDateTime = GetDateTime()
if (MyText == MyText2):
# file hasn't changed...print to screen only
print CurDateTime
sleep(5)
ReadFile(MyText2)
else:
# print to screen and printer
print CurDateTime +MyText2
# send CurDateTime +MyText2 to printer
sleep(5)
ReadFile(MyText2)
ReadFile(MyText)
Hope this helps somebody