When I run the code below I get this error message "EOFError: Ran out of input"
what does it mean?? How it can be corrected?? and how to output the records details on the screen.
import pickle # this library is required to create binary files
class CarRecord:
def __init__(self):
self.VehicleID = " "
self.Registration = " "
self.DateOfRegistration = " "
self.EngineSize = 0
self.PurchasePrice = 0.00
ThisCar = CarRecord()
Car = [ThisCar for i in range(2)] # list of 2 car records
Car[0].VehicleID = "CD333"
Car[0].Registration = "17888"
Car[0].DateOfRegistration = "18/2/2017"
Car[0].EngineSize = 2500
Car[0].PurchasePrice = 22000.00
Car[1].VehicleID = "AB123"
Car[1].Registration = "16988"
Car[1].DateOfRegistration = "19/2/2017"
Car[1].EngineSize = 2500
Car[1].PurchasePrice = 20000.00
CarFile = open ('Cars.TXT', 'wb' ) # open file for binary write
for j in range (2): # loop for each array element
pickle.dump (Car[j], CarFile) # write a whole record to the binary file
CarFile.close() # close file
CarFile = open ('Cars.TXT','rb') # open file for binary read
Car = [] # start with empty list
while True: #check for end of file
Car.append(pickle.load(CarFile)) # append record from file to end of list
CarFile.close()
Short answer: The simplest solution is to write the complete list to file using pickle.dump(). There's no need to write all objects one by one in a loop. Pickle is designed to do this for you.
Example code and alternative solutions:
Below is a fully working example. Some notes:
I've updated your __init__ function a bit to make the initialization code a lot easier and shorter.
I've also added a __repr__ function. This could be used to print the record details to screen, which you also asked. (Note that you could also implement a __str__ function, but I chose to implement __repr__ for this example).
This code example uses standard Python coding styles (PEP-8).
This code uses a context manager to open the file. This is safer and avoid the need to manually close the file.
If you really want to write the objects manually, for whatever reason, there are a few alternatives to do that safely. I'll explain them after this code example:
import pickle
class CarRecord:
def __init__(self, vehicle_id, registration, registration_date, engine_size, purchase_price):
self.vehicle_id = vehicle_id
self.registration = registration
self.registration_date = registration_date
self.engine_size = engine_size
self.purchase_price = purchase_price
def __repr__(self):
return "CarRecord(%r, %r, %r, %r, %r)" % (self.vehicle_id, self.registration,
self.registration_date, self.engine_size,
self.purchase_price)
def main():
cars = [
CarRecord("CD333", "17888", "18/2/2017", 2500, 22000.00),
CarRecord("AB123", "16988", "19/2/2017", 2500, 20000.00),
]
# Write cars to file.
with open('Cars.TXT', 'wb') as car_file:
pickle.dump(cars, car_file)
# Read cars from file.
with open('Cars.TXT', 'rb') as car_file:
cars = pickle.load(car_file)
# Print cars.
for car in cars:
print(car)
if __name__ == '__main__':
main()
Output:
CarRecord('CD333', '17888', '18/2/2017', 2500, 22000.0)
CarRecord('AB123', '16988', '19/2/2017', 2500, 20000.0)
Instead of dumping the list at once, you could also do it in a loop. The following code snippets are alternative implementations to "Write cars to file" and "Read cars from file".
Alternative 1: write number of objects to file
At the start of the file, write the number of cars. This can be used to read the same amount of cars from the file.
# Write cars to file.
with open('Cars.TXT', 'wb') as car_file:
pickle.dump(len(cars), car_file)
for car in cars:
pickle.dump(car, car_file)
# Read cars from file.
with open('Cars.TXT', 'rb') as car_file:
num_cars = pickle.load(car_file)
cars = [pickle.load(car_file) for _ in range(num_cars)]
Alternative 2: use an "end" marker
At the end of the file, write some recognizable value, for example None. When reading this object can be used to detect the end of file.
# Write cars to file.
with open('Cars.TXT', 'wb') as car_file:
for car in cars:
pickle.dump(car, car_file)
pickle.dump(None, car_file)
# Read cars from file.
with open('Cars.TXT', 'rb') as car_file:
cars = []
while True:
car = pickle.load(car_file)
if car is None:
break
cars.append(car)
You can change you while loop to this:
this will break out of your while loop at the end of the input when it recieves the EOFError
while True: #check for end of file
try:
Car.append(pickle.load(CarFile)) # append record from file to end of list
except EOFError:
break
CarFile.close()
You get that error when the file you are trying to load with pickle is empty.So make sure that there's things written into '.pkl file'
Related
I am new to python and I m learning by doing.
At this moment, my code is running quite slow and it seems to take longer and longer by each time I run it.
The idea is to download an employee list as CSV, then to check the location of each Employee ID by running it trough a specific page then writing it to an excel file.
We have around 600 associates on site each day and I need to find their location and to keep refreshing it each 2-4 minutes.
EDIT:
For everyone to have a better understanding, I have a CSV file ( TOT.CSV ) that contains Employee ID's, Names and other information of the associates that I have on site.
In order to get their location, I need to run each employee ID from that CSV file trough https://guided-coaching-dub.corp.amazon.com/api/employee-location-svc/GetLastSeenLocationOfEmployee?employeeId= 1 by 1 and at the same time to write it in another CSV file ( Location.csv ). Right now, it does in about 10 minutes and I want to understand if the way I did it is the best possible way, or if there is something else that I could try.
My code looks like this:
# GET EMPLOYEE ID FROM THE CSV
data = read_csv("Z:\\_Tracker\\Dump\\attendance\\TOT.csv")
# converting column data to list
TOT_employeeID = data['Employee ID'].tolist()
# Clean the Location Sheet
with open("Z:\\_Tracker\\Dump\\attendance\\Location.csv", "w") as f:
pass
print("Previous Location data cleared ... ")
# go through EACH employee ID to find out location
for x in TOT_employeeID:
driver.get(
"https://guided-coaching-dub.corp.amazon.com/api/employee-location-svc/GetLastSeenLocationOfEmployee?employeeId=" + x)
print("Getting Location data for EmployeeID: " + x)
locData = driver.find_element(By.TAG_NAME, 'body').text
aaData = str(locData)
realLoc = aaData.split('"')
# write to excel
with open("Z:\\_Tracker\\Dump\\attendance\\Location.csv",
"a") as f:
writer = csv.writer(f)
writer.writerow(realLoc)
time.sleep(5)
print("Employee Location data downloaded...")
Is there a way I can do this faster?
Thank you in advance!
Regards,
Alex
Something like this.
import concurrent.futures
def process_data(data: pd.DataFrame) -> None:
associates = data['Employee ID'].unique()
with concurrent.futures.ProcessPoolExecutor() as executer:
executer.map(get_location, associates)
def get_location(associate: str) -> None:
driver.get(
"https://guided-coaching-dub.corp.amazon.com/api/employee-location-svc/GetLastSeenLocationOfEmployee?"
f"employeeId={associate}")
print(f"Getting Location data for EmployeeID: {associate}")
realLoc = str(driver.find_element(By.TAG_NAME, 'body').text).split('"')
with open("Z:\\_Tracker\\Dump\\attendance\\Location.csv", "a") as f:
writer = csv.writer(f)
writer.writerow(realLoc)
if __name__ == "__main__":
data = read_csv("Z:\\_Tracker\\Dump\\attendance\\TOT.csv")
process_data(data)
You could try separating the step of reading the information and writing the information to your CSV file, like below:
# Get Employee Location Information
# Create list for employee information, to be used below
employee_Locations = []
for x in TOT_employeeID:
driver.get("https://guided-coaching-dub.corp.amazon.com/api/employee-location-svc/GetLastSeenLocationOfEmployee?employeeId=" + x)
print("Getting Location data for EmployeeID: " + x)
locData = driver.find_element(By.TAG_NAME, 'body').text
aaData = str(locData)
realLoc = [aaData.split('"')]
employee_Locations.extend(realLoc)
# Write to excel - Try this as a separate step
with open("Z:\\_Tracker\\Dump\\attendance\\Location.csv","a") as f:
writer = csv.writer(f, delimiter='\n')
writer.writerow(employee_Locations)
print("Employee Location data downloaded...")
You may see some performance gains by collecting all your information first, then writing to your CSV file
I'm trying to make a phone book in python and I want to save all contacts in a file, encoded as JSON, but when I try to read the exported JSON data from the file, I get an error:
Extra data: line 1 column 103 - line 1 column 210 (char 102 - 209)
(It works fine when I have only one object in "list.txt")
This is my code:
class contacts:
def __init__(self, name="-", phonenumber="-", address="-"):
self.name= name
self.phonenumber= phonenumber
self.address= address
self.jsonData=json.dumps(vars(self),sort_keys=False, indent=4)
self.writeJSON(self.jsonData)
def writeJSON(self, jsonData):
with open("list.txt", 'a') as f:
json.dump(jsonData, f)
ted=contacts("Ted","+000000000","Somewhere")
with open('list.txt') as p:
p = json.load(p)
print p
The output in list.txt:
"{\n \"phonenumber\": \"+000000000\", \n \"name\": \"Ted\", \n \"address\": \"Somewhere\"\n}"
Now, if I add another object, it can't read the JSON data anymore. If my way of doing it is wrong, how else should I export the JSON code of every object in a class, so it can be read back when I need to?
The reason this isn't working is that this code path gives you an invalid JSON structure. With one contact you get this:
{"name":"", "number":""}
While with 2 contacts you would end up with this:
{"name":"", "number":""}{"name":"", "number":""}
The second one is invalid json because 2 objects should be encoded in an array, like this:
[{"name":"", "number":""},{"name":"", "number":""}]
The problem with your code design is that you're writing to the file every time you create a contact. A better idea is to create all contacts and then write them all to the file at once. This is cleaner, and will run more quickly since file I/O is one of the slowest things a computer can do.
My suggestion is to create a new class called Contact_Controller and handle your file IO there. Something like this:
import json
class Contact_Controller:
def __init__(self):
self.contacts = []
def __repr__(self):
return json.dumps(self)
def add_contact(self, name="-", phonenumber="-", address="-"):
new_contact = Contact(name,phonenumber,address)
self.contacts.append(new_contact)
return new_contact
def save_to_file(self):
with open("list.txt", 'w') as f:
f.write(str(self.contacts))
class Contact:
def __init__(self, name="-", phonenumber="-", address="-"):
self.name= name
self.phonenumber= phonenumber
self.address= address
def __repr__(self):
return json.dumps({"name": self.name, "phonenumber": self.phonenumber, "address": self.address})
contact_controller = Contact_Controller()
ted = contact_controller.add_contact("Ted","+000000000","Somewhere")
joe = contact_controller.add_contact("Joe","+555555555","Somewhere Else")
contact_controller.save_to_file()
with open('list.txt') as p:
p = json.load(p)
print(p)
I've also changed it to use the built in __repr__() class method. Python will call that method whenever it needs a string representation of the object.
in writeJSON, you opened the file for append (mode='a'), which works fine the first time, but not the subsequent calls. To fix this problem, open the file with overwrite mode ('w'):
with open("list.txt", 'w') as f:
I am making a program that will open multiple files, they are all very similar. All contains a few one word lines in lowercase on Notepad. I do not want to repeat the code multiple times. Ideally I want to use a while loop to repeat the code but change what file it opens each repeat. Is there a way to do it?
This is the current code:
File = open("Key Words\Audio.txt","r") #This will open the file called Audio.
Audio = [] #This creates the Audio list
Audio = File.read().splitlines() #This saves everything on each line of the Audio file to a diffrent section of the Audio list.
File = open("Key Words\Calls.txt","r") #This will open the file called Calls.
Calls = [] #This creates the Calls list
Calls = File.read().splitlines() #This saves everything on each line of the Calls file to a diffrent section of the Calls list.
File = open("Key Words\Charging.txt","r") #This will open the file called Charging.
Charging = [] #This creates the Charging list
Charging = File.read().splitlines() #This saves everything on each line of the Charging file to a diffrent section of the Charging list.
File.close() #This closes the File(s).
This is what functions are for:
def readfile(filepath):
with open(filepath, 'r') as f:
return f.read().splitlines()
audio = readfile('Key Words\Audio.txt')
calls = readfile('Key Words\Calls.txt')
charging = readfile('Key Words\Charging.txt')
Make a list of the files you need to open:
files_to_open = [
'file_1.txt',
'file_2.txt'
]
calls_info = {}
Iterate over the list, and open and process:
for file_ in files_to_open:
with open(file_) as f:
calls_info[file_] = f.read().splitlines()
Here, I created a calls_info variable. What this will do is store everything in a dictionary. These hold keys and values - to access the value of a file, simply index it like so:
calls_info[file_path] # Make sure file_path is the right path you put in the list!
Put the code in a function:
def openandread(filename):
# No need to close the file if you use with:
with open(filename,"r") as File:
return_this = File.read().splitlines()
return return_this
and then just call this function multiple times:
Audio = openandread("Key Words\Audio.txt")
Calls = openandread("Key Words\Calls.txt")
Charging = openandread("Key Words\Charging.txt")
or if you want to make it even shorter:
Audio, Calls, Charging = (openandread(i) for i in ["Key Words\Audio.txt", "Key Words\Calls.txt", "Key Words\Charging.txt"])
Try this
Audio = []
Calls = []
Charging = []
FILES_LISTS = (
( "Key Words\Audio.txt", Audio ),
( "Key Words\Calls.txt", Calls ),
( "Key Words\Charging.txt", Charging )
)
for file_name, list_var in FILES_LISTS:
File = open( file_name, 'r' )
list_var += File.read().splitlines()
File.close()
Make sure to type list_var += and not list_var =. This works because lists are mutable and because python works with references.
You can try unipath
# Install
$easy_install unipath
# In python
from unipath import Path
t1 = Path('Key Words\Audio.txt')
...
I currently have a program that appends to an already existing file called "ConcentrationData.txt". However, I would like to create a new text file every time the program is run, preferably with a file name that has the date and time. This is what my current script looks like:
def measureSample(self):
sys.stdout.flush()
freqD1, trandD1, absoD1 = dev.getMeasurement(LED_TO_COLOR='D1'])
freqD2, trandD2, absoD2 = dev.getMeasurement(LED_TO_COLOR='D2'])
absoDiff= absoD1 - absoD2
Coeff= 1
Conc = absoDiff/Coeff
Conc3SD = '{Value:1.{digits}f'.format(Value = Conc, digits=3)
self.textEdit.clear()
self.textEdit.setText('Concentration is {0}'.format(Conc3SD))
timeStr = time.strftime('%m-%d-%Y %H:%M:%S %Z')
outFile = open('ConcentrationData.txt','a')
outFile.write('{0} || Concentration: {1}'.format(timeStr, Conc3SD))
outFile.close()
How would I go about doing that?
(Also, I'm pretty new to python so I'm sorry if this sounds like a silly question).
You can do something on the lines of the following
class my_class:
_data_fd = None
def __init__(self,create,filename):
if(create):
self._data_fd = open(filename,'w')
def __del__(self):
if(self._data_fd != None):
self._data_fd.close()
def measureSample(self):
##do something here
outFile = self._data_fd
outFile.write('{0} || Concentration: {1}'.format(timeStr, Conc3SD))
if __name__ == '__main__':
timeStr = time.strftime('%m-%d-%Y_%H_%M_%S_%Z') #use unerscore instead of spaces
filename = "{0}.{1}".format("Data.txt",timeStr)
imy_class = my_class(1,filename)
imy_class.measureSample()
imy_class.measureSample() ##call multiple times the fd remains open for the lifetime of the object
del imy_class ### the file closes now and you will have multiple lines of data
I want to learn Python so I started writing my first program which is a phone book directory.
It has the options to add a name and phone number, remove numbers, and search for them.
Ive been stuck on the remove part for about 2 days now and just can't get it working correctly. I've been in the Python IRC and everything, but haven't been able to figure it out.
Basically, my program stores the numbers to a list in a file. I cannot figure out how to remove a particular line in the file but keep the rest of the file intact. Can someone please help me with this?
Some people have advised that it will be easier to do if I create a temp file, remove the line, then copy the remaining lines from the original file over to the temp file. Then write over the original file over with the temp file. So I have been trying this...
if ui == 'remove':
coname = raw_input('What company do you want to remove? ') # company name
f = open('codilist.txt', 'r') # original phone number listing
f1 = open('codilist.tmp', 'a') # open a tmp file
for line in f:
if line.strip() != coname.strip():
for line in f:
f1.write(line)
break # WILL LATER OVERWRITE THE codilist.txt WITH THE TMP FILE
else:
f1.write(line)
else:
print 'Error: That company is not listed.'
f1.close()
f.close()
continue
I assume your file contains something like <name><whitespace><number> on each line? If that's the case, you could use something like this for your if statement (error handling not included!):
name, num = line.strip().split()
if name != coname.strip():
# write to file
Suggestion:
Unless there is some specific reason for you to use a custom format, the file format json is quite good for this kind of task. Also note the use of the 'with' statement in these examples, which saves you having to explicitly close the file.
To write the information:
import json
# Somehow build a dict of {coname: num,...}
info = {'companyA': '0123456789', 'companyB': '0987654321'}
with open('codilist.txt', 'w') as f:
json.dump(info, f, indent=4) # Using indent for prettier files
To read/amend the file:
import json
with open('codilist.txt', 'r+') as f:
info = json.load(f)
# Remove coname
if coname in info:
info.pop(coname)
else:
print 'No record exists for ' + coname
# Add 'companyC'
info['companyC'] = '0112233445'
# Write back to file
json.dump(info, f, indent=4)
You'll need python2.6 or later for these examples. If you're on 2.5, you'll need these imports:
import simplejson as json
from __future__ import with_statement
Hope that helps!
Here is a pretty extensively rewritten version:
all the phone data is wrapped into a Phonebook class; data is kept in memory (instead of being saved and reloaded for every call)
it uses the csv module to load and save data
individual actions are turned into short functions or methods (instead of One Big Block of Code)
commands are abstracted into a function-dispatch dictionary (instead of a cascade of if/then tests)
This should be much easier to understand and maintain.
import csv
def show_help():
print('\n'.join([
"Commands:",
" help shows this screen",
" load [file] loads the phonebook (file name is optional)",
" save [file] saves the phonebook (file name is optional)",
" add {name} {number} adds an entry to the phonebook",
" remove {name} removes an entry from the phonebook",
" search {name} displays matching entries",
" list show all entries",
" quit exits the program"
]))
def getparam(val, prompt):
if val is None:
return raw_input(prompt).strip()
else:
return val
class Phonebook(object):
def __init__(self, fname):
self.fname = fname
self.data = []
self.load()
def load(self, fname=None):
if fname is None:
fname = self.fname
try:
with open(fname, 'rb') as inf:
self.data = list(csv.reader(inf))
print("Phonebook loaded")
except IOError:
print("Couldn't open '{}'".format(fname))
def save(self, fname=None):
if fname is None:
fname = self.fname
with open(fname, 'wb') as outf:
csv.writer(outf).writerows(self.data)
print("Phonebook saved")
def add(self, name=None, number=None):
name = getparam(name, 'Company name? ')
number = getparam(number, 'Company number? ')
self.data.append([name,number])
print("Company added")
def remove(self, name=None):
name = getparam(name, 'Company name? ')
before = len(self.data)
self.data = [d for d in self.data if d[0] != name]
after = len(self.data)
print("Deleted {} entries".format(before-after))
def search(self, name=None):
name = getparam(name, 'Company name? ')
found = 0
for c,n in self.data:
if c.startswith(name):
found += 1
print("{:<20} {:<15}".format(c,n))
print("Found {} entries".format(found))
def list(self):
for c,n in self.data:
print("{:<20} {:<15}".format(c,n))
print("Listed {} entries".format(len(self.data)))
def main():
pb = Phonebook('phonebook.csv')
commands = {
'help': show_help,
'load': pb.load,
'save': pb.save,
'add': pb.add,
'remove': pb.remove,
'search': pb.search,
'list': pb.list
}
goodbyes = set(['quit','bye','exit'])
while True:
# get user input
inp = raw_input("#> ").split()
# if something was typed in
if inp:
# first word entered is the command; anything after that is a parameter
cmd,args = inp[0],inp[1:]
if cmd in goodbyes:
# exit the program (can't be delegated to a function)
print 'Goodbye.'
break
elif cmd in commands:
# "I know how to do this..."
try:
# call the appropriate function, and pass any parameters
commands[cmd](*args)
except TypeError:
print("Wrong number of arguments (type 'help' for commands)")
else:
print("I didn't understand that (type 'help' for commands)")
if __name__=="__main__":
main()
Something simple like this will read all of f, and write out all the lines that don't match:
for line in f:
if line.strip() != coname.strip():
f1.write(line)
Ned's answer looks like it should work. If you haven't tried this already, you can set python's interactive debugger above the line in question. Then you can print out the values of line.strip() and coname.strip() to verify you are comparing apples to apples.
for line in f:
import pdb
pdb.set_trace()
if line.strip() != coname.strip():
f1.write(line)
Here's a list of pdb commands.
You probably don't want to open the temp file in append ('a') mode:
f1 = open('codilist.tmp', 'a') # open a tmp file
also, be aware that
for line in f:
...
f1.write(line)
will write everything to the file without newlines.
The basic structure you want is:
for line in myfile:
if not <line-matches-company>:
tmpfile.write(line + '\n') # or print >>tmpfile, line
you'll have to implement <line-matches-company> (there isn't enough information in the question to know what it should be -- perhaps if you showed a couple of lines from your data file..?)
I got this working...
if ui == 'remove':
coname = raw_input('What company do you want to remove? ') # company name
f = open('codilist.txt')
tmpfile = open('codilist.tmp', 'w')
for line in f:
if coname in line:
print coname + ' has been removed.'
else:
tmpfile.write(line)
f.close()
tmpfile.close()
os.rename('codilist.tmp', 'codilist.txt')
continue