I need to make a program that receives a integer and stores it on a file. When it has 15 (or 20, the exact number doesn't matter) it will overwrite the first one that it wrote. They may be on the same line or each one in a new line.
This program reads temperature from a sensor and then i will show that on a site with a php chart.
I thought about writing a value every half an hour maybe, and when it has 15 values and a new one comes it overwrites the oldest one.
I'm having troubles saving the values, i dont know how to save the list as a string with new lines, it saves double new lines, i'm new at python and i get really lost.
This doesn't work but it is a "sample" of what i want to do:
import sys
import os
if not( sys.argv[1:] ):
print "No parameter"
exit()
# If file doesn't exist, create it and save the value
if not os.path.isfile("tempsHistory"):
data = open('tempsHistory', 'w+')
data.write( ''.join( sys.argv[1:] ) + '\n' )
else:
data = open('tempsHistory', 'a+')
temps = []
for line in data:
temps += line.split('\n')
if ( len( temps ) < 15 ):
data.write( '\n'.join( sys.argv[1:] ) + '\n' )
else:
#Maximum amount reached, save new, delete oldest
del temps[ 0 ]
temps.append( '\n'.join( sys.argv[1:] ) )
data.truncate( 0 )
data.write( '\n'.join(str(e) for e in temps) )
data.close( )
Im getting lost with the ''.join and \n etc... I mean, i have to write with join to make the list save as a string and not with the [ '', '']. If i use '\n'.join, it saves double space, i think.
Thank you in advance!
I think what you want is something like this:
import sys
fileTemps = 'temps'
with open(fileTemps, 'rw') as fd:
temps = fd.readlines()
if temps.__len__() >= 15:
temps.pop(0)
temps.append(' '.join(sys.argv[1:]) + '\n')
with open(fileTemps, 'w') as fd:
for l in temps:
fd.write(l)
First you open the file for reading. The fd.readlines() call will give you the lines in the file. Then you check the size, and if the number of lines is greater than 15, then you pop the first value and append the new line. Then you write everything to a file.
In Python, generally, when you read from a file (e.g. using readline()) gives you the line with an '\n' at the end, that is why you get double line breaks.
Hope this helps.
You want something like
values = open(target_file, "r").read().split("\n")
# ^ this solves your original problem as readline() will keep the \n in returned list items
if len(values) >= 15:
# keep the values at 15
values.pop()
values.insert(0, new_value)
# push new value at the start of the list
tmp_fd, tmp_fn = tempfile.mkstemp()
# ^ this part is important
os.write(tmp_fd, "\n".join(values))
os.close(tmp_fd)
shutil.move(tmp_fn, target_file)
# ^ as here, the operation of actual write to the file, your webserver is reading, is atomic
# this is eg. how text editors save files
But anyway, I'd suggest you to consider using a database, be it postgresql, redis, sqlite or whatever floats your boat
You should try to not confuse storing data in lists with formatting in strings. Data does not require the "\n"s
So just temps.append(sys.argv[1:]) is enough.
In addition you should not serialize / deserialize the data on your own. Have a look into pickle. This is much simpler to use than reading / writing lists on your own.
Related
FYI I am new to Python and this website!
I have a csv file:
Product Number,Account Number,Transactions,Year Number,Left Output,Mid Output
43854835,12345,23123,12,12,45
4353454,23456,123213213,4,23,56
7657657,34567,321321,5,34,67
21321312,45678,321321,8,45,78
21312313,56789,2131233,3,56,89
If I want to refer to column 2 as the one where I need to conduct left and mid in Python, what is the best approach without libraries? I also want to append at the last column of the data as seen in the image.
This takes in a .csv file, reads the lines into a list, appends new data using the LEFT and MID functions and saves it to a new file (newFile.csv). This works according to the data in the imgur link.
Note: the script is hardly optimised; it was tested on ~2 million lines and it took a couple of minutes and ALOT of ram (2-3GB), so be careful about running this (backup original csv file, save work, close programs, etc ...)
I could modify this to batch process lines so memory is freed as well as maybe some sort of cache, but since I'm assuming it will be used sparingly, it should probably be fine.
filename = "myFile.csv"
# don't want to overwrite original
new_filename = "newFile.csv"
def LEFT(s, length):
# example: LEFT("apple",3) returns "app".
return str(s[:length])
def MID(s, start, length):
# example: MID("apple",2,3) returns "ppl"
return str(s[start - 1: start - 1 + length])
# read file contents into list
with open(filename, 'r') as file:
# store file data in a list
file_data = file.read().splitlines()
# loop and append new data to list
for i, line in enumerate(file_data):
# ignore header
if (i == 0): continue;
# parse 2nd column
second_column = line.split(",")[1]
# append 5th and 6th column
file_data[i] += "," + LEFT(second_column, 2) \
+ "," + MID(second_column, 4, 2)
# write modified list to new file
with open(new_filename, 'w') as file:
for line in file_data:
file.write(line + '\n')
I would like to insert a string at a specific column of a specific line in a file.
Suppose I have a file file.txt
How was the English test?
How was the Math test?
How was the Chemistry test?
How was the test?
I would like to change the last line to say How was the History test? by adding the string History at line 4 column 13.
Currently I read in every line of the file and add the string to the specified position.
with open("file.txt", "r+") as f:
# Read entire file
lines = f.readlines()
# Update line
lino = 4 - 1
colno = 13 -1
lines[lino] = lines[lino][:colno] + "History " + lines[lino][colno:]
# Rewrite file
f.seek(0)
for line in lines:
f.write(line)
f.truncate()
f.close()
But I feel like I should be able to simply add the line to the file without having to read and rewrite the entire file.
This is possibly a duplicate of below SO thread
Fastest Way to Delete a Line from Large File in Python
In above it's a talk about delete, which is just a manipulation, and yours is more of a modification. So the code would get updated like below
def update(filename, lineno, column, text):
fro = open(filename, "rb")
current_line = 0
while current_line < lineno - 1:
fro.readline()
current_line += 1
seekpoint = fro.tell()
frw = open(filename, "r+b")
frw.seek(seekpoint, 0)
# read the line we want to update
line = fro.readline()
chars = line[0: column-1] + text + line[column-1:]
while chars:
frw.writelines(chars)
chars = fro.readline()
fro.close()
frw.truncate()
frw.close()
if __name__ == "__main__":
update("file.txt", 4, 13, "History ")
In a large file it make sense to not make modification till the lineno where the update needs to happen, Imagine you have file with 10K lines and update needs to happen at 9K, your code will load all 9K lines of data in memory unnecessarily. The code you have would work still but is not the optimal way of doing it
The function readlines() reads the entire file. But it doesn't have to. It actually reads from the current file cursor position to the end, which happens to be 0 right after opening. (To confirm this, try f.tell() right after with statement.) What if we started closer to the end of the file?
The way your code is written implies some prior knowledge of your file contents and layouts. Can you place any constraints on each line? For example, given your sample data, we might say that lines are guaranteed to be 27 bytes or less. Let's round that to 32 for "power of 2-ness" and try seeking backwards from the end of the file.
# note the "rb+"; need to open in binary mode, else seeking is strictly
# a "forward from 0" operation. We need to be able to seek backwards
with open("file.txt", "rb+") as f:
# caveat: if file is less than 32 bytes, this will throw
# an exception. The second parameter, 2, says "from end of file"
f.seek(-32, 2)
last = f.readlines()[-1].decode()
At which point the code has only read the last 32 bytes of the file.1 readlines() (at the byte level) will look for the line end byte (in Unix, \n or 0x0a or byte value 10), and return the before and after. Spelled out:
>>> last = f.readlines()
>>> print( last )
[b'hemistry test?\n', b'How was the test?']
>>> last = last[-1]
>>> print( last )
b'How was the test?'
Crucially, this works robustly under UTF-8 encoding by exploiting the UTF-8 property that ASCII byte values under 128 do not occur when encoding non-ASCII bytes. In other words, the exact byte \n (or 0x0a) only ever occurs as a newline and never as part of a character. If you are using a non-UTF-8 encoding, you will need to check if the code assumptions still hold.
Another note: 32 bytes is arbitrary given the example data. A more realistic and typical value might be 512, 1024, or 4096. Finally, to put it back to a working example for you:
with open("file.txt", "rb+") as f:
# caveat: if file is less than 32 bytes, this will throw
# an exception. The second parameter, 2, says "from end of file"
f.seek(-32, 2)
# does *not* read while file, unless file is exactly 32 bytes.
last = f.readlines()[-1]
last_decoded = last.decode()
# Update line
colno = 13 -1
last_decoded = last_decoded[:colno] + "History " + last_decoded[colno:]
last_line_bytes = len( last )
f.seek(-last_line_bytes, 2)
f.write( last_decoded.encode() )
f.truncate()
Note that there is no need for f.close(). The with statement handles that automatically.
1 The pedantic will correctly note that the computer and OS will likely have read at least 512 bytes, if not 4096 bytes, relating to the on-disk or in-memory page size.
You can use this piece of code :
with open("test.txt",'r+') as f:
# Read the file
lines=f.readlines()
# Gets the column
column=int(input("Column:"))-1
# Gets the line
line=int(input("Line:"))-1
# Gets the word
word=input("Word:")
lines[line]=lines[line][0:column]+word+lines[line][column:]
# Delete the file
f.seek(0)
for i in lines:
# Append the lines
f.write(i)
This answer will only loop through the file once and only write everything after the insert. In cases where the insert is at the end there is almost no overhead and where the insert at the beginning it is no worse than a full read and write.
def insert(file, line, column, text):
ln, cn = line - 1, column - 1 # offset from human index to Python index
count = 0 # initial count of characters
with open(file, 'r+') as f: # open file for reading an writing
for idx, line in enumerate(f): # for all line in the file
if idx < ln: # before the given line
count += len(line) # read and count characters
elif idx == ln: # once at the line
f.seek(count + cn) # place cursor at the correct character location
remainder = f.read() # store all character afterwards
f.seek(count + cn) # move cursor back to the correct character location
f.write(text + remainder) # insert text and rewrite the remainder
return # You're finished!
I'm not sure whether you were having problems changing your file to contain the word "History", or whether you wanted to know how to only rewrite certain parts of a file, without having to rewrite the whole thing.
If you were having problems in general, here is some simple code which should work, so long as you know the line within the file that you want to change. Just change the first and last lines of the program to read and write statements accordingly.
fileData="""How was the English test?
How was the Math test?
How was the Chemistry test?
How was the test?""" # So that I don't have to create the file, I'm writing the text directly into a variable.
fileData=fileData.split("\n")
fileData[3]=fileData[3][:11]+" History"+fileData[3][11:] # The 3 referes to the line to add "History" to. (The first line is line 0)
storeData=""
for i in fileData:storeData+=i+"\n"
storeData=storeData[:-1]
print(storeData) # You can change this to a write command.
If you wanted to know how to change specific "parts" to a file, without rewriting the whole thing, then (to my knowledge) that is not possible.
Say you had a file which said Ths is a TEST file., and you wanted to correct it to say This is a TEST file.; you would technically be changing 17 characters and adding one on the end. You are changing the "s" to an "i", the first space to an "s", the "i" (from "is") to a space, etc... as you shift the text forward.
A computer can't actually insert bytes between other bytes. It can only move the data, to make room.
I'm somewhat new to python. I'm trying to sort through a list of strings and integers. The lists contains some symbols that need to be filtered out (i.e. ro!ad should end up road). Also, they are all on one line separated by a space. So I need to use 2 arguments; one for the input file and then the output file. It should be sorted with numbers first and then the words without the special characters each on a different line. I've been looking at loads of list functions but am having some trouble putting this together as I've never had to do anything like this. Any takers?
So far I have the basic stuff
#!/usr/bin/python
import sys
try:
infilename = sys.argv[1] #outfilename = sys.argv[2]
except:
print "Usage: ",sys.argv[0], "infile outfile"; sys.exit(1)
ifile = open(infilename, 'r')
#ofile = open(outfilename, 'w')
data = ifile.readlines()
r = sorted(data, key=lambda item: (int(item.partition(' ')[0])
if item[0].isdigit() else float('inf'), item))
ifile.close()
print '\n'.join(r)
#ofile.writelines(r)
#ofile.close()
The output shows exactly what was in the file but exactly as the file is written and not sorted at all. The goal is to take a file (arg1.txt) and sort it and make a new file (arg2.txt) which will be cmd line variables. I used print in this case to speed up the editing but need to have it write to a file. That's why the output file areas are commented but feel free to tell me I'm stupid if I screwed that up, too! Thanks for any help!
When you have an issue like this, it's usually a good idea to check your data at various points throughout the program to make sure it looks the way you want it to. The issue here seems to be in the way you're reading in the file.
data = ifile.readlines()
is going to read in the entire file as a list of lines. But since all the entries you want to sort are on one line, this list will only have one entry. When you try to sort the list, you're passing a list of length 1, which is going to just return the same list regardless of what your key function is. Try changing the line to
data = ifile.readlines()[0].split()
You may not even need the key function any more since numbers are placed before letters by default. I don't see anything in your code to remove special characters though.
since they are on the same line you dont really need readlines
with open('some.txt') as f:
data = f.read() #now data = "item 1 item2 etc..."
you can use re to filter out unwanted characters
import re
data = "ro!ad"
fixed_data = re.sub("[!?#$]","",data)
partition maybe overkill
data = "hello 23frank sam wilbur"
my_list = data.split() # ["hello","23frank","sam","wilbur"]
print sorted(my_list)
however you will need to do more to force numbers to sort maybe something like
numbers = [x for x in my_list if x[0].isdigit()]
strings = [x for x in my_list if not x[0].isdigit()]
sorted_list = sorted(numbers,key=lambda x:int(re.sub("[^0-9]","",x))) + sorted(strings(
Also, they are all on one line separated by a space.
So your file contains a single line?
data = ifile.readlines()
This makes data into a list of the lines in your file. All 1 of them.
r = sorted(...)
This makes r the sorted version of that list.
To get the words from the line, you can .read() the entire file as a single string, and .split() it (by default, it splits on whitespace).
I have a certain check to be done and if the check satisfies, I want the result to be printed. Below is the code:
import string
import codecs
import sys
y=sys.argv[1]
list_1=[]
f=1.0
x=0.05
write_in = open ("new_file.txt", "w")
write_in_1 = open ("new_file_1.txt", "w")
ligand_file=open( y, "r" ) #Open the receptor.txt file
ligand_lines=ligand_file.readlines() # Read all the lines into the array
ligand_lines=map( string.strip, ligand_lines ) #Remove the newline character from all the pdb file names
ligand_file.close()
ligand_file=open( "unique_count_c_from_ac.txt", "r" ) #Open the receptor.txt file
ligand_lines_1=ligand_file.readlines() # Read all the lines into the array
ligand_lines_1=map( string.strip, ligand_lines_1 ) #Remove the newline character from all the pdb file names
ligand_file.close()
s=[]
for i in ligand_lines:
for j in ligand_lines_1:
j = j.split()
if i == j[1]:
print j
The above code works great but when I print j, it prints like ['351', '342'] but I am expecting to get 351 342 (with one space in between). Since it is more of a python question, I have not included the input files (basically they are just numbers).
Can anyone help me?
Cheers,
Chavanak
To convert a list of strings to a single string with spaces in between the lists's items, use ' '.join(seq).
>>> ' '.join(['1','2','3'])
'1 2 3'
You can replace ' ' with whatever string you want in between the items.
Mark Rushakoff seems to have solved your immediate problem, but there are some other improvements that could be made to your code.
Always use context managers (with open(filename, mode) as f:) for opening files rather than relying on close getting called manually.
Don't bother reading a whole file into memory very often. Looping over some_file.readilines() can be replaced with looping over some_file directly.
For example, you could have used map(string.strip, ligland_file) or better yet [line.strip() for line in ligland_file]
Don't choose names to include the type of the object they refer to. This information can be found other ways.
For exmaple, the code you posted can be simplified to something along the lines of
import sys
from contextlib import nested
some_real_name = sys.argv[1]
other_file = "unique_count_c_from_ac.txt"
with nested(open(some_real_name, "r"), open(other_file, "r")) as ligand_1, ligand_2:
for line_1 in ligand_1:
# Take care of the trailing newline
line_1 = line_1.strip()
for line_2 in ligand_2:
line_2 = line2.strip()
numbers = line2.split()
if line_1 == numbers[1]:
# If the second number from this line matches the number that is
# in the user's file, print all the numbers from this line
print ' '.join(numbers)
which is more reliable and I believe more easily read.
Note that the algorithmic performance of this is far from ideal because of these nested loops. Depending on your need, this could potentially be improved, but since I don't know exactly what data you need to extract to tell you whether you can.
The time this takes currently in my code and yours is O(nmq), where n is the number of lines in one file, m is the number of lines in the other, and q is the length of lines in unique_count_c_from_ac.txt. If two of these are fixed/small, then you have linear performance. If two can grow arbitrarily (I sort of imagine n and m can?), then you could look into improving your algorithm, probably using sets or dicts.
I want to read a line in a file and insert the new line ("\n") character in the n position on a line, so that a 9-character line, for instance, gets converted into three 3-character lines, like this:
"123456789" (before)
"123\n456\n789" (after)
I've tried with this:
f = open(file, "r+")
f.write("123456789")
f.seek(3, 0)
f.write("\n")
f.seek(0)
f.read()
-> '123\n56789'
I want it not to substitute the character in position n, but only to insert another ("\n") char in that position.
Any idea about how to do this?
Thanks
I don't think there is any way to do that in the way you are trying to: you would have to read in to the end of the file from the position you want to insert, then write your new character at the position you wish it to be, then write the original data back after it. This is the same way things would work in C or any language with a seek() type API.
Alternatively, read the file into a string, then use list methods to insert your data.
source_file = open("myfile", "r")
file_data = list(source_file.read())
source_file.close()
file_data.insert(position, data)
open("myfile", "wb").write(file_data)
with open(file, 'r+') as f:
data = f.read()
f.seek(0)
for i in range(len(data)): # could also use 'for i, chara in enumerate(data):' and then 'f.write(chara)' instead of 'f.write(data[i])'
if (i + 1) % 3 == 0: # could also do 'if i % 3 == 2:', but that may be slightly confusing
f.write('\n')
else:
f.write(data[i])
I don't think it's all that Pythonic (due to the range(len(data))), but it should work, unless your data file is really really large (in which case you'll have to process the data in the file part by part and store the results in another file to prevent overwriting data you haven't processed yet).
(More on the with statement.)
You can think a file is just an array of characters, and if you want to insert a new element in the middle of an array, then you have to shift all the elements that are after it.
You could do what you say if the file contained a "linked list" of chars or "extends", but then you would need a special editor to see it sequentially.