Only the last saved dataset exists when using h5py - python

I am trying to save several datasets into a hdf5 file by h5py module, but it seems only the last one is saved. I think that because when a break statement was added, the first dataset is saved instead.
The code in problem is below. How can I fix it?
set_num = 0
for cur in data["init"]:
'''
got result as a list
'''
ipt = h5py.File(output_file, "w")
s = str(set_num)
ipt[s] = result
'''
create an attribute for ipt[s]
'''
set_num += 1
ipt.close()
#break
I apologize if there's any silly mistake.

You are opening, closing the file on each pass of the for: loop, and you have the attribute set to "w", meaning that it is overwriting the existing file during each pass.
My recommendation is to instead open the file using the with clause, and nest the for: loop inside of that, which would make the intent clearer and obviate the need to explicitly close the file. This example might help (though it is not tested, so may need modifications):
with h5py.File(output_file, "w") as ipt:
for set_num, curr in enumerate(data["init"]):
s = str(set_num)
ipt[s] = result

You only get the last dataset because you are opening the file in write mode ('w'), inside your loop. Simple solution is use append mode ( a'). Better, move the file open outside the loop and use the with...as: context manager.

Related

python script does not rewrite the file on itself

so first and foremost i wanted to connect multiple lines into one and add ","
so in example
line1
line2
line2
to
line1,line2,line3
i managed to make it work with this script right here
filelink = input("Enter link here ")
fix = file = open(filelink, "r")
data=open(filelink).readlines()
for n,line in enumerate(data):
if line.startswith("line"):
data[n] = "\n"+line.rstrip()
else:
data[n]=line.rstrip()
print(','.join(data))
HOWEVER in the terminal itself it shows it executed perfectly but in the text file itself it's still remains the same no connected lines and no commas
side note. i would love some explanations how does the loop work and what "enumerate" stands for and why specifically this code i tried googling each one separately and understand the code but i didn't manage to find what i was looking for if anyone keen to explain the code line by line shortly i would be very appreciative
Thanks in advance <3
This is somewhat superfluous:
fix = file = open(filelink, "r")
That's assigning two names to the same file object, and you don't even use fix, so at least drop that part.
For handling files, you would be better using a context manager. That means that you can open a resource and they will automatically get closed for you once you're done (usually).
In any case, you opened in read mode with open(filelink, "r") so you'll never change the file contents. print(','.join(data)) will probably show you what you expect, but print() writes to stdout and the change will only be in your terminal. You will not modify the base file with this. But, I think you're sufficiently close that I'll try close the missing connection.
In this case, you need to:
open the file first in read mode to pull the data out.
Do the transform in python to the data
Open the file again in write mode (which wipes the existing contents)
Write the transformed data
So, like this:
filelink = input("Enter link here ")
with open(filelink) as infile: # context manager, by default in "r" mode
data = [item.strip() for item in infile.readlines()]
data = ','.join(data)
# Now write it back out
with open(filelink, "w") as outfile:
outfile.write(data)

Is It Possible to Make a File with a Function in Python?

I am trying to make a file with a function in python, but I have no idea how. This is what I have, and what I need.
def file_maker():
file_number = input("What number player are you? ")
#Insert however you make a file in code, naming it ('inventory.' + filenumber + '.txt')
I only need to know how I would initiate the file-making process. I tried googling it, but the only thing that comes up is how to access a function within a different file. I am an amateur programmer, any and all suggestions are welcome. Thanks for your time.
def file_maker():
file_number = input("What number player are you? ")
with open("inventory.%s.txt" % file_number, "w") as f:
# Do whatever you need with the file using the 'f' to refer to the file object
pass # in case you don't want to do anything with the file, but just create it
Read more regarding open function here: Open function
FYI, this will overwrite the file if it already exists.
To create a file, just open in it write mode.
file_handle=open ('inventory.' + filenumber + '.txt', "w")
file_handle is now an object that you can use various methods on to add content to the file. Read the documentation here.
Make sure you close the file when you are done with it using file_handle.close()
Note: Although this method works, it is usually considered better practice to use with, as shown in the other answer. It uses less code and automatically closes the file when done.

Delete a line in multiple text files with the same line beginning but varying line ending using Python v3.5

I have a folder full of .GPS files, e.g. 1.GPS, 2.GPS, etc...
Within each file is the following five lines:
Trace #1 at position 0.004610
$GNGSA,A,3,02,06,12,19,24,25,,,,,,,2.2,1.0,2.0*21
$GNGSA,A,3,75,86,87,,,,,,,,,,2.2,1.0,2.0*2C
$GNVTG,39.0304,T,39.0304,M,0.029,N,0.054,K,D*32
$GNGGA,233701.00,3731.1972590,S,14544.3073733,E,4,09,1.0,514.675,M,,,0.49,3023*27
...followed by the same data structure, with different values, over the next five lines:
Trace #6 at position 0.249839
$GNGSA,A,3,02,06,12,19,24,25,,,,,,,2.2,1.0,2.0*21
$GNGSA,A,3,75,86,87,,,,,,,,,,2.2,1.0,2.0*2C
$GNVTG,247.2375,T,247.2375,M,0.081,N,0.149,K,D*3D
$GNGGA,233706.00,3731.1971997,S,14544.3075178,E,4,09,1.0,514.689,M,,,0.71,3023*2F
(I realise the values after the $GNGSA lines don't vary in the above example. This is just a bad example... in the real dataset they do vary!)
I need to remove the lines that begin with "$GNGSA" and "$GNVTG" (i.e. I need to delete lines 2, 3, and 4 from each group of five lines within each .GPS file).
This five-line pattern continues for a varying number of times throughout each file (for some files, there might only be two five-line groups, while other files might have hundreds of the five-line groups). Hence, deleting these lines based on the line number will not work (because the line number would be variable).
The problem I am having (as seen in the above examples) is that the text that follows the "$GNGSA" or "$GNVTG" varies.
I'm currently learning Python (I'm using v3.5), so figured this would make for a good project for me to learn a few new tricks...
What I've tried already:
So far, I've managed to create the code to loop through the entire folder:
import os
indir = '/Users/dhunter/GRID01/' # input directory
for i in os.listdir(indir): # for each "i" (iteration) within the indir variable directory...
if i.endswith('.GPS'): # if the filename of an iteration ends with .GPS, then...
print(i + ' loaded') # print the filename to CLI, simply for debugging purposes.
with open(indir + i, 'r') as my_file: # open the iteration file
file_lines = my_file.readlines() # uses the readlines method to create a list of all lines in the file.
print(file_lines) # this prints the entire contents of each file to CLI for debugging purposes.
Everything in the above works perfectly.
What I need help with:
How do I detect and delete the lines themselves, and then save the file (to the same location; there is no need to save to a different filename)?
The filenames - which usually end with ".GPS" - sometimes end with ".gps" instead (the only difference being the case). My above code will only work with the uppercase files. Besides completely duplicating the code and changing the endswith argument, how do I make it work with both cases?
In the end, my file needs to look something like this:
Trace #1 at position 0.004610
$GNGGA,233701.00,3731.1972590,S,14544.3073733,E,4,09,1.0,514.675,M,,,0.49,3023*27
Trace #6 at position 0.249839
$GNGGA,233706.00,3731.1971997,S,14544.3075178,E,4,09,1.0,514.689,M,,,0.71,3023*2F
Any suggestions, please? Thanks in advance. :)
You're almost there.
import os
indir = '/Users/dhunter/GRID01/' # input directory
for i in os.listdir(indir): # for each "i" (iteration) within the indir variable directory...
if i.endswith('.GPS'): # if the filename of an iteration ends with .GPS, then...
print(i + ' loaded') # print the filename to CLI, simply for debugging purposes.
with open(indir + i, 'r') as my_file: # open the iteration file
for line in my_file:
if not line.startswith('$GNGSA') and not line.startswith('$GNVTG'):
print(line)
As per what the others have said, you're on the right track! Where you're going wrong is in the case-sensitive file extension check, and in reading in the entire file contents at once (this isn't per se wrong, but it's probably adding complexity we won't need).
I've commented your code, removing all the debug stuff for simplicity, to illustrate what I mean:
import os
indir = '/path/to/files'
for i in os.listdir(indir):
if i.endswith('.GPS'): #This CASE SENSITIVELY checks the file extension
with open(indir + i, 'r') as my_file: # Opens the file
file_lines = my_file.readlines() # This reads the ENTIRE file at once into an array of lines
So we need to fix the case sensitivity issue, and instead of reading in all the lines, we'll instead read the file line-by-line, check each line to see if we want to discard it or not, and write the lines we're interested in into an output file.
So, incorporating #tdelaney's case-insensitive fix for file name, we replace line #5 with
if i.lower().endswith('.gps'): # Case-insensitively check the file name
and instead of reading in the entire file at once, we'll instead iterate over the file stream and print each desired line out
with open(indir + i) as in_file, open(indir + i + 'new.gps') as out_file: # Open the input file for reading and creates + opens a new output file for writing - thanks #tdelaney once again!
for line in in_file # This reads each line one-by-one from the in file
if not line.startswith('$GNGSA') and not line.startswith('$GNVTG'): # Check the line has what we want (thanks Avinash)
out_file.write(line + "\n") # Write the line to the new output file
Note that you should make certain that you open the output file OUTSIDE of the 'for line in in_file' loop, or else the file will be overwritten on every iteration which will erase what you've already written to it so far (I suspect this is the issue you've had with the previous answers). Open both files at the same time and you can't go wrong.
Alternatively, you can specify the file access mode when you open the file, as per
with open(indir + i + 'new.gps', 'a'):
which will open the file in append-mode, which is a specialised from of write-mode that preserves the original contents of the file, and appends new data to it instead of overwriting existing data.
Ok, based on suggestions by Avinash Raj, tdelaney, and Sampson Oliver, here on Stack Overflow, and another friend who helped privately, here is the solution that is now working:
import os
indir = '/Users/dhunter/GRID01/' # input directory
for i in os.listdir(indir): # for each "i" (iteration) within the indir variable directory...
if i.lower().endswith('.gps'): # if the filename of an iteration ends with .GPS, then...
if not i.lower().endswith('.gpsnew.gps'): # if the filename does not end with .gpsnew.gps, then...
print(i + ' loaded') # print the filename to CLI.
with open (indir + i, 'r') as my_file:
for line in my_file:
if not line.startswith('$GNGSA'):
if not line.startswith('$GNVTG'):
with open(indir + i + 'new.gps', 'a') as outputfile:
outputfile.write(line)
outputfile.write('\r\n')
(You'll see I had to add in another layer of if statement to stop it from using the output files from previous uses of the script "if not i.lower().endswith('.gpsnew.gps'):", but this line can easily be deleted for anyone who uses these instructions in future)
We switched the open mode on the third-last line to "a" for append, so that it would save all the right lines to the file, rather than overwriting each time.
We also added in the final line to add a line break at the end of each line.
Thanks everyone for their help, explanations, and suggestions. Hopefully this solution will be useful to someone in future. :)
2. The filenames:
The if accepts any expression returning a truth value, and you can combine expressions with the standart boolean operators: if i.endswith('.GPS') or i.endswith('.gps').
You can also put the ... and ... expression after the if in brackets, to feel more sure, but it's not neccessary.
Alternatively, as a less universal solution, (but since you wanted to learn a few tricks :)) you can use string manipulation in this case: an object of type string has a lot of methods. '.gps'.upper() gives '.GPS' -- try, if you can make use of this! (even a printed string is a string object, but your variables behave the same).
1. Finding the Lines:
As you can see in the other solution, you need not read out all of your lines, you can check if want to have them 'on the fly'. But I will stick to your approach with readlines. It gives you a list, and lists support indexing and slicing. Try:
anylist[stratindex, endindex, stride], for any values, so for example try: newlist = range(100)[1::5].
It's always helpfull to try out the easy basic operations in interactive mode, or at the beginning of your script. Here range(100) is just some sample list. Here you see, how the python for-syntax works, differently than in other languages: you can iterate over any list, and if you just need integers, you create a list with integers with range().
So this will work the same with any other list -- e.g. the one you get from readlines()
This selects a slice from the list, beginnig with the second element, ending at the end (since the end index is omitted), and taking every 5th element. Now you have this sub-list, you can just revome it from the original. So for the example with the range:
a = range(100)
del(a[1::5])
print a
So you see, that the appropriate items have been removed. Now do the same with your file_lines, and then proceed to remove the other lines you want to remove.
Then, in a new with block, open the file for writing and do writelines(file_lines), so the remainig lines are written back to the file.
Of course you can also take the approach to look for the content of each line with a for loop over your list and startswith(). Or you can combine the approaches, and check, if deleting lines by number leaves the right starts, so you can print an error if something is unexpected...
3. Saving the file
You can close your file after you have the lines saved in the readlines(). In fact this is done automatically at the end of the with-block. Then just open it in 'w' mode instead of 'r' and do yourfilename.writelines(yourlist). You don't need to save, it's saven on closing.

Python, run commands in specific order

I'm writing a script that gets the most recently modified file from a unix directory.
I'm certain it works, but I have to create a unittest to prove it.
The problem is the setUp function. I want to be able to predict the order the files are created in.
self.filenames = ["test1.txt", "test2.txt", "test3.txt", "filename.txt", "test4"]
newest = ''
for fn in self.filenames:
if pattern.match(fn): newest = fn
with open(fn, "w") as f: f.write("some text")
The pattern is "test.*.txt" so it just matches the first three in the list. In multiple tests, newest sometimes returns 'test3.txt' and sometimes 'test1.txt'.
Use os.utime to explicitly set modified time on the files that you have created. That way your test will run faster.
I doubt that the filesystem you are using supports fractional seconds on file create time.
I suggest you insert a call to time.sleep(1) in your loop so that the filesystem actually has a different timestamp on each created file.
It could be due to syncing. Just because you call write() on files in a certain order, it doesn't mean the data will be updated by the OS in that order.
Try calling f.flush() followed by os.fsync() on your file object before going to the next file. Giving some time between calls (using sleep()) might help also

Reading from a file using pickle and for loop in python

I have a file in which I have dumped a huge number of lists.Now I want to load this file into memory and use the data inside it.I tried to load my file using the "load" method of "pickle", However, for some reason it just gives me the first item in the file. actually I noticed that it only load the my first list into memory and If I want to load my whole file(a number of lists) then I have to iterate over my file and use "pickle.load(filename)" in each of the iterations i take.
The problem is that I don't know how to actually implement it with a loop(for or while), because I don't know when I reach the end of my file.
an example would help me a lot.
thanks
How about this:
lists = []
infile = open('yourfilename.pickle', 'r')
while 1:
try:
lists.append(pickle.load(infile))
except (EOFError, UnpicklingError):
break
infile.close()

Categories