We Keep Coding

Python OSError: [Errno 9] Bad file descriptor after opening big json file

I just tried to read in a big json file (the Wikipedia json dump) in Python line by line and got the Error:
Traceback (most recent call last):
File "C:/.../test_json_wiki_file.py", line 19, in <module>
test_fct()
File "C:/.../test_json_wiki_file.py", line 12, in test_fct
for line in f:
OSError: [Errno 9] Bad file descriptor
Here is my code:
import json
def test_fct():
data = []
i = 0
with open('E:/.../20200713.json/20200713.json') as f:
for line in f:
data.append(json.loads(line))
i = i + 1
if i > 1:
input_file.close()
return data
test_data = test_fct()
The file size is around 700GB and the description (https://www.wikidata.org/wiki/Wikidata:Database_download) of the file states that it can be read line by line. I don't know if this is important but the E:/ hard drive is an external one.
Thank you for your help in advance :)

I don't have any firsthand knowledge on opening large files in python, but did you mean to have the path as 20200713.json/20200713.json. Is the first one actually a directory that has a .json extension? I'd also suggest trying to first load a smaller sample of the file (opening might be hard, so maybe just use the more command in terminal?).

How do I change/choose the file path in Python?

I'm trying to access a .txt file in Python and I can't figure out how to open the file. I ended up copying the contents into a list directly but I would like to know how to open a file for the future.
If I run this nothing prints. I think it's because Python is looking in the wrong folder/directory but I don't know how to change file paths.
sourcefile = open("CompletedDirectory.txt").read()
print(sourcefile)

The file CompletedDirectory.txt is probably empty.
If Python could not find the file, you would get a FileNotFoundError exception:
>>> sourcefile = open("CompletedDirectory.txt").read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'CompletedDirectory.txt'
Note that using read() in this way is not recommended. You're not closing the file properly. Use a context manager:
with open("CompletedDirectory.txt") as infile:
sourcefile = infile.read()
This will automatically close infile on leaving the with block.

You can get the current working directory:
import os
os.getcwd()
Then just concat it with the file container directory
os.path.join("targetDir", "fileName")

IO error in savetxt while using numpy

Im trying to read a dataset and collect meta features from it.
I get the following error after executing the python file.
Traceback (most recent call last):
File "runmeta.py", line 79, in <module>
np.savetxt('datasets/'+str(i)+'/metafeatures',meta[i],delimiter=',')
File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 940, in savetxt
fh = open(fname, 'w')
IOError: [Errno 2] No such file or directory: 'datasets/2/metafeatures'

the error you're getting is simply telling you it didn't find the file. i would suggest looking into absolute and relative file paths.
advice in error handling:
the error is triggered on this line
fh = open(fname, 'w')
so as you debug your program, look at the line python shows you. maybe change the variable fname. that is where i would start.
currently
fname = 'datasets/2/metafeatures'

Cannot determine type of file

Hi i have just started learning image processing using python.
When i tried to open an image that i downloaded from the net, I keep getting this error and I have no idea about how to resolve it. Can anyone please help me with this?
>>> dna=mahotas.imread('dna.jpeg')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\mahotas\io\freeimage.py", line 773, in imread
img = read(filename)
File "C:\Python27\lib\site-packages\mahotas\io\freeimage.py", line 444, in read
bitmap = _read_bitmap(filename, flags)
File "C:\Python27\lib\site-packages\mahotas\io\freeimage.py", line 490, in _read_bitmap
'mahotas.freeimage: cannot determine type of file %s' % filename)
ValueError: mahotas.freeimage: cannot determine type of file dna.jpeg

Hello this looks like a pretty old thread but I found it recently because I had the same problem.
I think that the error message is misleading because it implies that the type of file is incorrect.
I fixed the problem by including the full path to the image file. For example, it could look something like:
dna = mahotas.imread('C:\Documents\dna.jpeg')

Error in reading hdf file using h5py package for python

I want to extract data from hdf files that I downloaded from MODIS website. A sample file is provided in the link. I am reading the hdf file by using the following lines of code:
>>> import h5py
>>> f = h5py.File( 'MYD08_M3.A2002182.051.2008334061251.psgscs_000500751197.hdf', 'r' )
The error I am getting:
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
f = h5py.File( 'MYD08_M3.A2002182.051.2008334061251.psgscs_000500751197.hdf', 'r' )
File "C:\Python27\lib\site-packages\h5py\_hl\files.py", line 165, in __init__
fid = make_fid(name, mode, userblock_size, fapl)
File "C:\Python27\lib\site-packages\h5py\_hl\files.py", line 57, in make_fid
fid = h5f.open(name, h5f.ACC_RDONLY, fapl=fapl)
File "h5f.pyx", line 70, in h5py.h5f.open (h5py\h5f.c:1640)
IOError: unable to open file (File accessability: Unable to open file)
I have tried several other hdf files from different sources but I am getting the same error. What seems to be the fault here?

I think there could be two possible problems:
1) As the file extension is "hdf", maybe this is a HDF4 file. HDF5 files normally have ".hdf5" or ".h5·" extension. I am not sure if h5py is able to read HDF4 files.
2) Perhaps you have to change permissions to the file itself. If you are in a linux machine try: chmod +r file.hdf
You can try to open your file with HDFView. This software is available in several platforms. You can check the properties of the files very easily with it.

This sounds like a file permission error, or even file existence. Maybe add some checks such as
import os
hdf_file = 'MYD08_M3.A2002182.051.2008334061251.psgscs_000500751197.hdf'
if not os.path.isfile(hdf_file):
print 'file %s not found' % hdf_file
if not os.access(hdf_file, os.R_OK):
print 'file %s not readable' % hdf_file
f = h5py.File(hdf_file, 'r')

I had the same issue, and later identified that my file had only "read-only", which for some reason stopped the h5py to read it. After modifying the permission to "write", I was able to read it. Not sure why it was set up like this.