We Keep Coding

Why do I using satpy on Himawari-8 standard data failed?

When I use satpy to read Himawari-8 standard datas, the terminal always says " 'filenames' was provided but is empty. "
The Himawari-8 has 16 observation bands, and it take one full disk picture every 10 minutes so for one folder of observation result which contains 16 folders(one folder is one band), and each band is separated into 10 parts so in one band folder it has 10 files.
The format of Himawari-8 standard data is .dat, and the data named as
HS_aaa_yyyymmdd_hhnn_Bbb_cccc_Rjj_Skkll.DAT.
H08: Himawari-8
yyyy: Observation start time(year)
mm: Observation start time(month)
dd: Observation start time(day)
hh: Observation start time(hour)
nn: Observation start time(min.)
bb: Band number (01 – 16)
cccc: Observation area and number, FLDK: Full Disk
jj: Spatial resolution
kk: segment number (01 – 10)
ll: total number of segments (01 – 10)
e.g. HS_H08_20210518_1100_B01_FLDK_R10_S0110.DAT
These are the all message from vscode:
(my_satpy_env) E:\HK\Python>python Process_SatHima_Imagery.py
Traceback (most recent call last):
File "E:\HK\Python\Process_SatHima_Imagery.py", line 18, in <module>
scn = Scene(filenames=files, reader='ahi_hsd',filter_parameters={'start_time': datetime(2021,5,25,2,00), 'end_time': datetime(2021,5,25,2,10)})
File "C:\Users\RSFBioL\anaconda3\envs\my_satpy_env\lib\site-packages\satpy\scene.py", line 108, in __init__
self._readers = self._create_reader_instances(filenames=filenames,
File "C:\Users\RSFBioL\anaconda3\envs\my_satpy_env\lib\site-packages\satpy\scene.py", line 157, in _create_reader_instances
return load_readers(filenames=filenames,
File "C:\Users\RSFBioL\anaconda3\envs\my_satpy_env\lib\site-packages\satpy\readers\__init__.py", line 546, in load_readers
raise ValueError("'filenames' was provided but is empty.")
ValueError: 'filenames' was provided but is empty.
I use the code from Github and he uploaded the code " https://github.com/gSasikala/ftp-himawari8-hsd/blob/main/examples/Processing_Satellite_Imagery.ipynb "
I use the editor, vs code, to write and the python version is 3.9.7 and I have installed anaconda and satpy to my computer.
Sorry that I'm a newbie at Python and Himawari-8 standard data, if there anything I missed just remind me.
Thanks for any reply or recommand to me.

The error message means that you are passing an empty list to the Scene object. So in this line of code (you can see it in the error traceback):
scn = Scene(filenames=files, reader='ahi_hsd',filter_parameters={'start_time': datetime(2021,5,25,2,00), 'end_time': datetime(2021,5,25,2,10)})
your files variable is an empty list ([]). Satpy can't load anything because it wasn't given any files. The notebook you linked to creates the files variable with this line of code:
files = glob.glob(r'D:\ftp_h8_hsd_25Jul\*.dat')
Which means get a list of all of the files ending with .dat in the directory D:\ftp_h8_hsd_25Jul\. This is a directory (folder) on the author of the notebook's Windows machine. If you don't have these data files on your local machine then you will need to download them from somewhere. If you have the files, then you need to change the D:\ftp_h8_hsd_25Jul\*.dat part of the code to match your download location. For example, if you are on linux and downloaded the files using a browser they may be in /home/<username>/Downloads/*.dat.
It isn't clear to me what you are trying to do or what your starting point is so I'm not sure I can help much further. Stackoverflow isn't the most widely used support forum for Satpy questions. If you'd like to chat more I'd suggest joining the Pytroll slack (see http://pytroll.github.io/#getting-in-touch), we'd be happy to help.

How to read the .mat file from Visual SFM in Python Code?

Can someone help me with the Python code to read the .mat file generated from Visual SFM? You can download the .mat file from the link:
https://github.com/cvlab-epfl/tf-lift/tree/master/example
You can get a .mat file in the zip in the link and the file is what I am asking for help.
It seems to be an ASCII file. I do not know how to read the data in the file.
I tried to load the data in the .mat file with scipy.io.loadmat() but an error occurred as:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
raise ValueError('Unknown mat file type, version %s, %s' % ret)
ValueError: Unknown mat file type, version 20, 0
Can someone help me to load the data in the file with Python code?
Thanks for your help and replies sincerely.

If you mean this VisualSFM (http://ccwu.me/vsfm/doc.html), then the .mat file isn't a MATLAB .mat file, but a 'match' file.
From the website:
[name].sift stores all the detected SIFT features, and [name].mat stores the feature matches.
It seems there is C++ code for reading this file (http://ccwu.me/vsfm/MatchFile.zip) which you could use to write a python parser.
Additionally, it seems like there is a python socket interface to VSFM, which may allow you to do what you want https://github.com/nrhine1/vsfm_util

Can run multiple OS command using variable in Python?

Hello I'm trying to run multiple OS command using Python in Mac OS
I want to convert image to tif file by using convert command in Terminal. However I have 100 images to convert, I make Python program to make it easier.
import os
import subprocess
files = os.listdir("/Users/woonie/Downloads/test")
i=1
for file in files:
args=['convert',file,'-resize','100%','-type',"Grayscale","/Users/woonie/Downloads/test/kor.",i,"test.exp0.tif"]
subprocess.Popen(args)
i = i+1
convert filename -...- output_filename.exp0.tif is the form of convert command so I need to change file name and output_filename every time. I have filename list in files. And I want to put number of image after "test" so it becomes kor.test1.exp0.tif, kor.test2.exp0.tif, etc.
Traceback (most recent call last):
File "/Users/woonie/PycharmProjects/image_chage/change.py", line 9, in <module>
subprocess.Popen(args)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1637, in _execute_child
self.pid = _posixsubprocess.fork_exec(
TypeError: expected str, bytes or os.PathLike object, not int
But this error came out.
So I have changed that code to
args=['convert'+str(file)+'-resize','100%','-type',"Grayscale","/Users/woonie/Downloads/test/kor."+str(i)+"test.exp0.tif"]
But same error came out...
It was work when I put that command directly into Terminal.
Did I use subprocess wrong? or I can't make program what I want to

Use
subprocess.run(args)
instead of
subprocess.Popen(args)
That being said, your args array should probably look like
args=['convert', str(file), '-resize','100%','-type',"Grayscale","/Users/woonie/Downloads/test/kor."+str(i)+"test.exp0.tif"]
because the filename and the resize options are probably separate arguments.

There are quite a few issues with your code.
Rather than carry around long pathnames multi-levels deep in your input and output files, it is often better to just change directory to where the files are and deal with simple filenames - it is less error-prone and less of a maintenance nightmare.
So, consider starting your code with something more like:
import os
# Go to where the images are
os.chdir('/Users/woonie/Downloads/test')
Rather than use use os.listdir(), you get much more flexibility if you use globbing exactly the same as you would in the shell. So, if you want to process just JPEGs rather than TIFFs you produced on the previous run, use:
import glob
# Get list of JPEGs to process
JPEGs = glob.glob('*jpg')
I am unsure why you are using subprocess.Popen() - you would normally do that when you want to capture the output, you might as well use subprocess.run()
I don't know why you use -resize 100%. Where did that come from? The image should be 100% anyway.
When you want to formulate your output filename, you can more simply use f-strings. So, having got rid of all the long paths, your last parameter becomes:
f'kor.{i}test.exp0.tif'

Issues in copying files on iOS using NSStreams

I am trying to copy image and media files using NSStreams. I can not use NSFileManager copyItemAtPath, as I have to copy the file using streams.
The data is transferred over the network and the stream is read by a Python script that writes the data to a file. This worked fine on Mac OSX but when I tried in iOS,the file was not saved in the proper format.
I am able to copy all the files, but some of the metadata like dimensions (for image and media files), and duration (for media files) is missing in the copied file, and the kind is always Document. The other metadata is fine.
When I try to read the file attributes using the NSFileManager
[[NSFileManager defaultManager] attributesOfItemAtPath:#"filePath" error:&error];
It shows an error in the console:
The operation couldn't be completed. No such file or directory
I also observed that all the copied files, irrespective of the file extension (.png,.jpeg,.mov, .zip), has a kind of Document
How do I copy the source image metadata into the copied file?
Are there any Xcode optimizations I need to turn off?
OS : Mac OSX 10.8.4, iOS 6
Xcode : 4.6.3

This works for me for any type of file:
if (![NSFileManager.defaultManager copyItemAtPath: sourceFileName toPath: targetFileName error: &error]) {
NSAlert *alert = [NSAlert alertWithError: error];
[alert runModal];
return;
}

I found out it is an issue with file extension.Some junk characters appended after the file extension( something like 1.png\\\)

Decompress zip file with password fails - bug in Python?

I get a strange error in python. When I try to extract a password protected file using the zip module, I get an exception when trying to set "oy" as password. Everything else seems to work. A bug in ZipFile module?
import zipfile
zip = zipfile.ZipFile("file.zip", "r")
zip.setpassword("oy".encode('utf-8'))
zip.extractall() #Above password "oy" generates the error here
zip.close()
This is the exception I get:
Traceback (most recent call last):
File "unzip.py", line 4, in <module>
zip.extractall()
File "C:\Program Files\Python32\lib\zipfile.py", line 1002, in extrac
l
self.extract(zipinfo, path, pwd)
File "C:\Program Files\Python32\lib\zipfile.py", line 990, in extract
return self._extract_member(member, path, pwd)
File "C:\Program Files\Python32\lib\zipfile.py", line 1035, in _extra
member
shutil.copyfileobj(source, target)
File "C:\Program Files\Python32\lib\shutil.py", line 65, in copyfileo
buf = fsrc.read(length)
File "C:\Program Files\Python32\lib\zipfile.py", line 581, in read
data = self.read1(n - len(buf))
File "C:\Program Files\Python32\lib\zipfile.py", line 633, in read1
max(n - len_readbuffer, self.MIN_READ_SIZE)
zlib.error: Error -3 while decompressing: invalid block type
If I use UTF-16 as encoding I get this error:
zlib.error: Error -3 while decompressing: invalid distance too far back
EDIT
I have now tested on a virtual Linux machine with following stuff:
Python version: 2.6.5
I created a password protected zip file with zip -e file.zip
hello.txt
Now it seems the problem is something else. Now I can extract the zip file even if the password is wrong!
try:
zip.setpassword("ks") # "ks" is wrong password but it still extracts the zip
zip.extractall()
except RuntimeException:
print "wrong!"
Sometimes I can extract the zip file with an incorrect password. The file (inside the zip file) is then extracted but when I try to open it the information seems to be corrupted/decrypted.

If there's a problem with the password, usually you get the following exception:
RuntimeError: ('Bad password for file', <zipfile.ZipInfo object at 0xb76dec2c>)
Since your exception complains about block type, most probably your .zip archive is corrupted, have you tried to unpack it with standalone unzip utility?
Or maybe you have used something funny, like 7zip to create it, which makes incompatible .zip archives.
You don't provide enough information (OS version? Python version? ZIP archive creator and contents? are there many files in those archives or single file in single archive? do all those files give same errors, or you can unpack some of them?), so here's quick Q&A section, which should help you to find and remedy the problem.
Q1. Is this a bug in Python?
A1. Unlikely.
Q2. What might cause this behaviour?
A2. Broken zip files, incompatible zip compressors -- since you don't tell anything, it's hard to point the the exact cause.
Q3. How to find the cause?
A3. Try to isolate the problem, find the file which gives you an error, try to use zip.testzip() and/or decompress that particular file with different unzip utility, share the results. Only you have access to the problematic files, so nobody can help you unless you try to do something yourself.
Q4. How to fix this?
A4. You cannot. Use different zip extractor, ZipFile won't work.

Try using the testzip() method to check the file's integrity before extracting files.

It could be possibly a bug in zipfile, or a bug in your zip implementation. I noted that your line numbers do not match mine so I guess this is python 3.2 earlier than the current 3.2.3 release I have.
Now, as to your code, it does work for me on Python 3.2.3 on Linux. I suggest you update to the latest 3.2.x as there seem to be a number of bug fixes related to zipfile and zlib, including fixes for crashes.