We Keep Coding

Why do I using satpy on Himawari-8 standard data failed?

When I use satpy to read Himawari-8 standard datas, the terminal always says " 'filenames' was provided but is empty. "
The Himawari-8 has 16 observation bands, and it take one full disk picture every 10 minutes so for one folder of observation result which contains 16 folders(one folder is one band), and each band is separated into 10 parts so in one band folder it has 10 files.
The format of Himawari-8 standard data is .dat, and the data named as
HS_aaa_yyyymmdd_hhnn_Bbb_cccc_Rjj_Skkll.DAT.
H08: Himawari-8
yyyy: Observation start time(year)
mm: Observation start time(month)
dd: Observation start time(day)
hh: Observation start time(hour)
nn: Observation start time(min.)
bb: Band number (01 – 16)
cccc: Observation area and number, FLDK: Full Disk
jj: Spatial resolution
kk: segment number (01 – 10)
ll: total number of segments (01 – 10)
e.g. HS_H08_20210518_1100_B01_FLDK_R10_S0110.DAT
These are the all message from vscode:
(my_satpy_env) E:\HK\Python>python Process_SatHima_Imagery.py
Traceback (most recent call last):
File "E:\HK\Python\Process_SatHima_Imagery.py", line 18, in <module>
scn = Scene(filenames=files, reader='ahi_hsd',filter_parameters={'start_time': datetime(2021,5,25,2,00), 'end_time': datetime(2021,5,25,2,10)})
File "C:\Users\RSFBioL\anaconda3\envs\my_satpy_env\lib\site-packages\satpy\scene.py", line 108, in __init__
self._readers = self._create_reader_instances(filenames=filenames,
File "C:\Users\RSFBioL\anaconda3\envs\my_satpy_env\lib\site-packages\satpy\scene.py", line 157, in _create_reader_instances
return load_readers(filenames=filenames,
File "C:\Users\RSFBioL\anaconda3\envs\my_satpy_env\lib\site-packages\satpy\readers\__init__.py", line 546, in load_readers
raise ValueError("'filenames' was provided but is empty.")
ValueError: 'filenames' was provided but is empty.
I use the code from Github and he uploaded the code " https://github.com/gSasikala/ftp-himawari8-hsd/blob/main/examples/Processing_Satellite_Imagery.ipynb "
I use the editor, vs code, to write and the python version is 3.9.7 and I have installed anaconda and satpy to my computer.
Sorry that I'm a newbie at Python and Himawari-8 standard data, if there anything I missed just remind me.
Thanks for any reply or recommand to me.

The error message means that you are passing an empty list to the Scene object. So in this line of code (you can see it in the error traceback):
scn = Scene(filenames=files, reader='ahi_hsd',filter_parameters={'start_time': datetime(2021,5,25,2,00), 'end_time': datetime(2021,5,25,2,10)})
your files variable is an empty list ([]). Satpy can't load anything because it wasn't given any files. The notebook you linked to creates the files variable with this line of code:
files = glob.glob(r'D:\ftp_h8_hsd_25Jul\*.dat')
Which means get a list of all of the files ending with .dat in the directory D:\ftp_h8_hsd_25Jul\. This is a directory (folder) on the author of the notebook's Windows machine. If you don't have these data files on your local machine then you will need to download them from somewhere. If you have the files, then you need to change the D:\ftp_h8_hsd_25Jul\*.dat part of the code to match your download location. For example, if you are on linux and downloaded the files using a browser they may be in /home/<username>/Downloads/*.dat.
It isn't clear to me what you are trying to do or what your starting point is so I'm not sure I can help much further. Stackoverflow isn't the most widely used support forum for Satpy questions. If you'd like to chat more I'd suggest joining the Pytroll slack (see http://pytroll.github.io/#getting-in-touch), we'd be happy to help.

How to validate mp4 file or audio files in general with python?

I have a rest API built with Django rest framework, one of its serializers is to accept Base64file which is our audio file, now what I want is simply check and validate the decoded file so I can know if this a valid mp4 or any audio type in general or not.
the problem is, sometimes the audio file after upload and save is corrupted and can't be played, so doing this validation is essential to make sure that the file is sent correctly or it was sent corrupted at first place.
I have been digging on google and the Internet searching for anything can do this simple task but all I found was how to play audio or manipulate it, I didn't even find something that may raise an exception if the file is not valid when trying to open it.
for more info. I'm using django-extra-fields, I use Base64FileField to implement my Audio file field, they provided an example to do so for like PDF's, I'm trying to do this similar way for audio but what is holding me is doing the check for audio.
The example of PDF:
class PDFBase64File(Base64FileField):
ALLOWED_TYPES = ['pdf']
def get_file_extension(self, filename, decoded_file):
try:
PyPDF2.PdfFileReader(io.BytesIO(decoded_file))
except PyPDF2.utils.PdfReadError as e:
logger.warning(e)
else:
return 'pdf'
What is done so far:
class AudioBase64File(Base64FileField):
ALLOWED_TYPES = (
'amr',
'ogg',
'm4a',
'3gp',
'aac',
'mp4',
'mp3',
'flac'
)
INVALID_FILE_MESSAGE = ("Please upload a valid audio.")
INVALID_TYPE_MESSAGE = ("The type of the audio couldn't be determined.")
def get_file_extension(self, filename, decoded_file):
# missing validation
return 'mp4'

You can use ffmpeg.
You can read the file and see if there is any error or not. ffmpeg will report any error while reading the file.
You can also skip some parts of the video just to make it faster but reading a file without doing anything is pretty fast and should be good enough.
ffmpeg -v error -i file.mp4 -f null - 2>error.log
How can I check the integrity of a video file (avi, mpeg, mp4…)?
ffmpeg

cTrader decode protobuf message from Report API Events (tunnel)

i am dealing with cTrader Trading platform.
My project is written in python 3 on tornado.
And have issue in decoding the prtobuf message from report API Events.
Below will list everything what i achieved and where have the problem.
First cTrader have Rest API for Report
so i got the .proto file and generated it for python 3
proto file is called : cTraderReportingMessages5_9_pb2
from rest Report API getting the protobuf message and able to decode in the following way because i know which descriptor to pass for decoding
from models import cTraderReportingMessages5_9_pb2
from protobuf_to_dict import protobuf_to_dict
raw_response = yield async_client.fetch(base_url, method=method, body=form_data, headers=headers)
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(descriptors[endpoint]['decode'], raw_response.body)
descriptors[endpoint]['decode'] = is my descriptor know exactly which descriptor to pass to decode my message
my content from cTraderReportingMessages5_9_pb2
# here is .proto file generated for python 3 is too big cant paste content here
https://ufile.io/2p2d6
So until here using rest api and know exactly which descriptor to pass, i am able to decode protobuf message and go forward.
2. Now the issue i face
Connecting with python 3 to the tunnel on 127.0.0.:5672
i am listening for events and receiving this kind of data back
b'\x08\x00\x12\x88\x01\x08\xda\xc9\x06\x10\xb6\xc9\x03\x18\xa1\x8b\xb8\x01 \x00*\x00:\x00B\x00J\x00R\x00Z\x00b\x00j\x00r\x00z\x00\x80\x01\xe9\x9b\x8c\xb5\x99-\x90\x01d\x98\x01\xea\x9b\x8c\xb5\x99-\xa2\x01\x00\xaa\x01\x00\xb0\x01\x00\xb8\x01\x01\xc0\x0
1\x00\xd1\x01\x00\x00\x00\x00\x00\x00\x00\x00\xd9\x01\x00\x00\x00\x00\x00\x00\x00\x00\xe1\x01\x00\x00\x00\x00\x00\x00\x00\x00\xea\x01\x00\xf0\x01\x01\xf8\x01\x00\x80\x02\x00\x88\x02\x00\x90\x02\x00\x98\x02\x00\xa8\x02\x00\xb0\x02\x00\xb8\x02\x90N\xc0\x02\x00\xc8\x0
2\x00
as recommendation i got, i need to use same .proto file generated for python that i did in step 1 and decode the message but without any success because i don't know the descriptor need to be passed.
so in 1 step was doing and working perfect this way
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(descriptors[endpoint]['decode'], raw_response.body)
but in second step can not decode the message using in the same way, what i am missing or how to decode the message using same .proto file?

Finally found a workaround by my self, maybe is a primitive way but only this worked for me.
By the answer got from providers need to use same .proto file for both situations
SOLUTION:
1. Did list with all the descriptors from .proto file
here is .proto file generated for python 3 is too big cant paste content here
https://ufile.io/2p2d6
descriptors = [cTraderReportingMessages5_9_pb2.descriptor_1, cTraderReportingMessages5_9_pb2.descriptor_2]
2. Loop throw list and pass one by one
for d in descriptors:
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(d, raw_response.body)
3. Check if decoded_response is not blank
if decoded_response:
# descriptor was found
# response is decoded
else:
# no descriptor
4. After decoded response we go parse it into dict:
from protobuf_to_dict import protobuf_to_dict
decoded_response_to_dict = protobuf_to_dict(decoded_response)
This solution that spent weeks on it finally worked.

Winzip cannot open an archive created by python shutil.make_archive on windows. On ubuntu archive manager does fine

I am trying to return a zip file in django http response, the code goes something like...
archive = shutil.make_archive('testfolder', 'zip', MEDIA_ROOT, 'testfolder')
response = HttpResponse(FileWrapper(open(archive)),
content_type=mimetypes.guess_type(archive)[0])
response['Content-Length'] = getsize(archive)
response['Content-Disposition'] = "attachment; filename=test %s.zip" % datetime.now()
return response
Now when this code is executed on ubuntu the resulting downloaded file opens without any issue, but when its executed on windows the file created does not open in winzip (gives error 'Unsupported Zip Format').
Is there something very obvious I am missing here? Isn't python code supposed to be portable?
EDIT:
Thanks to J.F. Sebastian for his comment...
There was no problem in creating the archive, it was reading it back into the request. So, the solution is to change second line of my code from,
response = HttpResponse(FileWrapper(open(archive)),
content_type=mimetypes.guess_type(archive)[0])
to,
response = HttpResponse(FileWrapper(open(archive, 'rb')), # notice extra 'rb'
content_type=mimetypes.guess_type(archive)[0])
checkout, my answer to this question for more details...

The code you have written should work correctly. I've just run the following line from your snippet to generate a zip file and was able to extract on Linux and Windows.
archive = shutil.make_archive('testfolder', 'zip', MEDIA_ROOT, 'testfolder')
There is something funny and specific going on. I recommend you check the following:
Generate the zip file outside of Django with a script that just has that one liner. Then try and extract it on a Windows machine. This will help you rule out anything going on relating to Django, web server or browser
If that works then look at exactly what is in the folder you compressed. Do the files have any funny characters in their names, are there strange file types, or super long filenames.
Run a md5 checksum on the zip file in Windows and Linux just to make absolutely sure that the two files are byte by byte identical. To rule out any file corruption that might have occured.

Thanks to J.F. Sebastian for his comment...
I'll still write the solution here in detail...
There was no problem in creating the archive, it was reading it back into the request. So, the solution is to change second line of my code from,
response = HttpResponse(FileWrapper(open(archive)),
content_type=mimetypes.guess_type(archive)[0])
to,
response = HttpResponse(FileWrapper(open(archive, 'rb')), # notice extra 'rb'
content_type=mimetypes.guess_type(archive)[0])
because apparently, hidden somewhere in python 2.3 documentation on open:
The most commonly-used values of mode are 'r' for reading, 'w' for
writing (truncating the file if it already exists), and 'a' for
appending (which on some Unix systems means that all writes append to
the end of the file regardless of the current seek position). If mode
is omitted, it defaults to 'r'. The default is to use text mode, which
may convert '\n' characters to a platform-specific representation on
writing and back on reading. Thus, when opening a binary file, you
should append 'b' to the mode value to open the file in binary mode,
which will improve portability. (Appending 'b' is useful even on
systems that don’t treat binary and text files differently, where it
serves as documentation.) See below for more possible values of mode.
So, in simple terms while reading binary files, using open(file, 'rb') increases portability of your code (it certainly did in this case)
Now, it extracts without troubles, on windows...

os.stat failing on file with funky characters

Using Python 2.6.5 on Windows XP, I'm processing a directory of files by calling os.stat on them to obtain their size. The script is failing when it reaches a particular file that happens to have non-ASCII characters embedded within the name. The exception being thrown is that os.stat couldn't find the file specified. I know the file is there because I can play in iTunes or VLC Media Player.
The name of the file in question is
1-02 Só Danço Samba (Jazz Samba).m4a
Just in case the characters aren't being displayed, the string is
'1-02 So\xb4 Danc\xb8o Samba (Jazz Samba).m4a'
Is there something I should or could be doing to make the name acceptable to os.stat? BTW, trying to open the file in python also fails for the same reason.

Try inserting the line # coding=UTF-8 at the top of your python file (only makes a difference for unicode in your script, as Philipp points out), and make sure you are storing the file names as unicode instead of str.
Tested with the following:
# coding=UTF-8
import os
fname = u'/temp/1-02 Só Danço Samba (Jazz Samba).m4a'
print(os.stat(fname))