We Keep Coding

Parsing through data with variable values

I'm running into a confusing issue with a current project. Basically, I am analyzing a MIDI file byte by byte (yes, I have to do it this way). MIDI files are structured such that the data for any given event in a track is conveyed via a series of bytes called a 'message'. A MIDI device knows what type of message based on a status code, which is 1-3 bytes within the 'message' that have specific IDs for specific functions.
I need to be able to look through a MIDI file's data and record each 'message' as a self-contained object so that I can put them in sequence for later use. The problem I run into is that as I progress through the data, I need an efficient way to read each status code and match it with the function I've written that processes the data for that type of message. What I'm trying to do now is this:
First, at any given point in the overall MIDI file, I look at the next 3 bytes and compare them with my list of all of the possible 3 byte MIDI status codes. Here's my first problem. For all 3 byte status code (which is equivalent to 6 hexadecimal digits, which is the encoding that is often used in MIDI editing), the 5th-8th bits (aka the second hexadecimal digit) specifies the MIDI channel that the current message affects, which is variable and unpredictable. Since I can't predict the channel, I need to treat it as a wild card value. In other words, I need to somehow be able to take the 6 digits and ignore the 2nd when I reference it against my list of codes. I don't think I explained that well, so here's an example:
6 hex-digit code inside the file: Bn7800 where n is the channel number
List of 6 digit codes: ['Bn7800','Bn7900','FF2001','FF2F00',...]
I need to search through that list for the correct code while ignoring the 2nd digit since I have no way of telling what it is ahead of time.
My other issue is that once I have that code, I need to call the specific function that I've written that handles that particular status code. I've named all of the functions after the status code they correspond to, but the problem is, I don't know how to take the string that I find in the list above and use it to call a function. For instance:
I need: Bn7800
In the list, I find it! So I store it as a new variable called stat_code = 'Bn7800'.
But I can't then call my function in this way -> self.stat_code(data)
because stat_code isn't the name of the function I want. I want the function that's named after the value of stat_code.
I'm not sure how I can go about calling a function based on the value of a stored variable.
Sorry if this is confusing, I find it difficult to explain fully without trying to explain the whole MIDI file structure. Please ask me questions if something is unclear. Thanks!
EDIT: Here is some of the code:
self.event_status_codes = {'ff0002': ff_00_02,'ff01': ff_01,'ff02': ff_02,'ff03': ff03,'ff04': ff_04,'ff05': ff_05,'ff06': ff_06,'ff07': ff_07,'ff2001': ff_20_01,'ff2f00': ff_2f_00,'ff5103': ff_51_03,'ff5405': ff_54_05,'ff5804': ff_58_04,'ff5902': ff_59_02,'ff7f': ff_7f,'8n': _8n,'9n': _9n,'an': an,'bn': bn,'cn': cn,'dn': dn,'en': en,'bn7800': bn_78_00,'bn7900': bn_79_00,'bn7a': bn_7a,'bn7b00': bn_7b_00,'bn7c00': bn_7c_00,'bn7d00': bn7d00,'bn7e': bn_7e,'bn7f00': bn_7f_00,'f0': f0,'f7': f7}
if track_data[hbc:hbc+3] in self.event_status_codes:
status_code = track_data[hbc:hbc+3]
self.status_code(track_data)
For reference, track_data is the raw hexadecimal sequence that I'm parsing through and hbc is a simple pointer that iterates through the file one hex digit at a time. I posted the whole status code dictionary, but right now am mainly concerned about the 3 digit ones because they typically specify channel number. Also, it's technically a Python dictionary because I had to name each corresponding function something slightly different than the actual code to make it readable and functional, so you can ignore the values of each key in the dictionary.

I'm not sure how I can go about calling a function based on the value of a stored variable.
This is doable by referring to the function by name. In your question, you imply that you're calling the function on self, which makes it easier: you might be able to get the attribute containing the function pointer by using the getattr() function, and then call it:
stat_code = 'B67800'
func = getattr(self, f'function_name_{stat_code}')
value = func()
# do stuff with value
This also works for if your function is part of any other module - just replace self in that example with the name of the module you'd call it from.
If you need to call a globally-defined function (not part of a module), you might need to index into locals() or globals():
func = globals()[f'function_name_{stat_code}']
value = func()

how to open a file closest to the file specified python

I am using python and I wand to run a program the will open the file specified by the user. But the problem is that, if the user doesn't specify the exact file name, then it will give an error. If the user wants to open "99999-file-name.mp3"
and he has typed "filename.mp3", then how can the program open the file closest to the one specified?

First get a list of files in the particular folder
Then use difflib.get_close_matches like so:
difflib.get_close_matches(user_specified_file, list_of_files)
to find "good" matches.
N.B: Consider putting a providing a small cutoff i.e 0.1 as suggested by #tobias_k to ensure you do you get a match always as the default cutoff of 0.6 means sometimes nothing will be a "good match" for what the user entered.
Similarly if you need to get only one file name also pass in the optional parameter n=1 to get the closest match since if you don't specify it you will get the 3 best matches.

To answer this question, you need to first define "closest" because in computing this can mean very different things. If you want to compare strings and find the most similar, then one good way of doing that is checking the edit distance. There are Python libraries out there for that, i.e. https://pypi.python.org/pypi/editdistance.
You give it two strings and it tells you how much you have to change one string to get the other. As per the documentation:
>>> import editdistance
>>> editdistance.eval('banana', 'bahama')
2L
PS. Can't help but to mention that I think this is a bad idea. If you want to do sth with the file opened and the program starts opening random files, then either you're eventually gonna overwrite a file that is not meant to be overwritten or you try to process a file that can't be processed in your intended way. I would recommend using a file select box that you can easily use with tKinter for example (even though tKinter is cancer).

I can't delete cases from .sav files using spss with python

I have some .sav files that I want to check for bad data. What I mean by bad data is irrelevant to the problem. I have written a script in python using the spss module to check the cases and then delete them if they are bad. I do that within a datastep by defining a dataset object and then getting its case list. I then use
del datasetObj.cases[k]
to delete the problematic cases within the datastep.
Here is my problem:
Say I have a data set foo.sav and it is the active data set in spss, then I can run something like:
BEGIN PROGRAM PYTHON.
import spss
spss.StartDataStep()
datasetObj = spss.Dataset()
caselist = datasetObj.cases
del caselist[k]
spss.EndDataStep()
END PROGRAM.
from within the spss client and it will delete the case k from the data set foo.sav. But, if I run something like the following using the directory of foo.sav as the working directory:
import os, spss
pathname = os.curdir()
foopathname = os.path.join(pathname, 'foo.sav')
spss.Submit("""
GET FILE='%(foopathname)s'.
DATASET NAME file1.
DATASET ACTIVATE file1.
""" %locals())
spss.StartDataStep()
datasetObj = spss.Dataset()
caselist = datasetObj.cases
del caselist[3]
spss.EndDataStep()
from command line, then it doesn't delete the case k. Similar code which gets values will work fine. E.g.,
print caselist[3]
will print case k (when it is in the data step). I can even change the values for the various entries of a case. But it will not delete cases. Any ideas?
I am new to python and spss, so there may be something that I am not seeing which is obvious to others; hence why I am asking the question.

Your first piece of code did not work for me. I adjusted it as follows to get it working:
BEGIN PROGRAM PYTHON.
import spss
spss.StartDataStep()
datasetObj = spss.Dataset()
del datasetObj.cases[k]
spss.EndDataStep()
END PROGRAM.
Notice that, in your code, caselist is just a list, containing values taken from the datasetObj in SPSS. The attribute .cases belongs to datasetObj.
With spss.Submit, you can also delete cases (or actually, not select them) using the SPSS command SELECT IF. For example, if your file has a variable (column) named age, with values ranging from 0 to 100, you can delete all cases with an age lower than (in SPSS: lt or <) 25 using:
BEGIN PROGRAM PYTHON.
import spss
spss.Submit("""
SELECT IF age lt 25.
""")
END PROGRAM.
Don't forget to add some code to save the edited file.

caselist is not actually a regular list containing the dataset values. Although its interface is the list interface, it actually works directly with the dataset, so it does not contain a list of values. It just accesses operations on the SPSS side to retrieve, change, or delete values. The most important difference is that since Statistics is not keeping the data in memory, the size of the caselist is not limited by memory.
However, if you are trying to iterate over the cases with a loop using
range(spss.GetCaseCount())
and deleting some, the loop will eventually fail, because the actual case count reflects the deletions, but the loop limit doesn't reflect that. And datasetObj.cases[k] might not be the case you expect if an earlier case has been deleted. So you need to keep track of the deletions and adjust the limit or the k value appropriately.
HTH

Tips for reading in a complex file - Python

I have complex, variable text files that I want to read into Python, but I'm not sure what the best strategy would be. I'm not looking for you to code anything for me, just some tips about what modules would best suit my needs/tips etc.
The files look something like:
Program
Username: X Laser: X Em: X
exp 1
sample 1
Time: X Notes: X
Read 1 X data
Read 2 X data
# unknown number of reads
sample 2
Time: X Notes: X
Read 1 X data
...
# Unknown number of samples
exp 2
sample 1
...
# Unknown number of experiments, samples and reads
# The 4 spaces between certain words represent tabs
To analyse this data I need to get the data for each reading and know which sample and experiment it came from. Also, I can change the output file format but I think the way I have written it here is the easiest to read.
To read this file in to Python the best way I can think of would be to read it in row by row and search for key words with regular expressions. For example, search the row for the "exp" keyword and then record the number after it, then search for sample in the next line and so on. However, of course this would not work if a keyword was used in the 'notes' section.
So, I'm kind of stumped as to what would best suit my needs (it's hard to use something if you don't know it exists!)
Thanks for your time.

It's a typical task for a syntactic analyzer. In this case, since
lexical constructs do not cross line boundaries and there's a single construct ("statement") per line. In other words, each line is a single statement
full syntax for a single line can be covered by a set of regexes
the structure of compounds (=entities connecting multiple "statements" into something bigger) is simple and straightforward
a (relatively) simple scannlerless parser based on lines, DFA and the aforementioned set of regexes can be applied:
set up the initial parser state (=current position relative to various entities to be tracked) and the parse tree (=data structure representing the information from the file in a convenient way)
for each line
classify it, e.g. by matching against the regexes applicable to the current state
use the matched regex's groups to get the line's statement's meaningful parts
using these parts, update the state and the parse tree
See get the path in a file inside {} by python for an example. There, I do not construct a parse tree (wasn't needed) but only track the current state.

write table cell real-time python

I would like to loop trough a database, find the appropriate values and insert them in the appropriate cell in a separate file. It maybe a csv, or any other human-readable format.
In pseudo-code:
for item in huge_db:
for list_of_objects_to_match:
if itemmatch():
if there_arent_three_matches_yet_in_list():
matches++
result=performoperationonitem()
write_in_file(result, row=object_to_match_id, col=matches)
if matches is 3:
remove_this_object_from_object_to_match_list()
can you think of any way other than going every time through all the outputfile line by line?
I don't even know what to search for...
even better, there are better ways to find three matching objects in a db and have the results in real-time? (the operation will take a while, but I'd like to see the results popping out RT)

Assuming itemmatch() is a reasonably simple function, this will do what I think you want better than your pseudocode:
for match_obj in list_of_objects_to_match:
db_objects = query_db_for_matches(match_obj)
if len(db_objects) >= 3:
result=performoperationonitem()
write_in_file(result, row=match_obj.id, col=matches)
else:
write_blank_line(row=match_obj.id) # if you want
Then the trick becomes writing the query_db_for_matches() function. Without detail, I'll assume you're looking for objects that match in one particular field, call it type. In pymongo such a query would look like:
def query_db_for_matches(match_obj):
return pymongo_collection.find({"type":match_obj.type})
To get this to run efficiently, make sure your database has an index on the field(s) you're querying on by first calling:
pymongo_collection.ensure_index({"type":1})
The first time you call ensure_index it could take a long time for a huge collection. But each time after that it will be fast -- fast enough that you could even put it into query_db_for_matches before your find and it would be fine.