I want to write some code to do acoustic analysis and I'm trying to determine the proper tool(s) for the job. I would normally write something like this in Python using numpy and scipy and possibly Cython for the analysis part. I've discovered that the world of Python audio libraries is a bit chaotic, with scads of very limited packages in various states of development.
I've also come across a bunch of audio/acoustic specific languages like SuperCollider, Faust, etc. that seem to make the audio processing easy but may be limited in terms of IO and analysis capability.
I'm currently working on Linux with Alsa and PulseAudio installed by default. I would prefer not to involve and of the various and sundry other audio packages like Jack if possible, though that is not a hard requirement.
My primary interest in this question is to determine whether there is a domain specific language that will provide for quicker prototyping and testing or whether a general language like Python is more appropriate. Thanks.
I've got a lot of experience with SuperCollider and Python (with and without Numpy). I do a lot of audio analysis, and I'm afraid the answer depends on what you want to do.
If you want to create systems that will input OR output audio in real time, then Python is not a good choice. The audio I/O libraries (as you say) are a bit sketchy. There's also a fundamental issue that Python's garbage collector is not really designed for realtime stuff. You should use a system that is designed from the ground up for realtime. SuperCollider is nice for this, and as caseyanderson notes, some of the standard building-blocks for audio analysis are right there. There are other environments too.
If you want to do hardcore work such as applying various machine learning algorithms, not necessarily in real time (i.e. if you can get away with reading/writing WAV files rather than live audio), then you should use a general-purpose programming language with wide support, and an ecosystem of good libraries for the extra things you want. Using Python with libs such as numpy and scikits-learn works great for this. It's good for quick prototyping, but not only does it lack solid realtime audio, it also has far fewer of the standard audio building-blocks. Those are two important things which hold you back when prototyping audio pipelines.
So, then, you're caught between these two options. Depending on your application you may be able to combine the two by manipulating the audio I/O in a realtime environment, and using OSC messaging or shell scripts to communicate with an external Python process. The limitation there is that you can't really throw masses of data around between the two (you can't sensibly pipe all your audio across to some other process, that'd be silly).
SuperCollider has lots of support for things along these lines, both as externals/plugins or Quarks. That said, it depends exactly what you want to do. If you are simply looking to detect events, Onsets.kr would be fine. If you are looking for frequency/pitch information, Pitch or Tartini would work (I find Tartini to be more accurate). If you are trying to track amplitude, a combination of Amplitude.ar and some simple math would also work.
Similarly, there is SpecCentroid.kr (for a kind of brightness analysis), Loudness.kr, SpecFlatness.kr, etc.
The above are all pretty general, and there are lots more (the JoshUGens externals package has some interesting FFT-related acoustics stuff). So I would recommend downloading the program, joining the mailing list (if you have further questions), which lives here, and poking around in the Externals, Quarks, and Standard UGens.
Nonetheless, since I am not sure what you are trying to do, I cannot make more concrete recommendations than the above combined with my feeling that it makes the most sense to go to SC for this, rather than writing all of your own tools in Python from scratch.
I'm not 100% sure what you want to do, but as an additional suggestion I would put forth: Spear with scripting in Common Lisp. If what you are doing involves a great deal of spectral analysis, then you can do the heavy Lifting in Spear, and script all of this using Common List with Common Music. Spear has some great tools in terms of editing out very specific partials.
Related
TLDR: Is there a Python library that allows me to get a application window frame as an image and rewrite it to the said application?
So the whole story is that I want to write an application using Python that does something similar to Lossless Scaling and Magpie. I want to grab an application window (a videogame window, for example), get the current frame as an image, then use some Machine Learning/Deep Learning algorithm (like FSR or DLSS) to upscale said image, then rewrite the current frame from the application with said upscaled image.
So far, I have been playing around with some upscaling algorithms like the one from Real-ESRGAN, but now my main problem is how to upscale the video game images in real-time. The only thing I found that does something related to what I need to do is PyAutoGUI. But this package only allows you to take screenshots of an application but not rewrite the graphics of said application.
I hope I have clarified my problem; feel free to comment if you still have any questions.
Thank you for reading this post, and have a good day.
Doing this with Python is going to be very difficult. A lot of the performance involved in this sort of thing is in avoiding as many memory copies as possible, and Python's idiom for string and bytes processing unfortunately makes quite a few additional copies in the course of any idiomatic program. I say this as a die-hard Python fan who is constantly trying to cram Python in everywhere it doesn't belong: you'd be better off doing this in Rust.
Update: After receiving some feedback from some folks with more direct experience in this sort of thing, I may have overstated the difficulty here. Many ML tools in Python provide zero-copy access, you can easily access and manipulate memory-mapped data from numpy and there is even a CUDA protocol for doing this to data in GPU memory, so while it's not exactly easy, as long as your operations are implemented as numpy operations and not as pure-python pixel-by-pixel logic, it shouldn't be much harder than other python machine learning applications which require access to native APIs for accessing their source data.
However, there's no way to access framebuffer data directly from python, so step 1 is going to be writing your own bindings over the relevant DirectX APIs. Since Magpie is open source, you can see which APIs it's using, for example, in its various C++ "Frame Source" backends. For example, this looks relevant: https://github.com/Blinue/Magpie/blob/42cfcba1222b07e4cec282eaff639aead229f123/Runtime/GraphicsCaptureFrameSource.cpp#L87
You can then look those APIs up on MSDN; that one, for example, is here: https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.direct3d11captureframepool.createfreethreaded?view=winrt-22621
CFFI is a good choice for writing native wrappers: https://cffi.readthedocs.io/en/latest/
Gluing these together appropriately is left as an exercise for the reader :).
Looking at 4-note polyphony, where notes are sampled audio (wav/mp3/ogg files). Is there some asynchronous audio library where I can tell the library to play an audio file (to completion, or a certain length), without blocking ? While the previous audio segment is still playing, I could invoke the library again to play another note, also without blocking. Thus I achieve polyphony, and can play upto 4-note chords ?
I am looking for an approach suitable for a very low end ARM9 (260MHz) + 64MB RAM type of device, running Linux. I am trying to keep the software as lean as possible, and thus cannot imagine putting a full-scale software synth on it. Also the user interaction of my simple 4-note polyphony, "poor-man's synth", needs to be programmatically achieved.
Only decent options I've found so far, seems to be SDL, but not clear how well it might fit my needs and meets the low-eright constraints.
Edit:
Found this SO Q&A but I am hoping that there is a more elegant solution, that has emerged in the 3 years since.
There is a list if Python sound/music libraries at https://wiki.python.org/moin/PythonInMusic; several of these appear to suit your needs (in the section Music programming in Python). Those with MIDI support may be particularly useful (there is also a section MIDI Mania), but others may also meet your needs.
I am looking at developing on an FPGA, but it would be easier for me to write the code in Python or Scala and have it converted to VHDL or Verilog.
I want to have many sensors hooked up to a device, and as the data comes in, calculations are done very quickly so it can be displayed on a video wall, so the FPGA would have as input dozens of sensors and several video controllers for the wall.
This is a library for code written in Scala. For this one I am curious if the code is written in Java and Scala would that affect what it generates.
http://simplifide.com/drupal6/
This is a python to VHDL converter.
http://www.myhdl.org/doku.php
With both of these I am curious as to the limitations.
I would prefer simplifide, as I am stronger at Scala than Python, but it seems that myhdl may be a more robust platform, just from some basic looking around.
UPDATE:
The reason for the FPGA is that it can do multiple tasks at one time very well, so when the data comes in, depending on the needs of users, based on the experiment, it would be easy to change the code on the FPGA to be able to adapt to the needs.
So, for example, if you have 8 x 3 different weather sensors on each floor of an office building, so there are temperature, wind speed, barometric sensors (8 of each sensor one each floor), and add sensors to test the deformation of the walls, then a real-time interface to read these in at the same time and keep updating the visual display may be helpful.
This is a made up example, but it would explain why an FPGA would be useful, otherwise I would need many different DSPs and then feed that into a computer to do the visual display, whereas an FPGA can do is faster, since it is hardware, with lower power needs.
There are two open-source libraries that can help make this easier in development, but I am not certain which would be a better platform to use to convert a program to VHDL/Verilog.
This is just one example. If I want to do a quantum circuit simulation on an FPGA, as this article suggests (http://www.cc.gatech.edu/computing/nano/documents/Radecka%20-%20FPGA%20Emulation%20of%20Quantum%20Circuits.pdf) then it would be easier to do this as a program, than building up a large circuit by hand.
Yes there is a python style HDL available and its free. MYHDL
This will generate VHDL or verilog . It can also simulate the code and output .VDI and you can look that in gtkwave
alternately if you want to edit the VHDL code you can use GHDL. google it you get lots of resources. there is OS available Fedora Electronics Lab it has all the tools to develop modern electronics.
All these are open source. Build and simulate using these tools.
To flash into the FPGA you need either xilinx or Altera tool chains to generate the bitstreams and flash them. All the best !
If you can afford it, I don't think anything will make your life easier than National Instruments' FPGA add-on for LabView. The visual environment of LabView is a reasonable fit for FPGA programming, and it takes care of many of the annoying details for you (unless you must worry about them as part of the algorithm, e.g. by building pipelines to hit your clock speed targets). Also, you may find that NI's real-time (non-FPGA) or DSP or DAQ or other solutions are adequate for your needs.
This is a made up example, but it would explain why an FPGA would be
useful, otherwise I would need many different DSPs and then feed that
into a computer to do the visual display, whereas an FPGA can do is
faster, since it is hardware, with lower power needs.
This depends entirely on the exact nature of the algorithms you need to execute.
There are two open-source libraries that can help make this easier in
development, but I am not certain which would be a better platform to
use to convert a program to VHDL/Verilog.
This is just one example. If I want to do a quantum circuit simulation
on an FPGA, as this article suggests
(http://www.cc.gatech.edu/computing/nano/documents/Radecka%20-%20FPGA%20Emulation%20of%20Quantum%20Circuits.pdf)
then it would be easier to do this as a program, than building up a
large circuit by hand.
It looks like you're looking for a High Level Synthesis tool, which neither of those is. For generating RTL for algorithms, code generators can definitely help but you'll still have to get your hands dirty with HDLs for other things.
I think you are looking for some tooling that Modaë Technologies is offering here. You can start with either Ruby or Python code, at the algorithmic/behavioral level. Their tools are capable of inferring data types automatically and converting the code to HDL (currently VHDL) at the RTL level.
One or two years ago I worked with MyHDL and LabVIEW. I wrote HDL in MyHDL, exported it as VHDL and imported them as external IP in LabVIEW.
LabView
It's nice for FPGA development, because you can keep track of clocks consumed by each sequential branch because of it's graphical representation. Pipelining and slicing your algorithm graphically is worth a lot to make sure that correct values are processed together.
However, when it comes to generating constants, initializer lists or generate recursive structures, the mapping from visual to actual hardware this approach is ... sub-optimal.
MyHDL
It's basically python, but the syntax looks a lot like Verilog with the decorators dedicated to be used as #always, etc. statements.
You're free to use any python-valid code in your HDL code. For testing purposes it might be applicable to write a test function with python before acutally implementing it on register transfer level (RTL).
When generating recursive structures you have the usual for statement.
Need a look-up table (LUT) for your algorithm? One line of list comprehension: Done.
For a full list of features, see the website.
Summary
LabVIEW is great for beginning, because you can focus on the actual implementation.
AS soon as you mastered to think parallel and on RTL and look into implementing more complex algorithms you may find MyHDL being better in
managing your IP without being bound to a proprietary platform
testing your code with full python power
sharing, versioning, etc.
I want to learn about graphical libraries by myself and toy with them a bit. I built a small program that defines lines and shapes as lists of pixels, but I cannot find a way to access the screen directly so that I can display the points on the screen without any intermediate.
What I mean is that I do not want to use any prebuilt graphical library such as gnome, cocoa, etc. I usually use Python to code and my program uses Python, but I can also code and integrate C modules with it.
I am aware that accessing the screen hardware directly takes away the multiplatform side of Python, but I disregard it for the sake of learning. So: is there any way to access the hardware directly in Python, and if so, what is it?
No, Python isn't the best choice for this type of raw hardware access to the video card. I would recommend writing C in DOS. Well, actually, I don't recommend it. It's a horrible thing to do. But, it's how I learned to do it, and it's probably about as friendly as you are going to get for accessing hardware directly without any intermediate.
I say DOS rather than Linux or NT because neither of those will give you direct access to the video hardware without writing a driver. That means having to learn the whole driver API, and you need to invoke a lot of "magic," that won't be very obvious because writing a video driver in Windows NT is fairly complicated.
I say C rather than Python because it gives you real pointers, and the ability to do stupid things with them. Under DOS, you can write to arbitrary physical memory addresses in C, which seems to be what you want. Trying to get Python working at all under an OS terrible enough to allow you direct hardware access would be a frustration in itself, even if you only wanted to do simple stuff that Python is good at.
And, as others have said, don't expect to use anything that you learn with this project in the real world. It may be interesting, but if you tried to write a real application in this way, you'd promptly be shot by whoever would have to maintain your code.
This seems a great self-learning path and I would add my two-cents worth and suggest you consider looking at the GObject, Cairo and Pygame modules some time.
The Python GObject module may be at a higher level than your current interest, but it enables pixel level drawing with Cairo (see the home page) as well as providing a general base for portable GUI apps using Python
Pygame also has pixel level methods as well as access methods to the graphics drivers (at the higher level) - here is a quick code example
This is rather an old thread now, but I stumbled upon it while musing the same question.
I used to program in assembly language. In my day, drawing on screen was simply(?) a matter of poking a value into a memory location. The value turned a pixel on or off and defined its colour.
The term 'poke' comes from Basic by the way, not assembler. In assembler, you had to write a value into a data register then tell the processor where to put the data using another command and specifying an address register, usually in hexadecimal form! And each different processor had its own assembly language. But hec was the code fast!
As hardware progressed, I found that graphics hardware programming became more and more complex. There's much more to it than now simply defining a pixel. The graphics subsystem has its own processor -- or processors -- and it's that that you've got to learn to talk to. The processor doesn't just plonk stuff in memory locations. (I believe that what used to be the fastest supercomputer in the world for a while ran on graphics chips!) 'Plonk' is not a Basic command by the way.
Sorry; I digress. In answer to the original poster's query, I believe that the goal of understanding the graphics-drawing process could have been best achieved by experimenting with a Raspberry Pi. It's Python compatible and hence perfect for the job. Its hardware is well documented and it's cheap and easy to use.
Hope this helps someone, Cheers, M
I've been sitting on this idea of working on a "networked intelligence" to look into some interesting ideas on the nature of intelligence and computers. I've decided to go about doing this by designing small robotic agents that will utilize PDP across some medium, (i.e. wifi/IR or something, to be decided), to enable them to gather large quantities of data independently and then be able to process and find trends in data efficiently by utilizing them together as a "supercomputer" (I always think it's odd using that term, but it's apt, one is utilizing multiple independent processing units in unison). I'm aware that Python has some PDP libraries available, and I was hoping to program the robots onto little Arduinos, and I've got a strong idea of how to do every component of the system except for actually implementing the PDP architecture across the system.
TL;DR? I want to make a bunch of little robots that can essentially connect together to form a small supercomputer and share and amalgamate information across all the agents. Is it feasible to create a PDP program that will freely relinquish parts of its processing power and then add in new ones.
I'm a pretty strong programmer, so if it's a matter of complexity and time, I'm willing to apply myself, but if it's an issue of having to strip apart some BIOS software and writing in Assembly, then I'd rather not. I'm not as familiar with PDP ideas as I would like to, and if you have any recommended reading to get me started, much appreciated.
Another note, the languages or platform is completely up for changes, I'd just like to see concrete evidence that one is better than the other.
Interesting idea, remins me on sensor networks.
You may find that an ardunio is a little underpowered for what you want. Perhaps it would be more efficient and easier to send the data back to a PC for processing.
If you want to continue with the ardunio idea, you could implement MapReduce which is a fairly simple construct that allows you to write distributed programs very easily.
I have a write up on the basics of MapReduce.
There is the famous Haddop implementation as well as Disco (python/erlang) and a very simple shell implementation called BashReduce that Last.fm created.