Right now I'm in need of a really fast screenshotter to feed screenshot into a CNN that will update mouse movement based on the screenshot. I'm looking to model the same kind of behavior presented in this paper, and similarly do the steps featured in Figure 6 (without the polar conversion). As a result of needing really fast input, I've searched around a bit and have been able to get this script from here slightly modified that outputs 10fps
from PIL import ImageGrab
from datetime import datetime
while True:
im = ImageGrab.grab([320, 180, 1600, 900])
dt = datetime.now()
fname = "pic_{}.{}.png".format(dt.strftime("%H%M_%S"), dt.microsecond // 100000)
im.save(fname, 'png')
Can I expect anything faster? I'd be fine with using a different program if it's available.
Writing to disk is very slow, and is probably a big part of what's making your loop take so long. Try commenting out the im.save() line and seeing how many screenshots can be captured (add a counter variable or something similar to count how many screenshots are being captured).
Assuming the disk I/O is the bottleneck, you'll want to split these two tasks up. Have one loop that just captures the screenshots and stores them in memory (e.g. to a dictionary with the timestamp as the key), then in a separate thread pull elements out of the dictionary and write them to disk.
See this question for pointers on threading in Python if you haven't done much of that before.
Related
I noticed a lack of good soundfont-compatible synthesizers written in Python. So, a month or so ago, I started some work on my own (for reference, it's here). Making this was also a challenge that I set for myself.
I keep coming up against the same problem again and again and again, summarized by this:
To play sound, a stream of data with a more-or-less constant rate of flow must be sent to the audio device
To synthesize sound in real time based on user input, little-to-no buffering can be used
Thus, there is a cap on the amount of time one 'buffer generation loop' can take
Python, as a language, simply cannot run fast enough to do synthesize sound within this time limit
The problem is not my code, or at least, I've tried to optimize it to extreme levels - using local variables in time-sensitive parts of the code, avoiding using dots to access variables in loops, using itertools for iteration, using pre-compiled macros like max, changing thread switching parameters, doing as few calculations as possible, making approximations, this list goes on.
Using Pypy helps, but even that starts to struggle after not too long.
It's worth noting that (at best) my synth at the moment can play about 25 notes simultaneously. But this isn't enough. Fluidsynth, a synth written in C, has a cap on the number of notes per instrument at 128 notes. It also supports multiple instruments at a time.
Is my assertion that Python simply cannot be used to write a synthesizer correct? Or am I missing something very important?
I'm attempting to create an autonomous RC car and my Python program is supposed to query the live stream on a given interval and add it to a training dataset. The data I want to collect is the array of the current image from OpenCV and the current speed and angle of the car. I would then like it to be loaded into Keras for processing.
I found out that numpy.save() just saves one array to a file. What is the best/most efficient way of saving data for my needs?
As with anything regarding performance or efficiency, test it yourself. The problem with recommendations for the "best" of anything is that they might change from year to year.
First, you should determine if this is even an issue you should be tackling. If you're not experiencing performance issues or storage issues, then don't bother optimizing until it becomes a problem. What ever you do, don't waste your time on premature optimizations.
Next, assuming it actually is an issue, try out every method for saving to see which one yields the smallest results in the shortest amount of time. Maybe compression is the answer, but that might slow things down? Maybe pickling objects would be faster? Who knows until you've tried.
Finally, weigh the trade-offs and decide which method you can compromise on; You'll almost never have one silver bullet solution. While your at it, determine if just adding more CPU, RAM or disk space at the problem would solve it. Cloud computing affords you a lot of headroom in those areas.
The most simple way is np.savez_compressed(). This saves any number of arrays using the same format as np.save() but encapsulated in a standard Zip file.
If you need to be able to add more arrays to an existing file, you can do that easily, because after all the NumPy ".npz" format is just a Zip file. So open or create a Zip file using zipfile, and then write arrays into it using np.save(). The APIs aren't perfectly matched for this, so you can first construct a StringIO "file", write into it with np.save(), then use writestr() in zipfile.
I'm a beginner with Python, but I'm at the final stages of a project I've been working on for the past year and I need help at the final step.
If needed I'll post my code though it's not really relevant.
Here is my problem:
I have a database of images, say for example a 100 images. On each one of those images, I run an algorithm called ICA. This algorithm is very heavy to compute and each picture individually usually takes 7-10 seconds, so 100 pictures can take 700-1000 seconds, and that's way too long to wait.
Thing is, my database of images never changes. I never add pictures or delete pictures, and so the output of the ICA algorithm is always the same. So in reality, every time I run my code, I wait forever and gain the same output every time.
Is there a way to save the data to the hard disk, and extract it at a later time?
Say, I compute the ICA of the 100 images, it takes forever, and I save it and close my computer. Now when I run the program, I don't want it to recompute ICA, I want it to use the values I stored previously.
Would such a thing be possible in Python? if so - how?
Since you're running computation-heavy algorithms, I'm going to assume you're using Numpy. If not, you should be.
Numpy has a numpy.save() function that lets you save arrays in binary format. You can then load them using numpy.load().
EDIT: Docs for aforementioned functions can be found here under the "NPZ Files" section.
Im trying the global alignment method from the Biopython module. Using it on short sequences is easy and gives an alignment matrix straightaway. However I really need to run it on larger sequences I have (an average lenght of 2000 nucleatides (or) characters). However I keep running into the Out of Memory error. I looked on SO and found this previous question. The answers provided are not helpfull as they link to the this same website which cant be accessed now.Apart from this I have tried these steps:
I tried using a 64-bit python since my personal computer has 4gb RAM.
sshed to a small school server with 16gb RAM and tried running on that. Its still running after close to 4 hours.
Since its is a small script im unsure how to modify it. ANy help will be greatly appreciated.
My script:
import os
from Bio import pairwise2
from Bio.pairwise2 import format_alignment
file_list = []
file_list = [each for each in os.listdir(os.getcwd()) if each.endswith(".dna")]
align_file = open("seq_align.aln","w")
seq_list = []
for each_file in file_list:
f_o = open(each_file,"r")
seq_list.append(f_o.read())
for a in pairwise2.align.globalmx(seq_list[0],seq_list[1]):
align_file.write(format_alignment(*a))
align_file.close()
So the school server finally completed the task. What I realized was that for each alignment there were 1000 matrices constructed and calculated. The align.globalxx method has a variable MAX_ALIGNMENT which is by default set to 1000. Changing it via monkey patching dint really change anything . The documentation says that, the method tries all possible alignments (yes 1000) but in my case all the matrices have the same alignment score (as well as few test sequences I tried). Finally a small piece of comment in the documentation states that if you need only 1 score, use the optional parameter one_alignment_only which accepts a boolean value only. All i did was this:
for a in pairwise2.align.globalmx(seq_list[0],seq_list[1],one_alignment_only=True):
align_file.write(format_alignment(*a))
This reduced the time considerably. However my PC still crashed so I assume this is a very memory intensive task and requires much more RAM (16gb on a small server). So probably a more efficient way of reading the sequences in the matrix should be thought of.
I've got a program that combines multiple(up to 48 images) that are all color coded images. Only running through 32 of the 1024x768 images takes like 7.5 minutes to process through the images and give the output image. I'm doing each image separately using getdata(wxPython) and then going through each image pixel by pixel to get the underlying value and adding it to the final image list before creating/displaying the final image. The entire program works just like I want it to work with one SLOW exception. It takes 7.5 minutes to complete to run. If I want to do anything slightly different I have to start all over.
What I'm wandering if I were to go through the images using threading that should save time but I'm concerned about the possibility of running into one problem...calling the same exact code at the same time by different threads. Is that something to be worried about or not? As well is there something I'm not thinking about that I should be worried about?
Read about GIL - threading won't do the trick with your problem. It does it, when you can represent your computation as some kind of pipeline, or when it uses a lot of IO (because only then synchronization will be useful, if you want to run the same thing on several threads instead of single threaded loop, you'll end up with random order of computations, one thread at the time).
You probably want to map your data (maybe parts of picture, maybe whole pictures) to many processes, as they can work in parallel, without taking care of GIL, because they will be different interpreter processes. You can use Pool here.