I inadvertently ran across a phenomenon that has me a bit perplexed. I was using IDLE for some quick testing, and I had some very simple code like this (which I have simplified for the purpose of illustration):
from time import clock # I am presently using windows
def test_speedup():
c = clock()
for i in range(1000):
print i,
print '=>', clock() - c
Now I ran this code like so (several times, with the same basic results):
# without pressing enter
>>> test_speedup()
0 1 2 3 4 . . . 997 998 999 => 12.8300956124 # the time to run code in seconds
# pressing enter ONLY 3 TIMES while the code ran
>>> test_speedup()
0 1 2 3 4 . . . 997 998 999 => 4.8656890089
# Pressing enter several times while the code ran
>>> test_speedup()
0 1 2 3 4 . . . 997 998 999 => 1.91522580283
My first hunch was that perhaps the code ran faster because the output system perhaps did not have to collate as many strings when I pressed enter (beginning anew each time I pressed enter). In fact, the output system always seemed to receive a boost in speed immediately after I pressed enter.
I have also reviewed the documentation here, but I am still a little puzzled why three newline characters would speed things up so substantially.
This question is admittedly somewhat trivial, unless one would like to know how to speed up the output system in IDLE without aborting the script. Still, I would like to understand what is happening with the output system here.
(I am using Python 2.7.x.)
Under the covers, IDLE is simulating a terminal on top of a Tk widget, which I'm pretty sure is ultimately derived from Text.
The presence of long lines slows down that widget a little bit. And appending to long lines takes longer than appending to short ones. If you really want to understand why this happens, you need to look at the Tcl code underlying the Tk Text widget, which Tkinter.Text is just a thin wrapper around.
Meanwhile, the Tkinter loop that IDLE runs does some funky things to allow it to accept input without blocking the loop. When it thinks there's nothing else going on, it may sometimes block for a short time until it sees input or a Tk event, and all of those short blocks may add up; pressing Enter may just cancel a few of them.
I'm not actually sure which of these two is more relevant here. You'd have to test it with a program that just spams long lines vs. one that inserts newlines every, say, 10 numbers, and see how much of the performance improvement you get that way.
From a quick test on my Mac, the original program visibly slows down gradually, and also has two quantum jumps in sluggishness around 500 and 920. So, it makes sense that hitting enter every 333 or so would speed things up a lot—you likely avoid both of those quantum slowdowns. If I change it to just remove the comma, the problem goes away.
Printing a newline for every number could of course cause a different slowdown, because that makes the terminal long enough to need to scroll, increases the scrollback buffer, etc. I didn't see that cost in IDLE, but run the same thing on the Windows command line, and you'll see problems with too many newlines just as bad as the problems with too few in IDLE. The best tradeoff should probably come from "square"-ish data, or data that's as close to 80 columns as possible without going over.
I believe this has something to do with the IDE that you are using to execute your code.
I have run your code (with syntax alterations for 3.3) and got the same time to execute both times:
999
=> 8.09542283367021
edit: This was performed in stock IDLE with Python 3.3. I spammed the enter key with multiple experiments and witnessed no time difference between string output on the control v. the experiment.
On a hunch, I decided to remove a print function and condense it into one line.:
print(i, "=>", clock()-c)
this produced 999 => 7.141783124325343
On this basis, I believe that the time differential you are seeing is due to the IDE sucking up more threads to process your code which results in faster times. Clearly, the time at the end is the total amount of time to compute and print your for loop.
To confirm my suspicion, I decided to put everything into a list of tuples and then print that list.
def new_speed_test():
speed_list = []
c = clock()
for i in range(1000):
speed_list.append((i, clock()-c))
print(speed_list[-1])
which output: (999, 0.0005668934240929957)
To conclude: My experiment confirms it is how your IDE handles output and CPU consumption.
Related
I'm trying to decide which one to use when I need to acquire lines of input from STDIN, so I wonder how I need to choose them in different situations.
I found a previous post (https://codereview.stackexchange.com/questions/23981/how-to-optimize-this-simple-python-program) saying that:
How can I optimize this code in terms of time and memory used? Note that I'm using different function to read the input, as sys.stdin.readline() is the fastest one when reading strings and input() when reading integers.
Is that statement true ?
The builtin input and sys.stdin.readline functions don't do exactly the same thing, and which one is faster may depend on the details of exactly what you're doing. As aruisdante commented, the difference is less in Python 3 than it was in Python 2, when the quote you provide was from, but there are still some differences.
The first difference is that input has an optional prompt parameter that will be displayed if the interpreter is running interactively. This leads to some overhead, even if the prompt is empty (the default). On the other hand, it may be faster than doing a print before each readline call, if you do want a prompt.
The next difference is that input strips off any newline from the end of the input. If you're going to strip that anyway, it may be faster to let input do it for you, rather than doing sys.stdin.readline().strip().
A final difference is how the end of the input is indicated. input will raise an EOFError when you call it if there is no more input (stdin has been closed on the other end). sys.stdin.readline on the other hand will return an empty string at EOF, which you need to know to check for.
There's also a third option, using the file iteration protocol on sys.stdin. This is likely to be much like calling readline, but perhaps nicer logic to it.
I suspect that while differences in performance between your various options may exist, they're liky to be smaller than the time cost of simply reading the file from the disk (if it is large) and doing whatever you are doing with it. I suggest that you avoid the trap of premature optimization and just do what is most natural for your problem, and if the program is too slow (where "too slow" is very subjective), you do some profiling to see what is taking the most time. Don't put a whole lot of effort into deciding between the different ways of taking input unless it actually matters.
As Linn1024 says, for reading large amounts of data input() is much slower.
A simple example is this:
import sys
for i in range(int(sys.argv[1])):
sys.stdin.readline()
This takes about 0.25μs per iteration:
$ time yes | py readline.py 1000000
yes 0.05s user 0.00s system 22% cpu 0.252 total
Changing that to sys.stdin.readline().strip() takes that to about 0.31μs.
Changing readline() to input() is about 10 times slower:
$ time yes | py input.py 1000000
yes 0.05s user 0.00s system 1% cpu 2.855 total
Notice that it's still pretty fast though, so you only really need to worry when you are reading thousands of entries like above.
It checks if it is TTY every time as input() runs by syscall and it works much more slow than sys.stdin.readline()
https://github.com/python/cpython/blob/af2f5b1723b95e45e1f15b5bd52102b7de560f7c/Python/bltinmodule.c#L1981
import sys
def solve(N, A):
for in range (N):
A.append(int(sys.stdin.readline().strip()))
return A
def main():
N = int(sys.stdin.readline().strip())
A = []
result = solve(N, A):
print(result)
main()
I'm struggling to troubleshoot a strange problem I've been having since starting to use Sikuli over multiple projects. I've been using the IDE and later tried to branch out due to having strange things happening with code. If I were to debug code earlier with popups, I can save the code, even restart my pc, even check the code in other text editors but the now non-existent popups (and old code) sometimes, well, pop up. In the end normally I end up ditching original files, and having to sometimes strangely comment out lines and re-add them one at a time (even though in the grand scale of things the end script was the same as before i did all that). I'm at a real loss for words.
It's making me struggle to differentiate between bad code and something going wrong elsewhere. Does anyone know what might cause this "phantom code"? Because I'm really at a loss.
And i would like advice as to what's going wrong with the while i < (inputvariable). I can't figure out what might be going wrong at all, am i over looking something?
I'm running all scripts through Sikuli IDE at the moment. I did want to learn how to write scripts and include sikuli hoping i could package it neatly but i couldn't seem to wrap my head around it.
For the while loop, where it's being compared to "SSLoops" i can't see why it's not breaking out of the loop when the criteria is met. (prints out above and beyond the number.)
I've had to do strange workarounds such as commenting out whole sections of code, trying to get it to work, and then slowly one by one reintroduce it till it matched the old script entirely. If I copied the script to a new file to make a cleaner copy, in hopes that if there is some sort of caching issue(?) it'd resolve, but I'd normally have to tinker around with it again.
BP = getBundlePath()
print(BP)
setBundlePath(BP + "\images")
BP2 = getBundlePath()
print(BP2)
# Regions
gameRegion = Region(230, 138, 1442, 875)
matchSpeedRegion = Region(1282, 920, 162, 91)
rewardRegion = Region()
def main():
SSLoops = input("How many times would you like to run Super Smash?")
SuperSmash(SSLoops)
def SuperSmash(SSLoops):
print(SSLoops)
i = 1
while i < SSLoops:
print(i)
print(SSLoops)
if exists("btnEnterSuperSmash.PNG"):
click("btnEnterSuperSmash.PNG")
while True:
if exists("btnReward.png"):
print("Completed! On to Rewards.")
#selectRewards()
break
else:
pass
if matchSpeedRegion.exists("btnStart.png"):
matchSpeedRegion.click("btnStart.png")
matchSpeedRegion.wait("btnRetreat.png", 3600)
if matchSpeedRegion.exists("btnSpeedUp.png"):
matchSpeedRegion.click("btnSpeedUp.png")
print("clicked x1")
print("clicking retreat")
matchSpeedRegion.click("btnRetreat.png")
matchSpeedRegion.wait(Pattern("btnRetreat.png").similar(0.65), 3600)
print("clicking okay")
gameRegion.click("btnOK.png")
wait(2)
gameRegion.wait("btnOK.png", 3600)
gameRegion.click("btnOK.png")
print("Completed!")
i = i + 1
if __name__ == '__main__':
main()
I have been getting popups saying "hey" because i had a loop in while true btnRewards to run a function to say "hey" - this would have hopefully on on to pick a reward out of 5 images in the end. But after removing it, as i'm trying to trouble shoot the main loop, it still pops up.
The loop that compares the user input variable to i just keeps on increasing. The indentation looks okay to me? but i must be wrong? or is something else making it go wrong?
I've been making the program run on a folder so the pictures to break the loop are immediately up, so it should have theoretically ran the amount of times inputted without anything else (1). Any help is deeply appreciated.
====
1
1
1
[log] CLICK on L[889,656]#S(0) (568 msec)
Completed! On to Rewards.
Completed!
2
1
[log] CLICK on L[889,656]#S(0) (565 msec)
Completed! On to Rewards.
Completed!
3
1
[log] CLICK on L[889,656]#S(0) (584 msec)
Completed! On to Rewards.
Completed!
4
1
====
Your problem: input() returns a string like so "4"
you then compare it using
while i < SSLoops:
which is always True and hence the loop does not end.
using
SSLoops = int(input("How many times would you like to run Super Smash?")) will solve your problem.
Be aware: this will crash if the given input cannot be converted to an integer value.
Suggestion: debug prints should look like so:
print "SSLoops =", SSLoops
so the output is better readable.
RaiMan from SikuliX (greetings to your cat ;-)
Im trying to get wifite (https://github.com/derv82/wifite) working on my 16x2 Adafruit LCD (http://www.adafruit.com/product/1110).
But for some Reason, if I press the specified button nothing happens.
I want to replace all pieces of code that look like this:
try
....
except KeyboardInterrupt:
....
With the code for the LCD Buttons:
try
....
except lcd.buttonPressed(lcd.SELECT):
....
But for some reason nothing happens if I press the button, I don't get a error - And wifite just keeps doing it's thin.
Any Idea why this isn't working how it should ?
Or is there maybe a better way ?
As others have pointed out except KeyboardInterrupt ... is a special construct in Python ... because a [Ctrl]-[C] is handled by your terminal driver and presents an "Interrupt" signal to your program (SIGINT under Unix, Linux and similar operating systems). (There is similar handling under Microsoft operating systems, with different details and slightly different terminology; but the Python code works the same either way.
There are other ways of accessing various other forms of hardware event ... and the Python exception module is not a typical way for those to be implemented.
In the case of the AdaFruit, or other Rasberry Pi devices, you'd use the modules they include with their package. You've already seen it, and presumably done the required import in your code (based on your reference to lcd.buttonPressed()). The problem is that this isn't how you use that function at all.
Read this carefully: https://blog.soton.ac.uk/pi/modules-available/adafruit-rgb-lcd-display/
... it includes example which show how you should be using it. That should be something like:
#!python
# Set up stuff here ...
got_event = False
while not got_event:
if lcd.buttonPressed(lcd.SELECT)
got_event = True
break
# Do other stuff, perhaps
# Or time.sleep(0.1)?
if got_event:
# In case we had other exit conditions from doing other stuff?
subprocess.call(YOUR_OTHER_PROGRAM)
Of course their code is a complete running program. I'm only highlighting a couple of points here. You need to loop around until you get the event your looking for (or loop around forever processing these events for as long as your device is up).
The lcd.buttonPressed() method is checking to see if the button has been pressed since the last time it was cleared; but the method/function doesn't block ... it returns True or False (or possibly None --- that wouldn't affect these code examples --- any "false" value means the button has not been pressed).
You want to sleep for some amount of time between checks. They use a half second delay in their example; I used a tenth of a second. (People will typically perceive a response within a tenth of a second from a computerized device as "instantaneous" while a half second delay will, typically, be slightly annoying). Checking as fast as you can will just needlessly heat up the electronics. Even a 0.01 (one hundredth of a second) sleep is sufficient ... but 0.05-0.1 are probably the best practice for something like this.
If I understand correctly, you want to have one of the buttons on the Adafruit LCD panel interrupt the program at almost any stage. (It would have been great if you'd mentioned this yourself!)
KeyboardInterrupt is a signal sent to a process, usually when a user presses Ctrl + c on a keyboard. To be more precise, a signal is sent by the OS and caught by the Python runtime, which raises a KeyboardInterrupt exception.
However, Ctrl + c is special! In almost any other case, when a user presses a key or a button, this is not translated into a special signal.
I'm not sure how you could get the behavior you want; this may depend quite a bit on the operating system you are using.
What you need is event detection try something like this. You might have to get familiar with Tkinter
from Tkinter import *
root = Tk()
def callback_end(event)
# do whatever ending procedure you want here
quit()
def main()
# do everything in your main code here
if lcd.buttonPressed(lcd.SELECT):
callback_end("<End>")
root.after(Period,main)
root.bind("<End>",callback_end) # if you press the end key it will call the callback_end function
root.after(Period,main) # repeats main every Period in miliseconds
root.mainloop()
I realize that this is not a complete answer but I hope it gets you going in the right direction
I'm trying to decide which one to use when I need to acquire lines of input from STDIN, so I wonder how I need to choose them in different situations.
I found a previous post (https://codereview.stackexchange.com/questions/23981/how-to-optimize-this-simple-python-program) saying that:
How can I optimize this code in terms of time and memory used? Note that I'm using different function to read the input, as sys.stdin.readline() is the fastest one when reading strings and input() when reading integers.
Is that statement true ?
The builtin input and sys.stdin.readline functions don't do exactly the same thing, and which one is faster may depend on the details of exactly what you're doing. As aruisdante commented, the difference is less in Python 3 than it was in Python 2, when the quote you provide was from, but there are still some differences.
The first difference is that input has an optional prompt parameter that will be displayed if the interpreter is running interactively. This leads to some overhead, even if the prompt is empty (the default). On the other hand, it may be faster than doing a print before each readline call, if you do want a prompt.
The next difference is that input strips off any newline from the end of the input. If you're going to strip that anyway, it may be faster to let input do it for you, rather than doing sys.stdin.readline().strip().
A final difference is how the end of the input is indicated. input will raise an EOFError when you call it if there is no more input (stdin has been closed on the other end). sys.stdin.readline on the other hand will return an empty string at EOF, which you need to know to check for.
There's also a third option, using the file iteration protocol on sys.stdin. This is likely to be much like calling readline, but perhaps nicer logic to it.
I suspect that while differences in performance between your various options may exist, they're liky to be smaller than the time cost of simply reading the file from the disk (if it is large) and doing whatever you are doing with it. I suggest that you avoid the trap of premature optimization and just do what is most natural for your problem, and if the program is too slow (where "too slow" is very subjective), you do some profiling to see what is taking the most time. Don't put a whole lot of effort into deciding between the different ways of taking input unless it actually matters.
As Linn1024 says, for reading large amounts of data input() is much slower.
A simple example is this:
import sys
for i in range(int(sys.argv[1])):
sys.stdin.readline()
This takes about 0.25μs per iteration:
$ time yes | py readline.py 1000000
yes 0.05s user 0.00s system 22% cpu 0.252 total
Changing that to sys.stdin.readline().strip() takes that to about 0.31μs.
Changing readline() to input() is about 10 times slower:
$ time yes | py input.py 1000000
yes 0.05s user 0.00s system 1% cpu 2.855 total
Notice that it's still pretty fast though, so you only really need to worry when you are reading thousands of entries like above.
It checks if it is TTY every time as input() runs by syscall and it works much more slow than sys.stdin.readline()
https://github.com/python/cpython/blob/af2f5b1723b95e45e1f15b5bd52102b7de560f7c/Python/bltinmodule.c#L1981
import sys
def solve(N, A):
for in range (N):
A.append(int(sys.stdin.readline().strip()))
return A
def main():
N = int(sys.stdin.readline().strip())
A = []
result = solve(N, A):
print(result)
main()
I am trying to make a program which has a raw_input in a loop, if anyone presses a key while the long loop is running the next raw_input takes that as input, how do I avoid that?
I don't know what else to add to this simple question. Do let me know if more is required.
EDIT
Some code
for i in range(1000):
var = raw_input("Enter the number")
#.... do some long magic and stuff here which takes afew seconds
print 'Output is'+str(output)
So if someone presses something inside the magic phase, that is take as the input for the next loop. That is where the problem begins. (And yes the loop has to run for 1000 times).
This works for me with Windows 7 64bit, python 2.7.
import msvcrt
def flush_input():
while msvcrt.kbhit():
msvcrt.getch()
I put the OS in the title, window 7 64 bit to be specific. I saw the
answers there. They do apply but by god they are so big. Aren't there
other n00b friendly and safer ways to take inputs?
Let me try to explain why you need to do such an elaborate process. When you press a key it is stored in a section of computer memory called keyboard buffer (not to be confused with stdin buffer). This buffer stores the key's pressed until it is processed by your program. Python doesn't provide any platform independent wrapper to do this task. You have to rely on OS specific system calls to access this buffer, and flush it, read it or query it. msvcrt is a MS VC++ Runtime Library and python msvcrt provides a wrapper over it. Unless you wan't a platform independent solution, it is quite straight forward.
Use msvcrt getch to read a character from console. msvcrt.kbhit() to test if a key press is present in the keyboard buffer and so on. So as MattH has shown, it just a couple of lines code. And if you think you are a noob take this opportunity to learn something new.
Just collect your input outside of the loop (before you enter the loop). Do you really want the user to enter 1000 numbers? well maybe you do. but just include a loop at the top and collect the 1000 numbers at the start, and store them in an array.
then on the bottom half change your loop so it just does all the work. then if someone enters something no the keyboard, it doesn't really matter anymore.
something like this:
def getvars(top=1000):
vars = []
for i in range(0,top):
anum = int(raw_input('%d) Please enter another number: ' % i))
vars.append(anum)
return vars
def doMagic(numbers):
top = len(numbers)
for number in numbers:
# do magic number stuff
print 'this was my raw number %s' % number
if __name__ == "__main__":
numbers = getvars(top=10)
doMagic(numbers)
presented in a different sort of way and less os dependent
There is another way to do it that should work. I don't have a windows box handy to test it out on but its a trick i used to use and its rather undocumented. Perhaps I'm giving away secrets... but its basically like this: trick the os into thinking your app is a screensaver by calling the api that turns on the screensaver function at the start of your magic calculations. at the end of your magic calculations or when you are ready to accept input again, call the api again and turn off the screensaver functionality.
That would work.
There is another way to do it as well. Since you are in windows this will work too. but its a fair amount of work but not really too much. In windows, the window that is foreground (at the top of the Z order) that window gets the 'raw input thread'. The raw input thread receives the mouse and keyboard input. So to capture all input all you need to do is create a function that stands up a transparent or (non transparent) window that sits at the top of the Z order setWindowPos would do the trick , have it cover the entire screen and perhaps display a message such as Even Geduld or Please wait
when you are ready to prompt the user for more input, you use showwindow() to hide the window, show the previous results, get the input and then reshow the window and capture the keys/mouse all over again.
Of course all these solutions tie you to a particular OS unless you implement some sort of try/except handling and/or wrapping of the low level windows SDK calls.