I currently have a for-loop, which is going through an incredible number of iterations to check something, and when it goes to a new iteration, I need it to check whether or not a variable I have is the same size of the current iteration.
Here is an example code of what I'm doing:
import datetime
now = datetime.datetime.now()
printcounter = 0
for i in range(3,100000000000+1,2):
if (printcounter == 1000000000):
print(i,"at %d" %now.hour, "hours and %d" % now.minute, "minutes.")
printcounter = 0
else:
#Do operation
printcounter += 1
However, since it's going through possibly millions of math heavy operations before I get my answer, I noticed that by striping this code of the 'printcounter' variable and not giving me a progress report gave me a significant speedup, by whole minutes sometimes.
Is there any way of only checking whether or not the 'printercounter' variable is equal to 10000, however without making it check every single iteration?
I personally can't think of anyway without resorting to nesting for loops, which can get very dirty, and I'd rather not have.
By the way, I'm using Windows 8.1, Python 3.5.1, if that makes any difference.
Edit:
I understand that it takes a significant portion of time to print, however, if I instead print to a file; my harddisk being very fast, then I still get the same, albeit reduced, difference in time. Also, I have been wanting to get the solution to this implemented in a lot of other scripts, so even if it's not a major problem here, I'd still like to know how to do it.
Edit 2:
Perhaps it's my fault for not being clear. I was looking to see if it was possible to check a value every once in a while, not every single time. For example, I don't want my code to check if 'printcounter' is equal to 1000000000 when it's 1, that's ridiculous. I know machines operate ridiculously fast, and so it doesn't matter, but I was curious to see if it was possible to reduce the number of times it checks that way, rather than having a dumb code which allows itself to be sloppy or lazy just because it's quick enough to correct for it.
If you don't want to check the variable every iteration, make it unnecessary...
by doing something like this instead:
import datetime
iterations = 100000000000
subiterations = 10000
chunks, remaining = divmod(iterations, subiterations)
now = datetime.datetime.now()
printcounter = 0
for i in range(chunks):
for j in range(subiterations):
#Do operation
pass
printcounter += subiterations
print('{:,d} at {} hours {} minutes'.format(printcounter, now.hour, now.minute))
if remaining:
for j in range(remaining):
#Do operation
pass
printcounter += remaining
print('{:,d} at {} hours {} minutes'.format(printcounter, now.hour, now.minute))
The speedup isn't because of checking that variable. It's because of the print statement itself. So no, there's no way to speed it up further besides removing that statement.
And to answer your specific question explicitly: you could restructure your code such that it isn't necessary to make that check, for example, using nested for loops. But that will likely be slower. The time it takes to check that one boolean comparison is very small.
Since printcounter is incremented at every iteration, why not use nested for loops?
Something roughly like this:
import datetime
now = datetime.datetime.now()
for j in range(100):
print(j, "at %d" %now.hour, "hours and %d" % now.minute, "minutes.")
for i in range(1000000000):
#Do operation
It's not going to make much difference because the int math is small compared to the actual print statement, but:
import datetime
now = datetime.datetime.now()
step = 2
init = 3
for i in range(init, 100000000000+1, step):
if (i % 10000*step) == init: #since we start at 3 and step by 2
print(i,"at %d" %now.hour, "hours and %d" % now.minute, "minutes.")
# Do Stuff
This structure will eliminate a few operations, but none of them are slow operations. But in terms of code structuring this is how I'd do it.
Related
I am using the following code unchanged in form but changed in content:
import numpy as np
import matplotlib.pyplot as plt
import random
from random import seed
from random import randint
import math
from math import *
from random import *
import statistics
from statistics import *
n=1000
T_plot=[0];
X_relm=[0];
class Objs:
def __init__(self, xIn, yIn, color):
self.xIn= xIn
self.yIn = yIn
self.color = color
def yfT(self, t):
return self.yIn*t+self.yIn*t
def xfT(self, t):
return self.xIn*t-self.yIn*t
xi=np.random.uniform(0,1,n);
yi=np.random.uniform(0,1,n);
O1 = [Objs(xIn = i, yIn = j, color = choice(["Black", "White"])) for i,j
in zip(xi,yi)]
X=sorted(O1,key=lambda x:x.xIn)
dt=1/(2*n)
T=20
iter=40000
Black=[]
White=[]
Xrelm=[]
for i in range(1,iter+1):
t=i*dt
for j in range(n-1):
check=X[j].xfT(t)-X[j+1].xfT(t);
if check<0:
X[j],X[j+1]=X[j+1],X[j]
if check<-10:
X[j].color,X[j+1].color=X[j+1].color,X[j].color
if X[j].color=="Black":
Black.append(X[j].xfT(t))
else:
White.append(X[j].xfT(t))
Xrel=mean(Black)-mean(White)
Xrelm.append(Xrel)
plot1=plt.figure(1);
plt.plot(T_plot,Xrelm);
plt.xlabel("time")
plt.ylabel("Relative ")
and it keeps running (I left it for 10 hours) without giving output for some parameters simply because it's too big I guess. I know that my code is not faulty totally (in the sense that it should give something even if wrong) because it does give outputs for fewer time steps and other parameters.
So, I am focusing on trying to optimize my code so that it takes lesser time to run. Now, this is a routine task for coders but I am a newbie and I am coding simply because the simulation will help in my field. So, in general, any inputs of a general nature that give insights on how to make one's code faster are appreciated.
Besides that, I want to ask whether defining a function a priori for the inner loop will save any time.
I do not think it should save any time since I am doing the same thing but I am not sure maybe it does. If it doesn't, any insights on how to deal with nested loops in a more efficient way along with those of general nature are appreciated.
(I have tried to shorten the code as far as I could and still not miss relevant information)
There are several issues in your code:
the mean is recomputed from scratch based on the growing array. Thus, the complexity of mean(Black)-mean(White) is quadratic to the number of elements.
The mean function is not efficient. Using a basic sum and division is much faster. In fact, a manual mean is about 25~30 times faster on my machine.
The CPython interpreter is very slow so you should avoid using loops as much as possible (OOP code does not help either). If this is not possible and your computation is expensive, then consider using a natively compiled code. You can use tools like PyPy, Numba or Cython or possibly rewrite a part in C.
Note that strings are generally quite slow and there is no reason to use them here. Consider using enumerations instead (ie. integers).
Here is a code fixing the first two points:
dt = 1/(2*n)
T = 20
iter = 40000
Black = []
White = []
Xrelm = []
cur1, cur2 = 0, 0
sum1, sum2 = 0.0, 0.0
for i in range(1,iter+1):
t = i*dt
for j in range(n-1):
check = X[j].xfT(t) - X[j+1].xfT(t)
if check < 0:
X[j],X[j+1] = X[j+1],X[j]
if check < -10:
X[j].color, X[j+1].color = X[j+1].color, X[j].color
if X[j].color == "Black":
Black.append(X[j].xfT(t))
else:
White.append(X[j].xfT(t))
delta1, delta2 = sum(Black[cur1:]), sum(White[cur2:])
sum1, sum2 = sum1+delta1, sum2+delta2
cur1, cur2 = len(Black), len(White)
Xrel = sum1/cur1 - sum2/cur2
Xrelm.append(Xrel)
Consider resetting Black and White to an empty list if you do not use them later.
This is several hundreds of time faster. It now takes 2 minutes as opposed to >20h (estimation) for the initial code.
Note that using a compiled code should be at least 10 times faster here so the execution time should be no more than dozens of seconds.
As mentioned in earlier comments, this one is a bit too broad to answer.
To illustrate; your iteration itself doesn't take very long:
import time
start = time.time()
for i in range(10000):
for j in range(10000):
pass
end = time.time()
print (end-start)
On my not-so-great machine that takes ~2s to complete.
So the looping portion is only a tiny fraction of your 10h+ run time.
The detail of what you're doing in the loop is the key.
Whilst very basic, the approach I've shown in the code above could be applied to your existing code to work out which bit(s) are the least performant and then raise a new question with some more specific, actionable detail.
So this is my line of code so far,
def Adder (i,j,k):
if i<=j:
for x in range (i, j+1):
print(x**k)
else:
print (0)
What it's supposed to do is get inputs (i,j,k) so that each number between [i,j] is multiplied the power of k. For example, Adder(3,6,2) would be 3^2 + 4^2 + 5^2 + 6^2 and eventually output 86. I know how to get the function to output the list of numbers between i and j to the power of K but I don't know how to make it so that the function sums that output. So in the case of my given example, my output would be 9, 16, 25, 36.
Is it possible to make it so that under my if conditional I can generate an output that adds up the numbers in the range after they've been taken to the power of K?
If anyone can give me some advice I would really appreciate it! First week of any coding ever and I don't quite know how to ask this question so sorry for vagueness!
Question now Answered, thanks to everyone who responded so quickly!
You could use built-in function sum()
def adder(i,j,k):
if i <= j:
print(sum(x**k for x in range(i,j+1)))
else:
print(0)
The documentation is here
I'm not sure if this is what you want but
if i<=j:
sum = 0
for x in range (i, j+1):
sum = sum + x**k #sum += x**k for simplicity
this will give you the sum of the powers
Looking at a few of the answers posted, they do a good job of giving you pythonic code for your solution, I thought I could answer your specific questions:
How can I get my function to add together its output?
A perhaps reasonable way is to iteratively and incrementally perform your calculations and store your interim solutions in a variable. See if you can visualize this:
Let's say (i,j,k) = (3,7,2)
We want the output to be: 135 (i.e., the result of the calculation 3^2 + 4^2 + 5^2 + 6^2 + 7^2)
Use a variable, call it result and initialize it to be zero.
As your for loop kicks off with x = 3, perform x^2 and add it to result. So result now stores the interim result 9. Now the loop moves on to x = 4. Same as the first iteration, perform x^2 and add it to result. Now result is 25. You can now imagine that result, by the time x = 7, contains the answer to the calculation 3^2+4^2+5^2+6^2. Let the loop finish, and you will find that 7^2 is also added to result.
Once loop is finished, print result to get the summed up answer.
A thing to note:
Consider where in your code you need to set and initialize the _result_ variable.
If anyone can give me some advice I would really appreciate it! First week of any coding ever and I don't quite know how to ask this question so sorry for vagueness!
Perhaps a bit advanced for you, but helpful to be made aware I think:
Alright, let's get some nuance added to this discussion. Since this is your first week, I wanted to jot down some things I had to learn which have helped greatly.
Iterative and Recursive Algorithms
First off, identify that the solution is an iterative type of algorithm. Where the actual calculation is the same, but is executed over different cumulative data.
In this example, if we were to represent the calculation as an operation called ADDER(i,j,k), then:
ADDER(3,7,2) = ADDER(3,6,2)+ 7^2
ADDER(3,6,2) = ADDER(3,5,2) + 6^2
ADDER(3,5,2) = ADDER(3,4,2) + 5^2
ADDER(3,4,2) = ADDER(3,3,2) + 4^2
ADDER(3,3,2) = 0 + 3^2
Problems like these can be solved iteratively (like using a loop, be it while or for) or recursively (where a function calls itself using a subset of the data). In your example, you can envision a function calling itself and each time it is called it does the following:
calculates the square of j and
adds it to the value returned from calling itself with j decremented
by 1 until
j < i, at which point it returns 0
Once the limiting condition (Point 3) is reached, a bunch of additions that were queued up along the way are triggered.
Learn to Speak The Language before using Idioms
I may get down-voted for this, but you will encounter a lot of advice displaying pythonic idioms for standard solutions. The idiomatic solution for your example would be as follows:
def adder(i,j,k):
return sum(x**k for x in range(i,j+1)) if i<=j else 0
But for a beginner this obscures a lot of the "science". It is far more rewarding to tread the simpler path as a beginner. Once you develop your own basic understanding of devising and implementing algorithms in python, then the idioms will make sense.
Just so you can lean into the above idiom, here's an explanation of what it does:
It calls the standard library function called sum which can operate over a list as well as an iterator. We feed it as argument a generator expression which does the job of the iterator by "drip feeding" the sum function with x^k values as it iterates over the range (1, j+1). In cases when N (which is j-i) is arbitrarily large, using a standard list can result in huge memory overhead and performance disadvantages. Using a generator expression allows us to avoid these issues, as iterators (which is what generator expressions create) will overwrite the same piece of memory with the new value and only generate the next value when needed.
Of course it only does all this if i <= j else it will return 0.
Lastly, make mistakes and ask questions. The community is great and very helpful
Well, do not use print. It is easy to modify your function like this,
if i<=j:
s = 0
for x in range (i, j+1):
s += x**k
return s # print(s) if you really want to
else:
return 0
Usually functions do not print anything. Instead they return values for their caller to either print or further process. For example, someone may want to find the value of Adder(3, 6, 2)+1, but if you return nothing, they have no way to do this, since the result is not passed to the program. A side note, do not capitalize functions. Those are for classes.
Now i may be nitpicking here but i wanted to know which is computationally more efficient over a large number of iterations (assuming no restrictions on memory) purely in terms of time taken to run the overall programme.
i am asking mainly in terms of two languages python and c, since these are the two i use most.
for e.g. in c, something like:
int count, sum = 0, 0;
while (count < 99999){
if (((pow(count, 2))/10)%10 == 4) {// just some random operation
sum += pow(count, 2);
}
count++;
}
or
int count, sum, temp = 0, 0, 0;
while (count < 99999){
temp = pow(count, 2)
if (((temp/10)%10 == 4) {// just some random operation
sum += temp;
}
count++;
}
In python something like
for i in range (99999):
n = len(str(i))
print "length of string", str(i), "is", n
or
for i in range (99999):
temp = str(i)
n = len(temp)
print "length of string:, temp, "is", n
Now these are just random operations i thought on the fly, my main question is whether it is better create a new variable or just repeat the operation. i know the computing power required will change as i, count increases but just generalizing for a large number of iterations which is better.
I have tested both your codes in python with timeit, and the result was 1.0221271729969885 seconds for the first option, 1.0154028950055363 for the second option
I have done this test only with one iteration on each example and as you can see, they are extremely close to each other, too close to be reliable after only one iteration. If you want to learn more though, I suggest you do these tests yourself with timeit
However, the action you took here is to replace a variable assignment to temp with 2 calls for str(i), not one, and it can of course vary wildly for other functions that are more or less complicated and time consuming than str is
Caching does improve performance and readability of the code most of the time. There may be corner cases for some super fast operations, however I'd say the type of code used in second examples would always be better.
Cached value will most probably be stored in register, hence the one mov will be involved. According to this document there are not so many cheaper operations.
As for readability consider bigger function calls with more arguments, while reviewing code it will require more brainwork to check that calls are the same, moreover if function changes it may introduce bugs where one call has been modified and the other one has been forgotten (true mostly for Python).
All in all it should be readability driving your choices in such cases rather than performance gains. Unless profiler points at this piece of code.
So I am not a CS major and have hard time answering questions about a program's big(O) complexity.
I wrote the following routine to output the pairs of numbers in an array which sum to 0:
asd=[-3,-2,-3,2,3,2,4,5,8,-8,9,10,-4]
def sum_zero(asd):
for i in range(len(asd)):
for j in range(i,len(asd)):
if asd[i]+asd[j]==0:
print asd[i],asd[j]
Now if someone asks the complexity of this method I know since the first loop goes thorough all the n items it will be more than (unless I am wrong) but can someone explain how to find the correct complexity?
If there is better more efficient way of solving this?
I won't give you a full solution, but will try to guide you.
You should get a pencil and a paper, and ask yourself:
How many times does the statement print asd[i], asd[j] execute? (in worst case, meaning that you shouldn't really care about the condition there)
You'll find that it really depends on the loop above it, which gets executed len(asd) (denote it by n) times.
The only thing you need to know, how many times is the inner loop executed giving that the outer loop has n iterations? (i runs from 0 up to n)
If you still not sure about the result, just take a real example, say n=20, and calculate how many times is the lowest statement executed, this will give you a very good indication about the answer.
def sum_zero(asd):
for i in range(len(asd)): # len called once = o(1), range called once = o(1)
for j in range(i,len(asd)): # len called once per i times = O(n), range called once per i times = O(n)
if asd[i]+asd[j]==0: # asd[i] called twice per j = o(2*n²)
# adding is called once per j = O(n²)
# comparing with 0 is called once per j = O(n²)
print asd[i],asd[j] # asd[i] is called twice per j = O(2*n²)
sum_zero(asd) # called once, o(1)
Assuming the worst case scenario (the if-condition always being true):
Total:
O(1) * 3
O(n) * 2
O(n²) * 6
O(6n² + 2n + 3)
A simple program to demonstrate the complexity:
target= []
quadraditc = []
linear = []
for x in xrange(1,100):
linear.append(x)
target.append(6*(x**2) + 2*x + 3)
quadraditc.append(x**2)
import matplotlib.pyplot as plt
plt.plot(linear,label="Linear")
plt.plot(target,label="Target Function")
plt.plot(quadraditc,label="Quadratic")
plt.ylabel('Complexity')
plt.xlabel('Time')
plt.legend(loc=2)
plt.show()
EDIT:
As pointed out by #Micah Smith, the above answer is the worst case operations, the Big-O is actually O(n^2), since the constants and lower order terms are omitted.
A User will specify a time interval of n secs/mins/hours and then two times (start / stop).
I need to be able to take this interval, and then step through the start and stop times, in order to get a list of these times. Then after this, I will perform a database look up via a table.objects.filter, in order to retrieve the data corresponding to each time.
I'm making some ridiculously long algorithms at the moment and I'm positive there could be an easier way to do this. That is, a more pythonic way. Thoughts?
it fits nicely as a generator, too:
def timeseq(start,stop,interval):
while start <= stop:
yield start
start += interval
used as:
for t in timeseq(start,stop,interval):
table.objects.filter(t)
or:
data = [table.objects.filter(t) for t in timeseq(start,stop,interval)]
Are you looking for something like this? (pseudo code)
t = start
while t != stop:
t += interval
table.objects.filter(t)
What about ...
result = RelevantModel.objects.filter(relavant_field__in=[
start + interval * i
for i in xrange((start - end).seconds / interval.seconds)
])
... ?
I can't imagine this is very different from what you're already doing, but perhaps it's more compact (particularly if you weren't using foo__in=[bar] or a list comprehension). Of course start and end would be datetime.datetime objects and interval would be a datetime.timedelta object.