What I'm trying to figure out is when I run this code for smaller numbers it returns the list just fine, but for larger numbers (I would call this small in the context of what I'm working on.) like 29996299, it will run for a long time, I've waited for 45 minutes with no results and had to end up killing the program. What I was wondering was whether there was a more efficient way to handle numbers whose scale was larger than 4 or 5 digits. I've tested a few permutations of the range function to see if there was a better way to handle the limits of the list I want to produce but nothing seems to have any effect on the amount of time it takes to do the computation. I'm new to python and am not that experienced as a programmer. Thank you for your time.
ran the program again before submitting this post and it took an hour and a half or so.
function of the program is to take the User selected number, use it to generate a lower bound, find all primes between the bound and input and append to list, then generate a secound upper bound and find all primes and then append to list, to create a list that extends forwards and backwards from the initial number.
the program works like I expect it to but not as quickly as I need it to since the numbers I'm going to be dealing with are going to get large quickly, almost doubling at each phase.
initial_num = input("Please enter a number. ")
lower_1 = int(initial_num) - 1000
upper_1 = int(initial_num)
list_1 = []
for num in range(lower_1,upper_1):
if num > 1:
for i in range(2,num):
if (num % i) == 0:
break
else:
list_1.append(num)
lower_2 = list_1[-1]
upper_2 = list_1[-1] + 2000
list_2 = []
for num in range(lower_2,upper_2 +1):
if num > 1:
for i in range(2,num):
if (num % i) == 0:
break
else:
list_2.append(num)
list_3 = list_1 + list_2[1:]
print list_3
You can use a more efficient algorithm to generate the entire list of prime numbers up to N. This is the Sieve of Erathostenes. Please have a look at the linked article, it even includes an example pseudocode. The basic idea of the algorithm is:
maintain L, a list of potentially prime numbers (initially all numbers from 2 to N)
pick the next prime number (p) as the first element of L (intially 2)
remove all numbers that are a multiple of p, up to N, since they cannot be prime
repeat from step 2
At the end you are left with a list of prime numbers.
An implementation in Pyhton from here
def eratosthenes2(n):
multiples = set()
for i in range(2, n+1):
if i not in multiples:
yield i
multiples.update(range(i*i, n+1, i))
print(list(eratosthenes2(100)))
To reduce memory consumpution you could consider usgin a bitset, storing one bit for each number. That should reduce memory usage by between 32 - 64 times. A bitset implementation is available for python here.
Related
I'm new to both Python and StackOverflow so I apologise if this question has been repeated too much or if it's not a good question. I'm doing a beginner's Python course and one of the tasks I have to do is to make a function that finds the next prime number after a given input. This is what I have so far:
def nextPrime(n):
num = n + 1
for i in range(1, 500):
for j in range(2, num):
if num%j == 0:
num = num + 1
return num
When I run it on the site's IDE, it's fine and everything works well but then when I submit the task, it says the runtime was too long and that I should optimise my code. But I'm not really sure how to do this, so would it be possible to get some feedback or any suggestions on how to make it run faster?
When your function finds the answer, it will continue checking the same number hundreds of times. This is why it is taking so long. Also, when you increase num, you should break out of the nested loop to that the new number is checked against the small factors first (which is more likely to eliminate it and would accelerate progress).
To make this simpler and more efficient, you should break down your problem in areas of concern. Checking if a number is prime or not should be implemented in its own separate function. This will make the code of your nextPrime() function much simpler:
def nextPrime(n):
n += 1
while not isPrime(n): n += 1
return n
Now you only need to implement an efficient isPrime() function:
def isPrime(x):
p,inc = 2,1
while p*p <= x:
if x % p == 0: return False
p,inc = p+inc,2
return x > 1
Looping from 1 to 500, especially because another loop runs through it, is not only inefficient, but also confines the range of the possible "next prime number" that you're trying to find. Therefore, you should make use of while loop and break which can be used to break out of the loop whenever you have found the prime number (of course, if it's stated that the number is less than 501 in the prompt, your approach totally makes sense).
Furthermore, you can make use of the fact that you only need check the integers less than or equal to the square root of the designated integer (which in python, is represented as num**0.5) to determine if that integer is prime, as the divisors of the integers always come in pair and the largest of the smaller divisor is always a square root, if it exists.
Given this plain is_prime1 function which checks all the divisors from 1 to sqrt(p) with some bit-playing in order to avoid even numbers which are of-course not primes.
import time
def is_prime1(p):
if p & 1 == 0:
return False
# if the LSD is 5 then it is divisible by 5 (i.e. not a prime)
elif p % 10 == 5:
return False
for k in range(2, int(p ** 0.5) + 1):
if p % k == 0:
return False
return True
Versus this "optimized" version. The idea is to save all the primes we have found until a certain number p, then we iterate on the primes (using this basic arithmetic rule that every number is a product of primes) so we don't iterate through the numbers until sqrt(p) but over the primes we found which supposed to be a tiny bit compared to sqrt(p). We also iterate only on half the elements, because then the largest prime would most certainly won't "fit" in the number p.
import time
global mem
global lenMem
mem = [2]
lenMem = 1
def is_prime2(p):
global mem
global lenMem
# if p is even then the LSD is off
if p & 1 == 0:
return False
# if the LSD is 5 then it is divisible by 5 (i.e. not a prime)
elif p % 10 == 5:
return False
for div in mem[0: int(p ** 0.5) + 1]:
if p % div == 0:
return False
mem.append(p)
lenMem += 1
return True
The only idea I have in mind is that "global variables are expensive and time consuming" but I don't know if there is another way, and if there is, will it really help?
On average, when running this same program:
start = time.perf_counter()
for p in range(2, 100000):
print(f'{p} is a prime? {is_prime2(p)}') # change to is_prime1 or is_prime2
end = time.perf_counter()
I get that for is_prime1 the average time for checking the numbers 1-100K is ~0.99 seconds and so is_prime2 (maybe a difference of +0.01s on average, maybe as I said the usage of global variables ruin some performance?)
The difference is a combination of three things:
You're just not doing that much less work. Your test case includes testing a ton of small numbers, where the distinction between testing "all numbers from 2 to square root" and testing "all primes from 2 to square root" just isn't that much of a difference. Your "average case" is roughly the midpoint of the range, 50,000, square root of 223.6, which means testing 48 primes, or testing 222 numbers if the number is prime, but most numbers aren't prime, and most numbers have at least one small factor (proof left as exercise), so you short-circuit and don't actually test most of the numbers in either set (if there's a factor below 8, which applies to ~77% of all numbers, you've saved maybe two tests by limiting yourself to primes)
You're slicing mem every time, which is performed eagerly, and completely, even if you don't use all the values (and as noted, you almost never do for the non-primes). This isn't a huge cost, but then, you weren't getting huge savings from skipping non-primes, so it likely eats what little savings you got from the other optimization.
(You found this one, good show) Your slice of primes took a number of primes to test equal to the square root of number to test, not all primes less than the square root of the number to test. So you actually performed the same number of tests, just with different numbers (many of them primes larger than the square root that definitely don't need to be tested).
A side-note:
Your up-front tests aren't actually saving you much work; you redo both tests in the loop, so they're wasted effort when the number is prime (you test them both twice). And your test for divisibility by five is pointless; % 10 is no faster than % 5 (computers don't operate in base-10 anyway), and if not p % 5: is a slightly faster, more direct, and more complete (your test doesn't recognize multiples of 10, just multiples of 5 that aren't multiples of 10) way to test for divisibility.
The tests are also wrong, because they don't exclude the base case (they say 2 and 5 are not prime, because they're divisible by 2 and 5 respectively).
First of all, you should remove the print call, it is very time consuming.
You should just time your function, not the print function, so you could do it like this:
start = time.perf_counter()
for p in range(2, 100000):
## print(f'{p} is a prime? {is_prime2(p)}') # change to is_prime1 or is_prime2
is_prime1(p)
end = time.perf_counter()
print ("prime1", end-start)
start = time.perf_counter()
for p in range(2, 100000):
## print(f'{p} is a prime? {is_prime2(p)}') # change to is_prime1 or is_prime2
is_prime2(p)
end = time.perf_counter()
print ("prime2", end-start)
is_prime1 is still faster for me.
If you want to hold primes in global memory to accelerate multiple calls, you need to ensure that the primes list is properly populated even when the function is called with numbers in random order. The way is_prime2() stores and uses the primes assumes that, for example, it is called with 7 before being called with 343. If not, 343 will be treated as a prime because 7 is not yet in the primes list.
So the function must compute and store all primes up to √49 before it can respond to the is_prime(343) call.
In order to quickly build a primes list, the Sieve of Eratosthenes is one of the fastest method. But, since you don't know in advance how many primes you need, you can't allocate the sieve's bit flags in advance. What you can do is use a rolling window of the sieve to move forward by chunks (of let"s say 1000000 bits at a time). When a number beyond your maximum prime is requested, you just generate more primes chunk by chunk until you have enough to respond.
Also, since you're going to build a list of primes, you might as well make it a set and check if the requested number is in it to respond to the function call. This will require generating more primes than needed for divisions but, in the spirit of accelerating subsequent calls, that should not be an issue.
Here's an example of an isPrime() function that uses that approach:
primes = {3}
sieveMax = 3
sieveChunk = 1000000 # must be an even number
def isPrime(n):
if not n&1: return n==2
global primes,sieveMax, sieveChunk
while n>sieveMax:
base,sieveMax = sieveMax, sieveMax + sieveChunk
sieve = [True]* sieveChunk
for p in primes:
i = (p - base%p)%p
sieve[i::p]=[False]*len(sieve[i::p])
for i in range(0, sieveChunk,2):
if not sieve[i]: continue
p = i + base
primes.add(p)
sieve[i::p] = [False]*len(sieve[i::p])
return n in primes
On the first call to an unknown prime, it will perform slower than the divisions approach but as the prime list builds up, it will provide much better response time.
After some trial and error I have found a solution which works very quickly for the Project Euler Problem 5. (I have found another way which correctly solved the example case (numbers 1-10) but took an eternity to solve the actual Problem.) Here it goes:
def test(n):
for x in range(2,21):
if n % x != 0:
return False
return True
def thwart(n):
for x in range(2,21):
if test(n/x):
n /= x
return n
raise TypeError
num = 1
for x in range(1,21):
num *= x
while True:
try:
num = thwart(num)
except TypeError:
break
print(num)
My main problem is understanding why calling thwart(num) repeatedly is enough to result in the correct solution. (I.e. why is it able to find the SMALLEST number and doesnt just spit out any number divisible by the numbers 1-20?)
I only had some vague thoughts when programming it and was surprised at how quickly it worked. But now I have trouble figuring out why exactly it even works... The optimized solutions of other people on SO Ive found so far were all talking about prime factors which I can't see how that would fit with my program...?
Any help is appreciated! Thanks!
Well this isn't really a coding issue but a mathematical issue. If you look at all the numbers from 1-20 as the prime sthat make them you'll get the following:
1, 2,3,2^2,5,2^3,7,2^3....2^2*5.
the interesting part here is that once you multiply by the highest exponent of every single factor in these numbers you will get a number that can be divided by each of the numbers between one and twenty.
Once you realize that the problem is a simple mathematical one and approach it as such you can use this basic code:
import math
primes = [2]
for n in range(3,21): #get primes between 1 and 20
for i in primes:
if n in primes:
break
if n%i == 0:
break
if i> math.sqrt(n):
primes.append(n)
break
s = 1
for i in primes:
for j in range(10): # no reason for 10, could as well be 5 because 2^5 >20
if i**j > 20:
s = s*(i**(j-1))
break
print s
Additionally, the hint that the number 2520 is the smallest number that can be divided by all numbers should make you understand how 2520 is chosen:
I have taken a photo for you:
As you can caculate, when you take the biggest exponents and multiply them you get the number 2520.
What your solution does
your solution basically takes the number which is 1*2*3*4..*20 and tries dividing it by every number between 2 to 20 in such a way that it will still remain relevant. By running it over and over you remove the un-needed numbers from it. early on it will remove all the unnecessary 2's by dividing by 2, returning the number and then being called again and divided by 2 again. Once all the two's have been eliminated it will eliminate all the threes, once all the unnecessary threes will be eliminated it will try dividing by 4 and it will se it wont work, continue to 5, 6, 7... and when it finishes the loop without being able to divide it will raise a TypeError and you will finish your program with the correct number. This is not an efficient way to solve this problem but it will work with small numbers.
I'm relatively new to the python world, and the coding world in general, so I'm not really sure how to go about optimizing my python script. The script that I have is as follows:
import math
z = 1
x = 0
while z != 0:
x = x+1
if x == 500:
z = 0
calculated = open('Prime_Numbers.txt', 'r')
readlines = calculated.readlines()
calculated.close()
a = len(readlines)
b = readlines[(a-1)]
b = int(b) + 1
for num in range(b, (b+1000)):
prime = True
calculated = open('Prime_Numbers.txt', 'r')
for i in calculated:
i = int(i)
q = math.ceil(num/2)
if (q%i==0):
prime = False
if prime:
calculated.close()
writeto = open('Prime_Numbers.txt', 'a')
num = str(num)
writeto.write("\n" + num)
writeto.close()
print(num)
As some of you can probably guess I'm calculating prime numbers. The external file that it calls on contains all the prime numbers between 2 and 20.
The reason that I've got the while loop in there is that I wanted to be able to control how long it ran for.
If you have any suggestions for cutting out any clutter in there could you please respond and let me know, thanks.
Reading and writing to files is very, very slow compared to operations with integers. Your algorithm can be sped up 100-fold by just ripping out all the file I/O:
import itertools
primes = {2} # A set containing only 2
for n in itertools.count(3): # Start counting from 3, by 1
for prime in primes: # For every prime less than n
if n % prime == 0: # If it divides n
break # Then n is composite
else:
primes.add(n) # Otherwise, it is prime
print(n)
A much faster prime-generating algorithm would be a sieve. Here's the Sieve of Eratosthenes, in Python 3:
end = int(input('Generate primes up to: '))
numbers = {n: True for n in range(2, end)} # Assume every number is prime, and then
for n, is_prime in numbers.items(): # (Python 3 only)
if not is_prime:
continue # For every prime number
for i in range(n ** 2, end, n): # Cross off its multiples
numbers[i] = False
print(n)
It is very inefficient to keep storing and loading all primes from a file. In general file access is very slow. Instead save the primes to a list or deque. For this initialize calculated = deque() and then simply add new primes with calculated.append(num). At the same time output your primes with print(num) and pipe the result to a file.
When you found out that num is not a prime, you do not have to keep checking all the other divisors. So break from the inner loop:
if q%i == 0:
prime = False
break
You do not need to go through all previous primes to check for a new prime. Since each non-prime needs to factorize into two integers, at least one of the factors has to be smaller or equal sqrt(num). So limit your search to these divisors.
Also the first part of your code irritates me.
z = 1
x = 0
while z != 0:
x = x+1
if x == 500:
z = 0
This part seems to do the same as:
for x in range(500):
Also you limit with x to 500 primes, why don't you simply use a counter instead, that you increase if a prime is found and check for at the same time, breaking if the limit is reached? This would be more readable in my opinion.
In general you do not need to introduce a limit. You can simply abort the program at any point in time by hitting Ctrl+C.
However, as others already pointed out, your chosen algorithm will perform very poor for medium or large primes. There are more efficient algorithms to find prime numbers: https://en.wikipedia.org/wiki/Generating_primes, especially https://en.wikipedia.org/wiki/Sieve_of_Eratosthenes.
You're writing a blank line to your file, which is making int() traceback. Also, I'm guessing you need to rstrip() off your newlines.
I'd suggest using two different files - one for initial values, and one for all values - initial and recently computed.
If you can keep your values in memory a while, that'd be a lot faster than going through a file repeatedly. But of course, this will limit the size of the primes you can compute, so for larger values you might return to the iterate-through-the-file method if you want.
For computing primes of modest size, a sieve is actually quite good, and worth a google.
When you get into larger primes, trial division by the first n primes is good, followed by m rounds of Miller-Rabin. If Miller-Rabin probabilistically indicates the number is probably a prime, then you do complete trial division or AKS or similar. Miller Rabin can say "This is probably a prime" or "this is definitely composite". AKS gives a definitive answer, but it's slower.
FWIW, I've got a bunch of prime-related code collected together at http://stromberg.dnsalias.org/~dstromberg/primes/
I've got, what I think is a valid solution to problem 2 of Project Euler (finding all even numbers in the Fibonacci sequence up to 4,000,000). This works for lower numbers, but crashes when I run it with 4,000,000. I understand that this is computationally difficult, but shouldn't it just take a long time to compute rather than crash? Or is there an issue in my code?
import functools
def fib(limit):
sequence = []
for i in range(limit):
if(i < 3):
sequence.append(i)
else:
sequence.append(sequence[i-1] + sequence[i-2])
return sequence
def add_even(x, y):
if(y % 2 == 0):
return x + y
return x + 0
print(functools.reduce(add_even,fib(4000000)))
The problem is about getting the Fibonacci numbers that are smaller than 4000000. Your code tries to find the first 4000000 Fibonacci values instead. Since Fibonacci numbers grow exponentially, this will reach numbers too large to fit in memory.
You need to change your function to stop when the last calculated value is more than 4000000.
Another possible improvement is to add the numbers as you are calculating them instead of storing them in a list, but this won't be necessary if you stop at the appropriate time.