We Keep Coding

How to optimize and find output for large inputs?

For an input number N, I am trying to find the count of numbers of special pairs (x,y) such that the following conditions hold:
x != y
1 <= N <= 10^50
0 <= x <= N
0 <= y <= N
F(x) + F(y) is prime where F is sum of all digits of the number
finally print the output of the count modulo 1000000007
Sample Input: 2
Sample Output: 2
Explanation:
(0,2) Since F(0)+F(2)=2 which is prime
(1,2) Since F(1)+F(2)=3 which is prime
(2,1) is not considered as (1,2) is same as (2,1)
My code is:
def mod(x,y,p):
res=1
x=x%p
while(y>0):
if((y&1)==1):
res=(res*x)%p
y=y>>1
x=(x*x)%p
return res
def sod(x):
a=str(x)
res=0
for i in range(len(a)):
res+=int(a[i])
return res
def prime(x):
p=0
if(x==1 or x==0):
return False
if(x==2):
return True
else:
for i in range(2,(x//2)+1):
if(x%i==0):
p+=1
if(p==0):
return (True)
else:
return(False)
n=int(input())
res=[]
for i in range (n+1):
for j in range(i,n+1):
if(prime(sod(int(i))+sod(int(j)))):
if([i,j]!=[j,i]):
if([j,i] not in res):
res.append([i,j])
count=len(res)
a=mod(count,1,(10**9)+7)
print(res)
print(a)
I expect the output of 9997260736 to be 671653298 but the error is code execution timed out.

Already posted a bit too long comments, so changing it to answer:
When thinking of such problems, don't translate the problem directly to code but see what can you do only once or in different order.
As of now, you're doing N*N passes, each time calculating a sum of digits for x and y (not that bad) AND factoring each sum to check whether it's prime (really bad). That means for sum s you're checking whether it's prime s+1 times! (for 0+s, 1+(s-1), ..., (s-1)+1, s+0).
What you can do differently?
Let's see what we know:
Sum of digits is the same for many numbers.
Sum of sod(x) and sod(y) is the same for many values.
Number is prime during its 1st and nth check (and checking whether it's prime is costly).
So the best would be to calculate prime numbers only once, and each of the sum only once. But how to do that when we have many numbers?
Change the direction of thinking: get the prime number, split it into two numbers (sodx and sody), then generate x and y from those numbers.
Example:
Prime p = 7. That give possible sums as 0+7, 1+6, 2+5, 3+4.
Then for each sum you can generate a number, e.g. for N=101 and sod=1, you have 1, 10, 100, and for sod=2 you have 2, 11, 20, 101. You can possibly store this, but generating this should not be that bad.
Other optimisation:
You have to think how to limit generating prime numbers using your N:
given N with lenN digits (remember, lenN is ~log(N)), the biggest sum of digits possible is 9*lenN (for N consisting of only 9's). That means our sodx and sody are <= 9*lenN, so prime p = sodx + sody <= 18*lenN
Look: that means 18*lenN checking for whether number is prime vs N*N checks your previous algorithm had!

Prime numbers sum - for loop and big numbers

I'm running the following code to find the sum of the first 10,000,000 prime numbers.
How can I optimize it such that it doesn't take forever to obtain the result(the sum of prime numbers)?
sum=0
num=2
iterator=0
while iterator<10000000:
prime = True
for i in range(2,num):
if (num%i==0):
prime = False
if prime:
sum=sum+num
# print (num, sum, iterator)
iterator=iterator+1
num=num+1
print(sum)

the 10,000,000 th prime is approximately n * ln(n) + n * ln( ln(n) ) or ~188980383 ... then you can use a sieve to find all primes under that value (discard any extras ... (ie you will get about 50k extra prime numbers when using 10million, note this took approximately 8 seconds for me))
see also : Finding first n primes?
see also : Fastest way to list all primes below N

You can use the Sieve of Eratosthenes. It's a much faster method to find the first n prime numbers.

It turns out the Sieve of Eratosthenes is fairly simple to implement in Python by using correct slicing. So you can use it to recover the n first primes and sum them.
It was pointed out in Joran Beasley's answer that an upper bound for the n-th prime is n * ln(n) + n * ln( ln(n) ), which we can use and then discrard the extra primes. Note that this bound does not work for n smaller than 6.
from math import log
def sum_n_primes(n):
# Calculate the upper bound
upper = int(n * ( log(n) + log(log(n))))
# Prepare our Sieve, for readability we make index match the number by adding 0 and 1
primes = [False] * 2 + [True] * (upper - 1)
# Remove non-primes
for x in range(2, int(upper**(1/2) + 1)):
if primes[x]:
primes[2*x::x] = [False] * (upper // x - 1) # Replace // by / in Python2
# Sum the n first primes
return sum([x for x, is_prime in enumerate(primes) if is_prime][:n])
It takes a few seconds, but outputs that the sum of the 10,000,000 first primes is 870530414842019.
If you need n to be any higher, one solution would be to improve your upper bound.

Most efficient way to find all factors with GMPY2 (or GMP)?

I know there's already a question similar to this, but I want to speed it up using GMPY2 (or something similar with GMP).
Here is my current code, it's decent but can it be better?
Edit: new code, checks divisors 2 and 3
def factors(n):
result = set()
result |= {mpz(1), mpz(n)}
def all_multiples(result, n, factor):
z = mpz(n)
while gmpy2.f_mod(mpz(z), factor) == 0:
z = gmpy2.divexact(z, factor)
result |= {mpz(factor), z}
return result
result = all_multiples(result, n, 2)
result = all_multiples(result, n, 3)
for i in range(1, gmpy2.isqrt(n) + 1, 6):
i1 = mpz(i) + 1
i2 = mpz(i) + 5
div1, mod1 = gmpy2.f_divmod(n, i1)
div2, mod2 = gmpy2.f_divmod(n, i2)
if mod1 == 0:
result |= {i1, div1}
if mod2 == 0:
result |= {i2, div2}
return result
If it's possible, I'm also interested in an implementation with divisors only within n^(1/3) and 2^(2/3)*n(1/3)
As an example, mathematica's factor() is much faster than the python code. I want to factor numbers between 20 and 50 decimal digits. I know ggnfs can factor these in less than 5 seconds.
I am interested if any module implementing fast factorization exists in python too.

I just made some quick changes to your code to eliminate redundant name lookups. The algorithm is still the same but it is about twice as fast on my computer.
import gmpy2
from gmpy2 import mpz
def factors(n):
result = set()
n = mpz(n)
for i in range(1, gmpy2.isqrt(n) + 1):
div, mod = divmod(n, i)
if not mod:
result |= {mpz(i), div}
return result
print(factors(12345678901234567))
Other suggestions will need more information about the size of the numbers, etc. For example, if you need all the possible factors, it may be faster to construct those from all the prime factors. That approach will let you decrease the limit of the range statement as you proceed and also will let you increment by 2 (after removing all the factors of 2).
Update 1
I've made some additional changes to your code. I don't think your all_multiplies() function is correct. Your range() statement isn't optimal since 2 is check again but my first fix made it worse.
The new code delays computing the co-factor until it knows the remainder is 0. I also tried to use the built-in functions as much as possible. For example, mpz % integer is faster than gmpy2.f_mod(mpz, integer) or gmpy2.f_mod(integer, mpz) where integer is a normal Python integer.
import gmpy2
from gmpy2 import mpz, isqrt
def factors(n):
n = mpz(n)
result = set()
result |= {mpz(1), n}
def all_multiples(result, n, factor):
z = n
f = mpz(factor)
while z % f == 0:
result |= {f, z // f}
f += factor
return result
result = all_multiples(result, n, 2)
result = all_multiples(result, n, 3)
for i in range(1, isqrt(n) + 1, 6):
i1 = i + 1
i2 = i + 5
if not n % i1:
result |= {mpz(i1), n // i1}
if not n % i2:
result |= {mpz(i2), n // i2}
return result
print(factors(12345678901234567))
I would change your program to just find all the prime factors less than the square root of n and then construct all the co-factors later. Then you decrease n each time you find a factor, check if n is prime, and only look for more factors if n isn't prime.
Update 2
The pyecm module should be able to factor the size numbers you are trying to factor. The following example completes in about a second.
>>> import pyecm
>>> list(pyecm.factors(12345678901234567890123456789012345678901, False, True, 10, 1))
[mpz(29), mpz(43), mpz(43), mpz(55202177), mpz(2928109491677), mpz(1424415039563189)]

There exist different Python factoring modules in the Internet. But if you want to implement factoring yourself (without using external libraries) then I can suggest quite fast and very easy to implement Pollard-Rho Algorithm. I implemented it fully in my code below, you just scroll down directly to my code (at the bottom of answer) if you don't want to read.
With great probability Pollard-Rho algorithm finds smallest non-trivial factor P (not equal to 1 or N) within time of O(Sqrt(P)). To compare, Trial Division algorithm that you implemented in your question takes O(P) time to find factor P. It means for example if a prime factor P = 1 000 003 then trial division will find it after 1 000 003 division operations, while Pollard-Rho on average will find it just after 1 000 operations (Sqrt(1 000 003) = 1 000), which is much much faster.
To make Pollard-Rho algorithm much faster we should be able to detect prime numbers, to exclude them from factoring and don't wait unnecessarily time, for that in my code I used Fermat Primality Test which is very fast and easy to implement within just 7-9 lines of code.
Pollard-Rho algorithm itself is very short, 13-15 lines of code, you can see it at the very bottom of my pollard_rho_factor() function, the remaining lines of code are supplementary helpers-functions.
I implemented all algorithms from scratch without using extra libraries (except random module). That's why you can see my gcd() function there although you can use built-in Python's math.gcd() instead (which finds Greatest Common Divisor).
You can see function Int() in my code, it is used just to convert Python's integers to GMPY2. GMPY2 ints will make algorithm faster, you can just use Python's int(x) instead. I didn't use any specific GMPY2 function, just converted all ints to GMPY2 ints to have around 50% speedup.
As an example I factor first 190 digits of Pi!!! It takes 3-15 seconds to factor them. Pollard-Rho algorithm is randomized hence it takes different time to factor same number on each run. You can restart program again and see that it will print different running time.
Of course factoring time depends greatly on size of prime divisors. Some 50-200 digits numbers can be factoring within fraction of second, some will take months. My example 190 digits of Pi has quite small prime factors, except largest one, that's why it is fast. Other digits of Pi may be not that fast to factor. So digit-size of number doesn't matter very much, only size of prime factors matter.
I intentionally implemented pollard_rho_factor() function as one standalone function, without breaking it into smaller separate functions. Although it breaks Python's style guide, which (as I remember) suggests not to have nested functions and place all possible functions at global scope. Also style guide suggests to do all imports at global scope in first lines of script. I did single function intentionally so that it is easy copy-pastable and fully ready to use in your code. Fermat primality test is_fermat_probable_prime() sub-function is also copy pastable and works without extra dependencies.
In very rare cases Pollard-Rho algorithm may fail to find non-trivial prime factor, especially for very small factors, for example you can replace n inside test() with small number 4 and see that Pollard-Rho fails. For such small failed factors you can easily use your Trial Division algorithm that you implemented in your question.
Try it online!
def pollard_rho_factor(N, *, trials = 16):
# https://en.wikipedia.org/wiki/Pollard%27s_rho_algorithm
import math, random
def Int(x):
import gmpy2
return gmpy2.mpz(x) # int(x)
def is_fermat_probable_prime(n, *, trials = 32):
# https://en.wikipedia.org/wiki/Fermat_primality_test
import random
if n <= 16:
return n in (2, 3, 5, 7, 11, 13)
for i in range(trials):
if pow(random.randint(2, n - 2), n - 1, n) != 1:
return False
return True
def gcd(a, b):
# https://en.wikipedia.org/wiki/Greatest_common_divisor
# https://en.wikipedia.org/wiki/Euclidean_algorithm
while b != 0:
a, b = b, a % b
return a
def found(f, prime):
print(f'Found {("composite", "prime")[prime]} factor, {math.log2(f):>7.03f} bits... {("Pollard-Rho failed to fully factor it!", "")[prime]}')
return f
N = Int(N)
if N <= 1:
return []
if is_fermat_probable_prime(N):
return [found(N, True)]
for j in range(trials):
i, stage, y, x = 0, 2, Int(1), Int(random.randint(1, N - 2))
while True:
r = gcd(N, abs(x - y))
if r != 1:
break
if i == stage:
y = x
stage <<= 1
x = (x * x + 1) % N
i += 1
if r != N:
return sorted(pollard_rho_factor(r) + pollard_rho_factor(N // r))
return [found(N, False)] # Pollard-Rho failed
def test():
import time
# http://www.math.com/tables/constants/pi.htm
# pi = 3.
# 1415926535 8979323846 2643383279 5028841971 6939937510 5820974944 5923078164 0628620899 8628034825 3421170679
# 8214808651 3282306647 0938446095 5058223172 5359408128 4811174502 8410270193 8521105559 6446229489 5493038196
# n = first 190 fractional digits of Pi
n = 1415926535_8979323846_2643383279_5028841971_6939937510_5820974944_5923078164_0628620899_8628034825_3421170679_8214808651_3282306647_0938446095_5058223172_5359408128_4811174502_8410270193_8521105559_6446229489
tb = time.time()
print('N:', n)
print('Factors:', pollard_rho_factor(n))
print(f'Time: {time.time() - tb:.03f} sec')
test()
Output:
N: 1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489
Found prime factor, 1.585 bits...
Found prime factor, 6.150 bits...
Found prime factor, 20.020 bits...
Found prime factor, 27.193 bits...
Found prime factor, 28.311 bits...
Found prime factor, 545.087 bits...
Factors: [mpz(3), mpz(71), mpz(1063541), mpz(153422959), mpz(332958319), mpz(122356390229851897378935483485536580757336676443481705501726535578690975860555141829117483263572548187951860901335596150415443615382488933330968669408906073630300473)]
Time: 2.963 sec

Dynamic programming - save calculating times

I had an overflow error with this program here!, I realized the mistake of that program. I cannot use range or xrange when it came to really long integers. I tried running the program in Python 3 and it works. My code works but then responds after several times. Hence in order to optimize my code, I started thinking of strategies for the optimizing the code.
My problem statement is A number is called lucky if the sum of its digits, as well as the sum of the squares of its digits is a prime number. How many numbers between A and B are lucky?.
I started with this:
squarelist=[0,1,4,9,16,25,36,49,64,81]
def isEven(self, n):
return
def isPrime(n):
return
def main():
t=long(raw_input().rstrip())
count = []
for i in xrange(t):
counts = 0
a,b = raw_input().rstrip().split()
if a=='1':
a='2'
tempa, tempb= map(int, a), map(int,b)
for i in range(len(b),a,-1):
tempsum[i]+=squarelist[tempb[i]]
What I am trying to achieve is since I know the series is ordered, only the last number changes. I can save the sum of squares of the earlier numbers in the list and just keep changing the last number. This does not calculate the sum everytime and check if the sum of squares is prime. I am unable to fix the sum to some value and then keep changing the last number.How to go forward from here?
My sample inputs are provided below.
87517 52088
72232 13553
19219 17901
39863 30628
94978 75750
79208 13282
77561 61794

I didn't get what you want to achieve with your code at all. This is my solution to the question as I understand it: For all natural numbers n in a range X so that a < X < b for some natural numbers a, b with a < b, how many numbers n have the property that the sum of its digits and the sum of the square of its digits in decimal writing are both prime?
def sum_digits(n):
s = 0
while n:
s += n % 10
n /= 10
return s
def sum_digits_squared(n):
s = 0
while n:
s += (n % 10) ** 2
n /= 10
return s
def is_prime(n):
return all(n % i for i in xrange(2, n))
def is_lucky(n):
return is_prime(sum_digits(n)) and is_prime(sum_digits_squared(n))
def all_lucky_numbers(a, b):
return [n for n in xrange(a, b) if is_lucky(n)]
if __name__ == "__main__":
sample_inputs = ((87517, 52088),
(72232, 13553),
(19219, 17901),
(39863, 30628),
(94978, 75750),
(79208, 13282),
(77561, 61794))
for b, a in sample_inputs:
lucky_number_count = len(all_lucky_numbers(a, b))
print("There are {} lucky numbers between {} and {}").format(lucky_number_count, a, b)
A few notes:
The is_prime is the most naive implementation possible. It's still totally fast enough for the sample input. There are many better implementations possible (and just one google away). The most obvious improvement would be skipping every even number except for 2. That alone would cut calculation time in half.
In Python 3 (and I really recommend using it), remember to use //= to force the result of the division to be an integer, and use range instead of xrange. Also, an easy way to speed up is_prime is Python 3's #functools.lru_cache.
If you want to save some lines, calculate the sum of digits by casting them to str and back to int like that:
def sum_digits(n):
return sum(int(d) for d in str(a))
It's not as mathy, though.

Finding all divisors of a number optimization

I have written the following function which finds all divisors of a given natural number and returns them as a list:
def FindAllDivisors(x):
divList = []
y = 1
while y <= math.sqrt(x):
if x % y == 0:
divList.append(y)
divList.append(int(x / y))
y += 1
return divList
It works really well with the exception that it's really slow when the input is say an 18-digit number. Do you have any suggestions for how I can speed it up?
Update:
I have the following method to check for primality based on Fermat's Little Theorem:
def CheckIfProbablyPrime(x):
return (2 << x - 2) % x == 1
This method is really efficient when checking a single number, however I'm not sure whether I should be using it to compile all primes up to a certain boundary.

You can find all the divisors of a number by calculating the prime factorization. Each divisor has to be a combination of the primes in the factorization.
If you have a list of primes, this is a simple way to get the factorization:
def factorize(n, primes):
factors = []
for p in primes:
if p*p > n: break
i = 0
while n % p == 0:
n //= p
i+=1
if i > 0:
factors.append((p, i));
if n > 1: factors.append((n, 1))
return factors
This is called trial division. There are much more efficient methods to do this. See here for an overview.
Calculating the divisors is now pretty easy:
def divisors(factors):
div = [1]
for (p, r) in factors:
div = [d * p**e for d in div for e in range(r + 1)]
return div
The efficiency of calculating all the divisors depends on the algorithm to find the prime numbers (small overview here) and on the factorization algorithm. The latter is always slow for very large numbers and there's not much you can do about that.

I'd suggest storing the result of math.sqrt(x) in a separate variable, then checking y against it. Otherwise it will be re-calculated at each step of while, and math.sqrt is definitely not a light-weight operation.

I would do a prime factor decomposition, and then compute all divisors from that result.

I don't know if there's much of a performance hit, but I'm pretty sure that cast to an int is unnecessary. At least in Python 2.7, int x / int y returns an int.