We Keep Coding

Calculating a^b mod p for a large prime p

I'm trying to write a python code that calculates a^b mod p, where p = 10^9+7 for a list of pairs (a,b). The challenge is that the code has to finish the calculation and output the result in < 1 second. I've implemented successive squaring to calculate a^b mod p quickly. Please see my code below:
from sys import stdin, stdout
rl = stdin.readline
wo = stdout.write
m = 10**9+7
def fn(a,n):
t = 1
while n > 0:
if n%2 != 0: #exponent is odd
t = t*a %m
a = a*a %m
n = int(n/2)
return t%m
t = int(rl()) # number of pairs
I = stdin.read().split() # reading all pairs
I = list(map(int,I)) # making pairs a list of integers
# calculating a^b mod p. I used map because I read its faster than a for loop
s = list(map(fn,I[0:2*t:2],I[1:2*t:2]))
stdout.write('\n'.join(map(str,s))) # printing output
for 200000 pairs (a,b) with a,b<10^9, my code takes > 1 second. I'm new to python and was hoping someone could help me identify the time bottle neck in my code. Is it reading input and printing output or the calculation itself? Thanks for the help!

I don't see something wrong with your code from an efficiency standpoint, it's just unnecessarily complicated.
Here's what I'd call the straight-forward solution:
n = int(input())
for _ in range(n):
a, b = map(int, input().split())
print(pow(a, b, 10**9 + 7))
That did get accepted with PyPy3 but not with CPython3. And with PyPy3 it still took 0.93 seconds.
I'd say their time limit is inappropriate for Python. But try yours with PyPy3 if you haven't yet.
In case someone's wondering whether the map wastes time, the following got accepted in 0.92 seconds:
n = int(input())
for _ in range(n):
a, b = input().split()
print(pow(int(a), int(b), 10**9 + 7))

list of permutations of HEADS and TAILS in n tosses with exactly k HEADS in Python

I'm trying to write some python code to list the different permutations of heads and tails given n tosses and exactly k heads. E.g. for n=6 tosses and k=3 heads, the output that I want is ['000111', '001011', '001101', '001110', '010011', '010101', '010110', '011001', '011010', '011100', '100011', '100101', '100110', '101001', '101010', '101100', '110001', '110010', '110100', '111000']
(Note: this is not a question about calculating probability, no issues there. My issue is with generating all of the different permutations for n tosses and exactly k heads).
I know that the number of final strings is given by the binomial coefficient aka combination aka n_choose_k (e.g. scipy.special.comb). However, I'm having trouble generating each of these different options for more than a modest (n>10) number of tosses. My first naive approach is to generate all possible permutations of a string such as "1111100000" (using itertools.permutations), and then I remove duplicates using set(). This works for low n (n<=10). However, this is crazy inefficient. Even with n=10 & k=5, the number of permutations is 3,628,800 while number of combinations is only 252. After n>10 this grinds to a halt. I know that itertools also has combinations generator (and _with_replacements). However, I'm not sure if it's possible to use it to generate output in the way that I require, since my problem isn't exactly selecting a subset of k elements from a set of length n, where order does not matter.
My initial code is below.
from itertools import permutations
import scipy.special
n = 10
k = n//2
n_choose_k = scipy.special.comb(n, k)
print('{} choose {} = {}'.format(n, k, n_choose_k))
U,D = '1','0'
seed = U * k + D * (n-k)
permlist = [''.join(p) for p in permutations(seed)]
permset = sorted(list(set(permlist)))
print(permset)
print('len permutations:{}, len set:{}'.format(len(permlist), len(permset)))
Note: I'm interesting in n tosses and exactly k heads, though I would be curious to see how the solution could be expanded to at least k heads too.
UPDATE:
I've accepted karl-knechtel's answer. But for those curious, here are all three methods with run times (on laptop with i7-9750H #2.6Ghz)
Results:
10 choose 5 = 252.0
Permutations: 252 items. Calculated in 1.0340404510498047s
Combinations: 252 items. Calculated in 0.00103759765625s
Binary ints: 252 items. Calculated in 0.002962350845336914s
12 choose 6 = 924.0
Permutations: Gave up after 30 seconds.
Combinations: 924 items. Calculated in 0.001998424530029297s
Binary ints: 924 items. Calculated in 0.023003339767456055s
14 choose 7 = 3432.0
Permutations: Gave up after 30 seconds.
Combinations: 3432 items. Calculated in 0.011020183563232422s
Binary ints: 3432 items. Calculated in 0.10396194458007812s
16 choose 8 = 12870.0
Permutations: Gave up after 30 seconds.
Combinations: 12870 items. Calculated in 0.049001455307006836s
Binary ints: 12870 items. Calculated in 0.42699623107910156s
20 choose 10 = 184756.0
Permutations: Gave up after 30 seconds.
Combinations: 184756 items. Calculated in 0.6319949626922607s
Binary ints: 184756 items. Calculated in 7.8870697021484375s
22 choose 11 = 705432.0
Permutations: Gave up after 30 seconds.
Combinations: 705432 items. Calculated in 2.974030017852783s
Binary ints: Gave up after 30 seconds.
24 choose 12 = 2704156.0
Permutations: Gave up after 30 seconds.
Combinations: 2704156 items. Calculated in 11.795861721038818s
Binary ints: Gave up after 30 seconds.
26 choose 13 = 10400600.0
Permutations: Gave up after 30 seconds.
Combinations: 10400600 items. Calculated in 51.053600549697876s
Binary ints: Gave up after 30 seconds.
Code:
import numpy as np
from itertools import permutations, combinations
import scipy.special
import time
n = 26
k = n//2
test_permutations = n <= 10
test_combinations = True
test_binaryints = n <= 20
test_print_outputs = n <= 6
n_choose_k = scipy.special.comb(n, k)
print('{} choose {} = {}'.format(n, k, n_choose_k))
U,D = '1','0'
def calc_permutations():
seed = U * k + D * (n-k)
l = [''.join(p) for p in permutations(seed)]
return list(set(l))
def calc_combinations():
'''Suggestion from https://stackoverflow.com/users/523612/karl-knechtel
https://stackoverflow.com/a/63100745/214488
'''
def bit_strings(size, one_count):
for one_indices in combinations(range(size), one_count):
yield ''.join(U if i in one_indices else D for i in range(size))
return list(bit_strings(n, k))
def calc_binaryints():
'''
Suggestion from https://stackoverflow.com/users/9046247/aramakus
https://stackoverflow.com/a/63100262/214488
'''
def check_bits(num, k, n):
""" True if `k` ones inside the number.
From https://stackoverflow.com/users/7505395/patrick-artner
https://stackoverflow.com/a/63100497/214488
"""
return sum( (num & 2**rank)>>rank for rank in range(n)) == k
def to_bin(i):
return format(i, '0'+str(n) + 'b')
start_int = 2**k-1
end_int = 2**n
l = [to_bin(i) for i in range(start_int, end_int) if check_bits(i,k,n)]
return l
def test_method(name, fn, do_test, sort=True):
print(name + ':', end='')
if do_test:
start_time = time.time()
l = fn()
duration = time.time() - start_time
if sort: l.sort()
print(' {} items. Calculated in {}s'.format(len(l), duration))
if test_print_outputs: print(l)
else:
print(' Gave up after 30 seconds.')
test_method('Permutations', calc_permutations, test_permutations)
test_method('Combinations', calc_combinations, test_combinations)
test_method('Binary ints', calc_binaryints, test_binaryints)

Manifesting the whole list of permutations (as strings) is a big memory hog. You can use a modified version of #aramakus answer to push the boundaries of whats feasible a bit by using integers and a checker using bit-shifts to generate only those integers with k "ones":
def check(num, k, n):
""" True if `k` ones inside the number."""
# uses bit shifting and masking to get number of 1s
return sum( (num & 2**rank)>>rank for rank in range(n)) == k
def pretty_print(l, n):
"""Pretty printer for list n (integer elements) padded to n digits"""
for num in l:
print((f'{num:0{n}b}'))
# or without formatting mini language
# print(('000000000'+bin(num)[2:])[-n:])
n = 15 # tested for up to 18 (which takes ~25s on my laptop)
k = n//2
nums = [i for i in range(2**n) if check(i,k,n)]
pretty_print(nums,n)
# total numbers is just 2**n, the amount of them with k ones is countable
print(f'exactly {k} ones: {len(nums)} out of {2**n} numbers')
Output:
000000001111111
000000010111111
000000011011111
000000011101111
000000011110111
...
111111001000000
111111010000000
111111100000000
exactly 7 ones: 6435 out of 32768 numbers
Ballpark calculation times using time.time() (so not repeated/averaged timings of multiple runs and most assuredly quite some measure of error involved):
# You should do this using timeit.timeit() for serious measurements
n k time in seconds
5 2 0.0019998550415039062
6 3 0.004002809524536133
7 3 0.009006261825561523
8 4 0.020014286041259766
9 4 0.04403090476989746
10 5 0.09506702423095703
11 5 0.20834732055664062
12 6 0.4523327350616455
13 6 0.9736926555633545
14 7 2.0954811573028564
15 7 4.479296922683716
16 8 9.40683913230896
17 8 19.881306886672974
18 9 41.978920459747314
# somewhat "linear" calculation time

However, I'm not sure if it's possible to use it to generate output in the way that I require, since my problem isn't exactly selecting a subset of k elements from a set of length n, where order does not matter.
It is, actually, with a bit of lateral thinking. The set you want to select from is the set of indices where a 1 should appear in a given output.
So: use itertools.combinations to determine the indices where the 1s should go (we are choosing k values from the n possible index values - 0 to n-1, inclusive - without replacement; that is exactly what combination are), and then generate the string for each. For example, as a generator:
def bit_strings(size, one_count):
for one_indices in itertools.combinations(range(size), one_count):
yield ''.join('1' if i in one_indices else '0' for i in range(size))
>>> len(list(bit_strings(20, 10))) # takes a bit less than a second on my machine
184756
This is, of course, still exponentially (literally!) slower than just calculating the number of combinations directly.

Not a fair solution, but you can calculate a 2**k-1 and write all integer numbers between 0 and 2**k-1 in binary format. This list would contain all 0s and 1s combinations with a total of k numbers.

itertools combinations in tandem with looping

I have the following Python code. Because random is being used, it generates a new answer every time:
import random
import numpy as np
N = 64 # Given
T = 5 # Given
FinalLengths = []
for i in range(T):
c = range(1, N)
x = random.sample(c, 2) # Choose 2 random numbers between 1 and N-1
LrgstNode = max(x)
SmlstNode = min(x)
RopeLengths = [SmlstNode, LrgstNode - SmlstNode, N - LrgstNode]
S = max(RopeLengths)
N = S
FinalLengths.append(S)
avgS = np.mean(FinalLengths) # Find average
print("The mean of S is {}".format(avgS))
My research has led me to possibly using itertools combinations in order to produce all possible combinations within the range and get the avg to converge. If so, how?
Thank you.

It sounds like you're after something like this:
import random
import numpy as np
from itertools import combinations
N = 64 # Given
T = 5 # Given
FinalLengths = []
for i in range(T):
c = list(range(1, N))
for x in combinations(c, 2):
S = max([min(x), max(x) - min(x), N - max(x)])
N = S
FinalLengths.append(S)
avgS = np.mean(FinalLengths) # Find average
print("The mean of S is {}".format(avgS))
To use combinations(l, size) we can pass in a list l and the size of each combination tuple, and int size. That's all there is to it!

An Explanation for the totient finder in this program

I need an explanation for the program suggested in the edit in the first answer over here. It is a program to find the totients of a range of numbers. Can somebody provide a simple explanation? (Ignore the summation part for now, I need to find out how the init method finds the totients.) I know there is an explanation in the answer, but that is an explanation for different programs, I need an explanation for this particular one.
class Totient:
def __init__(self, n):
self.totients = [1 for i in range(n)]
for i in range(2, n):
if self.totients[i] == 1:
for j in range(i, n, i):
self.totients[j] *= i - 1
k = j / i
while k % i == 0:
self.totients[j] *= i
k /= i
def __call__(self, i):
return self.totients[i]
if __name__ == '__main__':
from itertools import imap
totient = Totient(10000)
print sum(imap(totient, range(10000)))

It's a variant of the Sieve of Eratosthenes for finding prime numbers.
If you want to know the totient of a single number n, the best way to find it is to factor n and take the product of 1 less than each factor; for instance, 30 = 2 * 3 * 5, and subtracting 1 from each factor, then multiplying, gives the totient 1 * 2 * 4 = 8. But if you want to find the totients of all the numbers less than a given n, a better approach than factoring each of them is sieving. The idea is simple: Set up an array X from 0 to n, store i in each Xi, then run through the array starting from 0 and whenever Xi = i loop over the multiples of i, multiplying each by i − 1.
Further discussion and code at my blog.

I'm not completely sure what the code is doing -- but frankly it looks pretty bad. It clearly is trying to use that Euler's totient function is multiplicative, meaning that a,b are relatively prime then t(a,b) = t(a)*t(b), together with the fact that if p is a prime then t(p) = p-1. But -- it seems to be using crude trial division to determine such things. If you really want to calculate the totient of all numbers in a given range then you should use an algorithm that sieves the numbers as you go along.
Here is a version which sieves as it goes and exploits the multiplicative nature to the hilt. At each pass through the main loop it starts with a prime, p which hasn't yet been processed. It determines all powers of p <= n and then uses a direct formula for these powers (see https://en.wikipedia.org/wiki/Euler%27s_totient_function ). Once these totients have been added, it forms all possible products <= n of these powers and the numbers for which the totients have been previously computed. This gives a whole slew of numbers to add to the list of previously determined numbers. At most sqrt(n) passes need to be made through the main loop. It runs almost instantly for n = 10000. It returns a list where the ith value is the totient of i (with t(0) = 0 for convenience):
def allTotients(n):
totients = [None]*(n+1) #totients[i] will contain the t(i)
totients[0] = 0
totients[1] = 1
knownTotients = [] #known in range 2 to n
p = 2
while len(knownTotients) < n - 1:
powers = [p]
k = 2
while p ** k <= n:
powers.append(p ** k)
k +=1
totients[p] = p - 1
for i in range(1,len(powers)):
totients[powers[i]] = powers[i] - powers[i-1]
#at this stage powers represent newly discovered totients
#combine with previously discovered totients to get still more
newTotients = powers[:]
for m in knownTotients:
for pk in powers:
if m*pk > n: break
totients[m*pk] = totients[m]*totients[pk]
newTotients.append(m*pk)
knownTotients.extend(newTotients)
#if there are any unkown totients -- the smallest such will be prime
if len(knownTotients) < n-1:
p = totients.index(None)
return totients
For completeness sake, here is a Python implementation of the algorithm to compute the totient of a single number which user448810 described in their answer:
from math import sqrt
#crude factoring algorithm:
small_primes = [2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,
53,59,61,67,71,73,79,83,89,97]
def factor(n):
#returns a list of prime factors
factors = []
num = n
#first pull out small prime factors
for p in small_primes:
while num % p == 0:
factors.append(p)
num = num // p
if num == 1: return factors
#now do trial division, starting at 101
k = 101
while k <= sqrt(num):
while num % k == 0:
factors.append(k)
num = num // k
k += 2
if num == 1:
return factors
else:
factors.append(num)
return factors
def totient(n):
factors = factor(n)
unique_factors = set()
t = 1
for p in factors:
if p in unique_factors:
t *= p
else:
unique_factors.add(p)
t *= (p-1)
return t

Optimise the solution to Project Euler 12 (Python)

I have the following code for Project Euler Problem 12. However, it takes a very long time to execute. Does anyone have any suggestions for speeding it up?
n = input("Enter number: ")
def genfact(n):
t = []
for i in xrange(1, n+1):
if n%i == 0:
t.append(i)
return t
print "Numbers of divisors: ", len(genfact(n))
print
m = input("Enter the number of triangle numbers to check: ")
print
for i in xrange (2, m+2):
a = sum(xrange(i))
b = len(genfact(a))
if b > 500:
print a
For n, I enter an arbitrary number such as 6 just to check whether it indeed returns the length of the list of the number of factors.
For m, I enter entered 80 000 000
It works relatively quickly for small numbers. If I enter b > 50 ; it returns 28 for a, which is correct.

My answer here isn't pretty or elegant, it is still brute force. But, it simplifies the problem space a little and terminates successfully in less than 10 seconds.
Getting factors of n:
Like #usethedeathstar mentioned, it is possible to test for factors only up to n/2. However, we can do better by testing only up to the square root of n:
let n = 36
=> factors(n) : (1x36, 2x18, 3x12, 4x9, 6x6, 9x4, 12x3, 18x2, 36x1)
As you can see, it loops around after 6 (the square root of 36). We also don't need to explicitly return the factors, just find out how many there are... so just count them off with a generator inside of sum():
import math
def get_factors(n):
return sum(2 for i in range(1, round(math.sqrt(n)+1)) if not n % i)
Testing the triangular numbers
I have used a generator function to yield the triangular numbers:
def generate_triangles(limit):
l = 1
while l <= limit:
yield sum(range(l + 1))
l += 1
And finally, start testing:
def test_triangles():
triangles = generate_triangles(100000)
for i in triangles:
if get_factors(i) > 499:
return i
Running this with the profiler, it completes in less than 10 seconds:
$ python3 -m cProfile euler12.py
361986 function calls in 8.006 seconds
The BIGGEST time saving here is get_factors(n) testing only up to the square root of n - this makes it heeeaps quicker and you save heaps of memory overhead by not generating a list of factors.
As I said, it still isn't pretty - I am sure there are more elegant solutions. But, it fits the bill of being faster :)

I got my answer to run in 1.8 seconds with Python.
import time
from math import sqrt
def count_divisors(n):
d = {}
count = 1
while n % 2 == 0:
n = n / 2
try:
d[2] += 1
except KeyError:
d[2] = 1
for i in range(3, int(sqrt(n+1)), 2):
while n % i == 0 and i != n:
n = n / i
try:
d[i] += 1
except KeyError:
d[i] = 1
d[n] = 1
for _,v in d.items():
count = count * (v + 1)
return count
def tri_number(num):
next = 1 + int(sqrt(1+(8 * num)))
return num + (next/2)
def main():
i = 1
while count_divisors(i) < 500:
i = tri_number(i)
return i
start = time.time()
answer = main()
elapsed = (time.time() - start)
print("result %s returned in %s seconds." % (answer, elapsed))
Here is the output showing the timedelta and correct answer:
$ python ./project012.py
result 76576500 returned in 1.82238006592 seconds.
Factoring
For counting the divisors, I start by initializing an empty dictionary and a counter. For each factor found, I create key of d[factor] with value of 1 if it does not exist, otherwise, I increment the value d[factor].
For example, if we counted the factors 100, we would see d = {25: 1, 2: 2}
The first while loop, I factor out all 2's, dividing n by 2 each time. Next, I begin factoring at 3, skipping two each time (since we factored all even numbers already), and stopping once I get to the square root of n+1.
We stop at the square_root of n because if there's a pair of factors with one of the numbers bigger than square_root of n, the other of the pair has to be less than 10. If the smaller one doesn't exist, there is no matching larger factor.
https://math.stackexchange.com/questions/1343171/why-only-square-root-approach-to-check-number-is-prime
while n % 2 == 0:
n = n / 2
try:
d[2] += 1
except KeyError:
d[2] = 1
for i in range(3, int(sqrt(n+1)), 2):
while n % i == 0 and i != n:
n = n / i
try:
d[i] += 1
except KeyError:
d[i] = 1
d[n] = 1
Now that I have gotten each factor, and added it to the dictionary, we have to add the last factor (which is just n).
Counting Divisors
Now that the dictionary is complete, we loop through each of the items, and apply the following formula: d(n)=(a+1)(b+1)(c+1)...
https://www.wikihow.com/Determine-the-Number-of-Divisors-of-an-Integer
All this formula means is taking all of the counts of each factor, adding 1, then multiplying them together. Take 100 for example, which has factors 25, 2, and 2. We would calculate d(n)=(a+1)(b+1) = (1+1)(2+1) = (2)(3) = 6 total divisors
for _,v in d.items():
count = count * (v + 1)
return count
Calculate Triangle Numbers
Now, taking a look at tri_number(), you can see that I opted to calculate the next triangle number in a sequence without manually adding each whole number together (saving me millions of operations). Instead I used T(n) = n (n+1) / 2
http://www.maths.surrey.ac.uk/hosted-sites/R.Knott/runsums/triNbProof.html
We are providing a whole number to the function as an argument, so we need to solve for n, which is going to be the whole number to add next. Once we have the next number (n), we simply add that single number to num and return
S=n(n+1)2
S=n2+n2
2S=n2+n
n2+n−2S=0
At this point, we use the quadratic formula for : ax2+bx+c=0.
n=−b±√b2−4ac / 2a
n=−1±√1−4(1)(−2S) / 2
n=−1±√1+8S / 2
https://socratic.org/questions/how-do-you-solve-for-n-in-s-n-n-1-2
So all tri_number() does is evaluate n=1+√1+8S / 2 (we ignore the negative equation here). The answer that is returned is the next triangle number in the sequence.
def tri_number(num):
next = 1 + int(sqrt(1+(8 * num)))
return num + (next/2)
Main Loop
Finally, we can look at main(). We start at whole number 1. We count the divisor of 1. If it is less than 500, we get the next triangle number, then try again and again until we get a number with > 500 divisors.
def main():
i = 1
while count_divisors(i) < 500:
i = tri_number(i)
return i
I am sure there are additional ways to optimize but I am not smart enough to understand those ways. If you find any better ways to optimize python, let me know! I originally solved project 12 in Golang, and that run in 25 milliseconds!
$ go run project012.go
76576500
2018/07/12 01:56:31 TIME: main() took 23.581558ms

one of the hints i can give is
def genfact(n):
t = []
for i in xrange(1, n+1):
if n%i == 0:
t.append(i)
return t
change that to
def genfact(n):
t=[]
for i in xrange(1,numpy.sqrt(n)+1):
if(n%i==0):
t.append(i)
t.apend(n/i)
since if a is a divisor than so is b=n/a, since a*b=a*n/b=n, That should help a part already (not sure if in your case a square is possible, but if so, add another case to exclude adding the same number twice)
You could devise a recursive thing too, (like if it is something like for 28, you get 1,28,2,14 and at the moment you are at knowing 14, you put in something to actually remember the divisors of 14 (memoize), than check if they are alraedy in the list, and if not, add them to the list, together with 28/d for each of the divisors of 14, and at the end just take out the duplicates
If you think my first answer is still not fast enough, ask for more, and i will check how it would be done to solve it faster with some more tricks (could probably make use of erastothenes sieve or so too, and some other tricks could be thought up as well if you would wish to really blow up the problem to huge proportions, like to check the first one with over 10k divisors or so)

while True:
c=0
n=1
m=1
for i in range(1,n+1):
if n%i==0:
c=c+1
m=m+1
n=m*(m+1)/2
if c>500:
break
print n

this is not my code but it is so optimized.
source: http://code.jasonbhill.com/sage/project-euler-problem-12/
import time
def num_divisors(n):
if n % 2 == 0: n = n / 2
divisors = 1
count = 0
while n % 2 == 0:
count += 1
n = n / 2
divisors = divisors * (count + 1)
p = 3
while n != 1:
count = 0
while n % p == 0:
count += 1
n = n / p
divisors = divisors * (count + 1)
p += 2
return divisors
def find_triangular_index(factor_limit):
n = 1
lnum, rnum = num_divisors(n), num_divisors(n + 1)
while lnum * rnum < 500:
n += 1
lnum, rnum = rnum, num_divisors(n + 1)
return n
start = time.time()
index = find_triangular_index(500)
triangle = (index * (index + 1)) / 2
elapsed = (time.time() - start)
print("result %s returned in %s seconds." % (triangle, elapsed))