We Keep Coding

String sorting problem with code execution time limit

I was recently trying to solve a HackerEarth problem. The code worked on the sample inputs and some custom inputs that I gave. But, when I submitted, it showed errors for exceeding the time limit. Can someone explain how I can make the code run faster?
Problem Statement: Cyclic shift
A large binary number is represented by a string A of size N and comprises of 0s and 1s. You must perform a cyclic shift on this string. The cyclic shift operation is defined as follows:
If the string A is [A0, A1,..., An-1], then after performing one cyclic shift, the string becomes [A1, A2,..., An-1, A0].
You performed the shift infinite number of times and each time you recorded the value of the binary number represented by the string. The maximum binary number formed after performing (possibly 0) the operation is B. Your task is to determine the number of cyclic shifts that can be performed such that the value represented by the string A will be equal to B for the Kth time.
Input format:
First line: A single integer T denoting the number of test cases
For each test case:
First line: Two space-separated integers N and K
Second line: A denoting the string
Output format:
For each test case, print a single line containing one integer that represents the number of cyclic shift operations performed such that the value represented by string A is equal to B for the Kth time.
Code:
import math
def value(s):
u = len(s)
d = 0
for h in range(u):
d = d + (int(s[u-1-h]) * math.pow(2, h))
return d
t = int(input())
for i in range(t):
x = list(map(int, input().split()))
n = x[0]
k = x[1]
a = input()
v = 0
for j in range(n):
a = a[1:] + a[0]
if value(a) > v:
b = a
v = value(a)
ctr = 0
cou = 0
while ctr < k:
a = a[1:] + a[0]
cou = cou + 1
if a == b:
ctr = ctr + 1
print(cou)

In the problem, the constraint on n is 0<=n<=1e5. In the function value(), you calculating integer from the binary string whose length can go up to 1e5. so the integer calculating by you can go as high as pow(2, 1e5). This surely impractical.
As mentioned by Prune, you must use some efficient algorithms for finding a subsequence, say sub1, whose repetitions make up the given string A. If you solve this by brute-force, the time complexity will be O(n*n), as maximum value of n is 1e5, time limit will exceed. so use some efficient algorithm.

I can't do much with the code you posted, since you obfuscated it with meaningless variables and a lack of explanation. When I scan it, I get the impression that you've made the straightforward approach of doing a single-digit shift in a long-running loop. You count iterations until you hit B for the Kth time.
This is easy to understand, but cumbersome and inefficient.
Since the cycle repeats every N iterations, you gain no new information from repeating that process. All you need to do is find where in the series of N iterations you encounter B ... which could be multiple times.
In order for B to appear multiple times, A must consist of a particular sub-sequence of bits, repeated 2 or more times. For instance, 101010 or 011011. You can detect this with a simple addition to your current algorithm: at each iteration, check to see whether the current string matches the original. The first time you hit this, simply compute the repetition factor as rep = len(a) / j. At this point, exit the shifting loop: the present value of b is the correct one.
Now that you have b and its position in the first j rotations, you can directly compute the needed result without further processing.
I expect that you can finish the algorithm and do the coding from here.
Ah -- taken as a requirements description, the wording of your problem suggests that B is a given. If not, then you need to detect the largest value.
To find B, append A to itself. Find the A-length string with the largest value. You can hasten this by finding the longest string of 1s, applying other well-known string-search algorithms for the value-trees after the first 0 following those largest strings.
Note that, while you iterate over A, you look for the first place in which you repeat the original value: this is the desired repetition length, which drives the direct-computation phase in the first part of my answer.

How does this bitwise manipulation change a single bit a particular index?

I understand what each of the individual operators does by itself, but I don't know how they interact in order to get the correct results.
def kill(n, k):
#Takes int n and replaces the bit k from right with 0. Returns the new number
return n & ~(1<<k-1)
I tested the program with the n as 37 and k as 3.
def b(n,s=""):
print (str(format(n, 'b')) +" "+ s)
def kill(n, k):
b(n, "n ")
b(1<<k-1, "1<<k-1")
b(~(1<<k-1), "~(1<<k-1) ")
b( n & ~(1<<k-1)," n & ~(1<<k-1) ")
return n & ~(1<<k-1)
#TESTS
kill(37, 3)
I decided to run through it step by step.
I printed both the binary representations of both n and ~(1<<k-1) but after that I was lost. ~(1<<k-1) gave me -101 and I'm not sure how to visualize that in binary. Can someone go through it step by step with visualizations for the binary?

All numbers below are printed in binary representation.
Say, n has m digits in binary representation. Observe that n & 11...1 (m ones) would return n. Indeed, working bitwise, if x is a one-bit digit (0 or 1), then x & 1 = x.
Moreover, observe that x & 0 = x. Therefore, to set up kth digit of number n to 0, we need to do operation and (&) with 11111..1011..1, where 0 is exactly on kth location from the end.
Now we need to generate 11111..1011..0. It has all ones except one digit. If we negate it, we get 00000..0100..1 which we get by 1 << k-1.
All in all: 1 << k-1 gives us 00000..0100..0. Its negation provides 11111..1011..1. Finally, we do & with the input.

Efficiently calculating mathematical formulas with exponents

I'm implementing a program that calculates an equation: F(n) = F(n-1) + 'a' + func1(func2(F(n-1))).
func1 takes every 'a' and makes it 'c' and every 'c' becomes 'a'.
func2 reverses the string (e.x. "xyz" becomes "zyx").
I want to calculate the Kth character of F(10**2017).
The basic rules are F(0) = "" (empty string), and examples are F(1) = "a", F(2) = "aac", and so on.
How do I do this efficiently?
The basic part of my code is this:
def op1 (str1):
if str1 == 'a':
return 'c'
else:
return 'a'
def op2 (str2):
return str2[::-1]
sinitial = ''
while (counter < 10**2017):
Finitial = Finitial + 'a' + op1(op2(Finitial))
counter += 1
print Finitial

Let's start by fixing your original code and defining a function to compute F(n) for small n. We'll also print out the first few values of F. All code below is for Python 3; if you're using Python 2, you'll need to make some minor changes, like replacing str.maketrans with string.maketrans and range with xrange.
swap_ac = str.maketrans({ord('a'): 'c', ord('c'): 'a'})
def F(n):
s = ''
for _ in range(n):
s = s + 'a' + s[::-1].translate(swap_ac)
return s
for n in range(7):
print("F({}) = {!r}".format(n, F(n)))
This gives the following output:
F(0) = ''
F(1) = 'a'
F(2) = 'aac'
F(3) = 'aacaacc'
F(4) = 'aacaaccaaaccacc'
F(5) = 'aacaaccaaaccaccaaacaacccaaccacc'
F(6) = 'aacaaccaaaccaccaaacaacccaaccaccaaacaaccaaaccacccaacaacccaaccacc'
A couple of observations at this point:
F(n) is a string of length 2**n-1. That means that F(n) grows fast. Computing F(50) would already require some serious hardware: even if we stored one character per bit, we'd need over 100 terabytes to store the full string. F(200) has more characters than there are estimated atoms in the solar system. So the idea of computing F(10**2017) directly is laughable: we need a different approach.
By construction, each F(n) is a prefix of F(n+1). So what we really have is a well-defined infinite string, where each F(n) merely gives us the first 2**n-1 characters of that infinite string, and we're looking to compute its kth character. And for any practical purpose, F(10**2017) might as well be that infinite string: for example, when we do our computation, we don't need to check that k < 2**(10**2017)-1, since a k exceeding this can't even be represented in normal binary notation in this universe.
Luckily, the structure of the string is simple enough that computing the kth character directly is straightforward. The major clue comes when we look at the characters at even and odd positions:
>>> F(6)[::2]
'acacacacacacacacacacacacacacacac'
>>> F(6)[1::2]
'aacaaccaaaccaccaaacaacccaaccacc'
The characters at even positions simply alternate between a and c (and it's straightforward to prove that this is true, based on the construction). So if our k is even, we can simply look at whether k/2 is odd or even to determine whether we'll get an a or a c.
What about the odd positions? Well F(6)[1::2] should look somewhat familiar: it's just F(5):
>>> F(6)[1::2] == F(5)
True
Again, it's straightforward to prove (e.g., by induction) that this isn't simply a coincidence, and that F(n+1)[1::2] == F(n) for all nonnegative n.
We now have an effective way to compute the kth character in our infinite string: if k is even, we just look at the parity of k/2. If k is odd, then we know that the character at position k is equal to that at position (k-1)/2. So here's a first solution to computing that character:
def char_at_pos(k):
"""
Return the character at position k of the string F(n), for any
n satisfying 2**n-1 > k.
"""
while k % 2 == 1:
k //= 2
return 'ac'[k//2%2]
And a check that this does the right thing:
>>> ''.join(char_at_pos(i) for i in range(2**6-1))
'aacaaccaaaccaccaaacaacccaaccaccaaacaaccaaaccacccaacaacccaaccacc'
>>> ''.join(char_at_pos(i) for i in range(2**6-1)) == F(6)
True
But we can do better. We're effectively staring at the binary representation of k, removing all trailing '1's and the next '0', then simply looking at the next bit to determine whether we've got an 'a' or a 'c'. Identifying the trailing 1s can be done by bit-operation trickery. This gives us the following semi-obfuscated loop-free solution, which I leave it to you to unwind:
def char_at_pos2(k):
"""
Return the character at position k of the string F(n), for any
n satisfying 2**n-1 > k.
"""
return 'ac'[k//(1+(k+1^k))%2]
Again, let's check:
>>> F(20) == ''.join(char_at_pos2(i) for i in range(2**20-1))
True
Final comments: this is a very well-known and well-studied sequence: it's called the dragon curve sequence, or the regular paper-folding sequence, and is sequence A014577 in the online encyclopaedia of integer sequences. Some Google searches will likely give you many other ways to compute its elements. See also this codegolf question.

Based on what you have already coded, here's my suggestion:
def main_function(num):
if num == 0:
return ''
previous = main_function(num-1)
return previous + 'a' + op1(op2(previous))
print(main_function(10**2017))
P.S: I'm not sure of the efficiency.

I need help finding a smart solution to shorten the time this code runs

2 days ago i started practicing python 2.7 on Codewars.com and i came across a really interesting problem, the only thing is i think it's a bit too much for my level of python knowledge. I actually did solve it in the end but the site doesn't accept my solution because it takes too much time to complete when you call it with large numbers, so here is the code:
from itertools import permutations
def next_bigger(n):
digz =list(str(n))
nums =permutations(digz, len(digz))
nums2 = []
for i in nums:
z =''
for b in range(0,len(i)):
z += i[b]
nums2.append(int(z))
nums2 = list(set(nums2))
nums2.sort()
try:
return nums2[nums2.index(n)+1]
except:
return -1
"You have to create a function that takes a positive integer number and returns the next bigger number formed by the same digits" - These were the original instructions
Also, at one point i decided to forgo the whole permutations idea, and in the middle of this second attempt i realized that there's no way it would work:
def next_bigger(n):
for i in range (1,11):
c1 = n % (10**i) / (10**(i-1))
c2 = n % (10**(i+1)) / (10**i)
if c1 > c2:
return ((n /(10**(i+1)))*10**(i+1)) + c1 *(10**i) + c2*(10**(i-1)) + n % (10**(max((i-1),0)))
break
if anybody has any ideas, i'm all-ears and if you hate my code, please do tell, because i really want to get better at this.

stolen from http://www.geeksforgeeks.org/find-next-greater-number-set-digits/
Following are few observations about the next greater number.
1) If all digits sorted in descending order, then output is always “Not Possible”. For example, 4321.
2) If all digits are sorted in ascending
order, then we need to swap last two digits. For example, 1234.
3) For
other cases, we need to process the number from rightmost side (why?
because we need to find the smallest of all greater numbers)
You can now try developing an algorithm yourself.
Following is the algorithm for finding the next greater number.
I)
Traverse the given number from rightmost digit, keep traversing till
you find a digit which is smaller than the previously traversed digit.
For example, if the input number is “534976”, we stop at 4 because 4
is smaller than next digit 9. If we do not find such a digit, then
output is “Not Possible”.
II) Now search the right side of above found digit ‘d’ for the
smallest digit greater than ‘d’. For “534976″, the right side of 4
contains “976”. The smallest digit greater than 4 is 6.
III) Swap the above found two digits, we get 536974 in above example.
IV) Now sort all digits from position next to ‘d’ to the end of
number. The number that we get after sorting is the output. For above
example, we sort digits in bold 536974. We get “536479” which is the
next greater number for input 534976.

"formed by the same digits" - there's a clue that you have to break the number into digits: n = list(str(n))
"next bigger". The fact that they want the very next item means that you want to make the least change. Focus on changing the 1s digit. If that doesn't work, try the 10's digit, then the 100's, etc. The smallest change you can make is to exchange two furthest digits to the right that will increase the value of the integer. I.e. exchange the two right-most digits in which the more right-most is bigger.
def next_bigger(n):
n = list(str(n))
for i in range(len(n)-1, -1, -1):
for j in range(i-1, -1, -1):
if n[i] > n[j]:
n[i], n[j] = n[j], n[i]
return int("".join(n))
print next_bigger(123)
Oops. This fails for next_bigger(1675). I'll leave the buggy code here for a while, for whatever it is worth.

How about this? See in-line comments for explanations. Note that the way this is set up, you don't end up with any significant memory use (we're not storing any lists).
from itertools import permutations
#!/usr/bin/python3
def next_bigger(n):
# set next_bigger to an arbitrarily large value to start: see the for-loop
next_bigger = float('inf')
# this returns a generator for all the integers that are permutations of n
# we want a generator because when the potential number of permutations is
# large, we don't want to store all of them in memory.
perms = map(lambda x: int(''.join(x)), permutations(str(n)))
for p in perms:
if (p > n) and (p <= next_bigger):
# we can find the next-largest permutation by going through all the
# permutations, selecting the ones that are larger than n, and then
# selecting the smallest from them.
next_bigger = p
return next_bigger
Note that this is still a brute-force algorithm, even if implemented for speed. Here is an example result:
time python3 next_bigger.py 3838998888
3839888889
real 0m2.475s
user 0m2.476s
sys 0m0.000s
If your code needs to be faster yet, then you'll need a smarter, non-brute-force algorithm.

You don't need to look at all the permutations. Take a look at the two permutations of the last two digits. If you have an integer greater than your integer, that's it. If not, take a look at the permutations of the last three digits, etc.
from itertools import permutations
def next_bigger(number):
check = 2
found = False
digits = list(str(number))
if sorted(digits, reverse=True) == digits:
raise ValueError("No larger number")
while not found:
options = permutations(digits[-1*check:], check)
candidates = list()
for option in options:
new = digits.copy()[:-1*check]
new.extend(option)
candidate = int(''.join(new))
if candidate > number:
candidates.append(candidate)
if candidates:
result = sorted(candidates)[0]
found = True
return result
check += 1

Project Euler 104: Need help in understanding the solution

Project Euler Q104 (https://projecteuler.net/problem=104) is as such:
The Fibonacci sequence is defined by the recurrence relation:
Fn = Fn−1 + Fn−2, where F1 = 1 and F2 = 1. It turns out that F541,
which contains 113 digits, is the first Fibonacci number for which the
last nine digits are 1-9 pandigital (contain all the digits 1 to 9,
but not necessarily in order). And F2749, which contains 575 digits,
is the first Fibonacci number for which the first nine digits are 1-9
pandigital.
Given that Fk is the first Fibonacci number for which the first nine
digits AND the last nine digits are 1-9 pandigital, find k.
And I wrote this simple code in Python:
def fibGen():
a,b = 1,1
while True:
a,b = b,a+b
yield a
k = 0
fibG = fibGen()
while True:
k += 1
x = str(fibG.next())
if (set(x[-9:]) == set("123456789")):
print x #debugging print statement
if(set(x[:9]) == set("123456789")):
break
print k
However, it was taking well.. forever.
After leaving it running for 30 mins, puzzled, I gave up and checked the solution.
I came across this code in C#:
long fn2 = 1;
long fn1 = 1;
long fn;
long tailcut = 1000000000;
int n = 2;
bool found = false;
while (!found) {
n++;
fn = (fn1 + fn2) % tailcut;
fn2 = fn1;
fn1 = fn;
if (IsPandigital(fn)) {
double t = (n * 0.20898764024997873 - 0.3494850021680094);
if (IsPandigital((long)Math.Pow(10, t - (long)t + 8)))
found = true;
}
}
Which.. I could barely understand. I tried it out in VS, got the correct answer and checked the thread for help.
I found these two, similar looking answers in Python then.
One here, http://blog.dreamshire.com/project-euler-104-solution/
And one from the thread:
from math import sqrt
def isPandigital(s):
return set(s) == set('123456789')
rt5=sqrt(5)
def check_first_digits(n):
def mypow( x, n ):
res=1.0
for i in xrange(n):
res *= x
# truncation to avoid overflow:
if res>1E20: res*=1E-10
return res
# this is an approximation for large n:
F = mypow( (1+rt5)/2, n )/rt5
s = '%f' % F
if isPandigital(s[:9]):
print n
return True
a, b, n = 1, 1, 1
while True:
if isPandigital( str(a)[-9:] ):
print a
# Only when last digits are
# pandigital check the first digits:
if check_first_digits(n):
break
a, b = b, a+b
b=b%1000000000
n += 1
print n
These worked pretty fast, under 1 minute!
I really need help understanding these solutions. I don't really know the meaning or the reason behind using stuff like log. And though I could easily do the first 30 questions, I cannot understand these tougher ones.
How is the best way to solve this question and how these solutions are implementing it?

These two solutions work on the bases that as fibonacci numbers get bigger, the ratio between two consecutive terms gets closer to a number known as the Golden Ratio, (1+sqrt(5))/2, roughly 1.618. If you have one (large) fibonacci number, you can easily calculate the next, just by multiplying it by that number.
We know from the question that only large fibonacci numbers are going to satisfy the conditions, so we can use that to quickly calculate the parts of the sequence we're interested in.
In your implementation, to calculate fib(n), you need to calculate fib(n-1), which needs to calculate fib(n-2) , which needs to calculate fib(n-3) etc, and it needs to calculate fib(n-2), which calculates fib(n-3) etc. That's a huge number of function calls when n is big. Having a single calculation to know what number comes next is a huge speed increase. A computer scientist would call the first method O(n^2)*: to calculate fib(n), you need n^2 sub calculations. Using the golden mean, the fibonacci sequence becomes (approximately, but close enouigh for what we need):
(using phi = (1+sqrt(5))/2)
1
1*phi
1*phi*phi = pow(phi, 2)
1*phi*phi*phi = pow(phi, 3)
...
1*phi*...*phi = pow(phi, n)
\ n times /
So, you can do an O(1) calculation: fib(n): return round(pow(golden_ratio, n)/(5**0.5))
Next, there's a couple of simplifications that let you use smaller numbers.
If I'm concerned about the last nine digits of a number, what happens further up isn't all that important, so I can throw anything after the 9th digit from the right away. That's what b=b%1000000000 or fn = (fn1 + fn2) % tailcut; are doing. % is the modulus operator, which says, if I divide the left number by the right, what's the remainder?
It's easiest to explain with equivalent code:
def mod(a,b):
while a > b:
a -= b
return a
So, there's a quick addition loop that adds together the last nine digits of fibonacci numbers, waiting for them to be pandigital. If it is, it calculates the whole value of the fibonacci number, and check the first nine digits.
Let me know if I need to cover anything in more detail.
* https://en.wikipedia.org/wiki/Big_O_notation