I have written a simple implementation of the Sieve of Eratosthenes, and I would like to know if there is a more efficient way to perform one of the steps.
def eratosthenes(n):
primes = [2]
is_prime = [False] + ((n - 1)/2)*[True]
for i in xrange(len(is_prime)):
if is_prime[i]:
p = 2*i + 1
primes.append(p)
is_prime[i*p + i::p] = [False]*len(is_prime[i*p + i::p])
return primes
I am using Python's list slicing to update my list of booleans is_prime. Each element is_prime[i] corresponds to an odd number 2*i + 1.
is_prime[i*p + i::p] = [False]*len(is_prime[i*p + i::p])
When I find a prime p, I can mark all elements corresponding to multiples of that prime False, and since all multiples smaller than p**2 are also multiples of smaller primes, I can skip marking those. The index of p**2 is i*p + i.
I'm worried about the cost of computing [False]*len(is_prime[i*p + 1::p]) and I have tried to compare it to two other strategies that I couldn't get to work.
For some reason, the formula (len(is_prime) - (i*p + i))/p (if positive) is not always equal to len(is_prime[i*p + i::p]). Is it because I've calculated the length of the slice wrong, or is there something subtle about slicing that I haven't caught?
When I use the following lines in my function:
print len(is_prime[i*p + i::p]), ((len(is_prime) - (i*p + i))/p)
is_prime[i*p + i::p] = [False]*((len(is_prime) - (i*p + i))/p)
I get the following output (case n = 50):
>>> eratosthenes2(50)
7 7
3 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 9, in eratosthenes2
ValueError: attempt to assign sequence of size 2 to extended slice of size 3
I also tried replacing the bulk updating line with the following:
for j in xrange(i*p + i, len(is_prime), p):
is_prime[j] = False
But this fails for large values of n because xrange doesn't take anything bigger than a long. I gave up on trying to wrestle itertools.count into what I needed.
Are there faster and more elegant ways to bulk-update the list slice? Is there anything I can do to fix the other strategies that I tried, so that I can compare them to the working one? Thanks!
Use itertools.repeat():
is_prime[i*p + 1::p] = itertools.repeat(False, len(is_prime[i*p + 1::p]))
The slicing syntax will iterate over whatever you put on the right-hand side; it doesn't need to be a full-blown sequence.
So let's fix that formula. I'll just borrow the Python 3 formula since we know that works:
1 + (hi - 1 - lo) / step
Since step > 0, hi = stop and lo = start, so we have:
1 + (len(is_prime) - 1 - (i*p + 1))//p
(// is integer division; this future-proofs our code for Python 3, but requires 2.7 to run).
Now, put it all together:
slice_len = 1 + (len(is_prime) - 1 - (i*p + 1))//p
is_prime[i*p + 1::p] = itertools.repeat(False, slice_len)
Python 3 users: Please do not use this formula directly. Instead, just write len(range(start, stop, step)). That gives the same result with similar performance (i.e. it's O(1)) and is much easier to read.
Related
I am trying to solve LeetCode problem 295. Find Median from Data Stream:
The median is the middle value in an ordered integer list. If the size of the list is even, there is no middle value and the median is the mean of the two middle values.
For example, for arr = [2,3,4], the median is 3.
For example, for arr = [2,3], the median is (2 + 3) / 2 = 2.5.
Implement the MedianFinder class:
MedianFinder() initializes the MedianFinder object.
void addNum(int num) adds the integer num from the data stream to the data structure.
double findMedian() returns the median of all elements so far.
Answers within 10-5 of the actual answer will be accepted.
Example 1:
[...]
MedianFinder medianFinder = new MedianFinder();
medianFinder.addNum(1); // arr = [1]
medianFinder.addNum(2); // arr = [1, 2]
medianFinder.findMedian(); // return 1.5 (i.e., (1 + 2) / 2)
medianFinder.addNum(3); // arr[1, 2, 3]
medianFinder.findMedian(); // return 2.0
For this question, I am adding numbers to a heap and then later use the smallest one to do some operations.
When I tried to do the operation, I found out that my heap returns a different value when doing heapq.heappop(self.small) than when doing self.small[0].
Could you please explain this to me? Any hint is much appreciated.
(Every number in self.small is added using heapq.heappush)
Here is my code when it works:
class MedianFinder:
def __init__(self):
self.small, self.large = [], []
def addNum(self, num):
heapq.heappush(self.small, -1 * num)
if (self.small and self.large) and -1 * self.small[0] > self.large[0]:
val = -1 * heapq.heappop(self.small)
heapq.heappush(self.large, val)
if len(self.small) > len(self.large) + 1:
val = -1 * heapq.heappop(self.small)
heapq.heappush(self.large, val)
if len(self.large) > len(self.small) + 1:
val = -1 * heapq.heappop(self.large)
heapq.heappush(self.small, val)
def findMedian(self):
if len(self.small) > len(self.large):
return -1 * self.small[0]
elif len(self.small) < len(self.large):
return self.large[0]
else:
return (-1 * self.small[0] + self.large[0]) / 2
For the last line, if I change:
-1 * self.small[0] + self.large[0]
into:
-1 * heapq.heappop(self.small) + heapq.heappop(self.large)
then the tests fail.
Why would that be any different?
When you change -1 * self.small[0] + self.large[0] into -1 * heapq.heappop(self.small) + heapq.heappop(self.large) then it will still work the first time findMedian is called, but when it is called again, it will return (in general) a different result. The reason is that with heappop, you remove a value from the heap. This should not happen, as this changes the data that a next call of findMedian will have to deal with. findMedian is supposed to leave the data structure unchanged.
Note how the challenge says this:
double findMedian() returns the median of all elements so far.
I highlight the last two words. These indicate that findMedian is not (only) called when the whole stream of data has been processed, but will be called several times during the processing of the data stream. That makes it crucial that findMedian does not modify the data structure, and so heappop should not be used.
I have a number and I want to find the sum of all of its possible substrings. Since the sum may be very large, I am taking modulo 1e9+7. Here is the code I wrote for it:
n = input()
total = 0
for i in range(len(n)):
for j in range(i, len(n)):
total = (total + int(n[i:j+1]))%(1e9+7)
print(int(total))
But this gives me Overflow error:
OverflowError: int too large to convert to float
Taking modulo inside also doesn't help:
total = (total%(1e9+7) + int(n[i:j+1])%(1e9+7))%(1e9+7)
Neither does converting total to int every step:
total = int((total%(1e9+7) + int(n[i:j+1])%(1e9+7))%(1e9+7))
I searched online, and many people were using decimal, so I tried that too:
import decimal
n = input()
total = 0
for i in range(len(n)):
for j in range(i, len(n)):
total = decimal.Decimal((int(total)%(1e9+7) + int(n[i:j+1])%(1e9+7))%(1e9+7))
print(int(total))
This also gave me the same error. So how can I fix it?
EDIT:
This is the input value causing the error:
630078954945407486971302572117011329116721271139829179349572383637541443562605787816061110360853600744212572072073871985233228681677019488795915592613136558538697419369158961413804139004860949683711756764106408843746324318507090165961373116110504156510362261390086415758369311946617855091322330974469225379475157034653986221430086928768825350265642050874148148762850633380318092218497923938934331494944977897218250062378381926065651168047990017214673422724320097123140849040864223795967322521445859865961967502883479331506850337535224430391571038073324911164139663006137417510539192705821435391823658411385338369730232377028300695738866310468334377735439392551346134885024889217010755804062396901380184592416137322133531990680941959600864409332647120219490414701983979339375751465609840801484592889925172867105767663865003474673877306782492180067353856493873352082672833242368295700361941406607094298524971010582848155295175876416280789802070376313832361148791530700602039387918303750966965311391574707837728570176384970704855124594407172251268098706978376090542912929344492513384183231040016207412648019561891411057151352984928184115181483534959666911309714744265773932487092170761893895469807486999330039447615795396834925983574737750806569360090695009597077440117397176004125384806886783639660621550162794222825503301866755919064102371025428970202609474167093725973377029019369542489008907399816344268271107251422642444394295031677496626574918192538987307032199141725692286015261369012945174380684144484086524852886270717368426832490990772158458075266904766542991903970407465284703623099975166486775916952691073355877964592340230731529088307662462168730966089062336436389277513929792989509527961806655782453847736236387259057358913693838332016851781407524359352813229072505380650711693049176679191021591021389372641247384964875287336650708846120446276882272810745622693709331578317206645848042706067557480758511553310824640020649394355322305550804086562784060919544289531008318716584762468479626254129440271062831239695632816441075920969278753806765451632278698182308325313693381046301850639648800754286747096420253430423489479466804681482044272737999450499555465823164480037422222367329404142745283822947899313540404366082279717101515216147594766392292429511243594117952923319141560834706050302935730124475680758933436231491685270482951411606790235850929380027076481544634203449136248177171198738003486838372217932769689898884784243826855286091288358239337707121625402050851090825870703676300673135330647766969648618275025084598123309185991065848687389917566591933063673989563554382295252021136544002390268843474613430643976640252484632736275857290476774458675146529714463940660919706163233969511685041545624837717646981487338708223589261847530514133271116906741091917303278220653192405195999010112864383206483703165777476499412818478890639921457227974733696980927826881905040085772367856544559431242332068267736163848514181227641896012840513431718789320462329400102516689122316306450427759800832003864806238880726778463445628851437837462460535312158694258263309302240517780411649608046974426356882853543894970861987774349994574781336617336532821610244134603281437218963727001383319023603953398010028490193243704729281796345148350290421966348930850029874490245386530847693496493617805444480239004190850309960252932678216332242871877009997232445612436261407704091991812835292853218055207198699990193596327522065871513324383404436885617489886675946278629169614212976734843602198938090909080423822416990482891694415707626880182335127385564845496968722502531064508236794490149037021373756875425278069621745860863912399196641388414399463171257761192133719058891540419926061937259684055960761838199081600923840918336924120526335236516500115108241855287430263809480233364084000105646409638910878994420820422552248809062853062971314598633808334114010418924133364933998209352785219452456089337721022144079478199256501302345218342858278235244199769192326879823000937772497828914123378185067698603690831733524939588396025784744225418501083799649691393873269892335315415985069239360869306287430161725433942167215211261881754762154607113521244775811112287675881463204557903372983296624502192459971356246676084633217778109344434581417296897250881570580436981350693342923679859228288541131389540894299576663507525370715066110292813780358628483132303131742983176445700582642966318721896316623598268218673189722517718549132701529540520838922283977941956346618584749724827265852568855500361230814706488134611068845316354598799199730845206624628172917488457386459633185969341459672532379007964850480186110477536989316248137632870890622881765455399233258931339410494872048786445380805620853716965362789859403153144908183575383939310217279705139721560019568071214255994974408572648841828625948484183872232281767754936910627565103682925375742419024815987856586166378115801243175559269046804610268271061187624472308223805466602642940260042682812756256714586960165383407898099487002894444062178839391272002262993374941547876859908138330758067507911277955632733866611899695647351688393078353839611712871637384462404921878353336617815170011477929312619557000783485114417555469412886951873941491122663330089140640237525839190435611147175201365580352057169422983019756479705286311737745688878300295162035783336721495176173005235943366036579342403031858371914608154767616552994518813852963325330926988565056420401996182426971074864830358200261362270644352471067742908740066312532706251619727693803689605359722994531913085746256816708333207607372464782479204187957800323115042281681129259667352353517828344865153265302260116463924808962369896134974192415072584064052318417051648079212739917316031264326241507504548923934897411859282836187869534528811502263549599774513219264816628970416773822011462914052436110107140947441642811202817152783793620844688755256396789417685977459433814911438287855016776285922198699082347185992693560300125534315526770599208589855117129892234808497648798749646357191091546146105862763759843591202631705539149848234511495065117372464217143148171544816412125072538951795020039403886647971400954941112844849901590156966236341935249533139330401618307950786879596469984119359421473188358955907773883294422386655263220117627516995224969119904423765978663514224958566699689223512229385859291888846405683717180332460248187496572144703276206259527743572333980326718238070708650626983118629971542697495485252436024567421318695854715454101364076644687730899594195633275321940815295981062932643316855398782785214003121345721054960499469768201901436790482699005065113381322434770359397558788285726356480844311358408385253074776523383477020175018046561198787369898157059096899613482593561519419424142071652533171543891165116251895092189352681035517633166882961672980048326420273966474172389658068362117989909317156664012999905834065753973024952538940459359134550258647169854000824881036941617642608336413135147747841565829503975928572807411306180458140011310004983383026486015252188390643855363612216856777424655584758055581172316113883725410980557553123430983725023598059269514271705141359723255705532679522179269076133226643135890378252461618715340762528062563692617084267038794880235151507089462270896512581989114649911930878095396167316626567923044257435256806444296060636631937672317042925789128926010875180848009703217786800815516725253240060242786574356358801695111745961187387934689215227487920216703113061886444488110070984015605138767865701169972101412815405111287993463590619141283476936740880771219789950164205754467253452729863863950188739206664510955140236732140158021974750113109733958553067094099406873014624519007182600409029981781192576220342072442404407525694905182026252480724976436388060662190514332680872481044874701969108472815273272711817597987293115253454183720993206057768497928765003643915107639254922781596562385649150660525460064354624202531015121109364122338886448890343464196409635269404582149158284742251021906623863213747664602076214326636400706220794772134432273627481233425177349936452817255847706031575002057972719912748242114947097609007914285108694438842246267149271162583225184547641087826552789896176698128504268589381137327318145707839809327397245651593444250042725815049799745201372918555659842435159140214766015206264721726961406530774240071612440812084893955728905341340722978959370951469131953687044216423231488696742678073353379700564119257217803190999447776766945656573316206535726981650249163828677908520393197985243387213795555465230391800618144495416908755212757986378350464035578468435976351162791656892497974733578066603931101486067508297960437234016531761812630581332141184120753306673976159208563797007731706257116416221162110419311205495221132895418746168121064205944498701002545540456478596919625174145565742316551390140063146675199648332585331762709727331999408582325796339606878970603814755703167837480442251068230531926699205980639895012239254327439300220930923024855090117769633765709714826339834535876565538006788679271916019685164920493416004883901394115173882206447090610228518660081719399539376075038070034810020211340293423040766585888292494537662913458772049817318437606250938129574541767841602448300145138073940696983979059415445451195212528787305894819950526282403144596367019708209116488019479785136064985877899731579692549619154770180888669752809069235496303872000398374241532650796235196695916947874428693058258893424293987791471045924421366356810359849529988292704148275922733760964828717638828532352066259757744446229184986951298617242419115550392412022848989840586920941416655818223507527601871564570641408192234786458318407574121105566577710381048872914691225665311980882110415222436817149605338657757168302854397900614706332327212921851066498300381053609841762266343669520372320628210114413639485761915162414609422533174459519006397814257737889054381038101112761472099883783780140565576787975042564664143611845913471061951575941338844337411203552878275342039192198521263603180368307831236175304268307887495444496370405314855591523212253
Convert it into Decimal
from decimal import Decimal
Decimal(630078954945407486971302572117011329116721271139829179349572383637541443562605787816061110360853600744212572072073871985233228681677019488795915592613136558538697419369158961413804139004860949683711756764106408843746324318507090165961373116110504156510362261390086415758369311946617855091322330974469225379475157034653986221430086928768825350265642050874148148762850633380318092218497923938934331494944977897218250062378381926065651168047990017214673422724320097123140849040864223795967322521445859865961967502883479331506850337535224430391571038073324911164139663006137417510539192705821435391823658411385338369730232377028300695738866310468334377735439392551346134885024889217010755804062396901380184592416137322133531990680941959600864409332647120219490414701983979339375751465609840801484592889925172867105767663865003474673877306782492180067353856493873352082672833242368295700361941406607094298524971010582848155295175876416280789802070376313832361148791530700602039387918303750966965311391574707837728570176384970704855124594407172251268098706978376090542912929344492513384183231040016207412648019561891411057151352984928184115181483534959666911309714744265773932487092170761893895469807486999330039447615795396834925983574737750806569360090695009597077440117397176004125384806886783639660621550162794222825503301866755919064102371025428970202609474167093725973377029019369542489008907399816344268271107251422642444394295031677496626574918192538987307032199141725692286015261369012945174380684144484086524852886270717368426832490990772158458075266904766542991903970407465284703623099975166486775916952691073355877964592340230731529088307662462168730966089062336436389277513929792989509527961806655782453847736236387259057358913693838332016851781407524359352813229072505380650711693049176679191021591021389372641247384964875287336650708846120446276882272810745622693709331578317206645848042706067557480758511553310824640020649394355322305550804086562784060919544289531008318716584762468479626254129440271062831239695632816441075920969278753806765451632278698182308325313693381046301850639648800754286747096420253430423489479466804681482044272737999450499555465823164480037422222367329404142745283822947899313540404366082279717101515216147594766392292429511243594117952923319141560834706050302935730124475680758933436231491685270482951411606790235850929380027076481544634203449136248177171198738003486838372217932769689898884784243826855286091288358239337707121625402050851090825870703676300673135330647766969648618275025084598123309185991065848687389917566591933063673989563554382295252021136544002390268843474613430643976640252484632736275857290476774458675146529714463940660919706163233969511685041545624837717646981487338708223589261847530514133271116906741091917303278220653192405195999010112864383206483703165777476499412818478890639921457227974733696980927826881905040085772367856544559431242332068267736163848514181227641896012840513431718789320462329400102516689122316306450427759800832003864806238880726778463445628851437837462460535312158694258263309302240517780411649608046974426356882853543894970861987774349994574781336617336532821610244134603281437218963727001383319023603953398010028490193243704729281796345148350290421966348930850029874490245386530847693496493617805444480239004190850309960252932678216332242871877009997232445612436261407704091991812835292853218055207198699990193596327522065871513324383404436885617489886675946278629169614212976734843602198938090909080423822416990482891694415707626880182335127385564845496968722502531064508236794490149037021373756875425278069621745860863912399196641388414399463171257761192133719058891540419926061937259684055960761838199081600923840918336924120526335236516500115108241855287430263809480233364084000105646409638910878994420820422552248809062853062971314598633808334114010418924133364933998209352785219452456089337721022144079478199256501302345218342858278235244199769192326879823000937772497828914123378185067698603690831733524939588396025784744225418501083799649691393873269892335315415985069239360869306287430161725433942167215211261881754762154607113521244775811112287675881463204557903372983296624502192459971356246676084633217778109344434581417296897250881570580436981350693342923679859228288541131389540894299576663507525370715066110292813780358628483132303131742983176445700582642966318721896316623598268218673189722517718549132701529540520838922283977941956346618584749724827265852568855500361230814706488134611068845316354598799199730845206624628172917488457386459633185969341459672532379007964850480186110477536989316248137632870890622881765455399233258931339410494872048786445380805620853716965362789859403153144908183575383939310217279705139721560019568071214255994974408572648841828625948484183872232281767754936910627565103682925375742419024815987856586166378115801243175559269046804610268271061187624472308223805466602642940260042682812756256714586960165383407898099487002894444062178839391272002262993374941547876859908138330758067507911277955632733866611899695647351688393078353839611712871637384462404921878353336617815170011477929312619557000783485114417555469412886951873941491122663330089140640237525839190435611147175201365580352057169422983019756479705286311737745688878300295162035783336721495176173005235943366036579342403031858371914608154767616552994518813852963325330926988565056420401996182426971074864830358200261362270644352471067742908740066312532706251619727693803689605359722994531913085746256816708333207607372464782479204187957800323115042281681129259667352353517828344865153265302260116463924808962369896134974192415072584064052318417051648079212739917316031264326241507504548923934897411859282836187869534528811502263549599774513219264816628970416773822011462914052436110107140947441642811202817152783793620844688755256396789417685977459433814911438287855016776285922198699082347185992693560300125534315526770599208589855117129892234808497648798749646357191091546146105862763759843591202631705539149848234511495065117372464217143148171544816412125072538951795020039403886647971400954941112844849901590156966236341935249533139330401618307950786879596469984119359421473188358955907773883294422386655263220117627516995224969119904423765978663514224958566699689223512229385859291888846405683717180332460248187496572144703276206259527743572333980326718238070708650626983118629971542697495485252436024567421318695854715454101364076644687730899594195633275321940815295981062932643316855398782785214003121345721054960499469768201901436790482699005065113381322434770359397558788285726356480844311358408385253074776523383477020175018046561198787369898157059096899613482593561519419424142071652533171543891165116251895092189352681035517633166882961672980048326420273966474172389658068362117989909317156664012999905834065753973024952538940459359134550258647169854000824881036941617642608336413135147747841565829503975928572807411306180458140011310004983383026486015252188390643855363612216856777424655584758055581172316113883725410980557553123430983725023598059269514271705141359723255705532679522179269076133226643135890378252461618715340762528062563692617084267038794880235151507089462270896512581989114649911930878095396167316626567923044257435256806444296060636631937672317042925789128926010875180848009703217786800815516725253240060242786574356358801695111745961187387934689215227487920216703113061886444488110070984015605138767865701169972101412815405111287993463590619141283476936740880771219789950164205754467253452729863863950188739206664510955140236732140158021974750113109733958553067094099406873014624519007182600409029981781192576220342072442404407525694905182026252480724976436388060662190514332680872481044874701969108472815273272711817597987293115253454183720993206057768497928765003643915107639254922781596562385649150660525460064354624202531015121109364122338886448890343464196409635269404582149158284742251021906623863213747664602076214326636400706220794772134432273627481233425177349936452817255847706031575002057972719912748242114947097609007914285108694438842246267149271162583225184547641087826552789896176698128504268589381137327318145707839809327397245651593444250042725815049799745201372918555659842435159140214766015206264721726961406530774240071612440812084893955728905341340722978959370951469131953687044216423231488696742678073353379700564119257217803190999447776766945656573316206535726981650249163828677908520393197985243387213795555465230391800618144495416908755212757986378350464035578468435976351162791656892497974733578066603931101486067508297960437234016531761812630581332141184120753306673976159208563797007731706257116416221162110419311205495221132895418746168121064205944498701002545540456478596919625174145565742316551390140063146675199648332585331762709727331999408582325796339606878970603814755703167837480442251068230531926699205980639895012239254327439300220930923024855090117769633765709714826339834535876565538006788679271916019685164920493416004883901394115173882206447090610228518660081719399539376075038070034810020211340293423040766585888292494537662913458772049817318437606250938129574541767841602448300145138073940696983979059415445451195212528787305894819950526282403144596367019708209116488019479785136064985877899731579692549619154770180888669752809069235496303872000398374241532650796235196695916947874428693058258893424293987791471045924421366356810359849529988292704148275922733760964828717638828532352066259757744446229184986951298617242419115550392412022848989840586920941416655818223507527601871564570641408192234786458318407574121105566577710381048872914691225665311980882110415222436817149605338657757168302854397900614706332327212921851066498300381053609841762266343669520372320628210114413639485761915162414609422533174459519006397814257737889054381038101112761472099883783780140565576787975042564664143611845913471061951575941338844337411203552878275342039192198521263603180368307831236175304268307887495444496370405314855591523212253)
The best solution to this question runs in linear time. Here's the idea:
let's look at a shorter example: 1234
consider the total sum of the substrings which end on the ith digit (0..3)
substringsum[0]: 1 (we have only a single substring)
substringsum[1]: 2 + 12 (two substrings)
substringsum[2]: 3+23+123 (three substrings)
substringsum[3]: 4 + 34+234+1234
see a pattern?
let's look at substringsum[2]:
3 + 23 + 123 = 3 + 20+3 + 120+3 = 3*3 + 20+120 = 3*3 + 10*(2+12) = 3*3 +10*substringsum[1]
in general:
substringsum[k] = (k+1)*digit[k] + 10 * substringsum[k-1]
This you can compute in linear time.
This type of ideas is called "Dynamic Programming"
The overflow error suggest that there's an undesired conversion to float, which I assume you did not intend. The conversion happens because the type of 1e9 is float. To fix this, use % int(1e9+7) instead of % 1e9+7
We have some large binary number N (large means millions of digits). We also have binary mask M where 1 means that we must remove digit in this position in number N and move all higher bits one position right.
Example:
N = 100011101110
M = 000010001000
Res 1000110110
Is it possible to solve this problem without cycle with some set of logical or arithmetical operations? We can assume that we have access to bignum arithmetic in Python.
Feels like it should be something like this:
Res = N - (N xor M)
But it doesn't work
UPD: My current solution with cycle is following:
def prepare_reduced_arrays(dict_of_N, mask):
'''
mask: string '0000011000'
each element of dict_of_N - big python integer
'''
capacity = len(mask)
answer = dict()
for el in dict_of_N:
answer[el] = 0
new_capacity = 0
for i in range(capacity - 1, -1, -1):
if mask[i] == '1':
continue
cap2 = (1 << new_capacity)
pos = (capacity - i - 1)
for el in dict_of_N:
current_bit = (dict_of_N[el] >> pos) & 1
if current_bit:
answer[el] |= cap2
new_capacity += 1
return answer, new_capacity
While this may not be possible without a loop in python, it can be made extremely fast with numba and just in time compilation. I went on the assumption that your inputs could be easily represented as boolean arrays, which would be very simple to construct from a binary file using struct. The method I have implemented involves iterating a few different objects, however these iterations were chosen carefully to make sure they were compiler optimized, and never doing the same work twice. The first iteration is using np.where to locate the indices of all the bits to delete. This specific function (among many others) is optimized by the numba compiler. I then use this list of bit indices to build the slice indices for slices of bits to keep. The final loop copies these slices to an empty output array.
import numpy as np
from numba import jit
from time import time
def binary_mask(num, mask):
num_nbits = num.shape[0] #how many bits are in our big num
mask_bits = np.where(mask)[0] #which bits are we deleting
mask_n_bits = mask_bits.shape[0] #how many bits are we deleting
start = np.empty(mask_n_bits + 1, dtype=int) #preallocate array for slice start indexes
start[0] = 0 #first slice starts at 0
start[1:] = mask_bits + 1 #subsequent slices start 1 after each True bit in mask
end = np.empty(mask_n_bits + 1, dtype=int) #preallocate array for slice end indexes
end[:mask_n_bits] = mask_bits #each slice ends on (but does not include) True bits in the mask
end[mask_n_bits] = num_nbits + 1 #last slice goes all the way to the end
out = np.empty(num_nbits - mask_n_bits, dtype=np.uint8) #preallocate return array
for i in range(mask_n_bits + 1): #for each slice
a = start[i] #use local variables to reduce number of lookups
b = end[i]
c = a - i
d = b - i
out[c:d] = num[a:b] #copy slices
return out
jit_binary_mask = jit("b1[:](b1[:], b1[:])")(binary_mask) #decorator without syntax sugar
###################### Benchmark ########################
bignum = np.random.randint(0,2,1000000, dtype=bool) # 1 million random bits
bigmask = np.random.randint(0,10,1000000, dtype=np.uint8)==9 #delete about 1 in 10 bits
t = time()
for _ in range(10): #10 cycles of just numpy implementation
out = binary_mask(bignum, bigmask)
print(f"non-jit: {time()-t} seconds")
t = time()
out = jit_binary_mask(bignum, bigmask) #once ahead of time to compile
compile_and_run = time() - t
t = time()
for _ in range(10): #10 cycles of compiled numpy implementation
out = jit_binary_mask(bignum, bigmask)
jit_runtime = time()-t
print(f"jit: {jit_runtime} seconds")
print(f"estimated compile_time: {compile_and_run - jit_runtime/10}")
In this example, I execute the benchmark on a boolean array of length 1,000,000 a total of 10 times for both the compiled and un-compiled version. On my laptop, the output is:
non-jit: 1.865583896636963 seconds
jit: 0.06370806694030762 seconds
estimated compile_time: 0.1652850866317749
As you can see with a simple algorithm like this, very significant performance gains can be seen from compilation. (in my case about 20-30x speedup)
As far as I know, this can be done without the use of loops if and only if M is a power of 2.
Let's take your example, and modify M so that it is a power of 2:
N = 0b100011101110 = 2286
M = 0b000000001000 = 8
Removing the fourth lowest bit from N and shifting the higher bits to the right would result in:
N = 0b10001110110 = 1142
We achieved this using the following algorithm:
Begin with N = 0b100011101110 = 2286
Iterate from the most-significant bit to the least-significant bit in M.
If the current bit in M is set to 1, then store the lower bits in some variable, x:
x = 0b1101110
Then, subtract every bit up to and including the current bit in M from N, so that we end up with the following:
N - (0b10000000 + x) = N - (0b10000000 + 0b1101110) = 0b100011101110 - 0b11101110 = 0b100000000000
This step can also be achieved by and-ing the bits with 0, which may be more efficient.
Next, we shift the result once to the right:
0b100000000000 >> 1 = 0b10000000000
Finally, we add back x to the shifted result:
0b10000000000 + x = 0b10000000000 + 0b1101110 = 0b10001101110 = 1142
There may be a possibility that this can somehow be done without loops, but it would actually be efficient if you were to simply iterate over M (from the most-significant bit to the least-significant bit) and performed this process on every set bit, as the time complexity would be O(M.bit_length()).
I wrote up the code for this algorithm as well, and I believe it's relatively efficient, but I don't have any big binary numbers to test it with:
def remove_bits(N, M):
bit = 2 ** (M.bit_length() - 1)
while bit != 0:
if M & bit:
ones = bit - 1
# Store lower `bit` bits.
temp = N & ones
# Clear lower `bit` bits.
N &= ~ones
# Shift once to the right.
N >>= 1
# Set stored lower `bit` bits.
N |= temp
bit >>= 1
return N
if __name__ == '__main__':
N = 0b100011101110
M = 0b000010001000
print(bin(remove_bits(N, M)))
Using your example, this returns your result: 0b1000110110
I don't think there's any way to do this in a constant number of calls to the built-in bitwise operators. Python would have to provide something like PEXT for that to be possible.
For literally millions of digits, you may actually get best performance by working in terms of sequences of bits, sacrificing the space advantages of Python ints and the time advantages of bitwise operations in favor of more flexibility in the operations you can perform. I don't know where the break-even point would be:
import itertools
bits = bin(N)[2:]
maskbits = bin(M)[2:].zfill(len(bits))
bits = bits.zfill(len(maskbits))
chosenbits = itertools.compress(bits, map('0'.__eq__, maskbits))
result = int(''.join(chosenbits), 2)
I want to get the length of a string including a part of the string that represents its own length without padding or using structs or anything like that that forces fixed lengths.
So for example I want to be able to take this string as input:
"A string|"
And return this:
"A string|11"
On the basis of the OP tolerating such an approach (and to provide an implementation technique for the eventual python answer), here's a solution in Java.
final String s = "A String|";
int n = s.length(); // `length()` returns the length of the string.
String t; // the result
do {
t = s + n; // append the stringified n to the original string
if (n == t.length()){
return t; // string length no longer changing; we're good.
}
n = t.length(); // n must hold the total length
} while (true); // round again
The problem of, course, is that in appending n, the string length changes. But luckily, the length only ever increases or stays the same. So it will converge very quickly: due to the logarithmic nature of the length of n. In this particular case, the attempted values of n are 9, 10, and 11. And that's a pernicious case.
A simple solution is :
def addlength(string):
n1=len(string)
n2=len(str(n1))+n1
n2 += len(str(n2))-len(str(n1)) # a carry can arise
return string+str(n2)
Since a possible carry will increase the length by at most one unit.
Examples :
In [2]: addlength('a'*8)
Out[2]: 'aaaaaaaa9'
In [3]: addlength('a'*9)
Out[3]: 'aaaaaaaaa11'
In [4]: addlength('a'*99)
Out[4]: 'aaaaa...aaa102'
In [5]: addlength('a'*999)
Out[5]: 'aaaa...aaa1003'
Here is a simple python port of Bathsheba's answer :
def str_len(s):
n = len(s)
t = ''
while True:
t = s + str(n)
if n == len(t):
return t
n = len(t)
This is a much more clever and simple way than anything I was thinking of trying!
Suppose you had s = 'abcdefgh|, On the first pass through, t = 'abcdefgh|9
Since n != len(t) ( which is now 10 ) it goes through again : t = 'abcdefgh|' + str(n) and str(n)='10' so you have abcdefgh|10 which is still not quite right! Now n=len(t) which is finally n=11 you get it right then. Pretty clever solution!
It is a tricky one, but I think I've figured it out.
Done in a hurry in Python 2.7, please fully test - this should handle strings up to 998 characters:
import sys
orig = sys.argv[1]
origLen = len(orig)
if (origLen >= 98):
extra = str(origLen + 3)
elif (origLen >= 8):
extra = str(origLen + 2)
else:
extra = str(origLen + 1)
final = orig + extra
print final
Results of very brief testing
C:\Users\PH\Desktop>python test.py "tiny|"
tiny|6
C:\Users\PH\Desktop>python test.py "myString|"
myString|11
C:\Users\PH\Desktop>python test.py "myStringWith98Characters.........................................................................|"
myStringWith98Characters.........................................................................|101
Just find the length of the string. Then iterate through each value of the number of digits the length of the resulting string can possibly have. While iterating, check if the sum of the number of digits to be appended and the initial string length is equal to the length of the resulting string.
def get_length(s):
s = s + "|"
result = ""
len_s = len(s)
i = 1
while True:
candidate = len_s + i
if len(str(candidate)) == i:
result = s + str(len_s + i)
break
i += 1
This code gives the result.
I used a few var, but at the end it shows the output you want:
def len_s(s):
s = s + '|'
b = len(s)
z = s + str(b)
length = len(z)
new_s = s + str(length)
new_len = len(new_s)
return s + str(new_len)
s = "A string"
print len_s(s)
Here's a direct equation for this (so it's not necessary to construct the string). If s is the string, then the length of the string including the length of the appended length will be:
L1 = len(s) + 1 + int(log10(len(s) + 1 + int(log10(len(s)))))
The idea here is that a direct calculation is only problematic when the appended length will push the length past a power of ten; that is, at 9, 98, 99, 997, 998, 999, 9996, etc. To work this through, 1 + int(log10(len(s))) is the number of digits in the length of s. If we add that to len(s), then 9->10, 98->100, 99->101, etc, but still 8->9, 97->99, etc, so we can push past the power of ten exactly as needed. That is, adding this produces a number with the correct number of digits after the addition. Then do the log again to find the length of that number and that's the answer.
To test this:
from math import log10
def find_length(s):
L1 = len(s) + 1 + int(log10(len(s) + 1 + int(log10(len(s)))))
return L1
# test, just looking at lengths around 10**n
for i in range(9):
for j in range(30):
L = abs(10**i - j + 10) + 1
s = "a"*L
x0 = find_length(s)
new0 = s+`x0`
if len(new0)!=x0:
print "error", len(s), x0, log10(len(s)), log10(x0)
I get this error when using a python script that calculates pi using the Gauss-Legendre algorithm. You can only use up to 1024 iterations before getting this:
C:\Users\myUsernameHere>python Desktop/piWriter.py
End iteration: 1025
Traceback (most recent call last):
File "Desktop/piWriter.py", line 15, in <module>
vars()['t' + str(sub)] = vars()['t' + str(i)] - vars()['p' + str(i)] * math.
pow((vars()['a' + str(i)] - vars()['a' + str(sub)]), 2)
OverflowError: long int too large to convert to float
Here is my code:
import math
a0 = 1
b0 = 1/math.sqrt(2)
t0 = .25
p0 = 1
finalIter = input('End iteration: ')
finalIter = int(finalIter)
for i in range(0, finalIter):
sub = i + 1
vars()['a' + str(sub)] = (vars()['a' + str(i)] + vars()['b' + str(i)])/ 2
vars()['b' + str(sub)] = math.sqrt((vars()['a' + str(i)] * vars()['b' + str(i)]))
vars()['t' + str(sub)] = vars()['t' + str(i)] - vars()['p' + str(i)] * math.pow((vars()['a' + str(i)] - vars()['a' + str(sub)]), 2)
vars()['p' + str(sub)] = 2 * vars()['p' + str(i)]
n = i
pi = math.pow((vars()['a' + str(n)] + vars()['b' + str(n)]), 2) / (4 * vars()['t' + str(n)])
print(pi)
Ideally, I want to be able to plug in a very large number as the iteration value and come back a while later to see the result.
Any help appreciated!
Thanks!
Floats can only represent numbers up to sys.float_info.max, or 1.7976931348623157e+308. Once you have an int with more than 308 digits (or so), you are stuck. Your iteration fails when p1024 has 309 digits:
179769313486231590772930519078902473361797697894230657273430081157732675805500963132708477322407536021120113879871393357658789768814416622492847430639474124377767893424865485276302219601246094119453082952085005768838150682342462881473913110540827237163350510684586298239947245938479716304835356329624224137216L
You'll have to find a different algorithm for pi, one that doesn't require such large values.
Actually, you'll have to be careful with floats all around, since they are only approximations. If you modify your program to print the successive approximations of pi, it looks like this:
2.914213562373094923430016933707520365715026855468750000000000
3.140579250522168575088244324433617293834686279296875000000000
3.141592646213542838751209274050779640674591064453125000000000
3.141592653589794004176383168669417500495910644531250000000000
3.141592653589794004176383168669417500495910644531250000000000
3.141592653589794004176383168669417500495910644531250000000000
3.141592653589794004176383168669417500495910644531250000000000
In other words, after only 4 iterations, your approximation has stopped getting better. This is due to inaccuracies in the floats you are using, perhaps starting with 1/math.sqrt(2). Computing many digits of pi requires a very careful understanding of the numeric representation.
As noted in previous answer, the float type has an upper bound on number size. In typical implementations, sys.float_info.max is 1.7976931348623157e+308, which reflects the use of 10 bits plus sign for the exponent field in a 64-bit floating point number. (Note that 1024*math.log(2)/math.log(10) is about 308.2547155599.)
You can add another half dozen decades to the exponent size by using the Decimal number type. Here is an example (snipped from an ipython interpreter session):
In [48]: import decimal, math
In [49]: g=decimal.Decimal('1e12345')
In [50]: g.sqrt()
Out[50]: Decimal('3.162277660168379331998893544E+6172')
In [51]: math.sqrt(g)
Out[51]: inf
This illustrates that decimal's sqrt() function performs correctly with larger numbers than does math.sqrt().
As noted above, getting lots of digits is going to be tricky, but looking at all those vars hurts my eyes. So here's a version of your code after (1) replacing your use of vars with dictionaries, and (2) using ** instead of the math functions:
a, b, t, p = {}, {}, {}, {}
a[0] = 1
b[0] = 2**-0.5
t[0] = 0.25
p[0] = 1
finalIter = 4
for i in range(finalIter):
sub = i + 1
a[sub] = (a[i] + b[i]) / 2
b[sub] = (a[i] * b[i])**0.5
t[sub] = t[i] - p[i] * (a[i] - a[sub])**2
p[sub] = 2 * p[i]
n = i
pi_approx = (a[n] + b[n])**2 / (4 * t[n])
Instead of playing games with vars, I've used dictionaries to store the values (the link there is to the official Python tutorial) which makes your code much more readable. You can probably even see an optimization or two now.
As noted in the comments, you really don't need to store all the values, only the last, but I think it's more important that you see how to do things without dynamically creating variables. Instead of a dict, you could also have simply appended the values to a list, but lists are always zero-indexed and you can't easily "skip ahead" and set values at arbitrary indices. That can occasionally be confusing when working with algorithms, so let's start simple.
Anyway, the above gives me
>>> print(pi_approx)
3.141592653589794
>>> print(pi_approx-math.pi)
8.881784197001252e-16
A simple solution is to install and use the arbitrary-precisionmpmath module which now supports Python 3. However, since I completely agree with DSM that your use ofvars()to create variables on the fly is an undesirable way to implement the algorithm, I've based my answer on his rewrite of your code and [trivially] modified it to make use ofmpmath to do the calculations.
If you insist on usingvars(), you could probably do something similar -- although I suspect it might be more difficult and the result would definitely harder to read, understand, and modify.
from mpmath import mpf # arbitrary-precision float type
a, b, t, p = {}, {}, {}, {}
a[0] = mpf(1)
b[0] = mpf(2**-0.5)
t[0] = mpf(0.25)
p[0] = mpf(1)
finalIter = 10000
for i in range(finalIter):
sub = i + 1
a[sub] = (a[i] + b[i]) / 2
b[sub] = (a[i] * b[i])**0.5
t[sub] = t[i] - p[i] * (a[i] - a[sub])**2
p[sub] = 2 * p[i]
n = i
pi_approx = (a[n] + b[n])**2 / (4 * t[n])
print(pi_approx) # 3.14159265358979