Im not sure about the best way to do this but I have a python script saved as a .py. The final output of this script is two files x1.txt and y1.txt.
Basically I want to run this script say 1000 times and each run write my two text files with new names i.e x1.txt + y1.txt then second run x2.txt and y2.txt.
Thinking about this it seems it might be better to start the whole script with something like
runs=xrange(:999)
for i in runs:
##run the script
and then finish with something that does
for i in runs:
filnameA=prefix += "a"+i
open("filnamea.txt", "w").write('\n'.join('\t'.join(x for x in g if x) for g in grouper(7, values)))
for i in runs:
filnameB=prefix += "a"+i
open("filnameB.txt", "w").write('\n'.join('\t'.join(x for x in g if x) for g in grouper(7, values)))
Is this really the best way to do it? I bet its not..better ideas?
I know you can import time and write a filename that mathes time but this would be annoying for processing later.
If your computer has the resources to run these in parallel, you can use multiprocessing to do it. Otherwise use a loop to execute them sequentially.
Your question isn't quite explicit about which part you're stuck with. Do you just need advice about whether you should use a loop? If yes, my answer is above. Or do you also need help with forming the filenames? You can do that part like this:
import sys
def myscript(iteration_number):
xfile_name = "x%d.txt" % iteration_number
yfile_name = "y%d.txt" % iteration_number
with open(xfile_name, "w") as xf:
with open(yfile_name, "w") as yf:
... whatever your script does goes here
def main(unused_command_line_args):
for i in xrange(1000):
myscript(i)
return 0
if __name__ == '__main__':
sys.exit(main(sys.argv))
import subprocess
import sys
script_name = 'dummy_file.py'
output_prefix = 'out'
n_iter = 5
for i in range(n_iter):
output_file = output_prefix + '_' + str(i) + '.txt'
sys.stdout = open(output_file, 'w')
subprocess.call(['python', script_name], stdout=sys.stdout, stderr=subprocess.STDOUT)
On running this, you'll get 5 output text files (out_0.txt, ..., out_4.txt)
I'm not sure, but maybe, it can help:
Suppose, I want to print 'hello' 10 times, without manually writing it 10 times. For doing this, I can define a function :
#Function for printing hello 10 times:
def func(x):
x="hello"
i=1
while i<10 :
print(x)
i += 1
else :
print(x)
print(func(1))
Related
Please be kind this is my second post and i hope you all like.
Here I have made a program that makes directories inside directories,
but the problem is I would like a way to make it self replicate.
Any ideas and help is greatly appreciated.
Before:
user/scripts
After:
user/scripts/worm1/worm2/worm3
The script is as follows:
import os, sys, string, random
worms_made = 0
stop = 20
patha = ''
pathb = '/'
pathc = ''
def fileworm(worms_made, stop, patha, pathb, pathc):
filename = (''.join(random.choice(string.ascii_lowercase
+string.ascii_uppercase + string.digits) for i in range(8)))
pathc = patha + filename + pathb
worms_made = worms_made + 1
os.system("mkdir %s" % filename)
os.chdir(pathc)
print "Worms made: %r" % worms_made
if worms_made == stop:
print "*Done"
exit(0)
elif worms_made != stop:
pass
fileworm(worms_made, stop, patha, pathb, pathc)
fileworm(worms_made, stop, patha, pathb, pathc)
To create a variable depth, you could do something like this:
import os
depth = 3
worms = ['worm{}'.format(x) for x in range(1, depth+1)]
path = os.path.join(r'/user/scripts', *worms)
os.path.makedirs(path)
As mentioned, os.path.makedirs() will create all the required folders in one call. You just need to build the full path.
Python has a function to help with creating paths called os.path.join(). This makes sure the correct / or \ is automatically added for the current operating system between each part.
worms is a list containing ["worm1", "worm2", "worm3"], it is created using a Python feature called a list comprehension. This is passed to the os.path.join() function using * meaning the each element of the list is passed as a separate parameter.
I suggest you try adding print worms or print path to see how it works.
The result is that a string looking something like as follows is passed to the function to create your folder structure:
/user/scripts/worm1/worm2/worm3
I have two functions prints() and clear(). The function prints() print lines of text and the function clear() delete the printed lines with this method:
def clear():
i = 0
while True:
if(i > globals['l']):
break
else:
sys.stdout.write("\033[F \033[K")
i += 1
where globals['l'] is the number of lines to clear.
Then after the function clear() runs and the lines are cleared, the function prints() run again etc...
I don't understand why the function clear() is clearing only 22 lines of 32 lines. But if I have, for example, 19 lines it is working perfectly. Where is the problem? How can I fix this?
Try this:
def clear(line_count):
sys.stdout.write("\033[F \033[K" * line_count)
sys.stdout.flush()
EDIT:
Works only on Unix systems
You can use:
print("\033c")
Or:
import subprocess
subprocess.call(["printf", "\033c"])
I am an almost new programmer learning python for a few months. For the last 2 weeks, I had been coding to make a script to search permutations of numbers that make magic squares.
Finally I succeeded in searching the whole 880 4x4 magic square numbers sets within 30 seconds. After that I made some different Perimeter Magic Square program. It finds out more than 10,000,000 permutations so that I want to store them part by part to files. The problem is that my program doesn't use all my processes that while it is working to store some partial data to a file, it stops searching new number sets. I hope I could make one process of my CPU keep searching on and the others store the searched data to files.
The following is of the similar structure to my magic square program.
while True:
print('How many digits do you want? (more than 20): ', end='')
ansr = input()
if ansr.isdigit() and int(ansr) > 20:
ansr = int(ansr)
break
else:
continue
fileNum = 0
itemCount = 0
def fileMaker():
global fileNum, itemCount
tempStr = ''
for i in permutationList:
itemCount += 1
tempStr += str(sum(i[:3])) + ' : ' + str(i) + ' : ' + str(itemCount) + '\n'
fileNum += 1
file = open('{0} Permutations {1:03}.txt'.format(ansr, fileNum), 'w')
file.write(tempStr)
file.close()
numList = [i for i in range(1, ansr+1)]
permutationList = []
itemCount = 0
def makePermutList(numList, ansr):
global permutationList
for i in numList:
numList1 = numList[:]
numList1.remove(i)
for ii in numList1:
numList2 = numList1[:]
numList2.remove(ii)
for iii in numList2:
numList3 = numList2[:]
numList3.remove(iii)
for iiii in numList3:
numList4 = numList3[:]
numList4.remove(iiii)
for v in numList4:
permutationList.append([i, ii, iii, iiii, v])
if len(permutationList) == 200000:
print(permutationList[-1])
fileMaker()
permutationList = []
fileMaker()
makePermutList(numList, ansr)
I added from multiprocessing import Pool at the top. And I replaced two 'fileMaker()' parts at the end with the following.
if __name__ == '__main__':
workers = Pool(processes=2)
workers.map(fileMaker, ())
The result? Oh no. It just works awkwardly. For now, multiprocessing looks too difficult for me.
Anybody, please, teach me something. How should my code be modified?
Well, addressing some things that are bugging me before getting to your asked question.
numList = [i for i in range(1, ansr+1)]
I know list comprehensions are cool, but please just do list(range(1, ansr+1)) if you need the iterable to be a list (which you probably don't need, but I digress).
def makePermutList(numList, ansr):
...
This is quite the hack. Is there a reason you can't use itertools.permutations(numList,n)? It's certainly going to be faster, and friendlier on memory.
Lastly, answering your question: if you are looking to improve i/o performance, the last thing you should do is make it multithreaded. I don't mean you shouldn't do it, I mean that it should literally be the last thing you do. Refactor/improve other things first.
You need to take all of that top-level code that uses globals, apply the backspace key to it, and rewrite functions that pass data around properly. Then you can think about using threads. I would personally use from threading import Thread and manually spawn Threads to do each unit of I/O rather than using multiprocessing.
I was playing with this sudoku solver, that I found.
Like quoted here it works perfect, but if I uncomment that single print a, that I commented out (line 13), then it stops before finding a full solution...?
import sys
from datetime import datetime # for datetime.now()
def same_row(i,j): return (i/9 == j/9)
def same_col(i,j): return (i-j) % 9 == 0
def same_block(i,j): return (i/27 == j/27 and i%9/3 == j%9/3)
def r(a):
i = a.find('.')
if i == -1: # All solved !
print a
else:
#print a
excluded_numbers = set()
for j in range(81):
if same_row(i,j) or same_col(i,j) or same_block(i,j):
excluded_numbers.add(a[j])
for m in '123456789':
if m not in excluded_numbers:
# At this point, m is not excluded by any row, column, or block, so let's place it and recurse
r(a[:i]+m+a[i+1:])
if __name__ == '__main__':
if len(sys.argv) == 2:
filI = open(sys.argv[1])
for pusI in filI:
pusI.strip()
print "pussle:\n",pusI
timStart = datetime.now()
r(pusI) # <- Calling the recursive solver ...
timEnd = datetime.now()
print "Duration (h:mm:ss.dddddd): "+str(timEnd-timStart)
else:
print str(len(sys.argv))
print 'Usage: python sudoku.py puzzle'
The program needs to be called with a file. That file should hold 1 sudoku per line.
For testing I used this:
25...1........8.6...3...4.1..48.6.9...9.4.8...1..29.4.9.53.7....6..5...7.........
QUESTION:
I can't understand how that single 'print a' manage to break the recursive loop, before it's done. Can anyone give an explanation?
Credit: I originally found the above sudoku solver code here:
http://www.scottkirkwood.com/2006/07/shortest-sudoku-solver-in-python.html
it's also shown here on StackOverflow:
Shortest Sudoku Solver in Python - How does it work?
It actually does find the solution.
I ran the program and get the solution
256491738471238569893765421534876192629143875718529643945387216162954387387612954
If you run with the uncommenting as you suggested and output that to a file:
python solver.py file.txt > output.txt
And search for the solution string, it is there. It's not the last line, for me it shows up 67% into the file.
The reason it does this is that the solver basically goes through a ton of combinations and it finds the solution but continues as long as there are any possible paths to go down to find a possible solution.
I am trying to understand how to get children to write to a parent's variables. Maybe I'm doing something wrong here, but I would have imagined that multiprocessing would have taken a fraction of the time that it is actually taking:
import multiprocessing, time
def h(x):
h.q.put('Doing: ' + str(x))
return x
def f_init(q):
h.q = q
def main():
q = multiprocessing.Queue()
p = multiprocessing.Pool(None, f_init, [q])
results = p.imap(h, range(1,5))
p.close()
-----Results-----:
1
2
3
4
Multiprocessed: 0.0695610046387 seconds
1
2
3
4
Normal: 2.78949737549e-05 seconds # much shorter
for i in range(len(range(1,5))):
print results.next() # prints 1, 4, 9, 16
if __name__ == '__main__':
start = time.time()
main()
print "Multiprocessed: %s seconds" % (time.time()-start)
start = time.time()
for i in range(1,5):
print i
print "Normal: %s seconds" % (time.time()-start)
#Blender basically already answered your question, but as a comment. There is some overhead associated with the multiprocessing machinery, so if you incur the overhead without doing any significant work, it will be slower.
Try actually doing some work that parallelizes well. For example, write Python code to open a file, scan it using a regular expression, and pull out matching lines; then make a list of ten big files and time how long it takes to do all ten with multiprocessing vs. plain Python. Or write code to compute an expensive function and try that.
I have used multiprocessing.Pool() just to run a bunch of instances of an external program. I used subprocess to run an audio encoder, and it ran four instances of the encoder at once for a noticeable speedup.