We Keep Coding

How to make progress bar for a function in python [duplicate]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I wrote a simple console app to upload and download files from an FTP server using the ftplib.
I would like the app to show some visualization of its download/upload progress for the user; each time a data chunk is downloaded, I would like it to provide a progress update, even if it's just a numeric representation like a percentage.
Importantly, I want to avoid erasing all the text that's been printed to the console in previous lines (i.e. I don't want to "clear" the entire terminal while printing the updated progress).
This seems a fairly common task – how can I go about making a progress bar or similar visualization that outputs to my console while preserving prior program output?

Python 3
A Simple, Customizable Progress Bar
Here's an aggregate of many of the answers below that I use regularly (no imports required).
Note: All code in this answer was created for Python 3; see end of answer to use this code with Python 2.
# Print iterations progress
def printProgressBar (iteration, total, prefix = '', suffix = '', decimals = 1, length = 100, fill = '█', printEnd = "\r"):
"""
Call in a loop to create terminal progress bar
#params:
iteration - Required : current iteration (Int)
total - Required : total iterations (Int)
prefix - Optional : prefix string (Str)
suffix - Optional : suffix string (Str)
decimals - Optional : positive number of decimals in percent complete (Int)
length - Optional : character length of bar (Int)
fill - Optional : bar fill character (Str)
printEnd - Optional : end character (e.g. "\r", "\r\n") (Str)
"""
percent = ("{0:." + str(decimals) + "f}").format(100 * (iteration / float(total)))
filledLength = int(length * iteration // total)
bar = fill * filledLength + '-' * (length - filledLength)
print(f'\r{prefix} |{bar}| {percent}% {suffix}', end = printEnd)
# Print New Line on Complete
if iteration == total:
print()
Sample Usage
import time
# A List of Items
items = list(range(0, 57))
l = len(items)
# Initial call to print 0% progress
printProgressBar(0, l, prefix = 'Progress:', suffix = 'Complete', length = 50)
for i, item in enumerate(items):
# Do stuff...
time.sleep(0.1)
# Update Progress Bar
printProgressBar(i + 1, l, prefix = 'Progress:', suffix = 'Complete', length = 50)
Sample Output
Progress: |█████████████████████████████████████████████-----| 90.0% Complete
Update
There was discussion in the comments regarding an option that allows the progress bar to adjust dynamically to the terminal window width. While I don't recommend this, here's a gist that implements this feature (and notes the caveats).
Single-Call Version of The Above
A comment below referenced a nice answer posted in response to a similar question. I liked the ease of use it demonstrated and wrote a similar one, but opted to leave out the import of the sys module while adding in some of the features of the original printProgressBar function above.
Some benefits of this approach over the original function above include the elimination of an initial call to the function to print the progress bar at 0% and the use of enumerate becoming optional (i.e. it is no longer explicitly required to make the function work).
def progressBar(iterable, prefix = '', suffix = '', decimals = 1, length = 100, fill = '█', printEnd = "\r"):
"""
Call in a loop to create terminal progress bar
#params:
iterable - Required : iterable object (Iterable)
prefix - Optional : prefix string (Str)
suffix - Optional : suffix string (Str)
decimals - Optional : positive number of decimals in percent complete (Int)
length - Optional : character length of bar (Int)
fill - Optional : bar fill character (Str)
printEnd - Optional : end character (e.g. "\r", "\r\n") (Str)
"""
total = len(iterable)
# Progress Bar Printing Function
def printProgressBar (iteration):
percent = ("{0:." + str(decimals) + "f}").format(100 * (iteration / float(total)))
filledLength = int(length * iteration // total)
bar = fill * filledLength + '-' * (length - filledLength)
print(f'\r{prefix} |{bar}| {percent}% {suffix}', end = printEnd)
# Initial Call
printProgressBar(0)
# Update Progress Bar
for i, item in enumerate(iterable):
yield item
printProgressBar(i + 1)
# Print New Line on Complete
print()
Sample Usage
import time
# A List of Items
items = list(range(0, 57))
# A Nicer, Single-Call Usage
for item in progressBar(items, prefix = 'Progress:', suffix = 'Complete', length = 50):
# Do stuff...
time.sleep(0.1)
Sample Output
Progress: |█████████████████████████████████████████████-----| 90.0% Complete
Python 2
To use the above functions in Python 2, set the encoding to UTF-8 at the top of your script:
# -*- coding: utf-8 -*-
And replace the Python 3 string formatting in this line:
print(f'\r{prefix} |{bar}| {percent}% {suffix}', end = printEnd)
With Python 2 string formatting:
print('\r%s |%s| %s%% %s' % (prefix, bar, percent, suffix), end = printEnd)

Writing '\r' will move the cursor back to the beginning of the line.
This displays a percentage counter:
import time
import sys
for i in range(100):
time.sleep(1)
sys.stdout.write("\r%d%%" % i)
sys.stdout.flush()

tqdm: add a progress meter to your loops in a second:
>>> import time
>>> from tqdm import tqdm
>>> for i in tqdm(range(100)):
... time.sleep(1)
...
|###-------| 35/100 35% [elapsed: 00:35 left: 01:05, 1.00 iters/sec]

Write a \r to the console. That is a "carriage return" which causes all text after it to be echoed at the beginning of the line. Something like:
def update_progress(progress):
print '\r[{0}] {1}%'.format('#'*(progress/10), progress)
which will give you something like: [ ########## ] 100%

It is less than 10 lines of code.
The gist here: https://gist.github.com/vladignatyev/06860ec2040cb497f0f3
import sys
def progress(count, total, suffix=''):
bar_len = 60
filled_len = int(round(bar_len * count / float(total)))
percents = round(100.0 * count / float(total), 1)
bar = '=' * filled_len + '-' * (bar_len - filled_len)
sys.stdout.write('[%s] %s%s ...%s\r' % (bar, percents, '%', suffix))
sys.stdout.flush() # As suggested by Rom Ruben

Try the click library written by the Mozart of Python, Armin Ronacher.
$ pip install click # both 2 and 3 compatible
To create a simple progress bar:
import click
with click.progressbar(range(1000000)) as bar:
for i in bar:
pass
This is what it looks like:
# [###-------------------------------] 9% 00:01:14
Customize to your hearts content:
import click, sys
with click.progressbar(range(100000), file=sys.stderr, show_pos=True, width=70, bar_template='(_(_)=%(bar)sD(_(_| %(info)s', fill_char='=', empty_char=' ') as bar:
for i in bar:
pass
Custom look:
(_(_)===================================D(_(_| 100000/100000 00:00:02
There are even more options, see the API docs:
click.progressbar(iterable=None, length=None, label=None, show_eta=True, show_percent=None, show_pos=False, item_show_func=None, fill_char='#', empty_char='-', bar_template='%(label)s [%(bar)s] %(info)s', info_sep=' ', width=36, file=None, color=None)

I realize I'm late to the game, but here's a slightly Yum-style (Red Hat) one I wrote (not going for 100% accuracy here, but if you're using a progress bar for that level of accuracy, then you're WRONG anyway):
import sys
def cli_progress_test(end_val, bar_length=20):
for i in xrange(0, end_val):
percent = float(i) / end_val
hashes = '#' * int(round(percent * bar_length))
spaces = ' ' * (bar_length - len(hashes))
sys.stdout.write("\rPercent: [{0}] {1}%".format(hashes + spaces, int(round(percent * 100))))
sys.stdout.flush()
Should produce something looking like this:
Percent: [############## ] 69%
... where the brackets stay stationary and only the hashes increase.
This might work better as a decorator. For another day...

Check this library: clint
it has a lot of features including a progress bar:
from time import sleep
from random import random
from clint.textui import progress
if __name__ == '__main__':
for i in progress.bar(range(100)):
sleep(random() * 0.2)
for i in progress.dots(range(100)):
sleep(random() * 0.2)
this link provides a quick overview of its features

Here's a nice example of a progressbar written in Python: http://nadiana.com/animated-terminal-progress-bar-in-python
But if you want to write it yourself. You could use the curses module to make things easier :)
[edit]
Perhaps easier is not the word for curses. But if you want to create a full-blown cui than curses takes care of a lot of stuff for you.
[edit]
Since the old link is dead I have put up my own version of a Python Progressbar, get it here: https://github.com/WoLpH/python-progressbar

import time,sys
for i in range(100+1):
time.sleep(0.1)
sys.stdout.write(('='*i)+(''*(100-i))+("\r [ %d"%i+"% ] "))
sys.stdout.flush()
output
[ 29% ] ===================

Install tqdm.(pip install tqdm)
and use it as follows:
import time
from tqdm import tqdm
for i in tqdm(range(1000)):
time.sleep(0.01)
That's a 10 seconds progress bar that'll output something like this:
47%|██████████████████▊ | 470/1000 [00:04<00:05, 98.61it/s]

and, just to add to the pile, here's an object you can use:
Add the following to a new file progressbar.py
import sys
class ProgressBar(object):
CHAR_ON = '='
CHAR_OFF = ' '
def __init__(self, end=100, length=65):
self._end = end
self._length = length
self._chars = None
self._value = 0
#property
def value(self):
return self._value
#value.setter
def value(self, value):
self._value = max(0, min(value, self._end))
if self._chars != (c := int(self._length * (self._value / self._end))):
self._chars = c
sys.stdout.write("\r {:3n}% [{}{}]".format(
int((self._value / self._end) * 100.0),
self.CHAR_ON * int(self._chars),
self.CHAR_OFF * int(self._length - self._chars),
))
sys.stdout.flush()
def __enter__(self):
self.value = 0
return self
def __exit__(self, *args, **kwargs):
sys.stdout.write('\n')
Can be included in your program with:
import time
from progressbar import ProgressBar
count = 150
print("starting things:")
with ProgressBar(count) as bar:
for i in range(count + 1):
bar.value += 1
time.sleep(0.01)
print("done")
Results in:
starting things:
100% [=================================================================]
done
This may be "over the top", but is handy when used frequently.

Run this at the Python command line (not in any IDE or development environment):
>>> import threading
>>> for i in range(50+1):
... threading._sleep(0.5)
... print "\r%3d" % i, ('='*i)+('-'*(50-i)),
Works fine on my Windows system.

Try to install this package: pip install progressbar2 :
import time
import progressbar
for i in progressbar.progressbar(range(100)):
time.sleep(0.02)
progresssbar github: https://github.com/WoLpH/python-progressbar

http://code.activestate.com/recipes/168639-progress-bar-class/ (2002)
http://code.activestate.com/recipes/299207-console-text-progress-indicator-class/ (2004)
http://pypi.python.org/pypi/progressbar (2006)
And a lot of tutorials waiting to be googled.

based on the above answers and other similar questions about CLI progress bar, I think I got a general common answer to all of them. Check it at https://stackoverflow.com/a/15860757/2254146
In summary, the code is this:
import time, sys
# update_progress() : Displays or updates a console progress bar
## Accepts a float between 0 and 1. Any int will be converted to a float.
## A value under 0 represents a 'halt'.
## A value at 1 or bigger represents 100%
def update_progress(progress):
barLength = 10 # Modify this to change the length of the progress bar
status = ""
if isinstance(progress, int):
progress = float(progress)
if not isinstance(progress, float):
progress = 0
status = "error: progress var must be float\r\n"
if progress < 0:
progress = 0
status = "Halt...\r\n"
if progress >= 1:
progress = 1
status = "Done...\r\n"
block = int(round(barLength*progress))
text = "\rPercent: [{0}] {1}% {2}".format( "#"*block + "-"*(barLength-block), progress*100, status)
sys.stdout.write(text)
sys.stdout.flush()
Looks like
Percent: [##########] 99.0%

I am using progress from reddit. I like it because it can print progress for every item in one line, and it shouldn't erase printouts from the program.
Edit: fixed link

A very simple solution is to put this code into your loop:
Put this in the body (i.e. top) of your file:
import sys
Put this in the body of your loop:
sys.stdout.write("-") # prints a dash for each iteration of loop
sys.stdout.flush() # ensures bar is displayed incrementally

I recommend using tqdm - https://pypi.python.org/pypi/tqdm - which makes it simple to turn any iterable or process into a progress bar, and handles all messing about with terminals needed.
From the documentation: "tqdm can easily support callbacks/hooks and manual updates. Here’s an example with urllib"
import urllib
from tqdm import tqdm
def my_hook(t):
"""
Wraps tqdm instance. Don't forget to close() or __exit__()
the tqdm instance once you're done with it (easiest using `with` syntax).
Example
-------
>>> with tqdm(...) as t:
... reporthook = my_hook(t)
... urllib.urlretrieve(..., reporthook=reporthook)
"""
last_b = [0]
def inner(b=1, bsize=1, tsize=None):
"""
b : int, optional
Number of blocks just transferred [default: 1].
bsize : int, optional
Size of each block (in tqdm units) [default: 1].
tsize : int, optional
Total size (in tqdm units). If [default: None] remains unchanged.
"""
if tsize is not None:
t.total = tsize
t.update((b - last_b[0]) * bsize)
last_b[0] = b
return inner
eg_link = 'http://www.doc.ic.ac.uk/~cod11/matryoshka.zip'
with tqdm(unit='B', unit_scale=True, miniters=1,
desc=eg_link.split('/')[-1]) as t: # all optional kwargs
urllib.urlretrieve(eg_link, filename='/dev/null',
reporthook=my_hook(t), data=None)

import sys
def progresssbar():
for i in range(100):
time.sleep(1)
sys.stdout.write("%i\r" % i)
progressbar()
NOTE: if you run this in interactive interepter you get extra numbers printed out

lol i just wrote a whole thingy for this
heres the code keep in mind you cant use unicode when doing block ascii i use cp437
import os
import time
def load(left_side, right_side, length, time):
x = 0
y = ""
print "\r"
while x < length:
space = length - len(y)
space = " " * space
z = left + y + space + right
print "\r", z,
y += "█"
time.sleep(time)
x += 1
cls()
and you call it like so
print "loading something awesome"
load("|", "|", 10, .01)
so it looks like this
loading something awesome
|█████ |

With the great advices above I work out the progress bar.
However I would like to point out some shortcomings
Every time the progress bar is flushed, it will start on a new line
print('\r[{0}]{1}%'.format('#' * progress* 10, progress))
like this:
[] 0%
[#]10%
[##]20%
[###]30%
2.The square bracket ']' and the percent number on the right side shift right as the '###' get longer.
3. An error will occur if the expression 'progress / 10' can not return an integer.
And the following code will fix the problem above.
def update_progress(progress, total):
print('\r[{0:10}]{1:>2}%'.format('#' * int(progress * 10 /total), progress), end='')

For python 3:
def progress_bar(current_value, total):
increments = 50
percentual = ((current_value/ total) * 100)
i = int(percentual // (100 / increments ))
text = "\r[{0: <{1}}] {2}%".format('=' * i, increments, percentual)
print(text, end="\n" if percentual == 100 else "")

function from Greenstick for 2.7:
def printProgressBar (iteration, total, prefix = '', suffix = '',decimals = 1, length = 100, fill = '#'):
percent = ("{0:." + str(decimals) + "f}").format(100 * (iteration / float(total)))
filledLength = int(length * iteration // total)
bar = fill * filledLength + '-' * (length - filledLength)
print'\r%s |%s| %s%% %s' % (prefix, bar, percent, suffix),
sys.stdout.flush()
# Print New Line on Complete
if iteration == total:
print()

Code for python terminal progress bar
import sys
import time
max_length = 5
at_length = max_length
empty = "-"
used = "%"
bar = empty * max_length
for i in range(0, max_length):
at_length -= 1
#setting empty and full spots
bar = used * i
bar = bar+empty * at_length
#\r is carriage return(sets cursor position in terminal to start of line)
#\0 character escape
sys.stdout.write("[{}]\0\r".format(bar))
sys.stdout.flush()
#do your stuff here instead of time.sleep
time.sleep(1)
sys.stdout.write("\n")
sys.stdout.flush()

The python module progressbar is a nice choice.
Here is my typical code:
import time
import progressbar
widgets = [
' ', progressbar.Percentage(),
' ', progressbar.SimpleProgress(format='(%(value_s)s of %(max_value_s)s)'),
' ', progressbar.Bar('>', fill='.'),
' ', progressbar.ETA(format_finished='- %(seconds)s -', format='ETA: %(seconds)s', ),
' - ', progressbar.DynamicMessage('loss'),
' - ', progressbar.DynamicMessage('error'),
' '
]
bar = progressbar.ProgressBar(redirect_stdout=True, widgets=widgets)
bar.start(100)
for i in range(100):
time.sleep(0.1)
bar.update(i + 1, loss=i / 100., error=i)
bar.finish()

i wrote a simple progressbar:
def bar(total, current, length=10, prefix="", filler="#", space=" ", oncomp="", border="[]", suffix=""):
if len(border) != 2:
print("parameter 'border' must include exactly 2 symbols!")
return None
print(prefix + border[0] + (filler * int(current / total * length) +
(space * (length - int(current / total * length)))) + border[1], suffix, "\r", end="")
if total == current:
if oncomp:
print(prefix + border[0] + space * int(((length - len(oncomp)) / 2)) +
oncomp + space * int(((length - len(oncomp)) / 2)) + border[1], suffix)
if not oncomp:
print(prefix + border[0] + (filler * int(current / total * length) +
(space * (length - int(current / total * length)))) + border[1], suffix)
as you can see, it have: length of bar, prefix and suffix, filler, space, text in bar on 100%(oncomp) and borders
here an example:
from time import sleep, time
start_time = time()
for i in range(10):
pref = str((i+1) * 10) + "% "
complete_text = "done in %s sec" % str(round(time() - start_time))
sleep(1)
bar(10, i + 1, length=20, prefix=pref, oncomp=complete_text)
out in progress:
30% [###### ]
out on complete:
100% [ done in 9 sec ]

Putting together some of the ideas I found here, and adding estimated time left:
import datetime, sys
start = datetime.datetime.now()
def print_progress_bar (iteration, total):
process_duration_samples = []
average_samples = 5
end = datetime.datetime.now()
process_duration = end - start
if len(process_duration_samples) == 0:
process_duration_samples = [process_duration] * average_samples
process_duration_samples = process_duration_samples[1:average_samples-1] + [process_duration]
average_process_duration = sum(process_duration_samples, datetime.timedelta()) / len(process_duration_samples)
remaining_steps = total - iteration
remaining_time_estimation = remaining_steps * average_process_duration
bars_string = int(float(iteration) / float(total) * 20.)
sys.stdout.write(
"\r[%-20s] %d%% (%s/%s) Estimated time left: %s" % (
'='*bars_string, float(iteration) / float(total) * 100,
iteration,
total,
remaining_time_estimation
)
)
sys.stdout.flush()
if iteration + 1 == total:
print
# Sample usage
for i in range(0,300):
print_progress_bar(i, 300)

Well here is code that works and I tested it before posting:
import sys
def prg(prog, fillchar, emptchar):
fillt = 0
emptt = 20
if prog < 100 and prog > 0:
prog2 = prog/5
fillt = fillt + prog2
emptt = emptt - prog2
sys.stdout.write("\r[" + str(fillchar)*fillt + str(emptchar)*emptt + "]" + str(prog) + "%")
sys.stdout.flush()
elif prog >= 100:
prog = 100
prog2 = prog/5
fillt = fillt + prog2
emptt = emptt - prog2
sys.stdout.write("\r[" + str(fillchar)*fillt + str(emptchar)*emptt + "]" + str(prog) + "%" + "\nDone!")
sys.stdout.flush()
elif prog < 0:
prog = 0
prog2 = prog/5
fillt = fillt + prog2
emptt = emptt - prog2
sys.stdout.write("\r[" + str(fillchar)*fillt + str(emptchar)*emptt + "]" + str(prog) + "%" + "\nHalted!")
sys.stdout.flush()
Pros:
20 character bar (1 character for every 5 (number wise))
Custom fill characters
Custom empty characters
Halt (any number below 0)
Done (100 and any number above 100)
Progress count (0-100 (below and above used for special functions))
Percentage number next to bar, and it's a single line
Cons:
Supports integers only (It can be modified to support them though, by making the division an integer division, so just change prog2 = prog/5 to prog2 = int(prog/5))

Here's my Python 3 solution:
import time
for i in range(100):
time.sleep(1)
s = "{}% Complete".format(i)
print(s,end=len(s) * '\b')
'\b' is a backslash, for each character in your string.
This does not work within the Windows cmd window.

Creating own .obj exporter for maya

I am creating my own .obj exporter for maya.
When i'm exporting just one mesh my code works just fine but when exporting several meshes / objects it fails to create the complete meshes.
I'm pretty sure that the problem is when i'm getting the
face.getVertices(), face.getUVIndex() and face.normalIndex() and printing them to the file. As i said the first mesh works fine but when it gets to the second mesh the codinates gets all wrong, they connect to the wrong triangles.
If anyone has any ideas on how to possibly loop them differently or change the values to the correct ones i would be forever greatful. Help would be very very appreciated!
Here is an example on how a multi object mesh ends out.
http://postimg.org/image/rr0fvs0v7/
import pymel.core as pm
import pymel.core.nodetypes as nt
planes = pm.ls(sl=True)
def meshFile():
def myRound(n):
return round(n, 6)
file = open("C:/Users/Blondiegirls/Desktop/test2.obj", "wb")
file.write("mtllib test2.mtl\r\n")
for p in planes[:]:
#pm.polyTriangulate(planes[0])
file.write("\r\ng default")
# Printa world kordinater
for index, point in enumerate(p.vtx):
temp = index,map(myRound, point.getPosition(space='world'))
file.write("\r\nv ")
file.write(str(' '.join(map(str, temp[1]))))
# Printa texture kordinater
mesh = pm.ls(g=True)[0]
U,V = mesh.getUVs()
UVs = zip(U,V)
for uv in UVs:
file.write("\r\nvt ")
file.write(str(uv[0])+" "+str(uv[1]))
#printa normals
for n in p.getNormals():
file.write("\r\nvn ")
file.write(str(n[0])+" "+str(n[1])+" "+str(n[2]))
file.write("\r\ns 1")
file.write("\r\ng ")
file.write(str(p))
file.write("\r\nusemtl test")
for faceIndex, face in enumerate(p.faces):
faceVertices = face.getVertices()
faceUV0 = face.getUVIndex(0)+1
faceUV1 = face.getUVIndex(1)+1
faceUV2 = face.getUVIndex(2)+1
faceNor0 = face.normalIndex(0)+1
faceNor1 = face.normalIndex(1)+1
faceNor2 = face.normalIndex(2)+1
file.write("\r\nf ")
faceVertices0 = int(faceVertices[0])+1
faceVertices1 = int(faceVertices[1])+1
faceVertices2 = int(faceVertices[2])+1
temp3 = (str(faceVertices0)) + "/" + (str(faceUV0)) +"/" + (str(faceNor0)) + " " + (str(faceVertices1)) + "/" + (str(faceUV1)) +"/" + (str(faceNor1)) + " " + (str(faceVertices2)) + "/" + (str(faceUV2)) +"/" + (str(faceNor2))
file.write(str(temp3))
file.close()
meshFile()
def MTLFile():
file2 = open("C:/Users/Blondiegirls/Desktop/test2.mtl", "wb")
object = cmds.ls(sl=1)[0].split(':')[0]
#print('object: '+object)
shipTX = pm.PyNode(object)
shadingGroups = shipTX.shadingGroups()
sg1 = shadingGroups[0]
material = sg1.listConnections(source=True, destination=False, type=nt.Lambert)[0]
file = material.color.listConnections(type=nt.File)[0]
filename = file.fileTextureName.get()
materialColor = material.getColor() #for Kd
materialAmbient = material.getAmbientColor() #for Ka
materialSpecular = material.getSpecularColor() #for Ks
refractiveIndex = material.getRefractiveIndex() #for Ni
file2.write("newmtl "+"test"+"\r\n")
file2.write("Ka "+str(materialAmbient[0])+" "+str(materialAmbient[1])+" "+str(materialAmbient[2])+"\r\n")
file2.write("Kd "+str(materialColor[0])+" "+str(materialColor[1])+" "+str(materialColor[2])+"\r\n")
file2.write("Ks "+str(materialSpecular[0])+" "+str(materialSpecular[1])+" "+str(materialSpecular[2])+"\r\n")
file2.write("d 1.0\r\n")
file2.write("Illum 2\r\n")
file2.write("map_Kd "+filename+"\r\n") #for map_Kd
file2.close()
MTLFile()

The problem is this line:
mesh = pm.ls(g=True)[0]
U,V = mesh.getUVs()
This is querying all of the geometry in the scene and returning the first object, but you should be operating on the current mesh in the iteration. I think what you want is:
U,V = p.getUVs()
Also, you should probably consider adding an argument to meshFile() rather than relying on a variable in the global scope.

Python calculations

I am trying to make a program that will display total stock of cement, brick, spanner, mirror, and hammer. But the problem Please help me
is thsi program keeps showing me' cementinVar is not defined' but i already defined it above as (cementinVar = Tkinter.IntVar()) .
def textboxvalue2():
Total_StockIn = 0
CementIn = cementinVar.get()
HammerIn = hammerinVar.get()
SpannerIn = spannerinVar.get()
BrickIn = brickinVar.get()
MirrorIn = mirrorinVar.get()
CementOut = cementoutVar.get()
HammerOur = hammeroutVar.get()
SpannerOut = spanneroutVar.get()
BrickOut = brickoutVar.get()
MirrorOut = mirroroutVar.get()
Total_StockIn = (CementIn + HammerIn + SpannerIn + BrickIn + MirrorIn)+Total_StockIn
StockInLabel = Tkinter.Label(sub,text='The total stock in is '+str(Total_StockIn))
StockInLabel.grid(row=7, column =2)

You define cementinVar in another scope. You have two possible solutions:
Use a global variable
Use a class variable

Python script to build least-cost paths between several polygons: How to speed up it?

I created a python program which uses the function "CostPath" of ArcGIS to automatically build least-cost paths (LCPs) between several polygons contained in the shapefile "selected_patches.shp". My python program seems to work but it is much too slow. I must build 275493 LCPs. Unfortunately, I don't know how to speed up my program (I am a beginner in Python programming language and ArcGIS). Or is there another solution to calculate rapidly least-cost paths between several polygons with ArcGIS (I use ArcGIS 10.1) ? Here is my code:
# Import system modules
import arcpy
from arcpy import env
from arcpy.sa import *
arcpy.CheckOutExtension("Spatial")
# Overwrite outputs
arcpy.env.overwriteOutput = True
# Set the workspace
arcpy.env.workspace = "C:\Users\LCP"
# Set the extent environment
arcpy.env.extent = "costs.tif"
rowsInPatches_start = arcpy.SearchCursor("selected_patches.shp")
for rowStart in rowsInPatches_start:
ID_patch_start = rowStart.getValue("GRIDCODE")
expressionForSelectInPatches_start = "GRIDCODE=%s" % (ID_patch_start) ## Define SQL expression for the fonction Select Layer By Attribute
# Process: Select Layer By Attribute in Patches_start
arcpy.MakeFeatureLayer_management("selected_patches.shp", "Selected_patch_start", expressionForSelectInPatches_start)
# Process: Cost Distance
outCostDist=CostDistance("Selected_patch_start", "costs.tif", "", "outCostLink.tif")
# Save the output
outCostDist.save("outCostDist.tif")
rowsInSelectedPatches_end = arcpy.SearchCursor("selected_patches.shp")
for rowEnd in rowsInSelectedPatches_end:
ID_patch_end = rowEnd.getValue("GRIDCODE")
expressionForSelectInPatches_end = "GRIDCODE=%s" % (ID_patch_end) ## Define SQL expression for the fonction Select Layer By Attribute
# Process: Select Layer By Attribute in Patches_end
arcpy.MakeFeatureLayer_management("selected_patches.shp", "Selected_patch_end", expressionForSelectInPatches_end)
# Process: Cost Path
outCostPath = CostPath("Selected_patch_end", "outCostDist.tif", "outCostLink.tif", "EACH_ZONE","FID")
# Save the output
outCostPath.save('P_' + str(int(ID_patch_start)) + '_' + str(int(ID_patch_end)) + ".tif")
# Writing in file .txt
outfile=open('P_' + str(int(ID_patch_start)) + '_' + str(int(ID_patch_end)) + ".txt", "w")
rowsTxt = arcpy.SearchCursor('P_' + str(int(ID_patch_start)) + '_' + str(int(ID_patch_end)) + ".tif")
for rowTxt in rowsTxt:
value = rowTxt.getValue("Value")
count = rowTxt.getValue("Count")
pathcost = rowTxt.getValue("PATHCOST")
startrow = rowTxt.getValue("STARTROW")
startcol = rowTxt.getValue("STARTCOL")
print value, count, pathcost, startrow, startcol
outfile.write(str(value) + " " + str(count) + " " + str(pathcost) + " " + str(startrow) + " " + str(startcol) + "\n")
outfile.close()
Thanks very much for your help.

The speed it takes to write to disc vs calculating your cost can be a bottleneck, consider adding a thread to handle all of your writes.
This:
for rowTxt in rowsTxt:
value = rowTxt.getValue("Value")
count = rowTxt.getValue("Count")
pathcost = rowTxt.getValue("PATHCOST")
startrow = rowTxt.getValue("STARTROW")
startcol = rowTxt.getValue("STARTCOL")
print value, count, pathcost, startrow, startcol
outfile.write(str(value) + " " + str(count) + " " + str(pathcost) + " " + str(startrow) + " " + str(startcol) + "\n")
Can be converted into a thread function by making rowsTxt a global variable, and having your thread write to disk from rowsTxt.
After you complete all of your processing you can have an additional global boolean so that your thread function can end when you are done writing everything and you can close your thread.
Example thread function I currently use:
import threading
class ThreadExample:
def __init__(self):
self.receiveThread = None
def startRXThread(self):
self.receiveThread = threading.Thread(target = self.receive)
self.receiveThread.start()
def stopRXThread(self):
if self.receiveThread is not None:
self.receiveThread.__Thread__stop()
self.receiveThread.join()
self.receiveThread = None
def receive(self):
while true:
#do stuff for the life of the thread
#in my case, I listen on a socket for data
#and write it out
So for your case, you could add a class variable to the thread class
self.rowsTxt
and then update your receive to check self.rowsTxt, and if it is not empty, handle it as u do in the code snippet i took from you above. After you handle it, set self.rowsTxt back to None. You could update your threads self.rowsTxt with your main function as it gets rowsTxt. Consider using a buffer like list for self.rowsTxt so you don't miss writing anything.

The most immediate change you can make to significant improve speed would be to switch to data access cursors (e.g. arcpy.da.SearchCursor()). To illustrate, I ran a benchmark test a while back to see the data access cursors perform compared to the old cursors.
The attached figure shows the results of a benchmark test on the new da method UpdateCursor versus the old UpdateCursor method. Essentially, the benchmark test performs the following workflow:
Create random points (10, 100, 1000, 10000, 100000)
Randomly sample from a normal distribution and add value to a new
column in the random points attribute table with a cursor
Run 5 iterations of each random point scenario for both the new and
old UpdateCursor methods and write the mean value to lists
Plot the results
import arcpy, os, numpy, time
arcpy.env.overwriteOutput = True
outws = r'C:\temp'
fc = os.path.join(outws, 'randomPoints.shp')
iterations = [10, 100, 1000, 10000, 100000]
old = []
new = []
meanOld = []
meanNew = []
for x in iterations:
arcpy.CreateRandomPoints_management(outws, 'randomPoints', '', '', x)
arcpy.AddField_management(fc, 'randFloat', 'FLOAT')
for y in range(5):
# Old method ArcGIS 10.0 and earlier
start = time.clock()
rows = arcpy.UpdateCursor(fc)
for row in rows:
# generate random float from normal distribution
s = float(numpy.random.normal(100, 10, 1))
row.randFloat = s
rows.updateRow(row)
del row, rows
end = time.clock()
total = end - start
old.append(total)
del start, end, total
# New method 10.1 and later
start = time.clock()
with arcpy.da.UpdateCursor(fc, ['randFloat']) as cursor:
for row in cursor:
# generate random float from normal distribution
s = float(numpy.random.normal(100, 10, 1))
row[0] = s
cursor.updateRow(row)
end = time.clock()
total = end - start
new.append(total)
del start, end, total
meanOld.append(round(numpy.mean(old),4))
meanNew.append(round(numpy.mean(new),4))
#######################
# plot the results
import matplotlib.pyplot as plt
plt.plot(iterations, meanNew, label = 'New (da)')
plt.plot(iterations, meanOld, label = 'Old')
plt.title('arcpy.da.UpdateCursor -vs- arcpy.UpdateCursor')
plt.xlabel('Random Points')
plt.ylabel('Time (minutes)')
plt.legend(loc = 2)
plt.show()

python script to click button

I have this cod working for long time for a game web site but now they changed the way it should be clicked :-
def formRaid():
#print formerID
one = 1
while one == 1:
try:
global _startTime
_startTime = datetime.datetime.now()
formpage = alo("http://"+server+".outwar.com/formraid.php?target=" + bossToRaidId+"&suid="+FORMER_ID).read()
codeID = formpage.split('codeid" value="')[1].split('">')[0]
Form = alo2("http://"+server+".outwar.com/formraid.php?target=" + bossToRaidId, "target=" + bossToRaidId + "&codeid=" + codeID + "&formtime=3&submit=Launch!").read()
global RAID_ID, RAID_NAME
RAID_ID = getRaidID(Form)
RAID_NAME = getRaidName(Form)
print ""
print "***" + RAID_NAME + " Formed."
one = 0
return True
except IndexError:
print "Forming Error, retrying...."
time.sleep(1)
Now i have to click a button to go one again and the information for this button is :-
<input type="submit" value="Join this Raid!" name="submit">
How i can make the click that used in :-
formpage = alo("http://"+server+".outwar.com/formraid.php?target=" + bossToRaidId+"&suid="+FORMER_ID).read()
codeID = formpage.split('codeid" value="')[1].split('">')[0]
or the one that used in :-
Form = alo2("http://"+server+".outwar.com/formraid.php?target=" + bossToRaidId, "target=" + bossToRaidId + "&codeid=" + codeID + "&formtime=3&submit=Launch!").read()
Click that button instead of what it did before ??

opener.open("http://"+server+".outwar.com/formraid.php?target=" + bossToRaidId + "&suid="+FORMER_ID, "formtime=3&message=&bomb=none&submit=Launch!")
only works for 2.7 python