We Keep Coding

Completion time of "_invalidate_internal" in Matplotlib increases linearly over extended periods. What is this function doing?

I am running a Python Tkinter GUI application for multiple days (ideally it will run continuously without needing to be restarted). After a few days, the program typically crashes with no explanation or error messages. I have used cProfile to profile the animation functionality of our Matplotlib graphs and search for possible slow-down points. The results clearly showed that the only function which increased in execution time over the program run period was this one:
/usr/lib/python3/dist-packages/matplotlib/transforms.py:134(_invalidate_internal)
I've looked at the Matplotlib source code (https://github.com/matplotlib/matplotlib/blob/main/lib/matplotlib/transforms.py), but I haven't been able to figure out what this function is doing. What does _invalidate_internal do, and is there anything I can do to prevent it from taking so long to execute?
For more context, our program has several animated matplotlib plots which graph inputs from sensor data over time. We plot the n most recent data points on each frame so that it gives the effect of a scrolling graph. Here is the animation code:
def animate(ii):
while True:
most_recent_time_graphed = param_dict[param_list[0]] #first, pulls up first plot
most_recent = reader.query_by_num(table="SensorData", num=1)
reader.commit() #if identical, do not animate
#then checks that plot's time list
if (len(most_recent) == 0):
break
#time_reader = datetime.strptime(most_recent[0][0], "%m/%d/%Y %H:%M:%S")
time_reader = datetime.datetime.fromtimestamp(most_recent[0][0])
if (len(most_recent_time_graphed.tList) != 0) and (time_reader == most_recent_time_graphed.tList[0]):
for i, param in enumerate(param_list, 1):
current_text = live_dict[param]
current_text.label.config(text=most_recent[0][i], fg="black", bg="white")
break #checks if the timestamp is exactly the same as prior, i.e. no new data points have been logged in this frame
#do I have to add an else?
else:
config_settings = csv_read()
c0, c1, c2 = config_dict['enable_text'], config_dict['num_config'], config_dict['provider_config']
c3, c4, c5 = config_dict['email_config'], config_dict['upper_config'], config_dict['lower_config']
for i, key in enumerate(param_dict, 1):
current_plot = param_dict[key]
current_param_val = float(most_recent[0][i])
current_text = live_dict[key] #update to live text data summary
if current_param_val > float(config_settings[c4][i-1]) or current_param_val < float(config_settings[c5][i-1]):
#only send text if enable_text is True
if config_settings[c0] == [str(True)]:
###sends text if new problem arises or every 5 minutes
if allIsGood[key] and Minute[key] == None:
print('new problem')
Minute[key] = datetime.now().minute
minuta[key] = Minute[key]
pCheck(float(config_settings[c4][i-1]),float(config_settings[c5][i-1]),key,current_param_val,config_settings[c1],config_settings[c2]) #uncomment to test emergency texts
elif allIsGood[key] == False and abs(Minute[key] - datetime.now().minute) % 5 == 0 and not (minuta[key] == datetime.now().minute):
print('same problem')
minuta[key] = datetime.now().minute
pCheck(float(config_settings[c4][i-1]),float(config_settings[c5][i-1]),key,current_param_val,config_settings[c1],config_settings[c2]) #uncomment to test emergency texts
#pass
#setting the parameter to not ok
allIsGood[key] = False
current_text.label.config(text=most_recent[0][i], fg="red", bg="white")
current_plot.plot_color = 'r'
else:
current_text.label.config(text=most_recent[0][i], fg="black", bg="white")
current_plot.plot_color = 'g'
###setting the parameter back to true and sending "ok" text
if allIsGood[key] == False:
Minute[key] = None
allOk(key,config_settings[c1],config_settings[c2])
pass
allIsGood[key] = True
data_stream = current_plot.incoming_data
time_stream = current_plot.tList
data_stream.insert(0, most_recent[0][i])
#time_f = datetime.strptime(most_recent[0][0], "%m/%d/%Y %H:%M:%S")
time_f = datetime.datetime.fromtimestamp(most_recent[0][0])
time_stream.insert(0, time_f)
if len(data_stream) < 20: #graph updates, growing to show 20 points
current_plot.make_plot()
else: #there are 20 points and more available, so animation occurs
data_stream.pop()
time_stream.pop()
current_plot.make_plot()
break

Python Chess Data (FEN) into Stockfish for Python

I am trying to use stockfish to evaluate a chess position using FEN notation all in Python. I am mainly using two libraries (pgnToFen I found on github here: https://github.com/SindreSvendby/pgnToFen and Stockfish the MIT licensed one here: https://github.com/zhelyabuzhsky/stockfish). After many bugs I have reached problem after problem. Stockfish not only can't analyse this FEN position (3b2k1/1p3pp1/8/3pP1P1/pP3P2/P2pB3/6K1/8 b f3 -) but it infinitely loops! "No worries!" and thought changing the source code would be accomplishable. Changed to _put(), but basically I am unable to put dummy values in because stdin.flush() won't execute once I give it those values! Meaning I don't even think I can skip to the next row in my dataframe. :( The code I changed is below.
def _put(self, command: str, tmp_time) -> None:
if not self.stockfish.stdin:
raise BrokenPipeError()
self.stockfish.stdin.write(f"{command}\n")
try:
self.stockfish.stdin.flush()
except:
if command != "quit":
self.stockfish.stdin.write('isready\n')
try:
time.sleep(tmp_time)
self.stockfish.stdin.flush()
except:
#print ('Imma head out', file=sys.stderr)
raise ValueError('Imma head out...')
#sys.stderr.close()
def get_evaluation(self) -> dict:
"""Evaluates current position
Returns:
A dictionary of the current advantage with "type" as "cp" (centipawns) or "mate" (checkmate in)
"""
evaluation = dict()
fen_position = self.get_fen_position()
if "w" in fen_position: # w can only be in FEN if it is whites move
compare = 1
else: # stockfish shows advantage relative to current player, convention is to do white positive
compare = -1
self._put(f"position {fen_position}", 5)
self._go()
x=0
while True:
x=x+1
text = self._read_line()
#print(text)
splitted_text = text.split(" ")
if splitted_text[0] == "info":
for n in range(len(splitted_text)):
if splitted_text[n] == "score":
evaluation = {
"type": splitted_text[n + 1],
"value": int(splitted_text[n + 2]) * compare,
}
elif splitted_text[0] == "bestmove":
return evaluation
elif x == 500:
evaluation = {
"type": 'cp',
"value": 10000,
}
return evaluation
and last but not least change to the init_ contructor below:
self._stockfish_major_version: float = float(self._read_line().split(" ")[1])
And the code where I am importing this code to is below, this is where errors pop up.
import pandas as pd
import re
import nltk
import numpy as np
from stockfish import Stockfish
import os
import sys
sys.path.insert(0, r'C:\Users\path\to\pgntofen')
import pgntofen
#nltk.download('punkt')
#Changed models.py for major version line 39 in stockfish from int to float
stockfish = Stockfish(r"C:\Users\path\to\Stockfish.exe")
file = r'C:\Users\path\to\selenium-pandas output.csv'
chunksize = 10 ** 6
for chunk in pd.read_csv(file, chunksize=chunksize):
for index, row in chunk.iterrows():
FullMovesStr = str(row['FullMoves'])
FullMovesStr = FullMovesStr.replace('+', '')
if "e.p" in FullMovesStr:
row.to_csv(r'C:\Users\MyName\Logger.csv', header=None, index=False, mode='a')
print('Enpassant')
continue
tokens = nltk.word_tokenize(FullMovesStr)
movelist = []
for tokenit in range(len(tokens)):
if "." in str(tokens[tokenit]):
try:
tokenstripped = re.sub(r"[0-9]+\.", "", tokens[tokenit])
token = [tokenstripped, tokens[tokenit+1]]
movelist.append(token)
except:
continue
else:
continue
DFMoves = pd.DataFrame(movelist, columns=[['WhiteMove', 'BlackMove']])
DFMoves['index'] = row['index']
DFMoves['Date'] = row['Date']
DFMoves['White'] = row['White']
DFMoves['Black'] = row['Black']
DFMoves['W ELO'] = row['W ELO']
DFMoves['B ELO'] = row['B ELO']
DFMoves['Av ELO'] = row['Av ELO']
DFMoves['Event'] = row['Event']
DFMoves['Site'] = row['Site']
DFMoves['ECO'] = row['ECO']
DFMoves['Opening'] = row['Opening']
pd.set_option('display.max_rows', DFMoves.shape[0]+1)
print(DFMoves[['WhiteMove', 'BlackMove']])
seqmoves = []
#seqmovesBlack = []
evalmove = []
pgnConverter = pgntofen.PgnToFen()
#stockfish.set_fen_position("rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1")
#rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
for index, row in DFMoves.iterrows():
try:
stockfish.set_fen_position("rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1")
except:
evalmove.append("?")
continue
#stockfish.set_fen_position("rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1")
pgnConverter.resetBoard()
WhiteMove = str(row['WhiteMove'])
BlackMove = str(row['BlackMove'])
if index == 0:
PGNMoves1 = [WhiteMove]
seqmoves.append(WhiteMove)
#seqmoves.append(BlackMove)
else:
seqmoves.append(WhiteMove)
#seqmoves.append(BlackMove)
PGNMoves1 = seqmoves.copy()
#print(seqmoves)
try:
pgnConverter.pgnToFen(PGNMoves1)
fen = pgnConverter.getFullFen()
except:
break
try:
stockfish.set_fen_position(fen)
print(stockfish.get_board_visual())
evalpos = stockfish.get_evaluation()
evalmove.append(evalpos)
except:
pass
try:
stockfish.set_fen_position("rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1")
except:
evalmove.append("?")
continue
pgnConverter.resetBoard()
if index == 0:
PGNMoves2 = [WhiteMove, BlackMove]
seqmoves.append(BlackMove)
else:
seqmoves.append(BlackMove)
PGNMoves2 = seqmoves.copy()
try:
pgnConverter.pgnToFen(PGNMoves2)
fen = pgnConverter.getFullFen()
except:
break
try:
stockfish.set_fen_position(fen)
print(stockfish.get_board_visual())
evalpos = stockfish.get_evaluation()
print(evalpos)
evalmove.append(evalpos)
except:
pass
#DFMoves['EvalWhite'] = evalwhite
#DFMoves['EvalBlack'] = evalblack
print(evalmove)
So the detailed question is getting stockfish.get_evalution() to just skip, or better yet fix the problem, for this ( 3b2k1/1p3pp1/8/3pP1P1/pP3P2/P2pB3/6K1/8 b f3 - ) FEN position. I have been working on this problem for quite a while so any insight into this would be very much appreciated.
My specs are Windows 10, Python 3.9, Processor:Intel(R) Core(TM) i9-10980XE CPU # 3.00GHz 3.00 GHz and RAM is 64.0 GB.
Thanks :)

Ok. It seems your fen is invalid (3b2k1/1p3pp1/8/3pP1P1/pP3P2/P2pB3/6K1/8 b f3 -). So check that. And python-chess (https://python-chess.readthedocs.io/en/latest/index.html) library allows you to use FEN AND chess engines. So, pretty cool no ? Here is an example of theses two fantastics tools :
import chess
import chess.engine
import chess.pgn
pgn = open("your_pgn_file.pgn")
game = chess.pgn.read_game(pgn)
engine = chess.engine.SimpleEngine.popen_uci("your_stockfish_path.exe")
# Iterate through all moves, play them on a board and analyse them.
board = game.board()
for move in game.mainline_moves():
board.push(move)
print(engine.analyse(board, chess.engine.Limit(time=0.1))["score"])

How to Auto scale y and x axis of a graph in real time python

I modified the code of this tutorial to create my own real time plot:
https://learn.sparkfun.com/tutorials/graph-sensor-data-with-python-and-matplotlib/speeding-up-the-plot-animation
I needed to plot the data from a proximity sensor in real time, the data is sent through USB cable to the computer and I read it with the serial port, so the code is already working how I wanted to, but I also want to modify the y-axis and x-axis, not let it static, because sometimes the peaks are 3000 and sometimes 2000 and when the sensor is not being touched, the peaks are at around 200 because it also detects the ambient light. any clue how can I make it?
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import serial
# Data from serial port
port = 'COM8'
baudrate = 9600
tout = 0.01 # Miliseconds
# Time to update the data of the sensor signal real time Rs=9600baud T=1/Rs
tiempo = (1 / baudrate) * 1000
# Parameters
x_len = 200 # Number of points to display
y_range = [20000, 40000] # Range of Y values to display
# Create figure for plotting
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
xs = list(range(0, x_len))
ys = [0] * x_len
ax.set_ylim(y_range)
# Create a blank line. We will update the line in animate
line, = ax.plot(xs, ys)
# Markers
startMarker = 60 # Start marker "<"
endMarker = 62 # End marker ">"
# Begin Arduino communication, Port COM8, speed 9600
serialPort = serial.Serial(port, baudrate, timeout=tout)
# Begin to save the arduino data
def arduinodata():
global startMarker, endMarker
ck = ""
x = "z" # any value that is not an end- or startMarker
bytecount = -1 # to allow for the fact that the last increment will be one too many
# wait for the start character
while ord(x) != startMarker:
x = serialPort.read()
# save data until the end marker is found
while ord(x) != endMarker:
if ord(x) != startMarker:
ck = ck + x.decode()
bytecount += 1
x = serialPort.read()
return ck
def readarduino():
# Wait until the Arduino sends '<Arduino Ready>' - allows time for Arduino reset
# It also ensures that any bytes left over from a previous message are discarded
msg = ""
while msg.find("<Arduino is ready>") == -1:
while serialPort.inWaiting() == 0:
pass
# delete for example the "\r\n" that may contain the message
msg = arduinodata()
msg = msg.split("\r\n")
msg = ''.join(msg)
# If the sensor send very big numbers over 90000 they will be deleted
if msg and len(msg) <= 5:
msg = int(msg)
return msg
elif msg and len(msg) >= 4:
msg = int(msg)
return msg
# This function is called periodically from FuncAnimation
def animate(i, ys):
# Read pulse from PALS2
pulse = readarduino()
# Add y to list
ys.append(pulse)
# Limit x and y lists to set number of items
ys = ys[-x_len:]
# Update line with new Y values
line.set_ydata(ys)
return line,
# Plot labels
plt.title('Heart frequency vs Time')
plt.ylabel('frequency ')
plt.xlabel('Samples')
# Set up plot to call animate() function periodically
ani = animation.FuncAnimation(fig, animate, fargs=(ys,), interval=tiempo, blit=True)
plt.show()
plt.close()
serialPort.close()
This is how the graph looks like, the x and y axis are always the same:

If you want to autoscale the y-axis, then you can simply adjust the y-axis limits in your animate() function:
def animate(i, ys):
# Read pulse from PALS2
pulse = readarduino()
# Add y to list
ys.append(pulse)
# Limit x and y lists to set number of items
ys = ys[-x_len:]
ymin = np.min(ys)
ymax = np.max(ys)
ax.set_ylim(ymin, ymax)
# Update line with new Y values
line.set_ydata(ys)
return line,
HOWEVER, the result will not be what you expect if you use blit=True. This is because blitting is attempting to only draw the parts of the graphs that have changed, and the ticks on the axes are excluded from that. If you need to change the limits and therefore the ticks, you should use blit=False in your call to FuncAnimation. Note that you will encounter a performance hit since matplotlib will have to redraw the whole plot at every frame, but if you want to change the limits, there is no way around that.

So I made some changes to the last code of this link https://www.learnpyqt.com/courses/graphics-plotting/plotting-pyqtgraph/
and I could solve the problem. x and Y axis are now autoscaling
import PyQt5
from PyQt5 import QtWidgets, QtCore
from pyqtgraph import PlotWidget, plot
import pyqtgraph as pg
import sys # We need sys so that we can pass argv to QApplication
import os
from random import randint
import serial
class MainWindow(QtWidgets.QMainWindow):
def __init__(self, *args, **kwargs):
super(MainWindow, self).__init__(*args, **kwargs)
self.graphWidget = pg.PlotWidget()
self.setCentralWidget(self.graphWidget)
# Data from serial port
self.port = 'COM8'
self.baudrate = 9600
self.tout = 0.01 # Miliseconds
# Time to update the data of the sensor signal Rs=9600baud T=1/Rs
self.tiempo = (1 / self.baudrate) * 1000
self.x_len = 200
self.x = list(range(0, self.x_len)) # 100 time points
self.y = [0] * self.x_len # 100 data points
self.graphWidget.setBackground('w')
# Markers
self.startMarker = 60 # Start marker "<"
self.endMarker = 62 # End marker ">"
# Begin Arduino communication, Port COM8, speed 9600
self.serialPort = serial.Serial(self.port, self.baudrate, timeout=self.tout)
pen = pg.mkPen(color=(255, 0, 0))
self.data_line = self.graphWidget.plot(self.x, self.y, pen=pen)
self.timer = QtCore.QTimer()
self.timer.setInterval(self.tiempo)
self.timer.timeout.connect(self.update_plot_data)
self.timer.start()
# Begin to save the arduino data
def arduinodata(self):
ck = ""
x = "z" # any value that is not an end- or startMarker
bytecount = -1 # to allow for the fact that the last increment will be one too many
# wait for the start character
while ord(x) != self.startMarker:
x = self.serialPort.read()
# save data until the end marker is found
while ord(x) != self.endMarker:
if ord(x) != self.startMarker:
ck = ck + x.decode()
bytecount += 1
x = self.serialPort.read()
return ck
def readarduino(self):
# Wait until the Arduino sends '<Arduino Ready>' - allows time for Arduino reset
# It also ensures that any bytes left over from a previous message are discarded
msg = ""
while msg.find("<Arduino is ready>") == -1:
while self.serialPort.inWaiting() == 0:
pass
# delete for example the "\r\n" that may contain the message
msg = self.arduinodata()
msg = msg.split("\r\n")
msg = ''.join(msg)
# If the sensor send very big numbers over 90000 they will be deleted
if msg and len(msg) <= 5:
msg = int(msg)
return msg
elif msg and len(msg) >= 4:
msg = int(msg)
return msg
def update_plot_data(self):
pulse = self.readarduino()
self.x = self.x[1:] # Remove the first y element.
self.x.append(self.x[-1] + 1) # Add a new value 1 higher than the last.
self.y = self.y[1:] # Remove the first
self.y.append(pulse) # Add a new random value.
self.data_line.setData(self.x, self.y) # Update the data.
app = QtWidgets.QApplication(sys.argv)
w = MainWindow()
w.show()
sys.exit(app.exec_())

Print a threading function in curses, Python

In my project I have to print some data stored in a database on the console. I have two functions that return string. I should print those data in two columns.
I thought of using the module curses in python to create those two columns.
So far all is good.
Another thing, is that my two function use the threading.Timer. So the string generated changes every 10 seconds for example if I set my Timer on 10.
So when I put the result of my function on the addstr() of a column, it prints the first String correctly, but nothing changes even if my string change. Otherwise when I resize for example the console, I notice that the string changes.
Here is my code :
import sys,os
import curses , curses.panel
import datetime
import time
import threading
def time_func():
printStr = threading.Timer(10,time_func)
printStr.start()
s = "Simple example that display datetime \n" + datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
return s
def draw_menu():
k = 0
cursor_x = 0
cursor_y = 0
# Clear and refresh the screen
stdscr = curses.initscr()
stdscr.clear()
stdscr.refresh()
# Initialize colors in curses
curses.start_color()
curses.init_pair(1, curses.COLOR_CYAN, curses.COLOR_BLACK)
curses.init_pair(2, curses.COLOR_RED, curses.COLOR_BLACK)
curses.init_pair(3, curses.COLOR_BLACK, curses.COLOR_WHITE)
curses.curs_set(0)
# Loop where k is the last character pressed
while (k != ord('q')):
# Declaration of strings
statusbarstr = "Press 'q' to exit | STATUS BAR "
'''Draw borders of the different columns'''
height, width = stdscr.getmaxyx()
stdscr.border()
stdscr.vline(1, 3 * height // 4, '|', width - 2)
stdscr.vline(1, height // 4, '|', width - 2)
stdscr.refresh()
'''Initialize the Statistics column'''
column_one = curses.newwin(height - 2, 3 * width // 4, 1,
1)
_, a_x = column_one.getmaxyx()
column_one.attron(curses.color_pair(1))
column_one.attron(curses.A_BOLD)
column_one.addstr(0, a_x // 2 - 2, "column one")
column_one.hline(1, 0, '-', a_x)
column_one.attroff(curses.color_pair(1))
column_one.attroff(curses.A_BOLD)
# I want to add my string here for example :
line = time_func()
column_one.addstr(3,1, line)
column_one.noutrefresh()
'''Initialize the Alerts column'''
column_two = curses.newwin(height - 2, width // 4, 1,
3 * width // 4 )
_, s_x = column_two.getmaxyx()
column_two.attron(curses.color_pair(1))
column_two.attron(curses.A_BOLD)
column_two.addstr(0, s_x // 2 - 5, "column two")
column_two.hline(1, 0, '-', s_x)
column_two.attroff(curses.color_pair(1))
column_two.attroff(curses.A_BOLD)
column_two.addstr(3, s_x // 2 - 5,"test")
column_two.noutrefresh()
# Render status bar
stdscr.attron(curses.color_pair(3))
stdscr.addstr(height-1, 0, statusbarstr)
stdscr.addstr(height-1, len(statusbarstr), " " * (width - len(statusbarstr) - 1))
stdscr.attroff(curses.color_pair(3))
# Turning on attributes for title
stdscr.attron(curses.color_pair(2))
stdscr.attron(curses.A_BOLD)
# Rendering title
# Turning off attributes for title
stdscr.attroff(curses.color_pair(2))
stdscr.attroff(curses.A_BOLD)
# Print rest of text
stdscr.move(height - 2, width - 2)
# Refresh the screen
stdscr.refresh()
# Wait for next input
k = stdscr.getch()
def main():
curses.wrapper(draw_menu())
if __name__ == "__main__":
main()

The curses function draw_menu is waiting for input at the end of the loop. If you resize the terminal, that
k = stdscr.getch()
returns a KEY_RESIZE, and runs through the loop again.
You could change that to wait a short time before giving up on the getch. That would return a -1.
Usually that's done using timeout (in curses), e.g., 50 milliseconds. There's a wrapper for that in Python (and by the way its description is incorrect). You should read the ncurses manual page for timeout.
Sometimes people say to use nodelay, which is usually poor advice since it would make the draw_menu thread use most of the CPU time.

Python script to build least-cost paths between several polygons: How to speed up it?

I created a python program which uses the function "CostPath" of ArcGIS to automatically build least-cost paths (LCPs) between several polygons contained in the shapefile "selected_patches.shp". My python program seems to work but it is much too slow. I must build 275493 LCPs. Unfortunately, I don't know how to speed up my program (I am a beginner in Python programming language and ArcGIS). Or is there another solution to calculate rapidly least-cost paths between several polygons with ArcGIS (I use ArcGIS 10.1) ? Here is my code:
# Import system modules
import arcpy
from arcpy import env
from arcpy.sa import *
arcpy.CheckOutExtension("Spatial")
# Overwrite outputs
arcpy.env.overwriteOutput = True
# Set the workspace
arcpy.env.workspace = "C:\Users\LCP"
# Set the extent environment
arcpy.env.extent = "costs.tif"
rowsInPatches_start = arcpy.SearchCursor("selected_patches.shp")
for rowStart in rowsInPatches_start:
ID_patch_start = rowStart.getValue("GRIDCODE")
expressionForSelectInPatches_start = "GRIDCODE=%s" % (ID_patch_start) ## Define SQL expression for the fonction Select Layer By Attribute
# Process: Select Layer By Attribute in Patches_start
arcpy.MakeFeatureLayer_management("selected_patches.shp", "Selected_patch_start", expressionForSelectInPatches_start)
# Process: Cost Distance
outCostDist=CostDistance("Selected_patch_start", "costs.tif", "", "outCostLink.tif")
# Save the output
outCostDist.save("outCostDist.tif")
rowsInSelectedPatches_end = arcpy.SearchCursor("selected_patches.shp")
for rowEnd in rowsInSelectedPatches_end:
ID_patch_end = rowEnd.getValue("GRIDCODE")
expressionForSelectInPatches_end = "GRIDCODE=%s" % (ID_patch_end) ## Define SQL expression for the fonction Select Layer By Attribute
# Process: Select Layer By Attribute in Patches_end
arcpy.MakeFeatureLayer_management("selected_patches.shp", "Selected_patch_end", expressionForSelectInPatches_end)
# Process: Cost Path
outCostPath = CostPath("Selected_patch_end", "outCostDist.tif", "outCostLink.tif", "EACH_ZONE","FID")
# Save the output
outCostPath.save('P_' + str(int(ID_patch_start)) + '_' + str(int(ID_patch_end)) + ".tif")
# Writing in file .txt
outfile=open('P_' + str(int(ID_patch_start)) + '_' + str(int(ID_patch_end)) + ".txt", "w")
rowsTxt = arcpy.SearchCursor('P_' + str(int(ID_patch_start)) + '_' + str(int(ID_patch_end)) + ".tif")
for rowTxt in rowsTxt:
value = rowTxt.getValue("Value")
count = rowTxt.getValue("Count")
pathcost = rowTxt.getValue("PATHCOST")
startrow = rowTxt.getValue("STARTROW")
startcol = rowTxt.getValue("STARTCOL")
print value, count, pathcost, startrow, startcol
outfile.write(str(value) + " " + str(count) + " " + str(pathcost) + " " + str(startrow) + " " + str(startcol) + "\n")
outfile.close()
Thanks very much for your help.

The speed it takes to write to disc vs calculating your cost can be a bottleneck, consider adding a thread to handle all of your writes.
This:
for rowTxt in rowsTxt:
value = rowTxt.getValue("Value")
count = rowTxt.getValue("Count")
pathcost = rowTxt.getValue("PATHCOST")
startrow = rowTxt.getValue("STARTROW")
startcol = rowTxt.getValue("STARTCOL")
print value, count, pathcost, startrow, startcol
outfile.write(str(value) + " " + str(count) + " " + str(pathcost) + " " + str(startrow) + " " + str(startcol) + "\n")
Can be converted into a thread function by making rowsTxt a global variable, and having your thread write to disk from rowsTxt.
After you complete all of your processing you can have an additional global boolean so that your thread function can end when you are done writing everything and you can close your thread.
Example thread function I currently use:
import threading
class ThreadExample:
def __init__(self):
self.receiveThread = None
def startRXThread(self):
self.receiveThread = threading.Thread(target = self.receive)
self.receiveThread.start()
def stopRXThread(self):
if self.receiveThread is not None:
self.receiveThread.__Thread__stop()
self.receiveThread.join()
self.receiveThread = None
def receive(self):
while true:
#do stuff for the life of the thread
#in my case, I listen on a socket for data
#and write it out
So for your case, you could add a class variable to the thread class
self.rowsTxt
and then update your receive to check self.rowsTxt, and if it is not empty, handle it as u do in the code snippet i took from you above. After you handle it, set self.rowsTxt back to None. You could update your threads self.rowsTxt with your main function as it gets rowsTxt. Consider using a buffer like list for self.rowsTxt so you don't miss writing anything.

The most immediate change you can make to significant improve speed would be to switch to data access cursors (e.g. arcpy.da.SearchCursor()). To illustrate, I ran a benchmark test a while back to see the data access cursors perform compared to the old cursors.
The attached figure shows the results of a benchmark test on the new da method UpdateCursor versus the old UpdateCursor method. Essentially, the benchmark test performs the following workflow:
Create random points (10, 100, 1000, 10000, 100000)
Randomly sample from a normal distribution and add value to a new
column in the random points attribute table with a cursor
Run 5 iterations of each random point scenario for both the new and
old UpdateCursor methods and write the mean value to lists
Plot the results
import arcpy, os, numpy, time
arcpy.env.overwriteOutput = True
outws = r'C:\temp'
fc = os.path.join(outws, 'randomPoints.shp')
iterations = [10, 100, 1000, 10000, 100000]
old = []
new = []
meanOld = []
meanNew = []
for x in iterations:
arcpy.CreateRandomPoints_management(outws, 'randomPoints', '', '', x)
arcpy.AddField_management(fc, 'randFloat', 'FLOAT')
for y in range(5):
# Old method ArcGIS 10.0 and earlier
start = time.clock()
rows = arcpy.UpdateCursor(fc)
for row in rows:
# generate random float from normal distribution
s = float(numpy.random.normal(100, 10, 1))
row.randFloat = s
rows.updateRow(row)
del row, rows
end = time.clock()
total = end - start
old.append(total)
del start, end, total
# New method 10.1 and later
start = time.clock()
with arcpy.da.UpdateCursor(fc, ['randFloat']) as cursor:
for row in cursor:
# generate random float from normal distribution
s = float(numpy.random.normal(100, 10, 1))
row[0] = s
cursor.updateRow(row)
end = time.clock()
total = end - start
new.append(total)
del start, end, total
meanOld.append(round(numpy.mean(old),4))
meanNew.append(round(numpy.mean(new),4))
#######################
# plot the results
import matplotlib.pyplot as plt
plt.plot(iterations, meanNew, label = 'New (da)')
plt.plot(iterations, meanOld, label = 'Old')
plt.title('arcpy.da.UpdateCursor -vs- arcpy.UpdateCursor')
plt.xlabel('Random Points')
plt.ylabel('Time (minutes)')
plt.legend(loc = 2)
plt.show()