We Keep Coding

Called Arcpy Script doesn't finish if called from C# Console App

Good Morning StackOverflowers,
There is another problem i cant solve without your help. I am working on a C# Console App (.NET 5) which is calling different Python Scripts to do ArcGis Stuff (=> "arcpy"-Lib :). The Scheme of calling them is always the same and there was never a problem. In the course of penetration tests i discovered an issue by one of the scripts:
The following script "foreaches" every row in the featureclass, zooms to it and exports it as an .png-File.
Executing this via a cmd (Non Admin & Admin) or via Python Gui (available with ArcGis Setup) it's works perfectly (=> creating 138 images), BUT if i execute it via C# App, it only creates 36 images, after that the process continues running but without creating images. The CPU Usage drops from 12% to 0% after creating the 36th images.
The second code snippet shows the called method but also describes the scheme of calling my python scripts. I am very much aware of the fact that this is not written well and i am going to do some code polish after fixing this problem :)
I hope there is someone out there with a tip.
Thank you very much in advance.
Kind regards,
Jan
import arcpy,os, logging
logging.basicConfig(filename='appPython.log', format='%(asctime)s - %(message)s', level=logging.INFO)
#Static Variables
mxdfileName = "D:\DigitalesFahrtenbuch_Datenpunkte\Templates\TemplateTelematik.mxd"
# Set the workspace for ListFeatureClasses
arcpy.env.workspace = str(sys.argv[1])
#arcpy.env.workspace = r"D:\DigitalesFahrtenbuch_Datenpunkte\DigFahrtenbuch_Datenpunkte.gdb"
featureclasses = arcpy.ListFeatureClasses()
try:
# Copy shapefiles to a file geodatabase
for fc in featureclasses:
featureName = os.path.splitext(fc)[0]
if "Dienstverrichtung_" in featureName and "_Projection" in featureName:
print(featureName)
#Global Variables
mxd = arcpy.mapping.MapDocument(mxdfileName)
df = arcpy.mapping.ListDataFrames(mxd,"*")[0]
#Create FeatureLayer
SelectionLayer = arcpy.management.MakeFeatureLayer(fc, "SelectionLayer").getOutput(0)
#Add Layer to mxd
arcpy.mapping.AddLayer(df, SelectionLayer, "TOP")
#Refresh TOC and DataFrames
arcpy.RefreshActiveView()
arcpy.RefreshTOC()
df = arcpy.mapping.ListDataFrames(mxd,"*")[0]
#Refresh TOC and DataFrames
arcpy.RefreshActiveView()
arcpy.RefreshTOC()
df = arcpy.mapping.ListDataFrames(mxd,"*")[0]
feature = arcpy.mapping.ListLayers(mxd, SelectionLayer, df)[0]
fields = ['OID#', 'SHAPE#', 'Name']
pngPath = r"D:\DigitalesFahrtenbuch_Datenpunkte\Images"
with arcpy.da.SearchCursor(feature, fields) as cursor:
for FID, Geometry, Name in cursor:
mxd.title = Name
print("{} in Bearbeitung.".format(mxd.title))
query = "ObjectID = {}".format(str(FID))
arcpy.management.SelectLayerByAttribute(feature, "NEW_SELECTION", query)
df.zoomToSelectedFeatures()
df.scale=2500
df.referenceScale = 3500
arcpy.RefreshActiveView()
png = "{}\\{}.png".format(pngPath, Name)
arcpy.mapping.ExportToPNG(mxd, png, df, df_export_width=2200, df_export_height=1300)
print("{} erfolgreich exportiert.".format(mxd.title))
print("Script beendet")
except Exception as e:
logging.error("Exception occurred", exc_info = True)
public static async Task<Tuple<string, bool>> ZoomToSelectedFeatures(string pPathToPythonExe, string pPathGeoDatabase)
{
Tuple<string, bool> resultTuple = null;
StringBuilder scriptMessageBuilder = new StringBuilder();
string scriptExceptions = string.Empty;
string scriptPrints = string.Empty;
string pythonPath = #"C:/Python27/ArcGIS10.8/python.exe";
try
{
await Task.Run(delegate
{
if (pPathToPythonExe != "")
{
pythonPath = pPathToPythonExe;
}
ProcessStartInfo start = new ProcessStartInfo();
//python interprater location
start.FileName = pythonPath;
//argument with file name and input parameters
start.Arguments =
$"{Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Python_Scripts\\Batch_ZoomToSelectedFeaturesAndExportPNG_2.py")}" +
$" {pPathGeoDatabase}";
start.UseShellExecute = false; // Do not use OS shell
start.CreateNoWindow = true; // We don't need new window
start.RedirectStandardOutput = true; // Any output, generated by application will be redirected back
start.RedirectStandardError = true; // Any error in standard output will be redirected back (for example exceptions)
start.LoadUserProfile = true;
using (Process process = Process.Start(start))
{
process.WaitForExit();
using (StreamReader reader = process.StandardOutput)
{
scriptExceptions = process.StandardError.ReadToEnd(); // Here are the exceptions from our Python script
scriptPrints = reader.ReadToEnd(); // Here is the result of StdOut(for example: print "test")
Debug.WriteLine("Batch_ZoomToSelectedFeaturesAndExportPNG_2.py meldet:");
Debug.WriteLine(scriptPrints);
Debug.WriteLine(scriptExceptions);
scriptMessageBuilder.AppendLine(scriptPrints);
scriptMessageBuilder.AppendLine(scriptExceptions);
}
}
resultTuple = new Tuple<string, bool>(scriptMessageBuilder.ToString(), true);
});
}
catch (Exception e)
{
Debug.WriteLine(e);
Debug.WriteLine(scriptExceptions);
resultTuple = new Tuple<string, bool>(scriptMessageBuilder.ToString(), false);
}
return resultTuple;
}

I solved it, by changing the output from "print messages" to "log entries". Now.... i honestly dont know why, the script creates all images correctly. Below the edited script.
Thanks anyway, Have a nice day!
import arcpy,os,logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG) # process everything, even if everything isn't printed
fh = logging.FileHandler('D:\\appPython.log')
fh.setLevel(logging.DEBUG) # or any level you want
logger.addHandler(fh)
#Define Variables
#Static Variables
mxdfileName = "D:\DigitalesFahrtenbuch_Datenpunkte\Templates\TemplateTelematik.mxd"
# Set the workspace for ListFeatureClasses
arcpy.env.workspace = str(sys.argv[1])
#arcpy.env.workspace = r"D:\DigitalesFahrtenbuch_Datenpunkte\DigFahrtenbuch_Datenpunkte.gdb"
# Use the ListFeatureClasses function to return a list of
# shapefiles.
featureclasses = arcpy.ListFeatureClasses()
try:
# Copy shapefiles to a file geodatabase
for fc in featureclasses:
featureName = os.path.splitext(fc)[0]
if "Dienstverrichtung_" in featureName and "_Projection" in featureName:
logger.info(featureName)
#Global Variables
mxd = arcpy.mapping.MapDocument(mxdfileName)
df = arcpy.mapping.ListDataFrames(mxd,"*")[0]
#Create FeatureLayer
SelectionLayer = arcpy.management.MakeFeatureLayer(fc, "SelectionLayer").getOutput(0)
#Add Layer to mxd
arcpy.mapping.AddLayer(df, SelectionLayer, "TOP")
#Refresh TOC and DataFrames
arcpy.RefreshActiveView()
arcpy.RefreshTOC()
df = arcpy.mapping.ListDataFrames(mxd,"*")[0]
#Refresh TOC and DataFrames
arcpy.RefreshActiveView()
arcpy.RefreshTOC()
df = arcpy.mapping.ListDataFrames(mxd,"*")[0]
feature = arcpy.mapping.ListLayers(mxd, SelectionLayer, df)[0]
fields = ['OID#', 'SHAPE#', 'Name']
pngPath = r"D:\DigitalesFahrtenbuch_Datenpunkte\Images"
with arcpy.da.SearchCursor(feature, fields) as cursor:
for FID, Geometry, Name in cursor:
mxd.title = Name
#print("{} in Bearbeitung.".format(mxd.title))
logger.info("{} in Bearbeitung.".format(mxd.title))
query = "ObjectID = {}".format(str(FID))
arcpy.management.SelectLayerByAttribute(feature, "NEW_SELECTION", query)
df.zoomToSelectedFeatures()
df.scale=2500
df.referenceScale = 3500
arcpy.RefreshActiveView()
png = "{}\\{}.png".format(pngPath, Name)
arcpy.mapping.ExportToPNG(mxd, png, df, df_export_width=2200, df_export_height=1300)
logger.info("{} erfolgreich exportiert.".format(mxd.title))
#print("{} erfolgreich exportiert.".format(mxd.title))
logger.info("Script beendet")
except Exception as e:
logger.error(e)

How to transcribe the recording for speech recognization

After downloading and uploading files related to the mozilla deeepspeech, I started using google colab. I am using mozilla/deepspeech for speech recognization. The code shown below is for recording my audio. After recording the audio, I want to use a function/method to transcribe the recording into text. Everything compiles, but the text does not come out correctly. Any thoughts in my code?
"""
To write this piece of code I took inspiration/code from a lot of places.
It was late night, so I'm not sure how much I created or just copied o.O
Here are some of the possible references:
https://blog.addpipe.com/recording-audio-in-the-browser-using-pure-html5-and-minimal-javascript/
https://stackoverflow.com/a/18650249
https://hacks.mozilla.org/2014/06/easy-audio-capture-with-the-mediarecorder-api/
https://air.ghost.io/recording-to-an-audio-file-using-html5-and-js/
https://stackoverflow.com/a/49019356
"""
from google.colab.output import eval_js
from base64 import b64decode
from scipy.io.wavfile import read as wav_read
import io
import ffmpeg
AUDIO_HTML = """
<script>
var my_div = document.createElement("DIV");
var my_p = document.createElement("P");
var my_btn = document.createElement("BUTTON");
var t = document.createTextNode("Press to start recording");
my_btn.appendChild(t);
//my_p.appendChild(my_btn);
my_div.appendChild(my_btn);
document.body.appendChild(my_div);
var base64data = 0;
var reader;
var recorder, gumStream;
var recordButton = my_btn;
var handleSuccess = function(stream) {
gumStream = stream;
var options = {
//bitsPerSecond: 8000, //chrome seems to ignore, always 48k
mimeType : 'audio/webm;codecs=opus'
//mimeType : 'audio/webm;codecs=pcm'
};
//recorder = new MediaRecorder(stream, options);
recorder = new MediaRecorder(stream);
recorder.ondataavailable = function(e) {
var url = URL.createObjectURL(e.data);
var preview = document.createElement('audio');
preview.controls = true;
preview.src = url;
document.body.appendChild(preview);
reader = new FileReader();
reader.readAsDataURL(e.data);
reader.onloadend = function() {
base64data = reader.result;
//console.log("Inside FileReader:" + base64data);
}
};
recorder.start();
};
recordButton.innerText = "Recording... press to stop";
navigator.mediaDevices.getUserMedia({audio: true}).then(handleSuccess);
function toggleRecording() {
if (recorder && recorder.state == "recording") {
recorder.stop();
gumStream.getAudioTracks()[0].stop();
recordButton.innerText = "Saving the recording... pls wait!"
}
}
// https://stackoverflow.com/a/951057
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
var data = new Promise(resolve=>{
//recordButton.addEventListener("click", toggleRecording);
recordButton.onclick = ()=>{
toggleRecording()
sleep(2000).then(() => {
// wait 2000ms for the data to be available...
// ideally this should use something like await...
//console.log("Inside data:" + base64data)
resolve(base64data.toString())
});
}
});
</script>
"""
def get_audio():
display(HTML(AUDIO_HTML))
data = eval_js("data")
binary = b64decode(data.split(',')[1])
process = (ffmpeg
.input('pipe:0')
.output('pipe:1', format='wav')
.run_async(pipe_stdin=True, pipe_stdout=True, pipe_stderr=True, quiet=True, overwrite_output=True)
)
output, err = process.communicate(input=binary)
riff_chunk_size = len(output) - 8
# Break up the chunk size into four bytes, held in b.
q = riff_chunk_size
b = []
for i in range(4):
q, r = divmod(q, 256)
b.append(r)
# Replace bytes 4:8 in proc.stdout with the actual size of the RIFF chunk.
riff = output[:4] + bytes(b) + output[8:]
sr, audio = wav_read(io.BytesIO(riff))
return audio, sr
audio, sr = get_audio()
def recordingTranscribe(audio):
data16 = np.frombuffer(audio)
return model.stt(data16)
recordingTranscribe(audio)

Try this
It is perfect!
note-required python 3.6 or below...
import speech_recognition as sr
def takeCommand():
r=sr.Recognizer()
with sr.Microphone() as source:
print("Listening...")
audio=r.listen(source)
try:
statement=r.recognize_google(audio,language='en-in')
print(f"user said:{statement}\n")
except Exception as e:
#speak("Sorry, please say that again")
print('Sorry, please say that again')
return "None"
return statement
if __name__=='__main__':
statement = takeCommand().lower()
print('detecting.....')
print(statement)

C# cannot call python and pass parameters

I am a newbie in programming, and I have an MVC project. I want to use C# call the Python py file and pass parameters in order to make a chart. I refer to this article How do I run a Python script from C#? . I have used this method to make lots of charts, and they can execute and pass parameters successfully. There's only two files cannot be successfully called and passed parameters, and I think it's Python has an error that can't call the py.
Here is a failure below. When I run proportion.py alone in Spyder, it can successfully use the Fixed parameters. BUT when i use C# to call it, there will be no response. The syntax in the file has been confirmed to be executed without problems, and methods i have been tried lots of methods, but still not resolved. Please save my projetct, I will be very thankful!!Thanks for any help.
Here is how i use C# to call Python below.
public ActionResult Index(string searchString, DateTime? startdate, DateTime? enddate)
{
run_sound("D:/Python/pie.py", "" + sd + "", "" + ed + "", "" + searchString + "");
run_Emoanalysis("picture/AAApy.py", "" + sd + "", "" + ed + "", "" + searchString + "");
run_proportion("D:/Microsoft Visual Studio/MVC project/MVC project/picture /proportion.py", "" + sd + "", "" + ed + "", "" + searchString + "");
}
//The following is the function of run_proportion,
//other functions(run_sound) are the same as this method, and carefully confirmed.
private string run_proportion(string cmd, string sdate, string edate, string condition)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = #"C:/Users/user/AppData/Local/Programs/Python/Python38-32/python.exe";
start.CreateNoWindow = true;
start.Arguments = string.Format("{0} {1} {2} {3}", cmd, sdate, edate, condition);
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
//Console.Write(result);
process.WaitForExit();
return result;
}
}
}
Here is proportion.py below that cannot be called and executed BY C#.
sd= sys.argv[1]
ed = sys.argv[2]
cdn = sys.argv[3]
sqlcom = "SELECT COUNT(DISTINCT url) FROM JIEBA WHERE (title LIKE '%" +str(cdn)+ "%') AND (post BETWEEN '" +str(sd)+ "' AND '" +str(ed)+ "')"
sqlcom2 = "SELECT COUNT(DISTINCT url) as KeyWordCount FROM JIEBA WHERE (title LIKE '%" +str(cdn)+ "%')"
df = pd.read_sql(sqlcom, con=cnxn)
df1 = np.array(df)
df0 = df1.tolist()
df2 = pd.read_sql(sqlcom2, con=cnxn)
df3 = np.array(df2)
df4 = df3.tolist()
df5 = str(df4[0][0])
print(df5)
df6 = str(df0[0][0])
print(df6)
c = int(df5)-int(df6)
# =============================================================================
count = float(df5)/float(df5)
print(count)
#
keyword = float(df6)/float(df5)
print(keyword)
#
keyword2 = str(round(float(df6)/float(df5)*100,2))+'%'
print(keyword2)
#
count2 = str(round((1-float(df6)/float(df5))*100,2))+'%'
print(count2)
# Change color
fig = plt.figure(figsize = (7,5))
ax = fig.add_subplot(111)
squarify.plot(sizes=[int(c),int(df6)], label=['期間"外"所佔筆數', '查詢後所佔比數'],value =(str(c)+'筆/'+str(df5)+'筆'+'\n'+'佔　'+str(count2),str(df6)+'筆/'+str(df5)+'筆'+'\n'+'佔　'+str(keyword2)), color=["red","blue"], alpha=.4)
plt.rcParams['font.sans-serif'] = 'Microsoft YaHei'
plt.rcParams['axes.unicode_minus'] = False
ax.set_title('關鍵字搜尋期間所佔比例',fontsize = 18)
plt.axis('off')
plt.tight_layout()
plt.savefig("D:\Microsoft Visual Studio\MVC project\MVC project\picture\keyproportion.png")

Also had some issues with running python, first would suggest replacing quotes in strings with a variable since it makes tracking them easier
var quote = '"';
also after doing the whole string do a
var commandUnescaped = Regex.Unescape(command);
pasting my way to call commands in case you want it, need to adapt it to windows , but same logic:
private (bool,string,string) RunCommand(string command, string args)
{
args = args.Replace("\"", "\\\"");
var process = new Process()
{
StartInfo = new ProcessStartInfo
{
FileName = "/bin/bash",
Arguments = $"-c \"{args}\"",
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true,
}
};
process.Start();
string output = process.StandardOutput.ReadToEnd();
string error = process.StandardError.ReadToEnd();
process.WaitForExit();
if (string.IsNullOrEmpty(error))
{
return (true,output,error);
}
else
{
return (false,output,error);
}
}

How can I save the headers and values in Html <script> as a table in the csv file?

I'm new to writing code. Using slenium and beautifulsoup, I managed to reach the script I want among dozens of scripts on the web page. I am looking for script [17]. When these codes are executed, the script [17] gives a result as follows.
the last part of my codes
html=driver.page_source
soup=BeautifulSoup(html, "html.parser")
scripts=soup.find_all("script")
x=scripts[17]
print(x)
result, output
note: The list of dates is ahead of the script [17]. slide the bar. Dummy Data
Dummy Data
<script language="JavaScript"> var theHlp='/yardim/matris.asp';var theTitle = 'Piyasa Değeri';var theCaption='Cam (TL)';var lastmod = '';var h='<a class=hisselink href=../Hisse/HisseAnaliz.aspx?HNO=';var e='<a class=hisselink href=../endeks/endeksAnaliz.aspx?HNO=';var d='<center><font face=symbol size=1 color=#FF0000><b>ß</b></font></center>';var u='<center><font face=symbol size=1 color=#008000><b>İ</b></font></center>';var n='<center><font face=symbol size=1 color=#00A000><b>=</b></font></center>';var fr='<font color=#FF0000>';var fg='<font color=#008000>';var theFooter=new Array();var theCols = new Array();theCols[0] = new Array('Hisse',4,50);theCols[1] = new Array('2012.12',1,60);theCols[2] = new Array('2013.03',1,60);theCols[3] = new Array('2013.06',1,60);theCols[4] = new Array('2013.09',1,60);theCols[5] = new Array('2013.12',1,60);theCols[6] = new Array('2014.03',1,60);theCols[7] = new Array('2014.06',1,60);theCols[8] = new Array('2014.09',1,60);theCols[9] = new Array('2014.12',1,60);theCols[10] = new Array('2015.03',1,60);theCols[11] = new Array('2015.06',1,60);theCols[12] = new Array('2015.09',1,60);theCols[13] = new Array('2015.12',1,60);theCols[14] = new Array('2016.03',1,60);theCols[15] = new Array('2016.06',1,60);theCols[16] = new Array('2016.09',1,60);theCols[17] = new Array('2016.12',1,60);theCols[18] = new Array('2017.03',1,60);theCols[19] = new Array('2017.06',1,60);theCols[20] = new Array('2017.09',1,60);theCols[21] = new Array('2017.12',1,60);theCols[22] = new Array('2018.03',1,60);theCols[23] = new Array('2018.06',1,60);theCols[24] = new Array('2018.09',1,60);theCols[25] = new Array('2018.12',1,60);theCols[26] = new Array('2019.03',1,60);theCols[27] = new Array('2019.06',1,60);theCols[28] = new Array('2019.09',1,60);theCols[29] = new Array('2019.12',1,60);theCols[30] = new Array('2020.03',1,60);var theRows = new Array();
theRows[0] = new Array ('<b>'+h+'30>ANA</B></a>','1,114,919,783.60','1,142,792,778.19','1,091,028,645.38','991,850,000.48','796,800,000.38','697,200,000.34','751,150,000.36','723,720,000.33','888,000,000.40','790,320,000.36','883,560,000.40','927,960,000.42','737,040,000.33','879,120,000.40','914,640,000.41','927,960,000.42','1,172,160,000.53','1,416,360,000.64','1,589,520,000.72','1,552,500,000.41','1,972,500,000.53','2,520,000,000.67','2,160,000,000.58','2,475,000,000.66','2,010,000,000.54','2,250,000,000.60','2,077,500,000.55','2,332,500,000.62','3,270,000,000.87','2,347,500,000.63');
theRows[1] = new Array ('<b>'+h+'89>DEN</B></a>','55,200,000.00','55,920,000.00','45,960,000.00','42,600,000.00','35,760,000.00','39,600,000.00','40,200,000.00','47,700,000.00','50,460,000.00','45,300,000.00','41,760,000.00','59,340,000.00','66,600,000.00','97,020,000.00','81,060,000.00','69,300,000.00','79,800,000.00','68,400,000.00','66,900,000.00','66,960,000.00','71,220,000.00','71,520,000.00','71,880,000.00','60,600,000.00','69,120,000.00','62,640,000.00','57,180,000.00','89,850,000.00','125,100,000.00','85,350,000.00');
theRows[2] = new Array ('<b>'+h+'269>SIS</B></a>','4,425,000,000.00','4,695,000,000.00','4,050,000,000.00','4,367,380,000.00','4,273,120,000.00','3,644,720,000.00','4,681,580,000.00','4,913,000,000.00','6,188,000,000.00','5,457,000,000.00','6,137,000,000.00','5,453,000,000.00','6,061,000,000.00','6,954,000,000.00','6,745,000,000.00','6,519,000,000.00','7,851,500,000.00','8,548,500,000.00','9,430,000,000.00','9,225,000,000.00','10,575,000,000.00','11,610,000,000.00','9,517,500,000.00','13,140,000,000.00','12,757,500,000.00','13,117,500,000.00','11,677,500,000.00','10,507,500,000.00','11,857,500,000.00','9,315,000,000.00');
theRows[3] = new Array ('<b>'+h+'297>TRK</B></a>','1,692,579,200.00','1,983,924,800.00','1,831,315,200.00','1,704,000,000.00','1,803,400,000.00','1,498,100,000.00','1,803,400,000.00','1,884,450,000.00','2,542,160,000.00','2,180,050,000.00','2,069,200,000.00','1,682,600,000.00','1,619,950,000.00','1,852,650,000.00','2,040,600,000.00','2,315,700,000.00','2,641,200,000.00','2,938,800,000.00','3,599,100,000.00','4,101,900,000.00','5,220,600,000.00','5,808,200,000.00','4,689,500,000.00','5,375,000,000.00','3,787,500,000.00','4,150,000,000.00','3,662,500,000.00','3,712,500,000.00','4,375,000,000.00','3,587,500,000.00');
var thetable=new mytable();thetable.tableWidth=650;thetable.shownum=false;thetable.controlaccess=true;thetable.visCols=new Array(true,true,true,true,true);thetable.initsort=new Array(0,-1);thetable.inittable();thetable.refreshTable();</script>
My purpose is to extract this output into a table and save it as a csv file. How can i extract this script as i want?
all dates should be on top, all names should be on the far right, all values should be between the two.
Hisse 2012.12 2013.3 2013.4 ...
ANA 1,114,919,783.60 1,142,792,778.19 1,091,028,645.38 ...
DEN 55,200,000.00 55,920,000.00 45,960,000.00 ....
.
.
.

Solution
The custom-function process_scripts() will produce what you are looking for. I am using the dummy data given below (at the end). First we check that the code does what it is expected and so we create a pandas dataframe to see the output.
You could also open this Colab Jupyter Notebook and run it on Cloud for free. This will allow you to not worry about any installation or setup and simply focus on examining the solution itself.
1. Processing A Single Script
## Define CSV file-output folder-path
OUTPUT_PATH = './output'
## Process scripts
dfs = process_scripts(scripts = [s],
output_path = OUTPUT_PATH,
save_to_csv = False,
verbose = 0)
print(dfs[0].reset_index(drop=True))
Output:
Name 2012.12 ... 2019.12 2020.03
0 ANA 1,114,919,783.60 ... 3,270,000,000.87 2,347,500,000.63
1 DEN 55,200,000.00 ... 125,100,000.00 85,350,000.00
2 SIS 4,425,000,000.00 ... 11,857,500,000.00 9,315,000,000.00
3 TRK 1,692,579,200.00 ... 4,375,000,000.00 3,587,500,000.00
[4 rows x 31 columns]
2. Processing All the Scripts
You can process all your scripts using the custom-define function process_scripts(). The code is given below.
## Define CSV file-output folder-path
OUTPUT_PATH = './output'
## Process scripts
dfs = process_scripts(scripts,
output_path = OUTPUT_PATH,
save_to_csv = True,
verbose = 0)
## To clear the output dir-contents
#!rm -f $OUTPUT_PATH/*
I did this on Google Colab and it worked as expected.
3. Making Paths in OS-agnostic Manner
Making paths for windows or unix based systems could be very different. The following shows you a method to achieve that without having to worry about which OS you will run the code. I have used the os library here. However, I would suggest you to look at the Pathlib library as well.
# Define relative path for output-folder
OUTPUT_PATH = './output'
# Dynamically define absolute path
pwd = os.getcwd() # present-working-directory
OUTPUT_PATH = os.path.join(pwd, os.path.abspath(OUTPUT_PATH))
4. Code: custom function process_scripts()
Here we use the regex (regular expression) library, along with pandas for organizing the data in a tabular format and then writing to csv file. The tqdm library is used to give you a nice progressbar while processing multiple scripts. Please see the comments in the code to know what to do if you are running it not from a jupyter notebook. The os library is used for path manipulation and creation of output-directory.
#pip install -U pandas
#pip install tqdm
import pandas as pd
import re # regex
import os
from tqdm.notebook import tqdm
# Use the following line if not using a jupyter notebook
# from tqdm import tqdm
def process_scripts(scripts,
output_path='./output',
save_to_csv: bool=False,
verbose: int=0):
"""Process all scripts and return a list of dataframes and
optionally write each dataframe to a CSV file.
Parameters
----------
scripts: list of scripts
output_path (str): output-folder-path for csv files
save_to_csv (bool): default is False
verbose (int): prints output for verbose>0
Example
-------
OUTPUT_PATH = './output'
dfs = process_scripts(scripts,
output_path = OUTPUT_PATH,
save_to_csv = True,
verbose = 0)
## To clear the output dir-contents
#!rm -f $OUTPUT_PATH/*
"""
## Define regex patterns and compile for speed
pat_header = re.compile(r"theCols\[\d+\] = new Array\s*\([\'](\d{4}\.\d{1,2})[\'],\d+,\d+\)")
pat_line = re.compile(r"theRows\[\d+\] = new Array\s*\((.*)\).*")
pat_code = re.compile("([A-Z]{3})")
# Calculate zfill-digits
zfill_digits = len(str(len(scripts)))
print(f'Total scripts: {len(scripts)}')
# Create output_path
if not os.path.exists(output_path):
os.makedirs(output_path)
# Define a list of dataframes:
# An accumulator of all scripts
dfs = []
## If you do not have tqdm installed, uncomment the
# next line and comment out the following line.
#for script_num, script in enumerate(scripts):
for script_num, script in enumerate(tqdm(scripts, desc='Scripts Processed')):
## Extract: Headers, Rows
# Rows : code (Name: single column), line-data (multi-column)
headers = script.strip().split('\n', 0)[0]
headers = ['Name'] + re.findall(pat_header, headers)
lines = re.findall(pat_line, script)
codes = [re.findall(pat_code, line)[0] for line in lines]
# Clean data for each row
lines_data = dict()
for line, code in zip(lines, codes):
line_data = line.replace("','", "|").split('|')
line_data[-1] = line_data[-1].replace("'", "")
line_data[0] = code
lines_data.update({code: line_data.copy()})
if verbose>0:
print('{}: {}'.format(script_num, codes))
## Load data into a pandas-dataframe
# and write to csv.
df = pd.DataFrame(lines_data).T
df.columns = headers
dfs.append(df.copy()) # update list
# Write to CSV
if save_to_csv:
num_label = str(script_num).zfill(zfill_digits)
script_file_csv = f'Script_{num_label}.csv'
script_path = os.path.join(output_path, script_file_csv)
df.to_csv(script_path, index=False)
return dfs
5. Dummy Data
## Dummy Data
s = """
<script language="JavaScript"> var theHlp='/yardim/matris.asp';var theTitle = 'Piyasa Değeri';var theCaption='Cam (TL)';var lastmod = '';var h='<a class=hisselink href=../Hisse/HisseAnaliz.aspx?HNO=';var e='<a class=hisselink href=../endeks/endeksAnaliz.aspx?HNO=';var d='<center><font face=symbol size=1 color=#FF0000><b>ß</b></font></center>';var u='<center><font face=symbol size=1 color=#008000><b>İ</b></font></center>';var n='<center><font face=symbol size=1 color=#00A000><b>=</b></font></center>';var fr='<font color=#FF0000>';var fg='<font color=#008000>';var theFooter=new Array();var theCols = new Array();theCols[0] = new Array('Hisse',4,50);theCols[1] = new Array('2012.12',1,60);theCols[2] = new Array('2013.03',1,60);theCols[3] = new Array('2013.06',1,60);theCols[4] = new Array('2013.09',1,60);theCols[5] = new Array('2013.12',1,60);theCols[6] = new Array('2014.03',1,60);theCols[7] = new Array('2014.06',1,60);theCols[8] = new Array('2014.09',1,60);theCols[9] = new Array('2014.12',1,60);theCols[10] = new Array('2015.03',1,60);theCols[11] = new Array('2015.06',1,60);theCols[12] = new Array('2015.09',1,60);theCols[13] = new Array('2015.12',1,60);theCols[14] = new Array('2016.03',1,60);theCols[15] = new Array('2016.06',1,60);theCols[16] = new Array('2016.09',1,60);theCols[17] = new Array('2016.12',1,60);theCols[18] = new Array('2017.03',1,60);theCols[19] = new Array('2017.06',1,60);theCols[20] = new Array('2017.09',1,60);theCols[21] = new Array('2017.12',1,60);theCols[22] = new Array('2018.03',1,60);theCols[23] = new Array('2018.06',1,60);theCols[24] = new Array('2018.09',1,60);theCols[25] = new Array('2018.12',1,60);theCols[26] = new Array('2019.03',1,60);theCols[27] = new Array('2019.06',1,60);theCols[28] = new Array('2019.09',1,60);theCols[29] = new Array('2019.12',1,60);theCols[30] = new Array('2020.03',1,60);var theRows = new Array();
theRows[0] = new Array ('<b>'+h+'30>ANA</B></a>','1,114,919,783.60','1,142,792,778.19','1,091,028,645.38','991,850,000.48','796,800,000.38','697,200,000.34','751,150,000.36','723,720,000.33','888,000,000.40','790,320,000.36','883,560,000.40','927,960,000.42','737,040,000.33','879,120,000.40','914,640,000.41','927,960,000.42','1,172,160,000.53','1,416,360,000.64','1,589,520,000.72','1,552,500,000.41','1,972,500,000.53','2,520,000,000.67','2,160,000,000.58','2,475,000,000.66','2,010,000,000.54','2,250,000,000.60','2,077,500,000.55','2,332,500,000.62','3,270,000,000.87','2,347,500,000.63');
theRows[1] = new Array ('<b>'+h+'89>DEN</B></a>','55,200,000.00','55,920,000.00','45,960,000.00','42,600,000.00','35,760,000.00','39,600,000.00','40,200,000.00','47,700,000.00','50,460,000.00','45,300,000.00','41,760,000.00','59,340,000.00','66,600,000.00','97,020,000.00','81,060,000.00','69,300,000.00','79,800,000.00','68,400,000.00','66,900,000.00','66,960,000.00','71,220,000.00','71,520,000.00','71,880,000.00','60,600,000.00','69,120,000.00','62,640,000.00','57,180,000.00','89,850,000.00','125,100,000.00','85,350,000.00');
theRows[2] = new Array ('<b>'+h+'269>SIS</B></a>','4,425,000,000.00','4,695,000,000.00','4,050,000,000.00','4,367,380,000.00','4,273,120,000.00','3,644,720,000.00','4,681,580,000.00','4,913,000,000.00','6,188,000,000.00','5,457,000,000.00','6,137,000,000.00','5,453,000,000.00','6,061,000,000.00','6,954,000,000.00','6,745,000,000.00','6,519,000,000.00','7,851,500,000.00','8,548,500,000.00','9,430,000,000.00','9,225,000,000.00','10,575,000,000.00','11,610,000,000.00','9,517,500,000.00','13,140,000,000.00','12,757,500,000.00','13,117,500,000.00','11,677,500,000.00','10,507,500,000.00','11,857,500,000.00','9,315,000,000.00');
theRows[3] = new Array ('<b>'+h+'297>TRK</B></a>','1,692,579,200.00','1,983,924,800.00','1,831,315,200.00','1,704,000,000.00','1,803,400,000.00','1,498,100,000.00','1,803,400,000.00','1,884,450,000.00','2,542,160,000.00','2,180,050,000.00','2,069,200,000.00','1,682,600,000.00','1,619,950,000.00','1,852,650,000.00','2,040,600,000.00','2,315,700,000.00','2,641,200,000.00','2,938,800,000.00','3,599,100,000.00','4,101,900,000.00','5,220,600,000.00','5,808,200,000.00','4,689,500,000.00','5,375,000,000.00','3,787,500,000.00','4,150,000,000.00','3,662,500,000.00','3,712,500,000.00','4,375,000,000.00','3,587,500,000.00');
var thetable=new mytable();thetable.tableWidth=650;thetable.shownum=false;thetable.controlaccess=true;thetable.visCols=new Array(true,true,true,true,true);thetable.initsort=new Array(0,-1);thetable.inittable();thetable.refreshTable();</script>
"""
## Make a dummy list of scripts
scripts = [s for _ in range(10)]

According to the provided <script> in your question, you can do something like code below to have a list of Dates for each name ANA, DEN ..:
for _ in range(1, len(aaa.split("<b>'"))-1):
s = aaa.split("<b>'")[_].split("'")
print(_)
lst = []
for i in s:
if "</B>" in i:
name = i.split('>')[1].split("<")[0]
print("{} = ".format(name), end="")
if any(j.isdigit() for j in i) and ',' in i:
lst.append(i)
print(lst)
It's just an example code, so it's not that beautiful :)
Hope this will help you.

parsing of javascript objects using python

friends!
I'm starting to learn python. I have a problem with obtaining the required value from javascript text. Here is the code, which I managed to download from website:
[<script src="//maps.google.com/maps?file=api&v=2&sensor=false&key=ABQIAAAAOjFUxXImJbfYejRUbw0-uBSoJppdodHXaiZe2O5Byw3T7kzYihSys_Exmi235-oDCy6xEhVelBMhBQ" type="text/javascript"></script>, <script type="text/javascript">
var map_shop = null;
var marker_shop = null;
function google_maps_shop_initialize()
{
if (GBrowserIsCompatible())
{
map_shop = new GMap2(document.getElementById("map_canvas_shop"));
point_center = new GLatLng(51.6663267, 39.1898874);
marker_shop = new GMarker(point_center);
map_shop.addOverlay(marker_shop);
map_shop.setCenter(point_center, 13);
//Create new Tile Layer
var gTileUrlTemplate = '//mt1.google.com/vt/lyrs=m#121,transit|vm:1&hl=ru&opts=r&x={X}&y={Y}&z={Z}';
var tileLayerOverlay = new GTileLayerOverlay(
new GTileLayer(null, null, null, {
tileUrlTemplate: gTileUrlTemplate,
isPng:true,
opacity:1
})
);
map_shop.addOverlay(tileLayerOverlay);
}
}
google_maps_shop_initialize();
</script>]
I want to print only one line from text, which contains coordinates point_center = new GLatLng(51.6663267, 39.1898874);
I'm trying decide it using re module, but the problem is that number of line may vary and I get empty output with this code:
if re.match("point_center = new GLatLng", line):
print (line)
Desirable output looks like this:
51.6663267, 39.1898874

If the Javascript is .txt format then you can simply do this:
from ast import literal_eval as make_tuple
with open("filename.txt") as f:
for line in f:
if "point_center = new GLatLng" in line:
linestring = line
linestring = linestring[26:]
linestring = make_tuple(linestring)
Your output should be a tuple.