We Keep Coding

Why is my Android app crashing every time I run a python script with chaquopy?

I am building an Android app that will allow the user to get a picture either by taking it in real time or uploading it from their saved images. Then, it will go through a machine learning script in python to determine their location. Before I completely connect to the algorithm, I am trying a test program that just returns a double.
from os.path import dirname, join
import csv
import random
filename = join(dirname(__file__), "new.csv")
def testlat():
return 30.0
def testlong():
return 30.0
These returned values are used in a Kotlin file that will then send those values to the Google Maps activity on the app for the location to be plotted.
class MainActivity : AppCompatActivity() {
var lat = 0.0
var long = 0.0
var dynamic = false
private val cameraRequest = 1888
lateinit var imageView: ImageView
lateinit var button: Button
private val pickImage = 100
private var imageUri: Uri? = null
var active = false
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
// Accesses the info image button
val clickMe = findViewById<ImageButton>(R.id.imageButton)
// Runs this function when the info icon is pressed by the user
// It will display the text in the variable infoText
clickMe.setOnClickListener {
Toast.makeText(this, infoText, Toast.LENGTH_LONG).show()
}
if (ContextCompat.checkSelfPermission(applicationContext, Manifest.permission.CAMERA)
== PackageManager.PERMISSION_DENIED) {
ActivityCompat.requestPermissions(
this,
arrayOf(Manifest.permission.CAMERA),
cameraRequest
)
}
imageView = findViewById(R.id.imageView)
val photoButton: Button = findViewById(R.id.button2)
photoButton.setOnClickListener {
val cameraIntent = Intent(MediaStore.ACTION_IMAGE_CAPTURE)
startActivityForResult(cameraIntent, cameraRequest)
dynamic = true
}
/*
The below will move to external photo storage once button2 is clicked
*/
button = findViewById(R.id.button)
button.setOnClickListener {
val gallery = Intent(Intent.ACTION_PICK, MediaStore.Images.Media.INTERNAL_CONTENT_URI)
startActivityForResult(gallery, pickImage)
}
// PYTHON HERE
if (! Python.isStarted()) {
Python.start(AndroidPlatform(this))
}
}
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)
if (resultCode == RESULT_OK && requestCode == pickImage) {
imageUri = data?.data
imageView.setImageURI(imageUri)
// PYTHON HERE
val py = Python.getInstance()
val pyobj = py.getModule("main")
this.lat = pyobj.callAttr("testlat").toDouble()
this.long = pyobj.callAttr("testlong").toDouble()
/* Open the map after image has been received from user
This will be changed later to instead call the external object recognition/pathfinding
scripts and then pull up the map after those finish running
*/
val mapsIntent = Intent(this, MapsActivity::class.java)
startActivity(mapsIntent)
}
}
}
I set up chaquopy and the gradle is building successfully, but everytime I get to the python part of emulating the app, it crashes. I'm not quite sure why that is; I thought maybe the program was too much for the phone to handle but it is a very basic python script so I doubt that's the issue.

If your app crashes, you can find the stack trace in the Logcat.
In this case, it's probably caused by the line return = 30.0. The correct syntax is return 30.0.

C# cannot call python and pass parameters

I am a newbie in programming, and I have an MVC project. I want to use C# call the Python py file and pass parameters in order to make a chart. I refer to this article How do I run a Python script from C#? . I have used this method to make lots of charts, and they can execute and pass parameters successfully. There's only two files cannot be successfully called and passed parameters, and I think it's Python has an error that can't call the py.
Here is a failure below. When I run proportion.py alone in Spyder, it can successfully use the Fixed parameters. BUT when i use C# to call it, there will be no response. The syntax in the file has been confirmed to be executed without problems, and methods i have been tried lots of methods, but still not resolved. Please save my projetct, I will be very thankful!!Thanks for any help.
Here is how i use C# to call Python below.
public ActionResult Index(string searchString, DateTime? startdate, DateTime? enddate)
{
run_sound("D:/Python/pie.py", "" + sd + "", "" + ed + "", "" + searchString + "");
run_Emoanalysis("picture/AAApy.py", "" + sd + "", "" + ed + "", "" + searchString + "");
run_proportion("D:/Microsoft Visual Studio/MVC project/MVC project/picture /proportion.py", "" + sd + "", "" + ed + "", "" + searchString + "");
}
//The following is the function of run_proportion,
//other functions(run_sound) are the same as this method, and carefully confirmed.
private string run_proportion(string cmd, string sdate, string edate, string condition)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = #"C:/Users/user/AppData/Local/Programs/Python/Python38-32/python.exe";
start.CreateNoWindow = true;
start.Arguments = string.Format("{0} {1} {2} {3}", cmd, sdate, edate, condition);
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
using (StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
//Console.Write(result);
process.WaitForExit();
return result;
}
}
}
Here is proportion.py below that cannot be called and executed BY C#.
sd= sys.argv[1]
ed = sys.argv[2]
cdn = sys.argv[3]
sqlcom = "SELECT COUNT(DISTINCT url) FROM JIEBA WHERE (title LIKE '%" +str(cdn)+ "%') AND (post BETWEEN '" +str(sd)+ "' AND '" +str(ed)+ "')"
sqlcom2 = "SELECT COUNT(DISTINCT url) as KeyWordCount FROM JIEBA WHERE (title LIKE '%" +str(cdn)+ "%')"
df = pd.read_sql(sqlcom, con=cnxn)
df1 = np.array(df)
df0 = df1.tolist()
df2 = pd.read_sql(sqlcom2, con=cnxn)
df3 = np.array(df2)
df4 = df3.tolist()
df5 = str(df4[0][0])
print(df5)
df6 = str(df0[0][0])
print(df6)
c = int(df5)-int(df6)
# =============================================================================
count = float(df5)/float(df5)
print(count)
#
keyword = float(df6)/float(df5)
print(keyword)
#
keyword2 = str(round(float(df6)/float(df5)*100,2))+'%'
print(keyword2)
#
count2 = str(round((1-float(df6)/float(df5))*100,2))+'%'
print(count2)
# Change color
fig = plt.figure(figsize = (7,5))
ax = fig.add_subplot(111)
squarify.plot(sizes=[int(c),int(df6)], label=['期間"外"所佔筆數', '查詢後所佔比數'],value =(str(c)+'筆/'+str(df5)+'筆'+'\n'+'佔　'+str(count2),str(df6)+'筆/'+str(df5)+'筆'+'\n'+'佔　'+str(keyword2)), color=["red","blue"], alpha=.4)
plt.rcParams['font.sans-serif'] = 'Microsoft YaHei'
plt.rcParams['axes.unicode_minus'] = False
ax.set_title('關鍵字搜尋期間所佔比例',fontsize = 18)
plt.axis('off')
plt.tight_layout()
plt.savefig("D:\Microsoft Visual Studio\MVC project\MVC project\picture\keyproportion.png")

Also had some issues with running python, first would suggest replacing quotes in strings with a variable since it makes tracking them easier
var quote = '"';
also after doing the whole string do a
var commandUnescaped = Regex.Unescape(command);
pasting my way to call commands in case you want it, need to adapt it to windows , but same logic:
private (bool,string,string) RunCommand(string command, string args)
{
args = args.Replace("\"", "\\\"");
var process = new Process()
{
StartInfo = new ProcessStartInfo
{
FileName = "/bin/bash",
Arguments = $"-c \"{args}\"",
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true,
}
};
process.Start();
string output = process.StandardOutput.ReadToEnd();
string error = process.StandardError.ReadToEnd();
process.WaitForExit();
if (string.IsNullOrEmpty(error))
{
return (true,output,error);
}
else
{
return (false,output,error);
}
}

How can I save the headers and values in Html <script> as a table in the csv file?

I'm new to writing code. Using slenium and beautifulsoup, I managed to reach the script I want among dozens of scripts on the web page. I am looking for script [17]. When these codes are executed, the script [17] gives a result as follows.
the last part of my codes
html=driver.page_source
soup=BeautifulSoup(html, "html.parser")
scripts=soup.find_all("script")
x=scripts[17]
print(x)
result, output
note: The list of dates is ahead of the script [17]. slide the bar. Dummy Data
Dummy Data
<script language="JavaScript"> var theHlp='/yardim/matris.asp';var theTitle = 'Piyasa Değeri';var theCaption='Cam (TL)';var lastmod = '';var h='<a class=hisselink href=../Hisse/HisseAnaliz.aspx?HNO=';var e='<a class=hisselink href=../endeks/endeksAnaliz.aspx?HNO=';var d='<center><font face=symbol size=1 color=#FF0000><b>ß</b></font></center>';var u='<center><font face=symbol size=1 color=#008000><b>İ</b></font></center>';var n='<center><font face=symbol size=1 color=#00A000><b>=</b></font></center>';var fr='<font color=#FF0000>';var fg='<font color=#008000>';var theFooter=new Array();var theCols = new Array();theCols[0] = new Array('Hisse',4,50);theCols[1] = new Array('2012.12',1,60);theCols[2] = new Array('2013.03',1,60);theCols[3] = new Array('2013.06',1,60);theCols[4] = new Array('2013.09',1,60);theCols[5] = new Array('2013.12',1,60);theCols[6] = new Array('2014.03',1,60);theCols[7] = new Array('2014.06',1,60);theCols[8] = new Array('2014.09',1,60);theCols[9] = new Array('2014.12',1,60);theCols[10] = new Array('2015.03',1,60);theCols[11] = new Array('2015.06',1,60);theCols[12] = new Array('2015.09',1,60);theCols[13] = new Array('2015.12',1,60);theCols[14] = new Array('2016.03',1,60);theCols[15] = new Array('2016.06',1,60);theCols[16] = new Array('2016.09',1,60);theCols[17] = new Array('2016.12',1,60);theCols[18] = new Array('2017.03',1,60);theCols[19] = new Array('2017.06',1,60);theCols[20] = new Array('2017.09',1,60);theCols[21] = new Array('2017.12',1,60);theCols[22] = new Array('2018.03',1,60);theCols[23] = new Array('2018.06',1,60);theCols[24] = new Array('2018.09',1,60);theCols[25] = new Array('2018.12',1,60);theCols[26] = new Array('2019.03',1,60);theCols[27] = new Array('2019.06',1,60);theCols[28] = new Array('2019.09',1,60);theCols[29] = new Array('2019.12',1,60);theCols[30] = new Array('2020.03',1,60);var theRows = new Array();
theRows[0] = new Array ('<b>'+h+'30>ANA</B></a>','1,114,919,783.60','1,142,792,778.19','1,091,028,645.38','991,850,000.48','796,800,000.38','697,200,000.34','751,150,000.36','723,720,000.33','888,000,000.40','790,320,000.36','883,560,000.40','927,960,000.42','737,040,000.33','879,120,000.40','914,640,000.41','927,960,000.42','1,172,160,000.53','1,416,360,000.64','1,589,520,000.72','1,552,500,000.41','1,972,500,000.53','2,520,000,000.67','2,160,000,000.58','2,475,000,000.66','2,010,000,000.54','2,250,000,000.60','2,077,500,000.55','2,332,500,000.62','3,270,000,000.87','2,347,500,000.63');
theRows[1] = new Array ('<b>'+h+'89>DEN</B></a>','55,200,000.00','55,920,000.00','45,960,000.00','42,600,000.00','35,760,000.00','39,600,000.00','40,200,000.00','47,700,000.00','50,460,000.00','45,300,000.00','41,760,000.00','59,340,000.00','66,600,000.00','97,020,000.00','81,060,000.00','69,300,000.00','79,800,000.00','68,400,000.00','66,900,000.00','66,960,000.00','71,220,000.00','71,520,000.00','71,880,000.00','60,600,000.00','69,120,000.00','62,640,000.00','57,180,000.00','89,850,000.00','125,100,000.00','85,350,000.00');
theRows[2] = new Array ('<b>'+h+'269>SIS</B></a>','4,425,000,000.00','4,695,000,000.00','4,050,000,000.00','4,367,380,000.00','4,273,120,000.00','3,644,720,000.00','4,681,580,000.00','4,913,000,000.00','6,188,000,000.00','5,457,000,000.00','6,137,000,000.00','5,453,000,000.00','6,061,000,000.00','6,954,000,000.00','6,745,000,000.00','6,519,000,000.00','7,851,500,000.00','8,548,500,000.00','9,430,000,000.00','9,225,000,000.00','10,575,000,000.00','11,610,000,000.00','9,517,500,000.00','13,140,000,000.00','12,757,500,000.00','13,117,500,000.00','11,677,500,000.00','10,507,500,000.00','11,857,500,000.00','9,315,000,000.00');
theRows[3] = new Array ('<b>'+h+'297>TRK</B></a>','1,692,579,200.00','1,983,924,800.00','1,831,315,200.00','1,704,000,000.00','1,803,400,000.00','1,498,100,000.00','1,803,400,000.00','1,884,450,000.00','2,542,160,000.00','2,180,050,000.00','2,069,200,000.00','1,682,600,000.00','1,619,950,000.00','1,852,650,000.00','2,040,600,000.00','2,315,700,000.00','2,641,200,000.00','2,938,800,000.00','3,599,100,000.00','4,101,900,000.00','5,220,600,000.00','5,808,200,000.00','4,689,500,000.00','5,375,000,000.00','3,787,500,000.00','4,150,000,000.00','3,662,500,000.00','3,712,500,000.00','4,375,000,000.00','3,587,500,000.00');
var thetable=new mytable();thetable.tableWidth=650;thetable.shownum=false;thetable.controlaccess=true;thetable.visCols=new Array(true,true,true,true,true);thetable.initsort=new Array(0,-1);thetable.inittable();thetable.refreshTable();</script>
My purpose is to extract this output into a table and save it as a csv file. How can i extract this script as i want?
all dates should be on top, all names should be on the far right, all values should be between the two.
Hisse 2012.12 2013.3 2013.4 ...
ANA 1,114,919,783.60 1,142,792,778.19 1,091,028,645.38 ...
DEN 55,200,000.00 55,920,000.00 45,960,000.00 ....
.
.
.

Solution
The custom-function process_scripts() will produce what you are looking for. I am using the dummy data given below (at the end). First we check that the code does what it is expected and so we create a pandas dataframe to see the output.
You could also open this Colab Jupyter Notebook and run it on Cloud for free. This will allow you to not worry about any installation or setup and simply focus on examining the solution itself.
1. Processing A Single Script
## Define CSV file-output folder-path
OUTPUT_PATH = './output'
## Process scripts
dfs = process_scripts(scripts = [s],
output_path = OUTPUT_PATH,
save_to_csv = False,
verbose = 0)
print(dfs[0].reset_index(drop=True))
Output:
Name 2012.12 ... 2019.12 2020.03
0 ANA 1,114,919,783.60 ... 3,270,000,000.87 2,347,500,000.63
1 DEN 55,200,000.00 ... 125,100,000.00 85,350,000.00
2 SIS 4,425,000,000.00 ... 11,857,500,000.00 9,315,000,000.00
3 TRK 1,692,579,200.00 ... 4,375,000,000.00 3,587,500,000.00
[4 rows x 31 columns]
2. Processing All the Scripts
You can process all your scripts using the custom-define function process_scripts(). The code is given below.
## Define CSV file-output folder-path
OUTPUT_PATH = './output'
## Process scripts
dfs = process_scripts(scripts,
output_path = OUTPUT_PATH,
save_to_csv = True,
verbose = 0)
## To clear the output dir-contents
#!rm -f $OUTPUT_PATH/*
I did this on Google Colab and it worked as expected.
3. Making Paths in OS-agnostic Manner
Making paths for windows or unix based systems could be very different. The following shows you a method to achieve that without having to worry about which OS you will run the code. I have used the os library here. However, I would suggest you to look at the Pathlib library as well.
# Define relative path for output-folder
OUTPUT_PATH = './output'
# Dynamically define absolute path
pwd = os.getcwd() # present-working-directory
OUTPUT_PATH = os.path.join(pwd, os.path.abspath(OUTPUT_PATH))
4. Code: custom function process_scripts()
Here we use the regex (regular expression) library, along with pandas for organizing the data in a tabular format and then writing to csv file. The tqdm library is used to give you a nice progressbar while processing multiple scripts. Please see the comments in the code to know what to do if you are running it not from a jupyter notebook. The os library is used for path manipulation and creation of output-directory.
#pip install -U pandas
#pip install tqdm
import pandas as pd
import re # regex
import os
from tqdm.notebook import tqdm
# Use the following line if not using a jupyter notebook
# from tqdm import tqdm
def process_scripts(scripts,
output_path='./output',
save_to_csv: bool=False,
verbose: int=0):
"""Process all scripts and return a list of dataframes and
optionally write each dataframe to a CSV file.
Parameters
----------
scripts: list of scripts
output_path (str): output-folder-path for csv files
save_to_csv (bool): default is False
verbose (int): prints output for verbose>0
Example
-------
OUTPUT_PATH = './output'
dfs = process_scripts(scripts,
output_path = OUTPUT_PATH,
save_to_csv = True,
verbose = 0)
## To clear the output dir-contents
#!rm -f $OUTPUT_PATH/*
"""
## Define regex patterns and compile for speed
pat_header = re.compile(r"theCols\[\d+\] = new Array\s*\([\'](\d{4}\.\d{1,2})[\'],\d+,\d+\)")
pat_line = re.compile(r"theRows\[\d+\] = new Array\s*\((.*)\).*")
pat_code = re.compile("([A-Z]{3})")
# Calculate zfill-digits
zfill_digits = len(str(len(scripts)))
print(f'Total scripts: {len(scripts)}')
# Create output_path
if not os.path.exists(output_path):
os.makedirs(output_path)
# Define a list of dataframes:
# An accumulator of all scripts
dfs = []
## If you do not have tqdm installed, uncomment the
# next line and comment out the following line.
#for script_num, script in enumerate(scripts):
for script_num, script in enumerate(tqdm(scripts, desc='Scripts Processed')):
## Extract: Headers, Rows
# Rows : code (Name: single column), line-data (multi-column)
headers = script.strip().split('\n', 0)[0]
headers = ['Name'] + re.findall(pat_header, headers)
lines = re.findall(pat_line, script)
codes = [re.findall(pat_code, line)[0] for line in lines]
# Clean data for each row
lines_data = dict()
for line, code in zip(lines, codes):
line_data = line.replace("','", "|").split('|')
line_data[-1] = line_data[-1].replace("'", "")
line_data[0] = code
lines_data.update({code: line_data.copy()})
if verbose>0:
print('{}: {}'.format(script_num, codes))
## Load data into a pandas-dataframe
# and write to csv.
df = pd.DataFrame(lines_data).T
df.columns = headers
dfs.append(df.copy()) # update list
# Write to CSV
if save_to_csv:
num_label = str(script_num).zfill(zfill_digits)
script_file_csv = f'Script_{num_label}.csv'
script_path = os.path.join(output_path, script_file_csv)
df.to_csv(script_path, index=False)
return dfs
5. Dummy Data
## Dummy Data
s = """
<script language="JavaScript"> var theHlp='/yardim/matris.asp';var theTitle = 'Piyasa Değeri';var theCaption='Cam (TL)';var lastmod = '';var h='<a class=hisselink href=../Hisse/HisseAnaliz.aspx?HNO=';var e='<a class=hisselink href=../endeks/endeksAnaliz.aspx?HNO=';var d='<center><font face=symbol size=1 color=#FF0000><b>ß</b></font></center>';var u='<center><font face=symbol size=1 color=#008000><b>İ</b></font></center>';var n='<center><font face=symbol size=1 color=#00A000><b>=</b></font></center>';var fr='<font color=#FF0000>';var fg='<font color=#008000>';var theFooter=new Array();var theCols = new Array();theCols[0] = new Array('Hisse',4,50);theCols[1] = new Array('2012.12',1,60);theCols[2] = new Array('2013.03',1,60);theCols[3] = new Array('2013.06',1,60);theCols[4] = new Array('2013.09',1,60);theCols[5] = new Array('2013.12',1,60);theCols[6] = new Array('2014.03',1,60);theCols[7] = new Array('2014.06',1,60);theCols[8] = new Array('2014.09',1,60);theCols[9] = new Array('2014.12',1,60);theCols[10] = new Array('2015.03',1,60);theCols[11] = new Array('2015.06',1,60);theCols[12] = new Array('2015.09',1,60);theCols[13] = new Array('2015.12',1,60);theCols[14] = new Array('2016.03',1,60);theCols[15] = new Array('2016.06',1,60);theCols[16] = new Array('2016.09',1,60);theCols[17] = new Array('2016.12',1,60);theCols[18] = new Array('2017.03',1,60);theCols[19] = new Array('2017.06',1,60);theCols[20] = new Array('2017.09',1,60);theCols[21] = new Array('2017.12',1,60);theCols[22] = new Array('2018.03',1,60);theCols[23] = new Array('2018.06',1,60);theCols[24] = new Array('2018.09',1,60);theCols[25] = new Array('2018.12',1,60);theCols[26] = new Array('2019.03',1,60);theCols[27] = new Array('2019.06',1,60);theCols[28] = new Array('2019.09',1,60);theCols[29] = new Array('2019.12',1,60);theCols[30] = new Array('2020.03',1,60);var theRows = new Array();
theRows[0] = new Array ('<b>'+h+'30>ANA</B></a>','1,114,919,783.60','1,142,792,778.19','1,091,028,645.38','991,850,000.48','796,800,000.38','697,200,000.34','751,150,000.36','723,720,000.33','888,000,000.40','790,320,000.36','883,560,000.40','927,960,000.42','737,040,000.33','879,120,000.40','914,640,000.41','927,960,000.42','1,172,160,000.53','1,416,360,000.64','1,589,520,000.72','1,552,500,000.41','1,972,500,000.53','2,520,000,000.67','2,160,000,000.58','2,475,000,000.66','2,010,000,000.54','2,250,000,000.60','2,077,500,000.55','2,332,500,000.62','3,270,000,000.87','2,347,500,000.63');
theRows[1] = new Array ('<b>'+h+'89>DEN</B></a>','55,200,000.00','55,920,000.00','45,960,000.00','42,600,000.00','35,760,000.00','39,600,000.00','40,200,000.00','47,700,000.00','50,460,000.00','45,300,000.00','41,760,000.00','59,340,000.00','66,600,000.00','97,020,000.00','81,060,000.00','69,300,000.00','79,800,000.00','68,400,000.00','66,900,000.00','66,960,000.00','71,220,000.00','71,520,000.00','71,880,000.00','60,600,000.00','69,120,000.00','62,640,000.00','57,180,000.00','89,850,000.00','125,100,000.00','85,350,000.00');
theRows[2] = new Array ('<b>'+h+'269>SIS</B></a>','4,425,000,000.00','4,695,000,000.00','4,050,000,000.00','4,367,380,000.00','4,273,120,000.00','3,644,720,000.00','4,681,580,000.00','4,913,000,000.00','6,188,000,000.00','5,457,000,000.00','6,137,000,000.00','5,453,000,000.00','6,061,000,000.00','6,954,000,000.00','6,745,000,000.00','6,519,000,000.00','7,851,500,000.00','8,548,500,000.00','9,430,000,000.00','9,225,000,000.00','10,575,000,000.00','11,610,000,000.00','9,517,500,000.00','13,140,000,000.00','12,757,500,000.00','13,117,500,000.00','11,677,500,000.00','10,507,500,000.00','11,857,500,000.00','9,315,000,000.00');
theRows[3] = new Array ('<b>'+h+'297>TRK</B></a>','1,692,579,200.00','1,983,924,800.00','1,831,315,200.00','1,704,000,000.00','1,803,400,000.00','1,498,100,000.00','1,803,400,000.00','1,884,450,000.00','2,542,160,000.00','2,180,050,000.00','2,069,200,000.00','1,682,600,000.00','1,619,950,000.00','1,852,650,000.00','2,040,600,000.00','2,315,700,000.00','2,641,200,000.00','2,938,800,000.00','3,599,100,000.00','4,101,900,000.00','5,220,600,000.00','5,808,200,000.00','4,689,500,000.00','5,375,000,000.00','3,787,500,000.00','4,150,000,000.00','3,662,500,000.00','3,712,500,000.00','4,375,000,000.00','3,587,500,000.00');
var thetable=new mytable();thetable.tableWidth=650;thetable.shownum=false;thetable.controlaccess=true;thetable.visCols=new Array(true,true,true,true,true);thetable.initsort=new Array(0,-1);thetable.inittable();thetable.refreshTable();</script>
"""
## Make a dummy list of scripts
scripts = [s for _ in range(10)]

According to the provided <script> in your question, you can do something like code below to have a list of Dates for each name ANA, DEN ..:
for _ in range(1, len(aaa.split("<b>'"))-1):
s = aaa.split("<b>'")[_].split("'")
print(_)
lst = []
for i in s:
if "</B>" in i:
name = i.split('>')[1].split("<")[0]
print("{} = ".format(name), end="")
if any(j.isdigit() for j in i) and ',' in i:
lst.append(i)
print(lst)
It's just an example code, so it's not that beautiful :)
Hope this will help you.

I am not able to obtain data from process.stdout.on globally

I am trying to obtain the value of category variable using a machine learning code in python. Although when i execute the code the category variable isn't changed at all and database stores the category as "A" which is defined outside globally. As far as i know, it is due to some asynchronous behavior but i don't know the actual solution.
var category = "A";
if (type == "lost") {
var spawn = require("child_process").spawn;
var process = spawn('python', ["./evaluate_lost.py", req.body.image]);
process.stdout.on('data', function(data) {
category += data.toString();
});
var newLost = {
name: name,
date: date,
time: time,
location: location,
phone: phone,
image: image,
description: desc,
category: category,
author: author
};
// Create a new lost and save to DB
Lost.create(newLost, function(err, newlyCreated) {
if (err) {
console.log(err);
} else {
//redirect back to items page
res.redirect("/items");
}
});
}
Well i am editing the question with the evaluate_lost.py script and the directory structure.
import sys
from keras import backend as K
import inception_v4
import numpy as np
import cv2
import os
import argparse
image=sys.argv[1]
# If you want to use a GPU set its index here
os.environ['CUDA_VISIBLE_DEVICES'] = ''
# This function comes from Google's ImageNet Preprocessing Script
def central_crop(image, central_fraction):
if central_fraction <= 0.0 or central_fraction > 1.0:
raise ValueError('central_fraction must be within (0, 1]')
if central_fraction == 1.0:
return image
img_shape = image.shape
depth = img_shape[2]
fraction_offset = int(1 / ((1 - central_fraction) / 2.0))
bbox_h_start = int(np.divide(img_shape[0], fraction_offset))
bbox_w_start = int(np.divide(img_shape[1], fraction_offset))
bbox_h_size = int(img_shape[0] - bbox_h_start * 2)
bbox_w_size = int(img_shape[1] - bbox_w_start * 2)
image = image[bbox_h_start:bbox_h_start+bbox_h_size, bbox_w_start:bbox_w_start+bbox_w_size]
return image
def get_processed_image(img_path):
# Load image and convert from BGR to RGB
im = np.asarray(cv2.imread(img_path))[:,:,::-1]
im = central_crop(im, 0.875)
im = cv2.resize(im, (299, 299))
im = inception_v4.preprocess_input(im)
if K.image_data_format() == "channels_first":
im = np.transpose(im, (2,0,1))
im = im.reshape(-1,3,299,299)
else:
im = im.reshape(-1,299,299,3)
return im
if __name__ == "__main__":
# Create model and load pre-trained weights
model = inception_v4.create_model(weights='imagenet', include_top=True)
# Open Class labels dictionary. (human readable label given ID)
classes = eval(open('validation_utils/class_names.txt', 'r').read())
# Load test image!
img_path = "../public/files/lost/" + image
img = get_processed_image(img_path)
# Run prediction on test image
preds = model.predict(img)
print("Class is: " + classes[np.argmax(preds)-1])
print("Certainty is: " + str(preds[0][np.argmax(preds)]))
sys.stdout.flush()
This is the directory structure which evaluates the python script on watch.jpg which is input through HTML form
I expect the category to be as returned from python machine learning code rather than what is already defined.

The data event handler runs asynchronously, you're not waiting for all the output to be consumed.
Use the end event to detect the end of the output, and run the code that saves the new Lost object there.
var category = "A";
if (type == "lost") {
var spawn = require("child_process").spawn;
var process = spawn('python', ["./evaluate_lost.py", req.body.image]);
process.stdout.on('data', function(data) {
category += data.toString();
});
process.stdout.on('end', function() {
var newLost = {
name: name,
date: date,
time: time,
location: location,
phone: phone,
image: image,
description: desc,
category: category,
author: author
};
// Create a new lost and save to DB
Lost.create(newLost, function(err, newlyCreated) {
if (err) {
console.log(err);
} else {
//redirect back to items page
res.redirect("/items");
}
});
});
}

Write gpxlogger data to the same file

I have some very basic gpxlogger code that writes all the data to file quite well. (below)
gpxlogger -d -f /home/pi/Desktop/EPQ/temp_gps/gpslog
However I would like this code to always write to the same file without overwriting it. So if possible when it started logging it would go to the bottom of the file and start logging the data there, below what has already been logged.
Thanks,
Dan.
Javascript to read xml file
<script src="http://maps.google.com/maps?file=api&v=2&key=ABQIAAAA7_kD1t_m22HBF9feCaDPZxQwcATY4FXmxYwkk9LNWGtAQdNKTBS1kBsTEqrRPg2kWxuNdmf2JVCIkQ" type="text/javascript"></script>
<script src="http://gmaps-utility-library.googlecode.com/svn/trunk/markermanager/release/src/markermanager.js"> </script><script>
var map;
function initialize () {
if (GBrowserIsCompatible()) {
map = new GMap2(document.getElementById("map_canvas"));
map.setCenter(new GLatLng(53.423027, -1.523462), 10);
map.addControl(new GLargeMapControl());
map.addControl(new GMapTypeControl());
map.addMapType(G_PHYSICAL_MAP);
map.setMapType(G_PHYSICAL_MAP);
addMarkersFromXML();
}
}
function addMarkersFromXML(){
var batch = [];
mgr = new MarkerManager(map);
var request = GXmlHttp.create();
request.open('GET', 'gpslog.xml', true);
request.onreadystatechange = function() {
if (request.readyState == 4 && request.status == 200) {
var xmlDoc = request.responseXML;
var xmlrows = xmlDoc.documentElement.getElementsByTagName("trkpt");
for (var i = 0; i < xmlrows.length; i++) {
var xmlrow = xmlrows[i];
var xmlcellLatitude = parseFloat(xmlrows[i].getAttribute("lat"));
var xmlcellLongitude = parseFloat(xmlrows[i].getAttribute("lon"));
var point = new GLatLng(xmlcellLatitude,xmlcellLongitude);
//get the time of the pin plot
var xmlcellplottime = xmlrow.getElementsByTagName("time")[0];
var celltextplottime = xmlcellplottime.firstChild.data;
//get the elevation of the pin plot
var xmlcellplotelevation = xmlrow.getElementsByTagName("ele")[0];
var celltextplotelevation = xmlcellplotelevation.firstChild.data;
//get the number of satellites at the time of the pin plot
var xmlcellplotsat = xmlrow.getElementsByTagName("sat")[0];
var celltextplotsat = xmlcellplotsat.firstChild.data;
var htmlString = "Time: " + celltextplottime + "<br>" + "Elevation: " + celltextplotelevation + "<br>" + "Satellites: " + celltextplotsat;
//var htmlString = 'yes'
var marker = createMarker(point,htmlString);
batch.push(marker);
}
mgr.addMarkers(batch,10);
mgr.refresh();
}
}
request.send(null);
}
function createMarker(point,html) {
var marker = new GMarker(point);
GEvent.addListener(marker, "click", function() {
marker.openInfoWindowHtml(html);
});
return marker;
}
</script>
</head>
<body onload="initialize()" onunload="GUnload()">
<div id="map_canvas" style="width: 1350px; height: 800px"></div>
<div id="message"></div>
</body>
</html>

Here's another option.
Look at gps3.py, put it, and the following script into a directory.
It reads data from the gpsd; creates the gpx log file if it doesn't exist; appends "trackpoint" data to it when data exists; while maintaining the same file and appending "trackpoint" data after a restart.
Place both in the same directory and then have you javascript read the file..or put the entire structure in the same script.
#!/usr/bin/env python
# coding=utf-8
""" gpx logger to create and append a gpx formatted log of gpsd data """
import os
import time
import gps3
from datetime import datetime
the_connection = gps3.GPSDSocket()
the_fix = gps3.Fix()
the_log = '/tmp/gpx3.gpx'
creation = datetime.utcnow()
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
genesis = creation.strftime(fmt)
if not os.path.isfile(the_log):
header = ('<?xml version = "1.0" encoding = "utf-8"?>\n'
'<gpx version = "1.1" '
'creator = "GPSD 3.9 - http://catb.org/gpsd" '
'client = "gps3.py - http://github.com/wadda/gps3"'
'xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance"'
'xmlns = "http://www.topografix.com/GPX/1/1"'
'xsi:schemaLocation = "http://www.topografix.com/GPX/1/1 http://www.topografix.com/GPX/1/1/gpx.xsd">\n '
'<metadata>\n '
' <time>{}\n'
'</metadata>\n').format(genesis)
f = open(the_log, 'w')
f.write(header)
f.close()
try:
for new_data in the_connection:
if new_data:
the_fix.refresh(new_data)
if not isinstance(the_fix.TPV['lat'], str): # lat determinate of when data is 'valid'
latitude = the_fix.TPV['lat']
longitude = the_fix.TPV['lon']
altitude = the_fix.TPV['alt']
time = the_fix.TPV['time']
mode = the_fix.TPV['mode']
tag = the_fix.TPV['tag']
sats = the_fix.satellites_used()
hdop = the_fix.SKY['hdop']
vdop = the_fix.SKY['vdop']
pdop = the_fix.SKY['pdop']
trackpoint = ('<trkpt lat = {} lon = {}>\n'
' <ele>{}</ele>\n'
' <time>{}</time>\n'
' <src>GPSD tag ="{}"</src>\n'
' <fix>{}</fix >\n'
' <sat>{}</sat>\n'
' <hdop>{}</hdop>\n'
' <vdop>{}</vdop>\n'
' <pdop>{}</pdop>\n'
'</trkpt>\n').format(latitude, longitude, altitude, time, tag, mode, sats[1], hdop, vdop, pdop)
addendum = open(the_log, 'a')
addendum.write(trackpoint)
addendum.close()
except Exception as error:
print('Danger-Danger',error)

You run into the problem of a daemonised gpxlogger requiring a -f flag for a file name and that will overwrite the file. This you know.
I see there are two options. Not run gpxlogger as a daemon
gpxlogger >> /home/pi/Desktop/EPQ/temp_gps/gpslog
or run it as a daemon and cat the file to an append-able file
gpxlogger -d -f /home/pi/Desktop/EPQ/temp_gps/gpslog & cat /home/pi/Desktop/EPQ/temp_gps/gpslog >> /home/pi/Desktop/EPQ/temp_gps/gpslog_concatenated

Another way to look at it would be to create sequential logs, and then concatenate them with gpsbable, but in order to do that you need to have a script, and an index.
Make the index echo 0 > ~/.gpxfilecount
Open a favourite editor and create a file including something like:
#! /usr/bin/bash
COUNT=`cat ~/.gpxfilecount`
echo $(($COUNT + 1 )) > ~/.gpxfilecount
filename="gpxlogfile${COUNT}.gpx"
exec gpxlogger -d -f $filename
Mark the script executable chmod +x ~/bin/gpxer.sh (or favourite name).
Every time you fire up the gpxlogger an incremented filename is created. Those file can then be concatenated without tears by gpsbable gpsbabel -i geo -f gpxlogfile1.gpx -f gpxlogfile2.gpx -f gpxlogfile3.gpx -o gpx -F biglogcat.gpx ...or however gpsbable works.

I was curious what building a gpx file from scratch would look like using only minidom. Unfortunately life intervened, sorry for the delay...
When this script (below) gpex3.py, still a little crude and inefficient (read/write every second), is placed in the same directory as gps3.py it creates an appendable gpx file at /tmp/gpx3.gpx
#! /usr/bin/python3
# coding=utf-8
"""banana"""
import xml.dom.minidom
import gps3
import time
from datetime import datetime, timezone, timedelta
import os
import sys
gps_connection = gps3.GPSDSocket()
gps_fix = gps3.Fix()
the_log = '/tmp/gpx3.gpx'
def start_time():
"""time in the beginning"""
timestart = str(datetime.utcnow().replace(tzinfo=(timezone(timedelta(0)))))
return timestart
def close(doc):
"""write file to disk and close"""
log_write = open(the_log, "w")
doc.writexml(log_write)
log_write.close()
if os.path.isfile(the_log):
doc = xml.dom.minidom.parse(the_log) # opens the pre-existing
gpx_element = doc.firstChild
else:
doc = xml.dom.minidom.Document()
gpx_element = doc.createElement("gpx")
doc.appendChild(gpx_element)
trk_element = doc.createElement("trkseg")
trk_element.setAttribute("began", start_time())
gpx_element.appendChild(trk_element)
utc = alt = hdop = vdop = pdop = mode = sats = tag = 'n/a'
try:
tpv_list = {'time': utc, 'ele': alt, 'tag': tag}
sky_list = {'hdop': hdop, 'vdop': vdop, 'pdop': pdop}
# misc_list = {'sat': sats, 'fix':mode} # just an account
element = {}
x = 1 # for the 'is it working?'
for new_data in gps_connection:
if new_data:
gps_fix.refresh(new_data)
if not isinstance(gps_fix.TPV['lat'], str):
trkpt_element = doc.createElement("trkpt")
trk_element.appendChild(trkpt_element)
trkpt_element.setAttribute('lat', str(gps_fix.TPV['lat']))
trkpt_element.setAttribute('lon', str(gps_fix.TPV['lon']))
# tpv_list[key]
for key in tpv_list:
if key == 'ele':
element[key] = '{}'.format(gps_fix.TPV['alt']) # because consistency with labels is a horrible.
else:
element[key] = '{}'.format(gps_fix.TPV[key])
# sky_list[key]
for key in sky_list:
element[key] = '{}'.format(gps_fix.SKY[key])
# Misc.
element['sat'] = '{}'.format(gps_fix.satellites_used()[1])
element['fix'] = '{}'.format(("ZERO", "NO_FIX", "2D", "3D")[gps_fix.TPV['mode']])
for key in element:
trkpt_data = doc.createElement(key)
trkpt_element.appendChild(trkpt_data)
new_value = doc.createTextNode(element[key])
trkpt_data.appendChild(new_value)
# print(doc.toprettyxml())
close(doc) # write to file with every trackpoint
print('Cycle', x) # Only an "is it working?"
x += 1
time.sleep(1)
except KeyboardInterrupt:
gps_connection.close()
print("\nTerminated by user\nGood Bye.\n")
if __name__ == '__main__':
pass