We Keep Coding

Why is my Android app crashing every time I run a python script with chaquopy?

I am building an Android app that will allow the user to get a picture either by taking it in real time or uploading it from their saved images. Then, it will go through a machine learning script in python to determine their location. Before I completely connect to the algorithm, I am trying a test program that just returns a double.
from os.path import dirname, join
import csv
import random
filename = join(dirname(__file__), "new.csv")
def testlat():
return 30.0
def testlong():
return 30.0
These returned values are used in a Kotlin file that will then send those values to the Google Maps activity on the app for the location to be plotted.
class MainActivity : AppCompatActivity() {
var lat = 0.0
var long = 0.0
var dynamic = false
private val cameraRequest = 1888
lateinit var imageView: ImageView
lateinit var button: Button
private val pickImage = 100
private var imageUri: Uri? = null
var active = false
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
// Accesses the info image button
val clickMe = findViewById<ImageButton>(R.id.imageButton)
// Runs this function when the info icon is pressed by the user
// It will display the text in the variable infoText
clickMe.setOnClickListener {
Toast.makeText(this, infoText, Toast.LENGTH_LONG).show()
}
if (ContextCompat.checkSelfPermission(applicationContext, Manifest.permission.CAMERA)
== PackageManager.PERMISSION_DENIED) {
ActivityCompat.requestPermissions(
this,
arrayOf(Manifest.permission.CAMERA),
cameraRequest
)
}
imageView = findViewById(R.id.imageView)
val photoButton: Button = findViewById(R.id.button2)
photoButton.setOnClickListener {
val cameraIntent = Intent(MediaStore.ACTION_IMAGE_CAPTURE)
startActivityForResult(cameraIntent, cameraRequest)
dynamic = true
}
/*
The below will move to external photo storage once button2 is clicked
*/
button = findViewById(R.id.button)
button.setOnClickListener {
val gallery = Intent(Intent.ACTION_PICK, MediaStore.Images.Media.INTERNAL_CONTENT_URI)
startActivityForResult(gallery, pickImage)
}
// PYTHON HERE
if (! Python.isStarted()) {
Python.start(AndroidPlatform(this))
}
}
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)
if (resultCode == RESULT_OK && requestCode == pickImage) {
imageUri = data?.data
imageView.setImageURI(imageUri)
// PYTHON HERE
val py = Python.getInstance()
val pyobj = py.getModule("main")
this.lat = pyobj.callAttr("testlat").toDouble()
this.long = pyobj.callAttr("testlong").toDouble()
/* Open the map after image has been received from user
This will be changed later to instead call the external object recognition/pathfinding
scripts and then pull up the map after those finish running
*/
val mapsIntent = Intent(this, MapsActivity::class.java)
startActivity(mapsIntent)
}
}
}
I set up chaquopy and the gradle is building successfully, but everytime I get to the python part of emulating the app, it crashes. I'm not quite sure why that is; I thought maybe the program was too much for the phone to handle but it is a very basic python script so I doubt that's the issue.

If your app crashes, you can find the stack trace in the Logcat.
In this case, it's probably caused by the line return = 30.0. The correct syntax is return 30.0.

Python InDesign scripting: Get overflowing textbox from preflight for automatic resizing

Thanks to this great answer I was able to figure out how to run a preflight check for my documents using Python and the InDesign script API. Now I wanted to work on automatically adjusting the text size of the overflowing text boxes, but was unable to figure out how to retrieve a TextBox object from the Preflight object.
I referred to the API specification, but all the properties only seem to yield strings which do not uniquely define the TextBoxes, like in this example:
Errors Found (1):
Text Frame (R=2)
Is there any way to retrieve the violating objects from the Preflight, in order to operate on them later on? I'd be very thankful for additional input on this matter, as I am stuck!

If all you need is to find and to fix the overset errors I'd propose this solution:
Here is the simple Extendscript to fix the text overset error. It decreases the font size in the all overflowed text frames in active document:
var doc = app.activeDocument;
var frames = doc.textFrames.everyItem().getElements();
var f = frames.length
while(f--) {
var frame = frames[f];
if (frame.overflows) resize_font(frame)
}
function resize_font(frame) {
app.scriptPreferences.enableRedraw = false;
while (frame.overflows) {
var texts = frame.parentStory.texts.everyItem().getElements();
var t = texts.length;
while(t--) {
var characters = texts[t].characters.everyItem().getElements();
var c = characters.length;
while (c--) characters[c].pointSize = characters[c].pointSize * .99;
}
}
app.scriptPreferences.enableRedraw = true;
}
You can save it in any folder and run it by the Python script:
import win32com.client
app = win32com.client.Dispatch('InDesign.Application.CS6')
doc = app.Open(r'd:\temp\test.indd')
profile = app.PreflightProfiles.Item('Stackoverflow Profile')
print('Profile name:', profile.name)
process = app.PreflightProcesses.Add(doc, profile)
process.WaitForProcess()
errors = process.processResults
print('Errors:', errors)
if errors[:4] != 'None':
script = r'd:\temp\fix_overset.jsx' # <-- here is the script to fix overset
print('Run script', script)
app.DoScript(script, 1246973031) # run the jsx script
# 1246973031 --> ScriptLanguage.JAVASCRIPT
# https://www.indesignjs.de/extendscriptAPI/indesign-latest/#ScriptLanguage.html
process = app.PreflightProcesses.Add(doc, profile)
process.WaitForProcess()
errors = process.processResults
print('Errors:', errors) # it should print 'None'
if errors[:4] == 'None':
doc.Save()
doc.Close()
input('\nDone... Press <ENTER> to close the window')

Thanks to the exellent answer of Yuri I was able solve my problem, although there are still some shortcomings.
In Python, I load my documents and check if there are any problems detected during the preflight. If so, I move on to adjusting the text frames.
myDoc = app.Open(input_file_path)
profile = app.PreflightProfiles.Item(1)
process = app.PreflightProcesses.Add(myDoc, profile)
process.WaitForProcess()
results = process.processResults
if "None" not in results:
# Fix errors
script = open("data/script.jsx")
app.DoScript(script.read(), 1246973031, variables.resize_array)
process.WaitForProcess()
results = process.processResults
# Check if problems were resolved
if "None" not in results:
info_fail(card.name, "Error while running preflight")
myDoc.Close(1852776480)
return FLAG_PREFLIGHT_FAIL
I load the JavaScript file stored in script.jsx, that consists of several components. I start by extracting the arguments and loading all the pages, since I want to handle them individually. I then collect all text frames on the page in an array.
var doc = app.activeDocument;
var pages = doc.pages;
var resizeGroup = arguments[0];
var condenseGroup = arguments[1];
// Loop over all available pages separately
for (var pageIndex = 0; pageIndex < pages.length; pageIndex++) {
var page = pages[pageIndex];
var pageItems = page.allPageItems;
var textFrames = [];
// Collect all TextFrames in an array
for (var pageItemIndex = 0; pageItemIndex < pageItems.length; pageItemIndex++) {
var candidate = pageItems[pageItemIndex];
if (candidate instanceof TextFrame) {
textFrames.push(candidate);
}
}
What I wanted to achieve was a setting where if one of a group of text frames was overflowing, the text size of all the text frames in this group are adjusted as well. E.g. text frame 1 overflows when set to size 8, no longer when set to size 6. Since text frame 1 is in the same group as text frame 2, both of them will be adjusted to size 6 (assuming the second frame does not overflow at this size).
In order to handle this, I pass an array containing the groups. I now check if the text frame is contained in one of these groups (which is rather tedious, I had to write my own methods since InDesign does not support modern functions like filter() as far as I am concerned...).
// Check if TextFrame overflows, if so add all TextFrames that should be the same size
for (var textFrameIndex = 0; textFrameIndex < textFrames.length; textFrameIndex++) {
var textFrame = textFrames[textFrameIndex];
// If text frame overflows, adjust it and all the frames that are supposed to be of the same size
if (textFrame.overflows) {
var foundResizeGroup = filterArrayWithString(resizeGroup, textFrame.name);
var foundCondenseGroup = filterArrayWithString(condenseGroup, textFrame.name);
var process = false;
var chosenGroup, type;
if (foundResizeGroup.length > 0) {
chosenGroup = foundResizeGroup;
type = "resize";
process = true;
} else if (foundCondenseGroup.length > 0) {
chosenGroup = foundCondenseGroup;
type = "condense";
process = true;
}
if (process) {
var foundFrames = findTextFramesFromNames(textFrames, chosenGroup);
adjustTextFrameGroup(foundFrames, type);
}
}
}
If this is the case, I adjust either the text size or the second axis of the text (which condenses the text for my variable font). This is done using the following functions:
function adjustTextFrameGroup(resizeGroup, type) {
// Check if some overflowing textboxes
if (!someOverflowing(resizeGroup)) {
return;
}
app.scriptPreferences.enableRedraw = false;
while (someOverflowing(resizeGroup)) {
for (var textFrameIndex = 0; textFrameIndex < resizeGroup.length; textFrameIndex++) {
var textFrame = resizeGroup[textFrameIndex];
if (type === "resize") decreaseFontSize(textFrame);
else if (type === "condense") condenseFont(textFrame);
else alert("Unknown operation");
}
}
app.scriptPreferences.enableRedraw = true;
}
function someOverflowing(textFrames) {
for (var textFrameIndex = 0; textFrameIndex < textFrames.length; textFrameIndex++) {
var textFrame = textFrames[textFrameIndex];
if (textFrame.overflows) {
return true;
}
}
return false;
}
function decreaseFontSize(frame) {
var texts = frame.parentStory.texts.everyItem().getElements();
for (var textIndex = 0; textIndex < texts.length; textIndex++) {
var characters = texts[textIndex].characters.everyItem().getElements();
for (var characterIndex = 0; characterIndex < characters.length; characterIndex++) {
characters[characterIndex].pointSize = characters[characterIndex].pointSize - 0.25;
}
}
}
function condenseFont(frame) {
var texts = frame.parentStory.texts.everyItem().getElements();
for (var textIndex = 0; textIndex < texts.length; textIndex++) {
var characters = texts[textIndex].characters.everyItem().getElements();
for (var characterIndex = 0; characterIndex < characters.length; characterIndex++) {
characters[characterIndex].setNthDesignAxis(1, characters[characterIndex].designAxes[1] - 5)
}
}
}
I know that this code can be improved upon (and am open to feedback), for example if a group consists of multiple text frames, the procedure will run for all of them, even though it need only be run once. I was getting pretty frustrated with the old JavaScript, and the impact is negligible. The rest of the functions are also only helper functions, which I'd like to replace with more modern version. Sadly and as already stated, I think that they are simply not available.
Thanks once again to Yuri, who helped me immensely!

Deadlock when combining Python with TBB

Here I would like to offer a complete test case showing a simple TBB parallel_for construct causing deadlock in a Python application. Python frontend is combined with TBB backend using pybind11:
void backend_tbb(vector<int>& result, std::function<int (int)>& callback)
{
int nthreads = tbb::task_scheduler_init::default_num_threads();
const char* cnthreads = getenv("TBB_NUM_THREADS");
if (cnthreads) nthreads = std::max(1, atoi(cnthreads));
tbb::task_group group;
tbb::task_arena arena(nthreads, 1);
tbb::task_scheduler_init init(nthreads);
group.run( [&] {
tbb::parallel_for(tbb::blocked_range<int>(0, result.size()),
[&](const tbb::blocked_range<int>& range)
{
for (int i = range.begin(); i != range.end(); i++)
result[i] = callback(i);
});
});
arena.execute( [&] { group.wait(); });
}
void backend_serial(vector<int>& result, std::function<int (int)>& callback)
{
for (int i = 0; i < result.size(); i++)
result[i] = callback(i);
}
PYBIND11_MODULE(python_tbb, m)
{
pybind11::bind_vector<std::vector<int> >(m, "stdvectorint");
m.def("backend_tbb", &backend_tbb, "TBB backend");
m.def("backend_serial", &backend_serial, "Serial backend");
}
With backend_tbb uncommented, app deadlocks infinitely:
from python_tbb import *
import numpy as np
def callback(a) :
return int(a) * 10
def main() :
length = 10
result1 = stdvectorint(np.zeros(length, np.int32))
result2 = stdvectorint(np.zeros(length, np.int32))
backend_serial(result1, callback)
# XXX Uncomment this to get the program hang
#backend_tbb(result2, callback)
for i in range(length) :
print("%d vs %d" % (result1[i], result2[i]))
if __name__ == "__main__" :
main()
I've tried gil_scoped_acquire/gil_scoped_release, but no change. Similar solution reportedly works for OpenMP loop - but again no luck when I try to do the same for TBB. Please kindly advice on this case, thanks!

The issue is that TBB tasks get spawned inside task_arena instance associated with the task_group, but the waiting is done inside another task_arena instance, called arena. This can lead to the deadlock. To fix the issue, try wrapping the call to group.run() into task_arena.execute() similarly as it is done for group.wait().
However, in this case, the latter wrapping seems superfluous. So, you might want to combine two wrappings into one
arena.execute() {
group.run( /* ... */ );
group.wait();
}
which, in this particular example, makes the use of task_group unnecessary since the master thread spawns the tasks and immediately joins for participating in their execution, similarly as it is done in tbb::parallel_for. Thus, task_group can be merely removed.

extract data from gmail add to spreadsheet- Google apps script

I have searched, copied and modified code, and tried to break down what others have done and I still can't get this right.
I have email receipts for an ecommerce webiste, where I am trying to harvest particular details from each email and save to a spreadsheet with a script.
Here is the entire script as I have now.
function menu(e) {
var ui = SpreadsheetApp.getUi();
ui.createMenu('programs')
.addItem('parse mail', 'grabreceipt')
.addToUi();
}
function grabreceipt() {
var ss = SpreadsheetApp.getActiveSheet();
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getSheetByName("Sheet1");
var threads = GmailApp.search("(subject:order receipt) and (after:2016/12/01)");
var a=[];
for (var i = 0; i<threads.length; i++)
{
var messages = threads[i].getMessages();
for (var j=0; j<messages.length; j++)
{
var messages = GmailApp.getMessagesForThread(threads[i]);
for (var j = 0; j < messages.length; j++) {
a[j]=parseMail(messages[j].getPlainBody());
}
}
var nextRow=s.getDataRange().getLastRow()+1;
var numRows=a.length;
var numCols=a[0].length;
s.getRange(nextRow,1,numRows,numCols).setValues(a);
}
function parseMail(body) {
var a=[];
var keystr="Order #,Subtotal:,Shipping:,Total:";
var keys=keystr.split(",");
var i,p,r;
for (i in keys) {
//p=keys[i]+(/-?\d+(,\d+)*(\.\d+(e\d+)?)?/);
p=keys[i]+"[\r\n]*([^\r^\n]*)[\r\n]";
//p=keys[i]+"[\$]?[\d]+[\.]?[\d]+$";
r=new RegExp(p,"m");
try {a[i]=body.match(p)[1];}
catch (err) {a[i]="no match";}
}
return a;
}
}
So the email data to pluck from comes as text only like this:
Order #89076
(body content, omitted)
Subtotal: $528.31
Shipping: $42.66 via Priority Mail®
Payment Method: Check Payment- Money order
Total: $570.97
Note: mywebsite order 456. Customer asked about this and that... etc.
The original code regex was designed to grab content, following the keystr values which were easily found on their own line. So this made sense:
p=keys[i]+"[\r\n]*([^\r^\n]*)[\r\n]";
This works okay, but results where the lines include more data that follows as in line Shipping: $42.66 via Priority Mail®.
My data is more blended, where I only wish to take numbers, or numbers and decimals. So I have this instead which validates on regex101.com
p=keys[i]+"[\$]?[\d]+[\.]?\d+$";
The expression only, [\$]?[\d]+[.]?\d+$ works great but I still get "no match" for each row.
Additionally, within this search there are 22 threads returned, and it populates 39 rows in the spreadsheet. I can not figure out why 39?

The reason for your regex not working like it should is because you are not escaping the "\" in the string you use to create the regex
So a regex like this
"\s?\$?(\d+\.?\d+)"
needs to be escaped like so:
"\\s?\\$?(\\d+\\.?\\d+)"
The below code is just modified from your parseEmail() to work as a snippet here. If you copy this to your app script code delete document.getElementById() lines.
Your can try your example in the snippet below it will only give you the numbers.
function parseMail(body) {
if(body == "" || body == undefined){
var body = document.getElementById("input").value
}
var a=[];
var keystr="Order #,Subtotal:,Shipping:,Total:";
var keys=keystr.split(",");
var i,p,r;
for (i in keys) {
p=keys[i]+"\\s?\\$?(\\d+\\.?\\d+)";
r=new RegExp(p,"m");
try {a[i]=body.match(p)[1];}
catch (err) {a[i]="no match";}
}
document.getElementById("output").innerHTML = a.join(";")
return a;
}
<textarea id ="input"></textarea>
<div id= "output"></div>
<input type = "button" value = "Parse" onclick = "parseMail()">
Hope that helps

Fixtures in Golang testing

Coming from the python world, fixtures are very useful (Fixtures defines a Python contract for reusable state / support logic, primarily for unit testing). I was wondering if there's similar support in Golang which can allow me to run my tests with some predefined fixtures like setting up server, tearing it down, doing some repeated tasks each time a test is run ? Can someone point me to some examples of doing the same in Golang ?

If you want to use the standard Go testing tools, you can define a function with the signature TestMain(m *testing.M) and put your fixture code in there.
From the testing package wiki:
It is sometimes necessary for a test program to do extra setup or teardown before or after testing. It is also sometimes necessary for a test to control which code runs on the main thread. To support these and other cases, if a test file contains a function:
func TestMain(m *testing.M)
then the generated test will call TestMain(m) instead of running the tests directly. TestMain runs in the main goroutine and can do whatever setup and teardown is necessary around a call to m.Run. It should then call os.Exit with the result of m.Run. When TestMain is called, flag.Parse has not been run. If TestMain depends on command-line flags, including those of the testing package, it should call flag.Parse explicitly.
A simple implementation of TestMain is:
func TestMain(m *testing.M) {
flag.Parse()
os.Exit(m.Run())
}

I know this is an old question, but this still came up in a search result so I thought I'd give a possible answer.
You can isolate code out to helper functions that return a "teardown" function to clean up after itself. Here's one possible way to go about starting a server and have it close at the end of the test case.
func setUpServer() (string, func()) {
h := func(w http.ResponseWriter, r *http.Request) {
code := http.StatusTeapot
http.Error(w, http.StatusText(code), code)
}
ts := httptest.NewServer(http.HandlerFunc(h))
return ts.URL, ts.Close
}
func TestWithServer(t *testing.T) {
u, close := setUpServer()
defer close()
rsp, err := http.Get(u)
assert.Nil(t, err)
assert.Equal(t, http.StatusTeapot, rsp.StatusCode)
}
This starts up a server with net/http/httptest and returns its URL along with a function that acts as the "teardown." This function is added to the defer stack so it's always called regardless of how the test case exits.
Optionally, you can pass in the *testing.T if you have a more complicated set up going on in there and you need to handle errors. This example shows the set up function returning a *url.URL instead of a URL formatted string, and the parse can possibly return an error.
func setUpServer(t *testing.T) (*url.URL, func()) {
h := func(w http.ResponseWriter, r *http.Request) {
code := http.StatusTeapot
http.Error(w, http.StatusText(code), code)
}
ts := httptest.NewServer(http.HandlerFunc(h))
u, err := url.Parse(ts.URL)
assert.Nil(t, err)
return u, ts.Close
}
func TestWithServer(t *testing.T) {
u, close := setUpServer(t)
defer close()
u.Path = "/a/b/c/d"
rsp, err := http.Get(u.String())
assert.Nil(t, err)
assert.Equal(t, http.StatusTeapot, rsp.StatusCode)
}

I wrote golang engine for use fixtures similar to pytest:
https://github.com/rekby/fixenv
Usage example:
package example
// db create database abd db struct, cached per package - call
// once and same db shared with all tests
func db(e Env)*DB{...}
// DbCustomer - create customer with random personal data
// but fixed name. Fixture result shared by test and subtests,
// then mean many calls Customer with same name will return same
// customer object.
// Call Customer with other name will create new customer
// and resurn other object.
func DbCustomer(e Env, name string) Customer {
// ... create customer
db(e).CustomerStore(cust)
// ...
return cust
}
// DbAccount create bank account for customer with given name.
func DbAccount(e Env, customerName, accountName string)Account{
cust := DbCustomer(e, customerName)
// ... create account
db(e).AccountStore(acc)
// ...
return acc
}
func TestFirstOwnAccounts(t *testing.T){
e := NewEnv(t)
// background:
// create database
// create customer bob
// create account from
accFrom := DbAccount(e, "bob", "from")
// get existed db, get existed bob, create account to
accTo := DbAccount(e, "bob", "to")
PutMoney(accFrom, 100)
SendMoney(accFrom, accTo, 20)
if accFrom != 80 {
t.Error()
}
if accTo != 20 {
t.Error()
}
// background:
// delete account to
// delete account from
// delete customer bob
}
func TestSecondTransferBetweenCustomers(t *testing.T){
e := NewEnv(t)
// background:
// get db, existed from prev test
// create customer bob
// create account main for bob
accFrom := DbAccount(e, "bob", "main")
// background:
// get existed db
// create customer alice
// create account main for alice
accTo := DbAccount(e, "alice", "main")
PutMoney(accFrom, 100)
SendMoney(accFrom, accTo, 20)
if accFrom != 80 {
t.Error()
}
if accTo != 20 {
t.Error()
}
// background:
// remove account of alice
// remove customer alice
// remove account of bob
// remove customer bob
}
// background:
// after all test finished drop database