We Keep Coding

Why Does Azure DevOps Server interleave output

Sorry in advance, I can't post actual code because of security restrictions at my job, but I'll try to make a contrived example.
I am working with python 3.6.1 and running a module in an Azure Pipeline (ADS 2019). In the module we have output done using a dictionary with the following structure
#dummy data, assume files could be in any order in any category
{
"compliant": ['file1.py', 'file2.py'], #list of files which pass
"non-compliant":['file3.py'], #list of files which fail
"incompatible":['file4.py'] #list of files which could not be tested due to exceptions
}
When a failure occurs one of our customers wants the script to output the command to call a script that can be run to correct the non-compliant files. The program is written similar to what follows
result = some_func() #returns the above dict
print('compliant:')
for file in result['compliant']:
print(file)
print('non-compliant:')
for file in result['non-compliant']:
print(file)
print('incompatible:')
for file in result['incompatible']:
print(file)
# prints a string to sys.stderr simillar to python -m script arg1 arg2 ...
# script that is output is based on the arguments used to call
print_command_to_fix(sys.argv)
When run normally I would get the correct output like follows:
#correct output: occurs on bash and cmd
compliant:
file1.py
file2.py
non-compliant:
file3.py
incompatible:
file4.py
python -m script arg1 arg2 arg_to_fix
when I run on the Azure Pipeline though, the output gets interleaved like follows
#incorrect output: occurs only on azure pipeline runs
compliant:
python -m script arg1 arg2 arg_to_fix
file1.py
file2.py
non-compliant:
file3.py
incompatible:
file4.py
Whether I try to use print or sys.stderr.write it doesn't seem to resolve the interleave, and I'm assuming the print_command_to_fix() is being called asynchronously somehow. But my guess probably isn't accurate since I haven't been working with ADS or python for very long.
TL;DR: What am I doing wrong to get the above interleaved output on Pipelines only?
Edit: clarified certain points and fixed typos

Discovered the answer after a few hours of troubleshooting and solutions.
ADS tracks both output streams in the program but does it asynchronously. The error was cause by outputting to both stdout and stderr. This being the case, outputting all output to one stream resolved the issue. The approach I took ended up being something like follows
result = some_func() #returns the above dict
output = []
output.append('compliant:')
output.extend(result['compliant'])
output.append(file)
output.extend(result['non-compliant'])
output.append('incompatible:')
output.extendresult['incompatible'])
# returns a string to simillar to python -m script arg1 arg2 ...
# script that is output is based on the arguments used to call
output.append(format_command_to_fix(sys.argv))
print('\n'.join(output))
Alternatively, I imagine other techniques for outputting async information should resolve as well.

Why Does Python Call a Subprocess Command Incorrectly?

This is a follow up on a previous question as I have made progress(which is irrelevant at this point). It is worth noting that I am learning python and barely know what I am doing, however, I am familiar with programming. I am trying to call an SCP command in the windows terminal through python. However, it is not doing the desired effect. The script runs smoothly with no errors and it prints the debug commands as I have written them. However, the SCP call does not actually go through on the other end. To make sure I have the right command, I have it set to print the same command that it called afterwards. When I copy this printed command and paste it into the windows command terminal, it gives the desired effect. Why is the same command not working correctly in python? Thanks. This is my script:
import subprocess
subprocess.run(['scp', 'c:/users/<name>/desktop/OOGA.txt', 'pi#<IP>:here/'], shell=True)
print ("done")
print ('scp', 'c:/users/<name>/desktop/OOGA.txt', 'pi#<IP>:here/')

Try using raw string if shell is set to True:
from subprocess import run as subrun
status = subrun(r'scp c:/users/<name>/desktop/OOGA.txt pi#<IP>:here/',shell=True)
print("Done")
print(status)

Using subprocess in Python, I get different results when in Python cmd and Python IDLE

I hope the title makes sense. To give specifics:
I am using csvtotable (https://github.com/vividvilla/csvtotable) to generate HTML tables from CSVs. I have installed via pip and am able to run a command line command:
csvtotable test1743.csv test1743.html
to generate a HTML page. All good so far.
I wanted to do this from within a Python script I had already written so I heard that subprocess was the way to do this. I looked up how to do it and understood that it can be done using the following:
subprocess.run('csvtotable test1743.csv test1743.html',shell=True)
So I tested this via the command line first by doing
python
from the command line and then running
import subprocess
subprocess.run('csvtotable test1743.csv test1743.html',shell=True)
Success! It worked. Fantastic.
However, when I try to do this from IDLE, it just returns a 1. I have checked the directory thinking that maybe the csv was missing from there, but it still doesn't work.
Am I misunderstanding how subprocess works?

Solved by finding a way to call the function without subprocess. I think the issue may have related to default arguments not being set when it is executed through python and hence why below I have had to specify so many arguments.
Code:
from csvtotable import convert
content = convert.convert("C:\\Users\\admin\\Google Drive\\test1743.csv",delimiter=",",quotechar='"',display_length=-1,overwrite=False,serve=False,pagination=True,virtual_scroll=1000, no_header=False, export=True, export_options=["copy","csv","json","print"])
convert.save("C:\\Users\\admin\\Google Drive\\test1743.html",content)
Note that the argument names had to be changed where they had a - in the name. I just changed any instance e.g. display-length to display_length in convert.py

Python code acting differently than when run in terminal

When I pass my mpirun command through terminal, the normal (and expected) outcome is an output file with a bunch of data in it.
However when I pass the code through my python script, all of the output files that are expected are created, however they contain no data. Is there any global explanation for this? I have tried the code many different ways, using both os.system and subprocess. I have also tried running the code in the background as well as just running. And I have also tried just having the program spit out the data vs. saving it to the output file, and all give the same result.
Here is the code:
os.system("mpirun -np 4 /home/mike/bin/Linux-ifort_XE_openmpi-1.6_emt64/v2_0_1/Pcrystal > mgo.out")

The simplest way to get that behavior is if mpirun is not being successfully run.
For instance if, from the command line, I run
not_actually_a_command > myFile.txt
myFile.txt will be created, but will be empty (the "command not found" message is printed to stderr so won't be caught by ">").
Try using the fully resolved path to mpirun
There doesn't seem to be something inherently wrong with your approach. When I do
os.system("echo hello, world >hello.txt")
it ends up with "hello, world" in it, so if you get your command to run it should work for you.

You should start with providing a complete path
os.system("/complete/path/to/mpirun
and print the result, print(os.system...etc.),
and post the error so we know what is wrong.
When using the subprocess module it may require a "shell=True"

Trying to use Gtk.Application with python, but failing to handle open

I am trying to have a Gtk.Application which stays unique and handles opening files. I am using python 2.7.3 on Ubuntu 12.04 with Gtk3 (fairly new to both python and Gtk)
The application runs fine without parameters, but it fails to get the file list when I run it trying to open a file. Here is the code, as minimalistic as I could make it:
#!/usr/bin/env python
import sys
from gi.repository import Gtk, Gio
def do_open(app, files, *hint):
print(app)
print(files)
print(hint)
def do_activate(app):
print "activate"
test = Gtk.Application(application_id="a.b", flags=Gio.ApplicationFlags.HANDLES_OPEN)
test.set_inactivity_timeout(10000)
test.connect("open", do_open)
test.connect("activate", do_activate)
test.run(sys.argv)
When I run the program without arguments it just prints "activate", which is fine. When I run it with a parameter (like ./test.py test.py) I get the following:
/usr/lib/python2.7/dist-packages/gi/types.py:43: Warning: g_value_get_boxed: assertion `G_VALUE_HOLDS_BOXED (value)' failed
return info.invoke(*args, **kwargs)
<Application object at 0x1c75230 (GtkApplication at 0x1cba0b0)>
[]
(1, '')
Does anyone understand why that assertion is failing and why I am getting an empty list of files?
As common as this task appears to be, I couldn't find any working example online either.

There is a bug in PyGObject. It is already reported in GNOME Bugzilla, check the bug report titled "Does not handle GFile in signal arguments".
Update: The bug was fixed in 2013. No more assertion and it returns the list of files (GFiles). In other words, the code works as expected (at least using 3.14):
$ python test.py test.py
<Application object at 0x7fcaf18f5d20 (GtkApplication at 0x11892b0)>
[<__main__.GLocalFile object at 0x7fcaf18a6050 (GLocalFile at 0x11aea00)>]
(1, '')