We Keep Coding

Output format of subprocess python

I'm trying to keep the format generated by a command to see all wordpress plugin list.
my script is :
import subprocess
r = open("/tmp/resultplugin", "w")
f = open("path/to/my/list", "r")
for x in f:
x = x.rstrip()
bashCommand = "wp plugin list --allow-root"
process = subprocess.Popen(bashCommand, cwd=x, shell=True, stdout=r)
output, error = process.communicate()
The output of the command when launched directly on the bash shell :
| name | status | update | version |
+---------------------------------+----------+-----------+------------+
| plugin name | inactive | none | 5.2.2 |
| plugin name | active | none | 10.4.0 |
| plugin name | inactive | none | 5.65 |
| plugin | inactive | none | 9.4.8 |
The output when i redirect the output in a file :
name status update version
plugin name inactive none 5.2.2
plugin name active none 10.4.0
plugin name inactive none 5.65
plugin name inactive none 9.4.8
I have hard time to find how to keep the same format, or at least have a little bit more visibility as it's very hard to read in the file
Can someone explain me how i can format the output to the file correctly ?
Thank you

Assign PCollection back to global window

I have a pipeline that takes a bounded PCollection, assigns timestamps to it and "windows" it into Sliding Windows. After a grouping transform, I want to assign the resulting PCollection back to the global window. I have not been able to figure out how to do this. See sample beam pseudo-code below:
import apache_beam as beam
with beam.Pipeline() as p:
(
p
| beam.io.ReadFromText()
| beam.ParDo(AddTimestampDoFn())
| beam.WindowInto(beam.window.SlidingWindows(60, 60))
| beam.GroupByKey()
| beam.ParDo(SomethingElse()
| beam.WindowInto(GlobalWindow()) # Here is where I want to bring back to global window
)
Any ideas on how to go about it?

Using beam.WindowInto(window.GlobalWindows()) should work. For example, with this quick test:
data = [{'message': 'Hi', 'timestamp': time.time()}]
events = (p
| 'Create Events' >> beam.Create(data) \
| 'Add Timestamps' >> beam.Map(lambda x: beam.window.TimestampedValue(x, x['timestamp'])) \
| 'Sliding Windows' >> beam.WindowInto(beam.window.SlidingWindows(60, 60)) \
| 'First window' >> beam.ParDo(DebugPrinterFn()) \
| 'global Window' >> beam.WindowInto(window.GlobalWindows()) \
| 'Second window' >> beam.ParDo(DebugPrinterFn()))
where DebugPrinterFn prints window information:
class DebugPrinterFn(beam.DoFn):
"""Just prints the element and window"""
def process(self, element, window=beam.DoFn.WindowParam):
logging.info("Received message %s in window=%s", element['message'], window)
yield element
I get the following output:
INFO:root:Received message Hi in window=[1575565500.0, 1575565560.0)
INFO:root:Received message Hi in window=GlobalWindow
Tested with the DirectRunner and 2.16.0 SDK. If it does not work for you:
Do you get any error?
Which runner and SDK are you using?
Full code here

How to select a sub menu from a context menu?

I am trying to click on a sub menu(BTDecoder) item from a context menu(send to) using pywinauto.
I could click on the menu item from context list and click on it. But when i try to click on sub menu, its not happening. its showing there is no item like that.
Here is my code :
path=os.path.realpath(path)
os.startfile(path) # open the folder named "FW"
app = pywinauto.Desktop(backend='uia').window(best_match='FW')
win = app.window(title_re='WRT_FW_27_12_2018_11_19_59_000001')
win.click_input(button='left')
win.click_input(button='right') # right click on one file listed there
app1 = pywinauto.Desktop(backend='uia').window(best_match='ContextMenu',top_level_only = True)
win1 = app1.window(title_re="Send to")
win1.click_input() # click on "Send to" context menu
app.print_control_identifiers()
app2 = pywinauto.Desktop(backend='uia').window(best_match='ContextMenuItem',top_level_only = True)
win2 = app2.window(title_re="BTDecoder")
win2.click_input() # trying to click on sub menu item called "BTDecoder" which not happening.
after clicking the "send to" contextmenu, sub menu context is appeared. after that for app.print_control_identifiers, am able to find the sub menu as shown below:
Dialog - 'FW' (L85, T151, R1250, B728)
['FW', 'FWDialog', 'Dialog', 'FW0', 'FW1']
child_window(title="FW", control_type="Window")
|
| Menu - 'Send to' (L31, T101, R468, B573)
| ['Menu', 'Send toMenu', 'Send to', 'Menu0', 'Menu1']
| child_window(title="Send to", control_type="Menu")
| |
| | MenuItem - 'Bluetooth device' (L34, T104, R465, B128)
| | ['Bluetooth device', 'MenuItem', 'Bluetooth deviceMenuItem', 'MenuItem0', 'MenuItem1']
| | child_window(title="Bluetooth device", auto_id="31011", control_type="MenuItem")
| |
| | MenuItem - 'BT Decoder CLI' (L34, T128, R465, B150)
| | ['BT Decoder CLI', 'BT Decoder CLIMenuItem', 'MenuItem2']
| | child_window(title="BT Decoder CLI", auto_id="31012", control_type="MenuItem")
| |
| | MenuItem - 'BT FW Trace Viewer' (L34, T150, R465, B172)
| | ['BT FW Trace ViewerMenuItem', 'MenuItem3', 'BT FW Trace Viewer']
| | child_window(title="BT FW Trace Viewer", auto_id="31013", control_type="MenuItem")
| |
| | MenuItem - 'BTDecoder' (L34, T172, R465, B194)
| | ['BTDecoderMenuItem', 'MenuItem4', 'BTDecoder']
| | child_window(title="BTDecoder", auto_id="31014", control_type="MenuItem")
| |
| | MenuItem - 'Compressed (zipped) folder' (L34, T194, R465, B216)
| | ['Compressed (zipped) folderMenuItem', 'MenuItem5', 'Compressed (zipped) folder']
| | child_window(title="Compressed (zipped) folder", auto_id="31015", control_type="MenuItem")
| |
| | MenuItem - 'Desktop (create shortcut)' (L34, T216, R465, B238)
| | ['Desktop (create shortcut)', 'Desktop (create shortcut)MenuItem', 'MenuItem6']
| | child_window(title="Desktop (create shortcut)", auto_id="31016", control_type="MenuItem")
```````````````````````````````````````````````````
how to click on this sub menu item?

you should be using backend ='uia' and below is the code you need to use to click on context menu's submenu item
popup_menu = Desktop(backend='uia').window(title="Context")
popup_menu[submenu].click_input()

path=os.path.realpath(path)
os.startfile(path) # open the folder named "FW"
app = pywinauto.Desktop(backend='uia').window(best_match='FW')
win = app.window(title_re='WRT_FW_27_12_2018_11_19_59_000001')
win.click_input(button='left')
win.click_input(button='right') # right click on one file listed there
app1 = pywinauto.Desktop(backend='uia').window(best_match='ContextMenu',top_level_only = True)
win1 = app1.window(title_re="Send to")
win1.click_input()
Add the below lines
app2 = Desktop(backend='win32')
app2.PopupMenu.menu_item('BTDecoder').click_input()

I was doing the similar kind of work to access the sub context menu for mmc (snap in console) type of app.
I was able to interact with the sub context menu by using below line of code-
# To access context menu item
app.ContextMenu.child_window(title='Send To', control_type='MenuItem').click_input()
# To access the sub context menu use the PopupMenu control type.
app.PopupMenu.child_window(title='BTDecoder', control_type='MenuItem').click_input()

pywinauto - Getting property text

I'm using pywinauto and I want to select text from one of the properties. However, I can't figure out how to access it.
import pywinauto
pwa_app = pywinauto.application.Application()
path = r'C:\Users\me\Cisco\Cisco AnyConnect Secure Mobility Client\vpnui.exe'
pwa_app.start(path)
w_handle = pywinauto.findwindows.find_windows(title=u'Cisco AnyConnect Secure Mobility Client', class_name='#32770')[0]
window = pwa_app.window_(handle=w_handle)
window.SetFocus()
window.print_control_identifiers(2)
Control Identifiers:
Dialog - 'Cisco AnyConnect Secure Mobility Client' (L1421, T493, R1854, B690)
['Cisco AnyConnect Secure Mobility Client', 'Cisco AnyConnect Secure Mobility ClientDialog', 'Dialog']
child_window(title="Cisco AnyConnect Secure Mobility Client", class_name="#32770")
|
| #32770 - '' (L1441, T535, R1834, B647)
| ['#32770', 'VPN:#32770']
| child_window(class_name="#32770")
|
| Static - 'VPN:' (L1527, T548, R1551, B561)
| ['Static', 'VPN:Static', 'VPN:', 'Static0', 'Static1']
| child_window(title="VPN:", class_name="Static")
|
| ComboBox - 'Brock AppDMZ VPN' (L1527, T594, R1719, B615)
| ['ComboBox', 'Connected to Brock AppDMZ VPN.ComboBox', 'VPN:ComboBox', 'ComboBox0', 'ComboBox1', 'Connected to Brock AppDMZ VPN.ComboBox0', 'Connected to Brock AppDMZ VPN.ComboBox1']
| child_window(title="Brock AppDMZ VPN", class_name="ComboBox")
|
| Edit - 'Brock AppDMZ VPN' (L1530, T597, R1699, B612)
| ['Connected to Brock AppDMZ VPN.Edit', 'Edit']
| child_window(title="Brock AppDMZ VPN", class_name="Edit")
|
| ComboBox - '' (L1527, T594, R1719, B615)
| ['ComboBox2', 'Connected to Brock AppDMZ VPN.ComboBox2']
| child_window(class_name="ComboBox")
I want to access the value "Connected to Brock AppDMZ VPN" in order to determine whether I'm connected to my VPN but the documentation on this library isn't super clear about how to do this.

The output of print_control_identifiers() is incomplete (the target text label is not listed here, there are 3 static texts on the window), but fortunately I have the same Cisco VPN client on my PC. The problem is that this "static" text is dynamic. :) There are 2 ways how to handle this situation.
1) Just create a window specification and check if this window exists:
connected_label = window.child_window(title="Connected to Brock AppDMZ VPN.", class_name="Static")
# default timeout is pywinauto.timings.Timings.window_find_timeout (5 sec.)
if connected_label.exists(timeout=10):
pass # do whatever you want
2) Rely on the number of this static text and check its text every time you need the status of connection:
label = window.child_window(class_name="Static", found_index=1).wait('exists')
if label.window_text() == "Connected to Brock AppDMZ VPN.":
pass # do whatever you want

label = window.child_window(class_name="Static", found_index=1).wait('exists')
print label.get_value()
Try this, <element>.get_value() hope this will get you the text required.

Python/Django/MySQL "Incorrect string value" error

I'm running a Django 1.4.2/Python 2.7.3/MySQL 5.5.28 site. One of the features of the site is that the admin can send an email to the server which calls a Python script via procmail that parses the email and tosses it into the DB. I maintain two versions of the site - a development and a production site. Both sites use different but identical vitualenvs (I even deleted them both and reinstalled all packages just to make sure).
I'm experiencing a weird issue. The exact same script succeeds on the dev server and fails on the production server. It fails with this error:
...django/db/backends/mysql/base.py:114: Warning: Incorrect string value: '\x92t kno...' for column 'message' at row 1
I'm well aware of the unicode issues Django has, and I know there are a ton of questions here on SO about this error, but I made sure to setup the database as UTF-8 from the beginning:
mysql> show variables like "character_set_database";
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| character_set_database | utf8 |
+------------------------+-------+
1 row in set (0.00 sec)
mysql> show variables like "collation_database";
+--------------------+-----------------+
| Variable_name | Value |
+--------------------+-----------------+
| collation_database | utf8_general_ci |
+--------------------+-----------------+
1 row in set (0.00 sec)
Additionally, I know that each column can have its own charset, but the message column is indeed UTF-8:
mysql> show full columns in listserv_post;
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| id | int(11) | NULL | NO | PRI | NULL | auto_increment | select,insert,update,references | |
| thread_id | int(11) | NULL | NO | MUL | NULL | | select,insert,update,references | |
| timestamp | datetime | NULL | NO | | NULL | | select,insert,update,references | |
| from_name | varchar(100) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| from_email | varchar(75) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| message | longtext | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
6 rows in set (0.00 sec)
Does anyone have any idea why I'm getting this error? Why is it happening under the production config but not the dev config?
Thanks!
[edit 1]
To be clear, the data are the same as well. I send a single email to the server, and procmail sends it off. This is what the .procmailrc looks like:
VERBOSE=off
:0
{
:0c
| <path>/dev/ein/scripts/process_new_mail.py dev > outputdev
:0
| <path>/prd/ein/scripts/process_new_mail.py prd > outputprd
}
There are 2 copies of process_new_mail.py, but that's just because it's version controlled so that I can maintain two separate environments. If I diff the two output files (which contain the message received), they're identical.
[edit 2]
I actually just discovered that both dev and prd configs are failing. The difference is that the dev config fails silently (maybe having to do with the DEBUG setting?). The problem is that there are some unicode characters in one of the messages, and Django is choking on them for some reason. I'm making progress....
I've tried editing the code to explicitly encode the message as ASCII and UTF-8, but it's still not working. I'm getting closer, though.

I fixed it! The problem was that I wasn't parsing the email correctly with respect to the charsets. My fixed email parsing code comes from this post and this post:
#get the charset of an email
#courtesy http://ginstrom.com/scribbles/2007/11/19/parsing-multilingual-email-with-python/
def get_charset(message, default='ascii'):
if message.get_content_charset():
return message.get_content_charset()
if message.get_charset():
return message.get_charset()
return default
#courtesy https://stackoverflow.com/questions/7166922/extracting-the-body-of-an-email-from-mbox-file-decoding-it-to-plain-text-regard
def get_body(message):
body = None
#Walk through the parts of the email to find the text body.
if message.is_multipart():
for part in message.walk():
#If part is multipart, walk through the subparts.
if part.is_multipart():
for subpart in part.walk():
if subpart.get_content_type() == 'text/plain':
#Get the subpart payload (i.e., the message body).
charset = get_charset(subpart, get_charset(message))
body = unicode(subpart.get_payload(decode=True), charset)
#Part isn't multipart so get the email body.
elif part.get_content_type() == 'text/plain':
charset = get_charset(subpart, get_charset(message))
body = unicode(part.get_payload(decode=True), charset)
#If this isn't a multi-part message then get the payload (i.e., the message body).
elif message.get_content_type() == 'text/plain':
charset = get_charset(subpart, get_charset(message))
body = unicode(message.get_payload(decode=True), charset)
return body
Thanks very much for the help!