We Keep Coding

Output format of subprocess python

I'm trying to keep the format generated by a command to see all wordpress plugin list.
my script is :
import subprocess
r = open("/tmp/resultplugin", "w")
f = open("path/to/my/list", "r")
for x in f:
x = x.rstrip()
bashCommand = "wp plugin list --allow-root"
process = subprocess.Popen(bashCommand, cwd=x, shell=True, stdout=r)
output, error = process.communicate()
The output of the command when launched directly on the bash shell :
| name | status | update | version |
+---------------------------------+----------+-----------+------------+
| plugin name | inactive | none | 5.2.2 |
| plugin name | active | none | 10.4.0 |
| plugin name | inactive | none | 5.65 |
| plugin | inactive | none | 9.4.8 |
The output when i redirect the output in a file :
name status update version
plugin name inactive none 5.2.2
plugin name active none 10.4.0
plugin name inactive none 5.65
plugin name inactive none 9.4.8
I have hard time to find how to keep the same format, or at least have a little bit more visibility as it's very hard to read in the file
Can someone explain me how i can format the output to the file correctly ?
Thank you

how to use jq tools or python as a output encoder to make result to translate to json formot is a difficult question

for a example
[root#test ~]# mysql -uroot -p'123123' -e"select user,host from mysql.user;"
+-------------------+-----------+
| user | host |
+-------------------+-----------+
| root | % |
| test | % |
| sqlaudit_test_mon | % |
| sysbase_test | % |
| mysql.session | localhost |
| mysql.sys | localhost |
+-------------------+-----------+
how to make search the result quick to convert to json format can you jq tools or python
such as that out put
[
{
"user":"root","host":"%"},
{
"user":"test","host":"%"},
{
"user":"sqlaudit_test_mon","host":"%"},
{
"user":"sysbase_test","host":"%"},
{
"user":"mysql.session","host":"localhost"},
{
"user":"mysql.sys","host":"localhost"}
]
i just want to know how to quick make search result to json,thank you!
it is better to user jq or python script it can make me search result to json format.

Just do it in your SELECT instead of pulling another program into a pipeline. MySQL has JSON functions. Ones of interest here are JSON_ARRAYAGG() and JSON_OBJECT(). Something like:
SELECT json_arrayagg(json_object('user', user, 'host', host)) FROM mysql.user;
should do it, plus whatever's needed to not print out that fancy table ascii art.

Here's an all-jq solution that assumes an invocation like this:
jq -Rcn -f program.jq sql.txt
Note in particular the -R ("raw input") and -n options.
def trim: sub(" *$";"") | sub("^ *";"");
# input: an array of values
def objectify($headers):
. as $in
| reduce range(0; $headers|length) as $i ({}; .[$headers[$i]] = ($in[$i]) ) ;
def preprocess:
select( startswith("|") )
| split("|")
| .[1:-1]
| map(trim) ;
reduce (inputs|preprocess) as $in (null;
if . == null then {header: $in}
else .header as $h
| .table += [$in|objectify($h)]
end )
| .table

How to update status on pretty table in python?

I am using python pretty table to print the status of each record on CLI.
How to display the status updates on the CLI in the same table.
Example:
+--------------+---------+
| Jobs | Status |
+--------------+---------+
| job1 | FAILED |
| job2 | SUCCESS |
+--------------+---------+
The jobs status will be updated by a thread. I want to display the updated status in the same table in CLI console.

I found ascii code to move the cursor to the previous line. And I am using the below logic to achieve the purpose
number_of_records = len(records) # number of jobs in a tables
total_lines = number_of_records + 3 + 1 # number of records + Borders + Header
if prev_lines != 0:
for i in range(prev_lines):
sys.stdout.write('\033[F')
prev_lines = total_lines
print status_table
Thanks :)

Can't seem to get pexpect to print data from command

I'm steadily working on how to ssh and parse data on a device by running a command. I have had a few questions along the way of this endeavor and much help with the questions I have asked. I'm now working with pexpect and I'm not seeing much in documentation with what I am doing. Basically I need to ssh in, as I said, and then run a command that will print out data, then get that data to print to my console.
Here is my code:
import pexpect
import pxssh
import getpass
child = pexpect.spawn('ssh www.example.com')
password = getpass.getpass('password: ')
child.sendline ('foo bar')
data = (child.read_nonblocking(size=1000, timeout=100))
print data
OUTPUT:
password:
foo bar
In the foo bar command the first line of the print out is foo bar so I am wondering if this is trying to print this data but only printing the first line. I add the read_nonblocking(size=1000, timeout=100) trying to set the size to be greater and a timeout to let the data print.
UPDATE with PXSSH
I have also tried to use the pxssh samples to do this and get only the list of commands that foo can run. I need to get the print out of foo bar which is the list of configs. My guess is that you can't have commands with spaces? Here is the code I have tried:
import pxssh
import getpass
try:
s = pxssh.pxssh()
s.force_password = True
hostname = raw_input('hostname: ')
username = raw_input('username: ')
password = getpass.getpass('password: ')
s.login (hostname, username, password)
s.sendline ('foo bar') # run a command
s.prompt() # match the prompt
print s.before # print everything before the prompt.
s.logout()
except pxssh.ExceptionPxssh, e:
print "pxssh failed on login."
print str(e)
Which gives me this back in console:
pxssh failed on login.
could not set shell prompt
:
Session idle time out is disabled
SSH> unset PROMPT_COMMAND
Error - Command [unset PROMPT_COMMAND] not found.
foo [ bar | bart | ran | up
| cmd | bee | hvac | monkey
| selective | list | help ]
check[v,nv,beep] [ list | help ]
delete [ all | bee | neewb | stuff
| up | cmd | fooconfig | root
| app | list | hvac | monkey
| selective | <filename> | confirmed | list | help ]
exit [ help ]
get [ vcf | nvcf | snmpcf | help ] [<filename>]
verbose [ help ]
help [ <command> | help ]
up arrow - brings up old command lines
down arrow - brings up newer command lines
right arrow - moves cursor to the right
left arrow - moves cursor to the left
insert - inserts a space at the cursor
delete - deletes character at the cursor
SSH> PS1='[PEXPECT]\$ '
Error - Command [PS1='[PEXPECT]\$ '] not found.
foo [ bar | bart | ran | up
| cmd | bee | hvac | monkey
| selective | list | help ]
check[v,nv,beep] [ list | help ]
delete [ all | bee | neewb | stuff
| up | cmd | fooconfig | root
| app | list | hvac | monkey
| selective | <filename> | confirmed | list | help ]
exit [ help ]
get [ vcf | nvcf | snmpcf | help ] [<filename>]
verbose [ help ]
help [ <command> | help ]
up arrow - brings up old command lines
down arrow - brings up newer command lines
right arrow - moves cursor to the right
left arrow - moves cursor to the left
insert - inserts a space at the cursor
delete - deletes character at the cursor
And as I mentioned I'm just trying to get the console to print out the foo bar command configs. This is the code I had working with python-exscript before finding out I needed to work in older Python 2.4.
CODE THAT I HAD WORKING IN EXSCRIPT THAT I NEED PEXPECT TO DO
account = read_login()
conn = SSH2()
conn.connect('example.com')
conn.login(account)
conn.execute('foo bar')
data = conn.response
conn.send('exit\r')
conn.close()
print data
Any help on how to get this code to work is greatly appreciated! Thanks!

Figured out the issue. I was missing a s.prompt()
try:
s = pxssh.pxssh(timeout=60, maxread=2000000)
s.force_password = True
hostname = raw_imput('hostname: ')
username = raw_input('password: ')
password = getpass.getpass('password: ')
s.PROMPT= 'SSH> '
s.login (hostname, username, password, auto_prompt_reset=False)
s.prompt()
s.sendline('foo bar')
s.prompt()
data = s.before
print data
s.logout()
except pxssh.ExceptionPxssh, e:
print "pxssh failed on login."
print str(e)

Python/Django/MySQL "Incorrect string value" error

I'm running a Django 1.4.2/Python 2.7.3/MySQL 5.5.28 site. One of the features of the site is that the admin can send an email to the server which calls a Python script via procmail that parses the email and tosses it into the DB. I maintain two versions of the site - a development and a production site. Both sites use different but identical vitualenvs (I even deleted them both and reinstalled all packages just to make sure).
I'm experiencing a weird issue. The exact same script succeeds on the dev server and fails on the production server. It fails with this error:
...django/db/backends/mysql/base.py:114: Warning: Incorrect string value: '\x92t kno...' for column 'message' at row 1
I'm well aware of the unicode issues Django has, and I know there are a ton of questions here on SO about this error, but I made sure to setup the database as UTF-8 from the beginning:
mysql> show variables like "character_set_database";
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| character_set_database | utf8 |
+------------------------+-------+
1 row in set (0.00 sec)
mysql> show variables like "collation_database";
+--------------------+-----------------+
| Variable_name | Value |
+--------------------+-----------------+
| collation_database | utf8_general_ci |
+--------------------+-----------------+
1 row in set (0.00 sec)
Additionally, I know that each column can have its own charset, but the message column is indeed UTF-8:
mysql> show full columns in listserv_post;
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| id | int(11) | NULL | NO | PRI | NULL | auto_increment | select,insert,update,references | |
| thread_id | int(11) | NULL | NO | MUL | NULL | | select,insert,update,references | |
| timestamp | datetime | NULL | NO | | NULL | | select,insert,update,references | |
| from_name | varchar(100) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| from_email | varchar(75) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| message | longtext | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
6 rows in set (0.00 sec)
Does anyone have any idea why I'm getting this error? Why is it happening under the production config but not the dev config?
Thanks!
[edit 1]
To be clear, the data are the same as well. I send a single email to the server, and procmail sends it off. This is what the .procmailrc looks like:
VERBOSE=off
:0
{
:0c
| <path>/dev/ein/scripts/process_new_mail.py dev > outputdev
:0
| <path>/prd/ein/scripts/process_new_mail.py prd > outputprd
}
There are 2 copies of process_new_mail.py, but that's just because it's version controlled so that I can maintain two separate environments. If I diff the two output files (which contain the message received), they're identical.
[edit 2]
I actually just discovered that both dev and prd configs are failing. The difference is that the dev config fails silently (maybe having to do with the DEBUG setting?). The problem is that there are some unicode characters in one of the messages, and Django is choking on them for some reason. I'm making progress....
I've tried editing the code to explicitly encode the message as ASCII and UTF-8, but it's still not working. I'm getting closer, though.

I fixed it! The problem was that I wasn't parsing the email correctly with respect to the charsets. My fixed email parsing code comes from this post and this post:
#get the charset of an email
#courtesy http://ginstrom.com/scribbles/2007/11/19/parsing-multilingual-email-with-python/
def get_charset(message, default='ascii'):
if message.get_content_charset():
return message.get_content_charset()
if message.get_charset():
return message.get_charset()
return default
#courtesy https://stackoverflow.com/questions/7166922/extracting-the-body-of-an-email-from-mbox-file-decoding-it-to-plain-text-regard
def get_body(message):
body = None
#Walk through the parts of the email to find the text body.
if message.is_multipart():
for part in message.walk():
#If part is multipart, walk through the subparts.
if part.is_multipart():
for subpart in part.walk():
if subpart.get_content_type() == 'text/plain':
#Get the subpart payload (i.e., the message body).
charset = get_charset(subpart, get_charset(message))
body = unicode(subpart.get_payload(decode=True), charset)
#Part isn't multipart so get the email body.
elif part.get_content_type() == 'text/plain':
charset = get_charset(subpart, get_charset(message))
body = unicode(part.get_payload(decode=True), charset)
#If this isn't a multi-part message then get the payload (i.e., the message body).
elif message.get_content_type() == 'text/plain':
charset = get_charset(subpart, get_charset(message))
body = unicode(message.get_payload(decode=True), charset)
return body
Thanks very much for the help!