We Keep Coding

Output format of subprocess python

I'm trying to keep the format generated by a command to see all wordpress plugin list.
my script is :
import subprocess
r = open("/tmp/resultplugin", "w")
f = open("path/to/my/list", "r")
for x in f:
x = x.rstrip()
bashCommand = "wp plugin list --allow-root"
process = subprocess.Popen(bashCommand, cwd=x, shell=True, stdout=r)
output, error = process.communicate()
The output of the command when launched directly on the bash shell :
| name | status | update | version |
+---------------------------------+----------+-----------+------------+
| plugin name | inactive | none | 5.2.2 |
| plugin name | active | none | 10.4.0 |
| plugin name | inactive | none | 5.65 |
| plugin | inactive | none | 9.4.8 |
The output when i redirect the output in a file :
name status update version
plugin name inactive none 5.2.2
plugin name active none 10.4.0
plugin name inactive none 5.65
plugin name inactive none 9.4.8
I have hard time to find how to keep the same format, or at least have a little bit more visibility as it's very hard to read in the file
Can someone explain me how i can format the output to the file correctly ?
Thank you

how to use jq tools or python as a output encoder to make result to translate to json formot is a difficult question

for a example
[root#test ~]# mysql -uroot -p'123123' -e"select user,host from mysql.user;"
+-------------------+-----------+
| user | host |
+-------------------+-----------+
| root | % |
| test | % |
| sqlaudit_test_mon | % |
| sysbase_test | % |
| mysql.session | localhost |
| mysql.sys | localhost |
+-------------------+-----------+
how to make search the result quick to convert to json format can you jq tools or python
such as that out put
[
{
"user":"root","host":"%"},
{
"user":"test","host":"%"},
{
"user":"sqlaudit_test_mon","host":"%"},
{
"user":"sysbase_test","host":"%"},
{
"user":"mysql.session","host":"localhost"},
{
"user":"mysql.sys","host":"localhost"}
]
i just want to know how to quick make search result to json,thank you!
it is better to user jq or python script it can make me search result to json format.

Just do it in your SELECT instead of pulling another program into a pipeline. MySQL has JSON functions. Ones of interest here are JSON_ARRAYAGG() and JSON_OBJECT(). Something like:
SELECT json_arrayagg(json_object('user', user, 'host', host)) FROM mysql.user;
should do it, plus whatever's needed to not print out that fancy table ascii art.

Here's an all-jq solution that assumes an invocation like this:
jq -Rcn -f program.jq sql.txt
Note in particular the -R ("raw input") and -n options.
def trim: sub(" *$";"") | sub("^ *";"");
# input: an array of values
def objectify($headers):
. as $in
| reduce range(0; $headers|length) as $i ({}; .[$headers[$i]] = ($in[$i]) ) ;
def preprocess:
select( startswith("|") )
| split("|")
| .[1:-1]
| map(trim) ;
reduce (inputs|preprocess) as $in (null;
if . == null then {header: $in}
else .header as $h
| .table += [$in|objectify($h)]
end )
| .table

Python's Subprocess removing Mysql columns and spacing

Specifically, I am using:
Python 2.4.3 (#1, May 24 2008, 13:47:28)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-14)] on linux2
I am trying to get the raw result of a mysql query with the column names and borders. Here is the raw command being run in bash:
[root#machine ~]# mysql -u root -e 'show databases;'
+--------------------+
| Database |
+--------------------+
| dbA |
| dbB |
| dbC |
+--------------------+
I am having trouble storing this value into a variable in Python:
import subprocess
cmd_array = ["mysql", "-u", "root", "-e", "show databases"]
p = subprocess.Popen(cmd_array)
raw_data = p.communicate()[0]
# Console outputs:
# +--------------------+
# | Database |
# +--------------------+
# | dbA |
# | dbB |
# | dbC |
# +--------------------+
#
# raw_data is None
p = subprocess.Popen(cmd_array, stdout=subprocess.PIPE)
rawData = p.communicate()[0]
print rawData
# Console outputs:
# Database
# dbA
# dbB
# dbC
#
# rawData is "Database\ndbA\ndbB\ndbC"
What is the best way to store the pretty printed version of the mysql output in a python variable?

You need to use -t:
p = check_output(["mysql" ,"-u" ," root", "-t" ,"-e", 'show databases;'])
print(p)
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| world |
+--------------------+
You can also use check_output to store the output.

mysql operates differently depending on whether it is being called from a console (pseudo-terminal, pty) or being called from another program. When called from another program, it is purposefully terse for easier parsing. One way to think about it is that
"Database\ndbA\ndbB\ndbC"
is the raw result and
+--------------------+
| Database |
+--------------------+
| dbA |
| dbB |
| dbC |
+--------------------+
is a pretty-printed result for people.
I have no idea what a "column visible string" is, but if you want to process the data in your program, you got the easily parsed result you want.
If you really do want the fancy print version, you can use the pexpect or pty modules to emulate a terminal. Here's an example:
>>> import pexpect
>>> import subprocess
>>> child = pexpect.spawn('mysql -e "show databases"')
>>> child.expect(pexpect.EOF)
0
>>> print child.before
+--------------------+
| Database |
+--------------------+
| information_schema |
| test |
+--------------------+

Looks like mysql uses isatty() to determine if STDOUT is a terminal which then modifies the output.
I can get the table borders by specifying the --table or -t option:
cmd_array = ["mysql", "-t", "-u", "root", "-e", "show databases"]

python boto - how to create health check for route 53

How do I create a health check for route53 using python boto? There are no examples and documentation is very lacking http://boto.readthedocs.org/en/latest/ref/route53.html
So.if given ip address, port, and path.....then what?

Even though boto documentation is lacking, you can understand it by the code and AWS API.
Take a look at boto.route53.healthcheck.HealthCheck and implement as
route = boto.connect_route53()
hc = boto.route53.healthcheck.HealthCheck(...)
route = create_health_check(hc)
... will be filled out by the help page of HealthCheck:
Help on HealthCheck in module boto.route53.healthcheck object:
class HealthCheck(__builtin__.object)
| An individual health check
|
| Methods defined here:
|
| __init__(self, ip_addr, port, hc_type, resource_path, fqdn=None, string_match=None, request_interval=30, failure_threshold=3)
| HealthCheck object
|
| :type ip_addr: str
| :param ip_addr: IP Address
|
| :type port: int
| :param port: Port to check
|
| :type hc_type: str
| :param ip_addr: One of HTTP | HTTPS | HTTP_STR_MATCH | HTTPS_STR_MATCH | TCP
|
| :type resource_path: str
| :param resource_path: Path to check
|
| :type fqdn: str
| :param fqdn: domain name of the endpoint to check
|
| :type string_match: str
| :param string_match: if hc_type is HTTP_STR_MATCH or HTTPS_STR_MATCH, the string to search for in the response body from the specified resource
|
| :type request_interval: int
| :param request_interval: The number of seconds between the time that Amazon Route 53 gets a response from your endpoint and the time that it sends the next health-check request.
|
| :type failure_threshold: int
| :param failure_threshold: The number of consecutive health checks that an endpoint must pass or fail for Amazon Route 53 to change the current status of the endpoint from unhealthy to healthy or vice versa.
|
| to_xml(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| POSTXMLBody = '\n <HealthCheckConfig>\n <IPAddr...il...
|
| XMLFQDNPart = '<FullyQualifiedDomainName>%(fqdn)s</FullyQualifiedDomai...
|
| XMLRequestIntervalPart = '<RequestInterval>%(request_interval)d</Reque...
|
| XMLStringMatchPart = '<SearchString>%(string_match)s</SearchString>'
|
| valid_request_intervals = (10, 30)

Python/Django/MySQL "Incorrect string value" error

I'm running a Django 1.4.2/Python 2.7.3/MySQL 5.5.28 site. One of the features of the site is that the admin can send an email to the server which calls a Python script via procmail that parses the email and tosses it into the DB. I maintain two versions of the site - a development and a production site. Both sites use different but identical vitualenvs (I even deleted them both and reinstalled all packages just to make sure).
I'm experiencing a weird issue. The exact same script succeeds on the dev server and fails on the production server. It fails with this error:
...django/db/backends/mysql/base.py:114: Warning: Incorrect string value: '\x92t kno...' for column 'message' at row 1
I'm well aware of the unicode issues Django has, and I know there are a ton of questions here on SO about this error, but I made sure to setup the database as UTF-8 from the beginning:
mysql> show variables like "character_set_database";
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| character_set_database | utf8 |
+------------------------+-------+
1 row in set (0.00 sec)
mysql> show variables like "collation_database";
+--------------------+-----------------+
| Variable_name | Value |
+--------------------+-----------------+
| collation_database | utf8_general_ci |
+--------------------+-----------------+
1 row in set (0.00 sec)
Additionally, I know that each column can have its own charset, but the message column is indeed UTF-8:
mysql> show full columns in listserv_post;
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| id | int(11) | NULL | NO | PRI | NULL | auto_increment | select,insert,update,references | |
| thread_id | int(11) | NULL | NO | MUL | NULL | | select,insert,update,references | |
| timestamp | datetime | NULL | NO | | NULL | | select,insert,update,references | |
| from_name | varchar(100) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| from_email | varchar(75) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| message | longtext | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
6 rows in set (0.00 sec)
Does anyone have any idea why I'm getting this error? Why is it happening under the production config but not the dev config?
Thanks!
[edit 1]
To be clear, the data are the same as well. I send a single email to the server, and procmail sends it off. This is what the .procmailrc looks like:
VERBOSE=off
:0
{
:0c
| <path>/dev/ein/scripts/process_new_mail.py dev > outputdev
:0
| <path>/prd/ein/scripts/process_new_mail.py prd > outputprd
}
There are 2 copies of process_new_mail.py, but that's just because it's version controlled so that I can maintain two separate environments. If I diff the two output files (which contain the message received), they're identical.
[edit 2]
I actually just discovered that both dev and prd configs are failing. The difference is that the dev config fails silently (maybe having to do with the DEBUG setting?). The problem is that there are some unicode characters in one of the messages, and Django is choking on them for some reason. I'm making progress....
I've tried editing the code to explicitly encode the message as ASCII and UTF-8, but it's still not working. I'm getting closer, though.

I fixed it! The problem was that I wasn't parsing the email correctly with respect to the charsets. My fixed email parsing code comes from this post and this post:
#get the charset of an email
#courtesy http://ginstrom.com/scribbles/2007/11/19/parsing-multilingual-email-with-python/
def get_charset(message, default='ascii'):
if message.get_content_charset():
return message.get_content_charset()
if message.get_charset():
return message.get_charset()
return default
#courtesy https://stackoverflow.com/questions/7166922/extracting-the-body-of-an-email-from-mbox-file-decoding-it-to-plain-text-regard
def get_body(message):
body = None
#Walk through the parts of the email to find the text body.
if message.is_multipart():
for part in message.walk():
#If part is multipart, walk through the subparts.
if part.is_multipart():
for subpart in part.walk():
if subpart.get_content_type() == 'text/plain':
#Get the subpart payload (i.e., the message body).
charset = get_charset(subpart, get_charset(message))
body = unicode(subpart.get_payload(decode=True), charset)
#Part isn't multipart so get the email body.
elif part.get_content_type() == 'text/plain':
charset = get_charset(subpart, get_charset(message))
body = unicode(part.get_payload(decode=True), charset)
#If this isn't a multi-part message then get the payload (i.e., the message body).
elif message.get_content_type() == 'text/plain':
charset = get_charset(subpart, get_charset(message))
body = unicode(message.get_payload(decode=True), charset)
return body
Thanks very much for the help!