We Keep Coding

Can't get working command line on prompt to work on subprocess

I need to extract text from a PDF. I tried the PyPDF2, but the textExtract method returned an encrypted text, even though the pdf is not encrypted acoording to the isEncrypted method.
So I moved on to trying accessing a program that does the job from the command prompt, so I could call it from python with the subprocess module. I found this program called textExtract, which did the job I wanted with the following command line on cmd:
"textextract.exe" "download.pdf" /to "download.txt"
However, when I tried running it with subprocess I couldn't get a 0 return code.
Here is the code I tried:
textextract = shlex.split(r'"textextract.exe" "download.pdf" /to "download.txt"')
subprocess.run(textextract)
I already tried it with shell=True, but it didn't work.
Can anyone help me?

I was able to get the following script to work from the command line after installing the PDF2Text Pilot application you're trying to use:
import shlex
import subprocess
args = shlex.split(r'"textextract.exe" "download.pdf" /to "download.txt"')
print('args:', args)
subprocess.run(args)
Sample screen output of running it from a command line session:
> C:\Python3\python run-textextract.py
args: ['textextract.exe', 'download.pdf', '/to', 'download.txt']
Progress:
Text from "download.pdf" has been successfully extracted...
Text extraction has been completed!
The above output was generated using Python 3.7.0.
I don't know if your use of spyder on anaconda affects things or not since I'm not familiar with it/them. If you continue to have problems with this, then, if it's possible, I suggest you see if you can get things working directly—i.e. running the the Python interpreter on the script manually from the command line similar to what's shown above. If that works, but using spyder doesn't, then you'll at least know the cause of the problem.

There's no need to build a string of quoted strings and then parse that back out to a list of strings. Just create a list and pass that:
command=["textextract.exe", "download.pdf", "/to", "download.txt"]
subprocess.run(command)
All that shlex.split is doing is creating a list by removing all of the quotes you had to add when creating the string in the first place. That's an extra step that provides no value over just creating the list yourself.

Allure not able to read output.xml file generated by AllureReportLibrary in Robot Framework

I am using Ride (RobotFramework IDE) and I have imported Library AllureReportLibrary in my project.
Using the Set Output Dir, I am creating a Directory C:/AutomationLogs/Allure and all the allure properties and xml files are getting generated in that path.
Set Output Dir C:/AutomationLogs/
Then I am using the "allure serve C:\AutomationLogs\Allure" command to try and generate the html report file in command prompt, but it shows the below error -
"Could not read result
C:\AutomationLogs\Allure\f56f4796-d30a-47f3-a988-d17f6c4e13ca-testsuite.xml:
{} com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot
deserialize va lue of type
ru.yandex.qatools.allure.model.SeverityLevel from String "None":
value not one of declared Enum instance names: [trivial, blocker,
minor, normal, critical]"
The xml file "f56f4796-d30a-47f3-a988-d17f6c4e13ca-testsuite.xml" was generated using the AllureReportLibrary
Also the index.html file which is generated after the command opens after this command and shows Allure Report unknown
unknown - unknown (Unknown) 0 test cases NaN%
I am using the below -
Allure version - 2.4.1
Ride version - RIDE 1.5.2.1 running on Python 2.7.12.
I am new to Robot Framework and Allure. Please let me know whether I have implemented it correctly and why I am facing the above error.
-Ryan M

I'm using the 1.1.1 version of Allure Adaptor for Robot Framework and the severity is picked from the test case tags and added as a label under the test-case element of the report.
However, it seems that Allure 2.6.0 is also expecting a valid value for the severity attribute of the test-case element.
In order to use Allure2 with the current reports I have altered AllureListener.py to also add the severity to the test case:
elif tag in SEVERITIES:
test.severity = tag
test.labels.append(TestLabel(
name='severity',
value=tag
))

If your output.xml has severity = None for any testcase then the allure-robotframework-adaptor will give the error that you have mentioned. Creating TestCase() object with severity='' in start_suitesetup method of AllureListener.py will do the trick.
def start_suitesetup(self, name, attributes):
....
....
test = TestCase(name=name,
description=description,
start=now(),
attachments=[],
labels=[],
parameters=[],
steps=[],
severity='')

How to create the Allure reports in Robot Framework ?
Initially, Download the Command line and UNzip the file and save the path of the bin folder in environment.
Link : http://repo.maven.apache.org/maven2/io/qameta/allure/allure-commandline/2.8.0/allure-commandline-2.8.0.zip
Unzip the above file then put it in the Environment folder.
Then Pip install the below modules
pip install allure-robotframework
pip install robotframework-allurereport
In robot file, Add the Library in Settings like,
Example :
Library AllureReportLibrary D:\eclipse\RobotFramework\results
Then Use the Below commands to run the robot code.
robot --listener allure_robotframework;D:\eclipse\RobotFramework\results
Example.txt
Finally,
Generate the HTML file by,
allure generate D:\eclipse\RobotFramework\results
Note : Use the same path what you used in the previous command to generate the HTml.file.
and
Open in Mozhila FireFox. It wont be work in Chrome. I dont know exactly why.
Regards,
Vijay

Serial Numbers from a Storage Controller over SSH

Background
I'm working on a bash script to pull serial numbers and part numbers from all the devices in a server rack, my goal is to be able to run a single script (inventory.sh) and walk away while it generates text files containing the information I need. I'm using bash for maximum compatibility, the RHEL 6.7 systems do have Perl and Python installed, however they have minimal libraries. So far I haven't had to use anything other than bash, but I'm not against calling a Perl or Python script from my bash script.
My Problem
I need to retrieve the Serial Numbers and Part numbers from the drives in a Dot Hill Systems AssuredSAN 3824, as well as the Serial numbers from the equipment inside. The only way I have found to get all the information I need is to connect over SSH and run the following three commands dumping the output to a local file:
show controllers
show frus
show disks
Limitations:
I don't have "sshpass" installed, and would prefer not to install it.
The Controller is not capable of storing SSH keys ( no option in custom shell).
The Controller also cannot write or transfer local files.
The Rack does NOT have access to the Internet.
I looked at paramiko, but while Python is installed I do not have pip.
I also cannot use CPAN.
For what its worth, the output comes back in XML format. (I've already written the code to parse it in bash)
Right now I think my best option would be to have a library for Python or Perl in the folder with my other scripts, and write a script to dump the commands' output to files that I can parse with my bash script. Which language is easier to just provide a library in a file? I'm looking for a library that is as small and simple as possible to use. I just need a way to get the output of those commands to XML files. Right now I am just using ssh 3 times in my script and having to enter the password each time.

Have a look at SNMP. There is a reasonable chance that you can use SNMP tools to remotely extract the information you need. The manufacturer should be able to provide you with the MIBs.

I ended up contacting the Manufacturer and asking my question. They said that the system isn't setup for connecting without a password, and their SNMP is very basic and won't provide the information I need. They said to connect to the system with FTP and use "get logs " to download an archive of the configuration and logs. Not exactly ideal as it takes 4 minutes just to run that one command but it seems to be my only option. Below is the script I wrote to retrieve the file automatically by adding the login credentials to the .netrc file. This works on RHEL 6.7:
#!/bin/bash
#Retrieve the logs and configuration from a Dot Hill Systems AssuredSAN 3824 automatically.
#Modify "LINE" and "HOST" to fit your configuration.
LINE='machine <IP> login manage password <password>'
HOST='<IP>'
AUTOLOGIN="/root/.netrc"
FILE='logfiles.zip'
#Check for and verify the autologin file
if [ -f $AUTOLOGIN ]; then
printf "Found auto-login file, checking for proper entry... \r"
READLINE=`cat $AUTOLOGIN | grep "$LINE"`
#Append the line to the end of .netrc if file exists but not the line.
if [ "$LINE" != "$READLINE" ]; then
printf "Proper entry not found, creating it... \r"
echo "$LINE" >> "$AUTOLOGIN"
else
printf "Proper entry found... \r"
fi
#Create the Autologin file if it doesn't exist
else
printf "Auto-Login file does not exist, creating it and setting permissions...\r"
echo "$LINE" > "$AUTOLOGIN"
chmod 600 "$AUTOLOGIN"
fi
#Start getting the information from the controller. (This takes a VERY long time)
printf "Retrieving Storage Controller data, this will take awhile... \r"
ftp $HOST << SCRIPT
get logs $FILE
SCRIPT
exit 0
This gave me a bunch of files in the zip, but all I needed was the "store_....logs" file. It was about 500,000 lines long, the first portion is the entire configuration in XML format, then the configuration in text format, followed by the logs from the system. I parsed the file and stripped off the logs at the end which cut the file down to 15,000 lines. From there I divided it into two files (config.xml and config.txt). I then pulled the XML output of the 3 commands that I needed and it to the 3 files my previously written script searches for. Now my inventory script pulls in everything it needs, albeit pretty slow due to waiting 4 minutes for the system to generate the zip file. I hope this helps someone in the future.
Edit:
Waiting 4 minutes for the system to compile was taking too long. So I ended up using paramiko and python scripts to dump output from the commands to files that my other code can parse. It accepts the IP of the Controller as a parameter. Here is the script for those interested. Thank you again for all the help.
#!/usr/bin/env python
#Saves output of "show disks" from the storage Controller to an XML file.
import paramiko
import sys
import re
import xmltodict
IP = sys.argv[1]
USERNAME = "manage"
PASSWORD = "password"
FILENAME = "./logfiles/disks.xml"
cmd = "show disks"
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
client.connect(IP,username=USERNAME,password=PASSWORD)
stdin, stdout, stderr = client.exec_command(cmd)
except Exception as e:
sys.exit(1)
data = ""
for line in stdout:
if re.search('#', line):
pass
else:
data += line
client.close()
f = open(FILENAME, 'w+')
f.write(data)
f.close()
sys.exit(0)

Installing a CGI Proxy on Tomcat

I want to setup a proxy for openlayers to use so I followed these steps:
Downloaded the proxy.cgi file from the OpenLayers web site:
http://trac.osgeo.org/openlayers/browser/trunk/openlayers/examples/proxy.cgi
Modify the proxy.cgi file to include my domain in the allowedHosts list:
allowedHosts = ['localhost:6901']
Copy the proxy.cgi file to the following folder:
$TOMCAT_PATH$/webapps/yourApp/WEB-INF/cgi/
Modify the file web.xml of your web app by adding the sections below. You find the file at
$TOMCAT_PATH$/webapps/yourApp/WEB-INF/web.xml
Comment: In case the web.xml file doesn’t exist for your webapp, just create it yourself or copy it from another webapp and modify it. (created!)
Comment: the “param-value” for the “executable” parameter has to contain the path to your Python installation. (it does!)
<servlet>
<servlet-name>cgi</servlet-name>
<servlet-class>org.apache.catalina.servlets.CGIServlet</servlet-class>
<init-param>
<param-name>debug</param-name>
<param-value>0</param-value>
</init-param>
<init-param>
<param-name>cgiPathPrefix</param-name>
<param-value>WEB-INF/cgi</param-value>
</init-param>
<init-param>
<param-name>executable</param-name>
<param-value>c:\python25\python.exe</param-value>
</init-param>
<init-param>
<param-name>passShellEnvironment</param-name>
<param-value>true</param-value>
</init-param>
<load-on-startup>5</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>cgi</servlet-name>
<url-pattern>/cgi-bin/*</url-pattern>
</servlet-mapping>
Modify the file context.xml of your web app by adding the element below. You find the file at $TOMCAT_PATH$/webapps/yourApp/META-INF/context.xml
Restart Tomcat
To use the proxy with OpenLayers, just include this single line into your code:
OpenLayers.ProxyHost = "/yourWebApp/cgi-bin/proxy.cgi?url=";
But when I try to use it like:
/webappname/cgi-bin/proxy.cgi?url=labs.metacarta.com
I get this error:
Some unexpected error occurred. Error text was: list index out of range
I think its related with os.environ["REQUEST_METHOD"] but I dont know how its related ..

You're asking for an environment variable that isn't defined.
You need to either catch and handle the exception or use os.environ.get:
try:
methodq = os.environ["REQUEST_METHOD"]
except KeyError:
methodq = "default value"
Or:
methodq = os.environ.get("REQUEST_METHOD", "default value")

You're submitting :
/webappname/cgi-bin/proxy.cgi?url=labs.metacarta.com
But the proxy.cgi script is trying to do this:
host = url.split("/")[2]
Try http://labs.metacarta.com for your url so proxy.cgi has some slashes to split on, or modify it to parse the url a little smarter.

The answer is: You don't install or use cgi proxy on Tomcat.
cgi is for apache server or IIS that are used as a front-end server. Tomcat may sit behind it. The configuration of Apache is detailed in: http://tomcat.apache.org/tomcat-6.0-doc/proxy-howto.html
Be warned that OpenLayers warns that its proxy.cgi is only an example and may not have good enough check to stop it from being exploited, i.e. it may run some malicious script.
If you are serving your OpenLayers client page on Tomcat alone and it contains layers from other GeoServer or Mapserver, you can use proxy servlet and specify it as:
OpenLayers.ProxyHost = "sevlet URL on the server that served this page";
http://wiki.apache.org/tomcat/ServletProxy
https://svn.openxdata.org/Experimental/openXmapper/trunk/gwt-openlayers-server/src/main/java/org/gwtopenmaps/openlayers/server/GwtOpenLayersProxyServlet.java
There are reverse proxy or rewrite sevlet solutions to it, too. Please Google on these.

automation : Script to take a mysqldump into a file named by date/time of backup

I tried fabric with a '>' in the command string. It always gives out an error code 2. Currently dabbling with subprocess.call, subprocess.check_output and keeping stdout="filesocket". Not working. The only thing that gets written in the file is the USAGE for mysqldump. Using shlex to parse 'mysqldump -uroot -ppassword database table1 table2'
All this because I don't know shell scripting with string variables from the 'date' utility. How do I take the current date and use it to name the backup file in shell script. OR how do I get this thing done in python?
Thanks in advance.
regards.

You can get a custom date out of date using the following syntax.
CUSTOM_DATE=$(date "+%Y-%m-%d_%H_%M_%S")
The easiest way to accomplish this is to put a script on the remote end that does 'everything'
#!/bin/bash
CUSTOM_DATE=$(date "+%Y-%m-%d_%H_%M_%S")
mysqldump -u admin -p password database table1 table2 >/path/to/backups/mysqldump.${CUSTOM_DATE}.db

"How do I take the current date and use it to name the backup file in shell script. OR how do I get this thing done in python?"
from datetime import datetime
filename = 'mysql_backup_{0:%Y%m%d_%H%M}.sql'.format(datetime.now())
# filename == 'mysql_backup_20120227_0952.sql'

My Answer from related stackoverflow post.
In Microsoft Windows, run below command in CMD
mysqldump -u USERNAME -pYOURPASSWORD --all-databases > "C:/mysql_backup_%date:~-10,2%-%date:~-7,2%-%date:~-4,4%-%time:~0,2%_%time:~3,2%_%time:~6,2%.sql"
Output file will look like,
mysql_backup_21-02-2015-13_07_18.sql
If you want to automate the backup process, then you can use Windows Task Scheduler, and put above command in .bat file - task scheduler will run the .bat file at specified interval.