We Keep Coding

Hex string type conversion in python

Since hours I try to solve this problem - hope anybody can help me. I parse a hex number out of the output of a program, which I run via Popen in Python. In the next step, this hex number is used as parameter for another call of the program via Popen. The problem is, that I am not able to pass the hex value to Popen, so that it works:
cmd = "./my_program"
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
response = p.stdout.read()
hexapattern = r'0x([0-9a-fA-F]+)'
hex_output = re.findall(hexapattern, str(response))[1] #e.g.: hex_string = 35fe9a30
hex_string = '\\x' + hex_output[6] + hex_output[7] + '\\x' + hex_output[4] + hex_output[5] + '\\x' + hex_output[2] + hex_output[3] + '\\x' + hex_output[0] + hex_output[1] #e.g.: hex_string = \x35\xfe\9a\x30
payload = '\x41\x41\x41' + hex_string
cmd = "echo -e -n " + payload + " | ./my_program"
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
response = p.stdout.read()
print(response)
The following line does not work the way it should. While the first part of the string is correctly interpreted (as 'AAA', ASCII character with number 41), the 'hex_string' is used in bash as '\x35\xfe\9a\x30'. It is not a problem, that some chars are not printable.
payload = '\x41\x41\x41' + hex_string
Output: AAA\x35\xfe\9a\x30
When I change the program to set the value manually to the variable (I don't want to do this), it works without any problems.
payload = '\x41\x41\x41' + '\x35\xfe\9a\x30'
Output: AAA[not printable char][not printable char][not printable char]
I already tried a lot of type conversions but failed.

ast.literal_eval is a way to make the string like if you typed it literally.
hex_output = "35fe9a30"
hex_string = '\\x' + hex_output[6] + hex_output[7] + '\\x' + hex_output[4] + hex_output[5] + '\\x' + hex_output[2] + hex_output[3] + '\\x' + hex_output[0] + hex_output[1] #e.g.: hex_string = \x35\xfe\9a\x30
payload = '\x41\x41\x41' + hex_string
import ast
result = ast.literal_eval('"{}"'.format(payload))
print('\x41\x41\x41' + '\x30\x9a\xfe\x35' == result)
prints True (note that hex_string is the reverted version of hex_output, which doesn't simplify the example...)
We just told ast.literal_eval to evaluate the string (hence the formatting with quotes) containing payload
There may be simpler solutions with codec, handling the whole data as bytes instead of str:
import codecs
print(b'\x41\x41\x41' + codecs.decode(hex_output.encode(),"hex"))
prints:
b'AAA5\xfe\x9a0'

Calling bash command inside Python returns error, but works in terminal

Here is the except of my code related to this:
def grd_commands(directory):
for filename in os.listdir(directory)[1:]:
print filename
new_filename = ''
first_letter = ''
second_letter = ''
bash_command = 'gmt grdinfo ' + filename + ' -I-'
print bash_command
coordinates = Popen(bash_command, stdout=PIPE, shell=True)
coordinates = coordinates.communicate()
latlong = re.findall(r'^\D*?([-+]?\d+)\D*?[-+]?\d+\D*?([-+]?\d+)', coordinates)
if '-' in latlong[1]:
first_letter = 'S'
else:
first_letter = 'N'
if '-' in latlong[0]:
second_letter = 'W'
else:
second_letter = 'E'
new_filename = first_letter + str(latlong[1]) + second_letter + str(latlong[0]) + '.grd'
Popen('gmt grdconvert ' + str(filename) + ' ' + new_filename, shell=True)
filenameis the name of the file that is is being passed to the function. When I run my code, I am receiving this error:
/bin/sh: gmt: command not found
Traceback (most recent call last):
File "/Users/student/Desktop/Code/grd_commands.py", line 38, in <module>
main()
File "/Users/student/Desktop/Code/grd_commands.py", line 10, in main
grd_commands(directory)
File "/Users/student/Desktop/Code/grd_commands.py", line 23, in grd_commands
latlong = re.findall(r'^\D*?([-+]?\d+)\D*?[-+]?\d+\D*?([-+]?\d+)', coordinates).split('\n')
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/re.py", line 177, in findall
return _compile(pattern, flags).findall(string)
TypeError: expected string or buffer
If I print out the string bash_command and try entering it into terminal it fully functions. Why doesn't it work when being called by my Python script?

The entire command line is being treated as a single command name. You need to either use shell=True to have the shell parse it as a command line:
coordinates = Popen(bash_command, stdout=PIPE, shell=True)
or preferably store the command name and its arguments as separate elements of a list:
bash_command = ['gmt', 'grdinfo', filename, '-I-']
coordinates = Popen(bash_command, stdout=PIPE)

Popen takes a list of arguments. There is a warning for using shell=True
Passing shell=True can be a security hazard if combined with untrusted input.
Try this:
from subprocess import Popen, PIPE
bash_command = 'gmt grdinfo ' + filename + ' -I-'
print(bash_command)
coordinates = Popen(bash_command.split(), stdout=PIPE)
print(coordinates.communicate()[0])
Ensure gmt is installed in a location specified by PATH in your /etc/environment file:
PATH=$PATH:/path/to/gmt
Alternatively, specify the path to gmt in bash_command:
bash_command = '/path/to/gmt grdinfo ' + filename + ' -I-'
You should be able to find the path with:
which gmt
As other people have suggested, an actual list would be the best approach instead of a string. Additionally, you must escape spaces with a '\' in order to actually access the file if there is a space in it.
for filename in os.listdir(directory)[1:]:
bash_command = ['gmt', 'grdinfo', filename.replace(" ", "\ "), '-I-']

'The filename, directory name, or volume label syntax is incorrect.\r\n', None

I am trying to have a Python script stage a file to a locally hosted Git:
gitExe = r"C:\Git\bin\git.exe"
gitdir = r' --git-dir="G:\QA\.git"'
worktree = r' --work-tree="G:\QA\"'
gitcmd = " add"
myCsv = "Reports.csv"
import subprocess
print gitdir
print gitcmd
print gitExe
cmd = 'C:\\Git\\bin\\git.exe --git-dir="G:\\QA\\.git" --work-tree="G:\\QA" add Reports.csv' ####WORKS####
#cmd = ['C:\\Git\\bin\\git.exe --git-dir="G:\\QA\\.git" --work-tree="G:\\QA" add Reports.csv'] ####DOESNT WORK####
#cmd = ['C:\\Git\\bin\\git.exe --git-dir="G:\\QA\\.git" --work-tree="G:\\QA" add Reports.csv'+""] ####DOESNT WORK####
#cmd = [gitExe + gitdir + worktree + gitcmd + myCsv] ####DOESNT WORK####
p = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
print cmd
print p.communicate()
When I pass the full command as a string (uncommendted line above) everything works fine but when I pass it as an array (the three commented out lines above) I get the following error message:
>>> ================================ RESTART ================================
>>>
--git-dir="G:\QA\.git"
add
C:\Git\bin\git.exe
['C:\\Git\\bin\\git.exe --git-dir="G:\\QA\\.git" --work-tree="G:\\QA\\" addReports.csv']
('The filename, directory name, or volume label syntax is incorrect.\r\n', None)
What gives?

When you pass the command as an array, each parameter must be a separate element.
cmd = ['C:\\Git\\bin\\git.exe', '--git-dir="G:\\QA\\.git"', '--work-tree="G:\\QA"',
'add', 'Reports.csv']
The first item in the array is the program name. In your case, python tried to make the entire command line into the program name.

Passing a string argument to a function - why isn't it working?

Trying to pass a string argument to a function, which will then be used as a variable throughout the function. For some reason, when I am trying to do this, it isn't working. What am I doing wrong?
import subprocess
def printerSetup(printer):
subprocess.call(r'Cscript c:/windows/System32/Printing_Admin_Scripts/en-US/Prnport.vbs -a -r "'printer'.print.web.com" -h "' + printer + '.print.web.com" -o raw')
if printer == 'saturn' or printer == 'jupiter' or printer == 'neptune':
subprocess.call(r'rundll32 printui.dll, PrintUIEntry /if /b "' + printer + '" /f w:\printers\toshibae3511\eng\est_c2.inf /r "' + printer + '.print.web.com" /m "TOSHIBA e-STUDIO Color PS3"')
if printer == 'mercury':
subprocess.call(r'rundll32 printui.dll, PrintUIEntry /if /b "' + printer + '" /f w:\printers\dell1720\drivers\print\dell1720\DKABJ740.inf /r "' + printer + '.print.web.com" /m "Dell Laser Printer 1720dn"')
printerSetup("neptune")
printerSetup("mercury")
Edited the program. After trying to run this new one, get this error:
C:\Python27\Projects\Printer Setup>c:\python27\python.exe saturn.py
File "saturn.py", line 3
subprocess.call(r'Cscript c:/windows/System32/Printing_Admin_Scripts/en-US/P
rnport.vbs -a -r "'printer'.print.web.com" -h "' + printer + '.print.web.c
om" -o raw')
^
SyntaxError: invalid syntax

You need to specify variable == value for each or statement, like so:
if printer == 'saturn' or printer == 'jupiter' or printer == 'neptune':
You also forgot the trailing colon on each if statement.
If you want to say "does this variable match this list of values?", the following might be cleaner for you:
if printer in ('saturn', 'jupiter', 'neptune'):
You also need to add variables to strings - you can't just place them adjacent:
'string' + variable + 'string'
# not
'string'variable'string'

How to print output using python?

When this .exe file runs it prints a screen full of information and I want to print a particular line out to the screen, here on line "6":
cmd = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
outputlist = outputstring.splitlines()
Output = outputlist[5]
print cmd
This works fine:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
This doesn't work:
cmd = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
I get an error saying Output isn't defined. But when I cut and paste:
outputstring = process.communicate()[0]
outputlist = outputstring.splitlines()
Output = outputlist[5]
before the cmd statement it tells me the process isn't defined. str(Output) should be what is printed on line 6 when the .exe is ran.

You're trying to append the result of a call into the call itself. You have to run the command once without the + str(Output) part to get the output in the first place.
Think about it this way. Let's say I was adding some numbers together.
z = 5 + b
b = z + 2
I have to define either z or b before the statements, depending on the order of the two statements. I can't use a variable before I know what it is. You're doing the same thing, using the Output variable before you define it.

It's not supposed to be a "dance" to move things around. It's a matter of what's on the left side of the "=". If it's on the left side, it's getting created; if it's on the right side it's being used.
As it is, your example can't work even a little bit because line one wants part of output, which isn't created until the end.
The easiest way to understand this is to work backwards. You want to see as the final result?
print output[5]
Right? So to get there, you have to get this from a larger string, right?
output= outputstring.splitlines()
print output[5]
So where did outputstring come from? It was from some subprocess.
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]
So where did process come from? It was created by subprocess Popen
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]
So where did cmd come from? I can't tell. Your example doesn't make sense on what command is being executed.
cmd = ?
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]

Just change your first line to:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
and the print statement at the end to:
print cmd + str(Output)
This is without knowing exactly what it is you want to print...
It -seems- as if your problem is trying to use Output before you actually define what the Output variable is (as the posts above)

Like you said, a variable has to be declared before you can use it. Therefore when you call str(Output) ABOVE Output = outputlist[5], Output doesn't exist yet. You need the actually call first:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
then you can print the output of that command:
cmd_return = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
should be the line directly above print cmd_return.