We Keep Coding

Hex string type conversion in python

Since hours I try to solve this problem - hope anybody can help me. I parse a hex number out of the output of a program, which I run via Popen in Python. In the next step, this hex number is used as parameter for another call of the program via Popen. The problem is, that I am not able to pass the hex value to Popen, so that it works:
cmd = "./my_program"
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
response = p.stdout.read()
hexapattern = r'0x([0-9a-fA-F]+)'
hex_output = re.findall(hexapattern, str(response))[1] #e.g.: hex_string = 35fe9a30
hex_string = '\\x' + hex_output[6] + hex_output[7] + '\\x' + hex_output[4] + hex_output[5] + '\\x' + hex_output[2] + hex_output[3] + '\\x' + hex_output[0] + hex_output[1] #e.g.: hex_string = \x35\xfe\9a\x30
payload = '\x41\x41\x41' + hex_string
cmd = "echo -e -n " + payload + " | ./my_program"
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
response = p.stdout.read()
print(response)
The following line does not work the way it should. While the first part of the string is correctly interpreted (as 'AAA', ASCII character with number 41), the 'hex_string' is used in bash as '\x35\xfe\9a\x30'. It is not a problem, that some chars are not printable.
payload = '\x41\x41\x41' + hex_string
Output: AAA\x35\xfe\9a\x30
When I change the program to set the value manually to the variable (I don't want to do this), it works without any problems.
payload = '\x41\x41\x41' + '\x35\xfe\9a\x30'
Output: AAA[not printable char][not printable char][not printable char]
I already tried a lot of type conversions but failed.

ast.literal_eval is a way to make the string like if you typed it literally.
hex_output = "35fe9a30"
hex_string = '\\x' + hex_output[6] + hex_output[7] + '\\x' + hex_output[4] + hex_output[5] + '\\x' + hex_output[2] + hex_output[3] + '\\x' + hex_output[0] + hex_output[1] #e.g.: hex_string = \x35\xfe\9a\x30
payload = '\x41\x41\x41' + hex_string
import ast
result = ast.literal_eval('"{}"'.format(payload))
print('\x41\x41\x41' + '\x30\x9a\xfe\x35' == result)
prints True (note that hex_string is the reverted version of hex_output, which doesn't simplify the example...)
We just told ast.literal_eval to evaluate the string (hence the formatting with quotes) containing payload
There may be simpler solutions with codec, handling the whole data as bytes instead of str:
import codecs
print(b'\x41\x41\x41' + codecs.decode(hex_output.encode(),"hex"))
prints:
b'AAA5\xfe\x9a0'

crontab does not run my python script

My python script is not running under my Crontab. But when i try to run it from the Terminal it works perfectly. I have placed this in the python script at the top:
#!/usr/bin/python
Also I tried:
#!/usr/bin/env python
I did my file executable:
chmod a+x vida.py
Added to my crontab and added PATH:
USER=gg
SHELL=/bin/sh
PATH=/usr/local/sbin/:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin/:/home/gg/DC.bin/:/home/gg/GNSSMET/DC/:usr/bin:/usr/bin/X11:/:/home/gg/GNSSMET/DC/bin/:/home/gg/:/usr/lib/python2.7/:
PYTHONPATH=/usr/bin/:/usr/lib/python2.7/
*/1 * * * * gg /usr/bin/python /home/gg/vida.py 2>&1 >>/home/gg/out1.txt
I checked the log via grep CRON /var/log/syslog
Jan 19 13:37:01 gg-pc CRON[26500]: (gg) CMD ( /usr/bin/python /home/gg/vida.py 2>&1 >>/home/gg/out1.txt)
I even run a dummy python script from using crontab and it worked like charm (simple Hello, World!). But when it comes to my script the output file out1.txt is created (which is empty) but does not run the actual script. I even checked all of the solutions presented on StackOverflow, none did work.
So here is my python script:
#!/usr/bin/env python
from datetime import *
import os
import sys
gamitRinexDir = '/home/gg/GAMIT/rinex'
stalist = ['ankr','argi','aut1','beug','brst','bucu','busk','ganm','gism','glsv','gmlk','gope','hofn','ingl','ista','joze',
'kiru','krcy','ktvl','mas1','mate','mets','mkps','morp','nico','onsa','orhn','orid','pdel','penc','polv','pots','puyv',
'sofi','vis0','vlns','wtzr','yebe','zeck','zimm']
letlist = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X']
seslist = ['0','1','2','3','4','5','6','7','8','9']
tnow = datetime.now()
dback = timedelta(hours=2)
tnow = tnow -dback
wlength = 4
os.system('rm ' + gamitRinexDir + '/*')
wlett = []
updir = []
doylist = []
yrlist = []
for i in range(wlength):
delta = timedelta(hours=i+1)
tback = tnow -delta
wlett.append(letlist[tback.hour])
doystr = 'doy ' + str(tnow.year) + ' ' + str(tnow.month) + ' ' + str(tnow.day) + ' ' + '> /home/gg/sil.sil'
os.system(doystr)
fid = open('/home/gg/sil.sil')
line = fid.readline().split()
doynum = '%03d' % (int(line[5]))
x = str(tnow.year)
yrnum = x[2:4]
updir.append(yrnum+doynum)
doylist.append(doynum)
yrlist.append(yrnum)
dirname = '/home/gg/REPO/nrtdata/'
for i in range(len(wlett)):
adirname = dirname + updir[i]+'/' + wlett[i]
for sta in stalist:
fname = adirname + '/' + sta + doylist[i] + wlett[i].lower() + '.' + yrlist[i]+'d.Z'
fname2 = gamitRinexDir + '/' + sta + doylist[i] + seslist[i] + '.' + yrlist[i]+'d.Z'
os.system('cp ' + fname + ' ' + fname2)
udoy = list(set(doylist))
dcmd = ''
for gun in udoy:
dcmd = dcmd + gun + ' '
CmdGamit = 'sh_gamit -d ' + x + ' ' + dcmd + ' ' + '-orbit IGSU -expt expt -eops usnd -gnss G -nopngs -metutil Z'
print(CmdGamit)
mainCmd = 'cd /home/gg/GAMIT/;'+CmdGamit
os.system(mainCmd)
filestocopy1 = 'met_*'
filestocopy2 = 'hexpta.*'
filestocopy3 = 'uexpt*'
ndirname = ' /home/gg/REPO_GAMIT/' + doynum + '_'+ wlett[-1]
os.system('mkdir ' + ndirname)
cleancmd1 = 'mv /home/gg/GAMIT/'+doynum +'/'+filestocopy1 + ' ' + ndirname
cleancmd2 = 'mv /home/gg/GAMIT/'+doynum +'/'+filestocopy2 + ' ' + ndirname
cleancmd3 = 'mv /home/gg/GAMIT/'+doynum +'/'+filestocopy3 + ' ' + ndirname
cleancmd4 = 'rm -r /home/gg/GAMIT/'+doynum
os.system(cleancmd1)
os.system(cleancmd2)
os.system(cleancmd3)
os.system(cleancmd4)
Please show me some pointers, I am seriously stuck here.

You should change you crontab line as such to get stdout and stderr saved to the file:
*/1 * * * * gg /usr/bin/python /home/gg/vida.py >> /home/gg/out1.txt 2>&1
Simply read out1.txt after crontab has run the line to see what's wrong
Edit after your comment:
Based on the error you've shared, I believe you're not actually writing anything in the /home/gg/sil.sil file:
doystr = 'doy ' + str(tnow.year) + ' ' + str(tnow.month) + ' ' + str(tnow.day) + ' ' + '> /home/gg/sil.sil'
os.system(doystr)
doystr does not evaluate to a shell command, I think you need to write the variable as below to write to the file.
doystr = 'echo "doy ' + str(tnow.year) + ' ' + str(tnow.month) + ' ' + str(tnow.day) + '" ' + '> /home/gg/sil.sil'

syntax:
minutes hour dom mon dow user command
55 16 * * * root /root/anaconda/bin/python /root/path/file_name.py &>> /root/output/output.log

python subprocess continuously feed stdin and read stdout which is possibly empty

I am trying to write a python script that optimises the parameters of a compiled c script that I wrote. The program (dups) is taking an input (using fgets) and prints the input if it is not in the set of previously seen inputs.
My python script should continuously feed input to dups followed by an evaluation of it's output (which can be either the input or nothing).
I am using subprocess writing to its stdin and reading from its stdout. However, using various solutions, the script hangs after reading the first line.
The C program only writes to stdout and I know that it is flushing. Furthermore I am able to reproduce my error using a python script simulating dups.
Sample code:
Sample dups for easy reproduction:
#!/usr/bin/python
import sys
names = set()
while True:
try:
x = raw_input()
except EOFError:
print "Unexpected EOF, quitting"
break
if x not in names:
print x
sys.stdout.flush()
names.add(x)
Main script:
import subprocess
import pty, os #for solution 4
inputs = ['Alice', 'Alice', 'Bob', 'Jane', 'Bob', 'Jane', 'Alice', 'Mike']
solution = 4
arguments = ['./sample_script.py']
popen_nargs = dict(stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
universal_newlines=True,
)
if solution == 3:
out_fname = "output.txt"
f_w = open(out_fname,"wt")
f_r = open(out_fname, "rt")
popen_nargs['stdout'] = f_w
popen_nargs['bufsize']=0
if solution == 4:
master, slave = pty.openpty()
popen_nargs['stdout'] = slave
stdout_handle = os.fdopen(master)
child = subprocess.Popen(arguments, **popen_nargs)
for inpt in inputs:
print "Testing '" + inpt + "'"
child.stdin.write(inpt +" \n")
child.stdin.flush()
print "Result:"
#http://stackoverflow.com/questions/4417546/constantly-print-subprocess-output-while-process-is-running
if solution == 0:
for stdout_line in iter(child.stdout.readline, ""):
print "-> '", stdout_line, "'"
#http://blog.endpoint.com/2015/01/getting-realtime-output-using-python.html
elif solution == 1:
while True:
output = child.stdout.readline()
if output == '' and child.poll() is not None:
break
if output:
print "-> '", output, "'"
#https://gist.github.com/zhenyi2697/7229421
elif solution == 2:
output=''
while True:
result = child.poll()
delta = child.stdout.read(1)
if result is not None:
print 'terminated'
break
if delta != ' ':
output = output + delta
else:
if '%' in output:
print 'percentage is:'
print output
elif '/s' in output:
print 'speed is:'
print output
print "-> '", output, "'"
output = ''
#http://stackoverflow.com/questions/5419888/reading-from-a-frequently-updated-file
elif solution == 3:
f_w.flush()
print "-> '", f_r.read(), "'"
print "end iteration"
#http://stackoverflow.com/questions/13603334/repeatedly-write-to-stdin-and-read-stdout-of-a-subprocess-without-closing-it
if solution == 4:
print "-> '", stdout_handle.readline(), "'"
if solution == 3:
f_w.close()
f_r.close()
# Close the program
child.communicate()
Output:
Solutions 0, 1 (don't terminate):
Testing 'Alice'
Result:
-> ' Alice
'
Solution 2 (doest terminate):
Testing 'Alice'
Result:
-> ' Alice '
Solution 3 (output.txt seems to be updated only in the end, despite bufsize=0):
Testing 'Alice'
Result:
-> ' '
end iteration
Testing 'Alice'
Result:
-> ' '
end iteration
...
Testing 'Mike'
Result:
-> ' '
end iteration
Solution 4 (seems promising, but read and readline do not work when there is no output hence it does not terminate):
Testing 'Alice'
Result:
-> ' Alice
'
end iteration
Testing 'Alice'
Result:
-> '

Python 2.7 IndexError: list index out of range, converting file

I have a code for convert Jmeter JTL FILE TO CSV, but when I run the code, I have the following error: IndexError: list index out of range in line 32
This is the code
import sys
import re
import datetime
import time
startTime = time.time()
cnt = 0
cnt2 = 0
failCnt = 0
reCompile = re.compile("\s([^\s]*?)=\"(.*?)\"")
delimiterCharacterOut = ","
def writeCSVLine(line):
x = reCompile.findall(line)
a = dict((row[0], row[1]) for row in x)
try:
a['ts1'] = str(int(int(a['ts'])/1000))
x = str(datetime.datetime.fromtimestamp(float(a['ts1'])))[0:19]
b = a['ts'] + ",\"" + x + "\"," + a['t'] + "," + a['lt'] + ",\"" + a['s'] + "\",\"" + a['lb'] + "\"," + a['rc'] + ",\"" + a['rm'] + "\",\"" + a['tn'] + "\",\"" + a['dt'] + "\"," + a['by'] + ",\"" + a['sc'] + "\"," + a['ec'] + ",\"" + a['ng'] + "\"," + a['na'] + ",\"" + a['hn'] + "\"," + a['in'] + "\n"
except:
return -1
o.write(b)
return 1
print "Splitting JTL file"
try:
runArgv = sys.argv #Save the command line
jtlInfile = str(sys.argv[1]) #Name of JTL input file
cvsOutfile = str(sys.argv[2]) # Name of CVS output file
reFilter = str(sys.argv[3]) # Filter the labels (lb) for the filter
except:
print "Error: Input format: <input file> <output file> <Filter by regular expression>"
raise
try:
f = open(jtlInfile, "r")
o = open(cvsOutfile, "w")
except:
raise
print "Filtering on regular expression : " + reFilter
cmpFilter = re.compile(reFilter)
# o.write("timestamp" + ",\""+ "datetime" + "\n")
o.write("timeStamp" + ",\"" + "datetime" + "\"," + "elapsed" + "," + "Latency" + ",\"" + "success" + "\",\"" + "label" + "\"," + "responseCode" + ",\"" + "responseMessage" + "\",\"" + "threadName"+ "\",\"" + "dataType" + "\"," + "bytes" + ",\"" + "SampleCount" + "\"," + "ErrorCount" + ",\"" + "grpThreads" + "\"," + "allThreads" + ",\"" + "Hostname" + "\"," + "IdleTime" + "\n")
for line in f:
try:
if cmpFilter.search(line):
returnVal = writeCSVLine(line)
if returnVal<0:
failCnt += 1
else:
cnt2 += 1
except:
print 'Error in line : ', cnt, line
raise
cnt += 1
endTime = time.time()
print "Time taken : ", str(endTime-startTime)
print "Lines processed : ", cnt
print "Lines that passed the filter : ", cnt2
print "Lines skipped (error?) : ", failCnt
f.close()
o.close()
Log de CMD
The base tutorial is in : http://balasoftwaretesting.blogspot.com/2012/03/converting-jmeter-jtl-file-to-csv-file.html?spref=bl

From the sys.argv docs, sys.argv is the list of command line arguments passed to a Python script.
Your command line log shows that you ran python JtltoCsv_Jmeter.py, which would result in an empty list for sys.argv. The tutorial provides a jtl file as an argument to JtltoCsv_Jmeter.py:
JtltoCsv_Jmeter.py C:\JtlToCsvConverter\input\sample.jtl
So it looks like maybe an error in copy/paste :)

Looking into the script, you need to pass 3 command line arguments:
Source JTL file
Target CSV file
Regular expression filter
So you need to execute the script like:
JtltoCsv_Jmeter.py example.jtl example.csv "(.*)"
Also there is an option to switch JMeter's results output format to CSV, in order to do so use one of the following steps:
Add jmeter.save.saveservice.output_format=csv line to user.properties file (lives under /bin folder of your JMeter installation)
Pass the property value via -J command line argument as:
jmeter -Jjmeter.save.saveservice.output_format=csv
See Apache JMeter Properties Customization Guide for more information on JMeter properties and ways of passing, setting and overriding them.

How to print output using python?

When this .exe file runs it prints a screen full of information and I want to print a particular line out to the screen, here on line "6":
cmd = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
outputlist = outputstring.splitlines()
Output = outputlist[5]
print cmd
This works fine:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
This doesn't work:
cmd = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
I get an error saying Output isn't defined. But when I cut and paste:
outputstring = process.communicate()[0]
outputlist = outputstring.splitlines()
Output = outputlist[5]
before the cmd statement it tells me the process isn't defined. str(Output) should be what is printed on line 6 when the .exe is ran.

You're trying to append the result of a call into the call itself. You have to run the command once without the + str(Output) part to get the output in the first place.
Think about it this way. Let's say I was adding some numbers together.
z = 5 + b
b = z + 2
I have to define either z or b before the statements, depending on the order of the two statements. I can't use a variable before I know what it is. You're doing the same thing, using the Output variable before you define it.

It's not supposed to be a "dance" to move things around. It's a matter of what's on the left side of the "=". If it's on the left side, it's getting created; if it's on the right side it's being used.
As it is, your example can't work even a little bit because line one wants part of output, which isn't created until the end.
The easiest way to understand this is to work backwards. You want to see as the final result?
print output[5]
Right? So to get there, you have to get this from a larger string, right?
output= outputstring.splitlines()
print output[5]
So where did outputstring come from? It was from some subprocess.
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]
So where did process come from? It was created by subprocess Popen
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]
So where did cmd come from? I can't tell. Your example doesn't make sense on what command is being executed.
cmd = ?
process = Popen(cmd, shell=True, stderr=STDOUT, stdout=PIPE)
outputstring = process.communicate()[0]
output= outputstring.splitlines()
print output[5]

Just change your first line to:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
and the print statement at the end to:
print cmd + str(Output)
This is without knowing exactly what it is you want to print...
It -seems- as if your problem is trying to use Output before you actually define what the Output variable is (as the posts above)

Like you said, a variable has to be declared before you can use it. Therefore when you call str(Output) ABOVE Output = outputlist[5], Output doesn't exist yet. You need the actually call first:
cmd = ' -a ' + str(a) + ' -b ' + str(b)
then you can print the output of that command:
cmd_return = ' -a ' + str(a) + ' -b ' + str(b) + str(Output)
should be the line directly above print cmd_return.