Condor quoting hell - python

I'm having trouble with passing a string representation of a dict to be read in by json.loads in Python through Condor.
On the command line I would type:
python script.py arg1 "{'x':50,'y':'a'}"
After much struggle with the docs, I understand Condor wants in the submit file:
arguments = "arg1 '""{''x'':50,''y'':''a''}""'"
However it is failing and claiming quadruple single quotes around the dict strings in the log, eg:
''''x'''':50
Is there any way to manipulate the quotes to get it to run as expected?

Related

Is there a way to convert a python dictionary string with some values as python commands?

How to convert this to a dictionary?
params = "{'cat_features':[X.columns.get_loc(i) for i in cat_vars],
'num_boost_round':100, 'eta':.01, 'reg_lambda':1.8, 'verbose':False,
'loss_function':'MultiClass','early_stopping_rounds':5}"
without the first part [X.columns.get_loc(i) for i in cat_vars] I can run ast.literal_eval(), but that doesn't work if there is python code in the string. Any idea how to solve this?
You can use plain eval.
However, using eval is is risky if the string comes from an non-trusted source because a properly crafted string could execute anything on the computer where the program is running.

Python3 doesnt detect tuple

I have this input:
python script.py --key '("music","aaa")' --date '("01/01/1990",0,0)'
And I do:
constrain = literal_eval(sys.argv[2])
print(type(constrain))
print(type(sys.argv[4]))
And all outputs are str while they should be tuples. The input cannot be changed!
You're command line should work perfectly from MSYS or Linux, but here you're running it from windows shell.
Windows shell doesn't treat simple quotes as syntatcic. They're passed literally to your python code. To top it all, the double quotes are removed which makes your second arg '(music,aaa)' when passed to python: no way you can literal_eval that. So your input string has to be changed (or your operating system :))
Do this to call your code:
python script.py --key "(""music"",""aaa"")" --date "(""01/01/1990"",0,0)"
You have to quote the arguments and double the quotes in the arguments. And use double quotes exclusively.
EDIT: or even better (would work on both Linux and Windows): use simple quotes inside your arguments, double quotes outside (literal_eval is not json: it understands both simple & double quotes!):
python script.py --key "('music','aaa')" --date "('01/01/1990',0,0)"
now I'm getting:
<type 'tuple'>
<type 'str'>
(you get str because the 4th argument is always a string, you probably forgot to literal_eval it)
and print(constrain) yields:
('music', 'aaa')
(so it's not a python issue, rather a CMD issue)

Python: Command line arguments not read?

I'm trying to read command line arguments in python in the form:
python myprogram.py string string string
I have tried using sys.argv[1-3] to get each string, but when I have a string such as $unny-Day, it does not process the entire string. How can I process strings like these entirely?
Are you using a shell? $ is a special character in the shell that is interpreted as a shell variable. Since the variable does not exist, it is textually substituted with an empty string.
Try using single quotes around your parameter, like > python myapp.py '$unny-Day'.

The meaning of the following bash script

I have a quick question. I got the following bash script from my friend, but I don't know what /inline/b64/ is, and how the following code segment works.
I have some experience with bash, and Python, but I cannot understand the following code fragment at all. Could anyone please give me some enlightenment?
More specifically,
1) What does /inline/b64 mean? I did some search on the web, but I couldn't find any clues.
2) What does the following command mean?
ENCODED_COMMAND=$(python <<EOF
3) What's the purpose of these kinds of encoding?
#!/bin/bash
COMMAND="FILTER file utterance_id /tmp/my_utt_list"
ENCODED_COMMAND=$(python <<EOF
import base64
print base64.urlsafe_b64encode('$COMMAND')
EOF
)
$BIN --edit_commands="/inline/b64/$ENCODED_COMMAND"
This depends on what the value of $BIN is. Presumably this is some other script which supports an --edit_commands flag. You would need to what that other script is expecting for this value to be able to interpret it.
This is combining a couple of bits of bash syntax. First, $(...) means "execute the enclosed command and capture its output as a string". Second, the <<EOF means that the following lines until the second EOF should be passed to the standard input of the command. So taken together, this is executing the Python script between the two EOFs, capturing its output, and assigning it to the ENCODED_COMMAND variable.
The script is taking some string, $COMMAND, and using the Python base64.urlsafe_b64encode function to encode it with Base64. The encoded string is then being passed to some unknown command, $BIN, which will presumably do something with it — perhaps decode and execute it in some way.

Pass binary data to os.system call

I need to call an executable in a python script and also pass binary data (generated in the same script) to this executable.
I have it working like so:
bin = make_config(data)
open('binaryInfo.bin', 'wb+').write(bin)
os.system("something.exe " + "binaryInfo.bin")
I thought I could avoid creating the binaryInfo.bin file altogether by passing 'bin' straight to the os.system call:
bin = make_config(data)
os.system("something.exe " + bin)
But in this case I get an error:
"Can't convert 'bytes' object to str implicitly"
Does anyone know the correct syntax here? Is this even possible?
Does anyone know the correct syntax here? Is this even possible?
Not like you're doing it. You can't pass arbitrary binary data on the UNIX command line, as each argument is inherently treated as null-terminated, and there's a maximum total length limit which is typically 64KB or less.
With some applications which recognize this convention, you may be able to pipe data on stdin using something like:
pipe = os.popen("something.exe -", "w")
pipe.write(bin)
pipe.close()
If the application doesn't recognize "-" for stdin, though, you will probably have to use a temporary file like you're already doing.
os.system(b"something.exe " + bin)
Should do it.. However, I'm not sure you should be sending binary data through the command line. There might be some sort of limit on character count. Also, does this something.exe actually accept binary data through the command line even?
how bout base64encoding it before sending and decoding on the other end... afaik command line arguments must be ascii range values (although this maynot be true... but I think it is..) ...
another option would be to do it the way you currently are and passing the file ...
or maybe see this Passing binary data as arguments in bash

Categories