We Keep Coding

lua run a shell command with parameters doesn't work

I am trying to execute a command from lua script. The command is to simply run a python script named "sha_compare.py" of which receives 3 arguments where two of them are variables from the lua script - dady_data and sha:
local method = ngx.var.request_method
local headers = ngx.req.get_headers()
if method == "POST" then
ngx.req.read_body()
local body_data = ngx.req.get_body_data()
local sha = headers['X-Hub-Signature-256']
ngx.print(os.execute("python3 sha_compare.py"..sha..body_data))
else
The script fails because of the way I call the arguments. The actual command if I would have ran it from cmd would have been something like:
python3 python3 sha_compare.py sha256=ffs8df aaaaa
Please tell me how should I change my code to call the python script with 3 vars properly.
If it is not possible or hard to implement, please let me know how can I call a .sh script which will receive those 3 params.

You're not providing spaces between the arguments: you're trying to execute
python3 sha_compare.pysha256=ffs8dfaaaaa
Do this:
os.execute("python3 sha_compare.py "..sha.." "..body_data)
It's often easier to build the command up as a table, and the concat it for execution:
local cmd = { 'python3', 'sha_compare.py', sha, body_data }
os.execute(table.concat(cmd, " "))

Python Subprocess call() does not execute shell command

I am trying to make a python program(python 3.6) that writes commands to terminal to download a specific youtube video(using youtube-dl).
If I go on terminal and execute the following command:
cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU"
It will download the video to my desktop. However, if I execute the below code, which should be doing the same command on terminal, it does not throw an error but also does not download that video.
import subprocess
cmd = ["cd;", "cd", "Desktop;", "youtube-dl", "\"https://www.youtube.com/watch?v=b91ovTKCZGU\""]
print(subprocess.call(cmd, stderr=subprocess.STDOUT,shell=True))
It seems that this just outputs 0. I do not think there is any kind of error 0 that exists(there are error 126 and 127). So if it is not throwing an error, why does it also not download the video?
Update:
I have fixed the above code by passing in a string, and have checked that youtube-dl is installed in my default python and is also in the folder where I want to download the videos, but its still throwing error 127, meaning command "youtube-dl" is not found.

cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU" is not a single command; it's a list (delimited by ;) of three separate commands.
subprocess.call(cmd, ..., shell=True) is effectively the same as
subprocess.call(['sh', '-c'] + cmd)
which is almost never what you want. Instead, just pass a single string and let the shell parse it.
subprocess.call('cd; cd Desktop; youtube-dl "https://www.youtube.com/watch?v=b91ovTKCZGU"', shell=True)
If you really want to use the list form (which is always a good idea), use the cwd parameter instead of running cd.
subprocess.call(['youtube-dl', 'https://www.youtube.com/watch?v=b91ovTKCZGU'],
cwd=os.path.expanduser("~/Desktop"))

I'll answer this with an example:
>>> subprocess.call(["echo $0 $2", "foo", "skipped", "bar"], shell=True)
foo bar
0
The first element of the list is the shell command (echo $0 $2), and the remaining elements are the positional parameters that the command can optionally use ($0, $1, ...).
In your example, you are creating a subshell that only runs the cd; command. The positional parameters are ignored. See the Popen and bash docs for details.
As noted in the comments, you should make the command a string (not a list).

Return value from shell script to python script

I am trying to achive following:
I am having a python script which is calling a shell script and passing a parameter as well. Shell script creates a tar.gz file using that parameter passed in some location.
Now shell script should pass the name and location of the tar.gz so created. Python script uses that to form a JSON and pass to some other code.
Along with this I want to add some check to make sure if tar.gz is generated then only value is returned to python otherwise not.
Here is the code:
Python script:
#!/usr/bin/python
import json
import subprocess
json_data='{"name": "StackOverflow", "uid": "8fa36334-ce51"}'
data = json.loads(json_data)
for keys,values in data.items():
print(keys)
print(values)
UID = data.get('uid')
rc = subprocess.check_output(["/home/cyc/Cyc-
Repo/cyc_core/cyc_platform/src/package/cyc_bsc/scripts/test.sh",
UID])
print rc
if rc != 0:
print "failed for passed in uid"
data_op = {}
data_op['pathTOCompressfile'] = 'value_should_be_return_from_shell'
data_op['status'] = 'OK'
json_data_op = json.dumps(data_op)
print json_data_op
shell script:
#!/bin/bash
if [ "$1" != "" ]; then
uid=$1
echo "Positional parameter 1 contains something $1"
else
echo "Positional parameter 1 is empty"
fi
LOG_TMP="tmp_log_file_location"
log_location="log_file_location"
filename="${log_location}/$uid.tar.gz"
echo $filename
tar -zcvf $filename -C $LOG_TMP/dc .
This is what i am not able to understand:
How to pass back the value of variable "filename" back to python script if tar -zcvf command is successful.
In python script how can i verify take value of filename and create JSON using that
In case value cannot be generated STATUS becomes fail in JSON ( within python ) so capture that as well.

Your shell script writes the name of the generated file to standard output, so your question boils down to how to catch stdandard output of a subprocess started from Python. This has been ansered here. BTW, when asking questions about Python, it would be a good idea to always specify the Python version you are using.
In your case however, I would redesign your shell script a bit:
Your script outputs not only the generated filename, but also messages about the "positional parameter", and this means that you would have to fiddle them apart in your script, whether it is an message or a valid output. You could send the messages to standard error, to keep them apart.
BTW, if there is no positional parameter, the generated file name is just .tar.gz. Is this really what you want to have? Wouldn't it better to do a exit 1, if there is no parameter?

How to pass App argument to os.system()

I am able to start application using quite simple syntax. Example
app="/Applications/MyApp/myAppExecutable"
file="/Users/userName/Pictures/myPicture.jpg"
cmd="open -a '%s' '%s'" % (app, file)
os.system(cmd)
Resulted cmd here is:
open -a '/Applications/MyApp/myAppExecutable' '/Users/userName/Pictures/myPicture.jpg'
and it runs just fine.
But the application I am running accepts an optional startup argument -proj filepath, so a full cmd string with this optional argument should look like:
open -a '/Applications/MyApp/myAppExecutable' -proj '/Users/userName/Pictures' '/Users/userName/Pictures/myPicture.jpg'
But if I feed os.system() with a such cmdstring I am getting:
open: invalid option -- p
How to pass an optional app argument without causing an error and crash?

Per the open man page, you have to pass --args before arguments to the program:
--args
All remaining arguments are passed to the opened application in the
argv parameter to main(). These arguments are not opened or inter-
preted by the open tool.
As an aside, you may want to look into using subprocess. Here's how the command looks like with subprocess:
subprocess.check_call(['open', '-a', app, file])
No fiddling with string interpolation required.

It is quite possible to submit the application's specific arguments to OS open command.
The syntax to open:
open -a /path/to/application/executable.file /filepath/to/the/file/to/open/with/app.ext
It appears it is OK to enclose both paths in double or single quotes such as:
open -a '/path/to/application/executable.file' '/filepath/to/the/file/to/open/with/app.ext'
As Ned has mentioned flag --args can be used to specify any Application's specific startup flags.
The App specific flags are placed after open's --args flag such as:
open -a /path/to/application/executable.file --args -proj
The problem is that the App specific flags (such as -proj) will only be passed at the time the App is starting up. If it is already running the open -a command will only open a file (if the file_to_be_opened is specified) but won't deliver the App's specific args. By other words the App is only able to receive its args at the time it starts.
There is -n flag available to be used with open -a command. When used open -a will start as many instances of Apps as needed. Each of App instances will get the App args properly.
open -a '/path/to/application/executable.file' '/filepath/to/the/file/to/open/with/app.ext -n --args -proj "SpecificToApp arg or command" '
It all translates if used with subprocess:
subprocess.check_call(['open', '-a', app, file, '-n', '--args', '-proj', 'proj_flag_values'])
or simply pass it as a string arg to:
os.system(cmdString)

how to pass an argument to a python script when starting with nohup

I need to start a python script with bash using nohup passing an arg that aids in defining a constant in a script I import. There are lots of questions about passing args but I haven't found a successful way using nohup.
a simplified version of my bash script:
#!/bin/bash
BUCKET=$1
echo $BUCKET
script='/home/path/to/script/script.py'
echo "starting $script with nohup"
nohup /usr/bin/python $script $BUCKET &
the relevant part of my config script i'm importing:
FLAG = sys.argv[0]
if FLAG == "b1":
AWS_ACCESS_KEY_ID = "key"
BUCKET = "bucket1"
AWS_SECRET_ACCESS_KEY = "secret"
elif FLAG == "b2":
AWS_ACCESS_KEY_ID = "key"
BUCKET = "bucket2"
AWS_SECRET_ACCESS_KEY = "secret"
else:
AWS_ACCESS_KEY_ID = "key"
BUCKET = "bucket3"
AWS_SECRET_ACCESS_KEY = "secret"
the script thats using it:
from config import BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
#do stuff with the values.
Frankly, since I'm passing the args to script.py, I'm not confident that they'll be in scope for the import script. That said, when I take a similar approach without using nohup, it works.

In general, the argument vector for any program starts with the program itself, and then all of its arguments and options. Depending on the language, the program may be sys.argv[0], argv[0], $0, or something else, but it's basically always argument #0.
Each program whose job is to run another program—like nohup, and like the Python interpreter itself—generally drops itself and all of its own options, and gives the target program the rest of the command line.
So, nohup takes a COMMAND and zero or more ARGS. Inside that COMMAND, argv[0] will be COMMAND itself (in this case, '/usr/bin/python'), and argv[1] and later will be the additional arguments ('/home/path/to/script/script.py' and whatever $BUCKET resolves to).
Next, Python takes zero or more options, a script, and zero or more args to that script, and exposes the script and its args as sys.argv. So, in your script, sys.argv[0] will be '/home/path/to/script/script.py', and sys.argv[1] will be whatever $BUCKET resolves to.
And bash works similarly to Python; $1 will be the first argument to the bash wrapper script ($0 will be the script itself), and so on. So, sys.argv[1] in the inner Python script will end up getting the first argument passed to the bash wrapper script.
Importing doesn't affect sys.argv at all. So, in both your config module and your top-level script, if you import sys, sys.argv[1] will hold the $1 passed to the bash wrapper script.
(On some platforms, in some circumstances argv[0] may not have the complete path, or may even be empty. But that isn't relevant here. What you care about is the eventual sys.argv[1], and bash, nohup, and python are all guaranteed to pass that through untouched.)

nohup python3 -u ./train.py --dataset dataset_directory/ --model model_output_directory > output.log &
Here Im executing train.py file with python3, Then -u is used to ignore buffering and show the logs on the go without storing, specifying my dataset_directory with argument style and model_output_directory then Greater than symbol(>)
then the logs is stored in output.log and them atlast and(&) symbol is used
To terminate this process
ps ax | grep train
then note the process_ID
sudo kill -9 Process_ID