I am trying to use fabric (v2.6) to run some commands that make use of bash's extglob and dotglob.
When I run:
c.run(f"shopt -s extglob dotglob && rm -Rf {project_path}* !(.|..|.venv) && shopt -u extglob dotglob")
I get this error:
`bash: -c: line 0: syntax error near unexpected token `('`
I am using the && because I found doing shopt -s extglob dotglob in a separate run call doesn't persist for the subsequent run calls. I'm pretty sure using && is enabling extglob and dotglob because when I do this:
`c.run("shopt -s extglob dotglob && shopt")`
It prints out the list of options and extglob and dotglob are both enabled.
Where am I going wrong here?
From the bash wiki:
extglob changes the way certain characters are parsed. It is necessary to have a newline (not just a semicolon) between shopt -s extglob and any subsequent commands to use it.
So you have to change your python code appropriately so that a newline is used instead of &&.
Or just do what the bash invocation does directly in python.
It seems extglob can't be used with Python Fabric unfortunately.
From the bash docs
extglob changes the way certain characters are parsed. It is necessary
to have a newline (not just a semicolon) between shopt -s extglob and
any subsequent commands to use it.
But from the Fabric docs
While Fabric can be used for many shell-script-like tasks, there’s a
slightly unintuitive catch: each run [...] has its own distinct shell session. This is required
in order for Fabric to reliably figure out, after your command has
run, what its standard out/error and return codes were.
Fortunately, a similar thing can be achieved using Bash's GLOBIGNORE shell variable instead
The GLOBIGNORE shell variable may be used to restrict the set of file
names matching a pattern. If GLOBIGNORE is set, each matching file
name that also matches one of the patterns in GLOBIGNORE is removed
from the list of matches. If the nocaseglob option is set, the
matching against the patterns in GLOBIGNORE is performed without
regard to case. The filenames . and .. are always ignored when
GLOBIGNORE is set and not null. However, setting GLOBIGNORE to a
non-null value has the effect of enabling the dotglob shell option, so
all other filenames beginning with a ‘.’ will match. To get the old
behavior of ignoring filenames beginning with a ‘.’, make ‘.*’ one of
the patterns in GLOBIGNORE. The dotglob option is disabled when
GLOBIGNORE is unset.
This also handily ignores . and .. when expanding wildcards, so to remove all files - except '.venv' - in a directory, we can do
c.run("GLOBIGNORE='.venv'; rm -Rf {project_path}*")
Related
Lets say I have the following python script
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--host", required=True)
parser.add_argument("--enabled", default=False, action="store_true")
args = parser.parse_args()
print("host: " + args.host)
print("enabled: " + str(args.enabled))
$ python3 test.py --host test.com
host: test.com
enabled: False
$ python3 test.py --host test.com --enabled
host: test.com
enabled: True
Now the script is used in a docker image and I want to pass the variables in docker run. For the host parameter it is quite easy
FROM python:3.10-alpine
ENV MY_HOST=default.com
#ENV MY_ENABLED=
ENV TZ=Europe/Berlin
WORKDIR /usr/src/app
COPY test.py .
CMD ["sh", "-c", "python test.py --host ${MY_HOST}"]
But how can I can make the --enabled flag to work? So when the/an ENV is unset or is 0 or off ore something, --enabled should be suppressed, otherwise it should be included in the CMD.
Is is possible without modify the python script?
For exactly the reasons you're showing here, I'd suggest modifying your script to be able to accept command-line options from environment variables. If you add a line
parser.set_defaults(
host=os.environ.get('MY_HOST'),
enabled=(os.environ.get('MY_ENABLED') == 'true')
)
then you can use docker run -e options to provide these values, without the complexity of trying to reconstruct the command line based on which options are and aren't present. (Also see Setting options from environment variables when using argparse.)
CMD ["./test.py"] # a fixed string, environment variables specified separately
docker run -e MY_HOST=example.com -e MY_ENABLED=true my-image
Conversely, you can provide the entire command line and its options when you run the container. (But depending on the context you might just be pushing the "how to construct the command" question up a layer.)
docker run my-image \
./test.py --host=example.com --enabled
In principle you can construct this using a separate shell script without modifying your Python script, but it will be somewhat harder and significantly less safe. That script could look something like
#!/bin/sh
TEST_ARGS="--host $MY_HOST"
if [ -n "$MY_ENABLED" ]; then
TEST_ARGS="$TEST_ARGS --enabled"
fi
exec ./test.py $TEST_ARGS
# ^^^^^^^^^^ without double quotes (usually a bug)
Expanding $TEST_ARGS without putting it in double quotes causes the shell to split the string's value on whitespace. This is usually a bug since it would cause directory names like /home/user/My Files to get split into multiple words. You're still at some risk if the environment variable values happen to contain whitespace or other punctuation, intentionally or otherwise.
There are safer but more obscure ways to approach this in shells with extensions like GNU bash, but not all Docker images contain these. Rather than double-check that your image has bash, and figure out bash array syntax, and write a separate script to do the argument handling, this is where I suggest handling it exclusively at the Python layer is a better approach.
Is it possible, to enable bash alias expansion in snakemake?
I'm writing a workflow that is taking a config file for execution parameters.
Let's assume in this config file, the location of a program executable must be defined and will be passed to the shell of a rule.
config.yaml:
myProgram: /very/long/path/to/executable/build/myprogram
Snakefile:
rule: runMyProgram
input: inputfile.txt
output: outputfile.txt
shell: "{config[myProgram]} -i {input} -o {output}"
But I would like to also enable the opportunity to directly call the program with the config.yaml:
myProgram: myprogram
In this case, if the user has set an alias instead of adding myprogram to the $PATH, the shell does not recognize the alias and the rule ends with an error. When testing shopt expand_aliases within the snakemake shell, I see it is turned off, but adding shopt -s expand_aliases to the shell directive of the rule also doesn't do the trick.
I also tried adding the shopt command to the shell.prefix(), with no success, obviously, as it just adds the prefix to the shell.
While all of us would agree, in this minimal example the user should just put the path to the executable to $PATH, there are circumstances where a user would e.g. use different program versions under different aliases.
Or phrased differently: I would like the program not to crash if a user puts an alias instead of adding it to the $PATH.
Hence I was wondering if there was another possibility to turn on expand_aliases globally for Snakemake?
I am trying to create a Docker image/container that will run on Windows 10/Linux and test a REST API. Is it possible to embed the function (from my .bashrc file) inside the DockerFile? The function pytest calls pylint before running the .py file. If the rating is not 10/10, then it prompts the user to fix the code and exits. This works fine on Linux.
Basically here is the pseudo-code inside the DockerFile I am attempting to build an image.
------------------------------------------
From: Ubuntu x.xx
install python
Install pytest
install pylint
copy test_file to the respective folder
Execute pytest test_file_name.py
if the rating is not 10\10:
prompt the user to resolve the rating issue and exit
------------here is the partial code snippet from the func------------------------
function pytest () {
argument1="$1"
# Extract the path and file name for pylint when method name is passed
pathfilename=`echo ${argument1} | sed 's/::.*//'`
clear && printf '\e[3J'
output=$(docker exec -t orch-$USER pylint -r n ${pathfilename})
if (echo "$output" | grep 'warning.*error' o&>/dev/null or
echo "${output}" | egrep 'warning|convention' &>/dev/null)
then
echo echo "${output}" | sed 's/\(warning\)/\o033[33m\1\o033[39m/;s/\(errors\|error\)/\o033[31m\1\o033[39m/'
YEL='\033[0;1;33m'
NC='\033[0m'
echo -e "\n ${YEL}Fix module as per pylint/PEP8 messages to achieve 10/10 rating before pusing to github\n${NC}"`
fi
Another option I can think of:
Step 1] Build the image (using DockerFile) with all the required software
Step 2] In a .py file, add the call for execution of pytest with the logic from the function.
Your thoughts?
You can turn that function into a standalone shell script. (Pretty much by just removing the function wrapper, and taking out the docker exec part of the tool invocation.) Once you've done that, you can COPY the shell script into your image, and once you've done that, you can RUN it.
...
COPY pylint-enforcer.sh .
RUN chmod +x ./pylint-enforcer.sh \
&& ./pylint-enforcer.sh
...
It looks like pylint will produce a non-zero exit code if it emits any messages. For purposes of a Dockerfile, it may be enough to just RUN pylint -r -n .; if it prints anything, it looks like it will return a non-zero exit code, which docker build will interpret as "failure" and not proceed.
You might consider whether you'll ever want the ability to build and push an image of code that isn't absolutely perfect (during a production-down event, perhaps), and whether you want to require root-level permissions to run simple code-validity tools (if you can docker anything you can edit arbitrary files on the host as root). I'd suggest running these tools out of a non-Docker virtual environment during your CI process, and neither place them in your Dockerfile nor depend on docker exec to run them.
I am using python2.7 and argparse for my script. I am executing script as below:
python2.7 script.py -a valuefora -b valueforb -c valueforc -d valueford
Now what I want is that,
if option -a is given, then only -b, -c, -d options should be asked.
In addition to above, I also want to make this group -a -b -c -d as a EITHER OR for -e i.e. ([-a -b -c -d] | -e )
Please correct me anywhere I am wrong.
Your best choice is to test for the presence of various combinations after parse_args and use parser.error to issue an argparse compatible error message. And write your own usage line. And make sure the defaults clearly indicate whether an option has been parsed or not.
If you can change the -a and -e options to command names like cmda or build, you could use subparsers. In this case you might define a command_a subparser that accepts -b, -c, and -d, and another command_e subparser that has none of these. This is closes argparse comes to 'required together' groups of arguments.
mutually exclusive groups can define something with a usage like [-a -b -c], but that just means -b cannot occur along with -a and -c. But there's nothing fancy about that mechanism. It just constructs a dictionary of such exclusions, and checks it each time it parses a new option. If there is a conflict it issues the error message and quits. It is not set up to handle fancy combinations, such as your (-e | agroup).
Custom actions can also check for the absence or presence of non-default values in the namespace, much as you would after parsing. But doing so during parsing isn't any simpler. And it raises questions about order. Do you want to handle -b -c -a the same way as -a -c -b? Should -a check for the presence of the others, or should -b check that -a has already been parsed? Who checks for the presence, or absence of -e.
The are a number of other stack questions about argparse groups, exclusive and inclusive, but I think these are the essential issues.
Is there any way to set up Python source checking on git server side?
Something like this (this is pre-commit hook):
#!/bin/sh
FILES=$(git diff --cached --name-only --diff-filter=ACM | grep -e '\.py$')
if [ -n "$FILES" ]; then
flake8 -r $FILES
fi
but on the server side(perhaps with update hook).
You can add an update hook, that will be getting input file in format of ${ref} ${oldrev} ${newrev}, e.g. something like refs/heads/master ddf343f635fe4440cad528e12f96f28bd50aa099 f59abbf28696389c91c2697c7db31f20cfa91d8a.
Armed with this knowledge, you can then make a diff between those two commits, list files that are there, check them for syntax, and fail in case you don't like them. If there is an entirely new branch pushed, the ${oldrev} will be 40 zeroes, so you'd have to probably check all the files in the new commit.