I'm having trouble with Python Fabric removing some local files. Yet, I found a related post on StackOverflow with a solution. But I am wondering why does adding 2>&1 at the end fix it?
I can run the following perfectly fine in my terminal:
$ find app/views/ -type f -name '*%%.php' -exec rm {} \;
However when I do a fabric call I get:
$ fab rmcache
[localhost] local: find app/views/ -type f -name '*%%.php' -exec rm {} \;
find: missing argument to `-exec'
Fatal error: local() encountered an error (return code 1) while executing 'find
app/views/ -type f -name '*%%.php' -exec rm {} \;'
0: Why does it require 2>&1 through Fabric, but not locally?
1: Why does this work through Fabric?
def rmcache():
local("find {0} -type f -name '*%%.php' -exec rm {{}} \ 2>&1;".format('app/views/'));
2: But this does not work through fabric?
def rmcache():
local("find {0} -type f -name '*%%.php' -exec rm {{}} \;".format('app/views/'));
0: 2>&1 redirects stderr to stdout which means that if your command is throwing an error fabric won't pick it up because it isn't being returned to fabric (See this answer for more details on 2>&1).
1 & 2: My guess is that your code is throwing an error because 'app/views' is a relative path and find requires the directory to exist, therefore you would have to run your fabric command from the directory that contains the app directory. Try using '/full/path/to/app/views' to ensure you are using the correct directory.
Related
I am trying to run xargs on multiple files at once:
sh -c 'find . -name "*.py" | xargs pylint'
This will give me a single pylint score for all py files in a repo. However when I try to modify it to do both black and pylint, it loops through each file individually and gives me the pylint score and black diff on each file:
find . -name "*.py" | xargs -I % sh -c 'pylint %; black --check --diff %;'
Any way to pass in the py files in batch rather than each individually?
If you use -I option it runs given command once for each entry in input.
You can do this instead:
find . -name "*.py" |
xargs sh -c 'pylint "$#"; black --check --diff "$#"'
So I have a remote Linux Server and would like to run a Python Script on my local machine to list all files and their modification dates in a specific folder on that remote server. That is my code so far:
command = "find \""+to_directory+'''\"* -type f -exec sh -c \"stat -c \'%y:%n\' \'{}\'\" \;'''
scp_process_ = subprocess.run("ssh "+to_user+"#"+to_host+" '"+command+"' ", shell=True, capture_output=False, text=True)
Now running the command
find "/shares/Public/Datensicherung/"* -type f -exec sh -c "stat -c '%y:%n' '{}'" \;
on the server itself works fine without any error.
But as soon I use a subprocess to run it remotely over ssh it has a problem with a file in a folder with spaces: "/shares/Public/Datensicherung/New folder/hi.txt" with a space in it:
stat: can't stat '/shares/Public/Datensicherung/New': No such file or directory
stat: can't stat 'folder/hi.txt': No such file or directory
I know it is messed up, but that is the best solution I could build.
I would like to stick with subprocess and ssh but if you have a better solution feel free to post it.
With shell=True you are invoking three shell instances, each of which requires a layer of quoting. This is possible to do, of course, but there are many reasons to avoid it if at all possible.
First off, you can easily avoid the local shell=True and this actually improves the robustness and clarity of your Python code.
command = "find \""+to_directory+'''\"* -type f -exec sh -c \"stat -c \'%y:%n\' \'{}\'\" \;'''
scp_process_ = subprocess.run(
["ssh", to_user+"#"+to_host, command],
capture_output=False, text=True)
Secondly, stat can easily accept multiple arguments, so you can take out the sh -c '...' too.
command = 'find "' + to_directory + '" -type f -exec stat -c "%y:%n" {} +'
The optimization also switches + for \; (so the sh -c '' wrapper was doubly useless anyway).
Sometimes the issue happening because malformed command string. For purpose of comunication with Unix shell was craeted shlex module. So basically you wrap your code with shlex and then pass it into supbrocess.run.
I don't see the actual final cmd to call but you could split it to proper command with shlex.split by yourself.
From your example it would be something like:
from shlex import join
cmd = join(['ssh',
f'{to_user}#{to_host}',
'find',
f'{to_directory}*',
'-type',
'f',
'-exec',
'sh',
'-c',
"stat -c '%y:%n' '{}'",
';']))
scp_process_ = subprocess.run(cmd, shell=True, capture_output=False, text=True)
Also, you maybe want to play around with shell=True option.
We've got a Python application and want to count all the lines of code under a specific directory and its subdirectories.
We don't need to ignore comments but we want to ignore all files containing test cases.
Path to test cases file always has /tests/ in the path (e.g. /python/trieus/persistence/service/tests/batch_service_tests.py).
I used below command to find the count but it did not exclude test files.
find . -name "*.py" -not -path "./tests*" | xargs wc -l | sort
What's the correct syntax here?
You can do:
find . -type d -name tests -prune -o -type f -name '*.py' \
-exec grep -hxc '.*' {} + | paste -sd+ | bc
find . -type d -name tests -prune excludes the tests directory
-type f -name '*.py' matches only .py files
-exec grep -hxc '.*' {} + get individual lines, you can modify the Regex pattern to meet you need here
paste -sd+ formats the output to put in a single line with + in between
bc does the addition on its STDIN data
I would suggest you to go one step at a time to understand the whole thing better.
As a side note, this will not get you the actual LOC, instead would get you the lines count only.
Example from running in an example directory on my system:
% find . -type d -name tests -prune -o -type f -name '*.py' -exec grep -hxc '.*' {} + | paste -sd+ | bc
5594
You can exclude a directory at ANY level by changing the first . to *. so find -name "*.py" -not -path "*/tests/*" would omit files from a directory named "tests" at any level of depth.
So the command will look like this:
find . -name "*.py" -not -path "*/tests/*" | xargs wc -l | sort
Running this on osx...
cd ${BUILD_DIR}/mydir && for DIR in $(find ./ '.*[^_].py' | sed 's/\/\//\//g' | awk -F "/" '{print $2}' | sort |uniq | grep -v .py); do
if [ -f $i/requirements.txt ]; then
pip install -r $i/requirements.txt -t $i/
fi
cd ${DIR} && zip -r ${DIR}.zip * > /dev/null && mv ${DIR}.zip ../../ && cd ../
done
cd ../
error:
(env) ➜ sh package_lambdas.sh find: .*[^_].py: No such file or directory
why?
find takes as an argument a list of directories to search. You provided what appears to be regular expression. Because there is no directory named (literally) .*[^_].py, find returns an error.
Below I have revised your script to correct that mistake (if I understand your intention). Because I see so many ill-written shell scripts these days, I've taken the liberty of "traditionalizing" it. Please see if you don't also find it more readable.
Changes:
use #!/bin/sh, guaranteed to be on an Unix-like system. Faster than bash, unless (like OS X) it is bash.
use lower case for variable names to distinguish from system variables (and not hide them).
eschew braces for variables (${var}); they're not needed in the simple case
do not pipe output to /usr/bin/true; route it to dev/null if that's what you mean
rm -f by definition cannot fail; if you meant || true, it's superfluous
put then and do on separate lines, easier to read, and that's how the Bourne shell language was meant to be used
Let && and || serve as line-continuation, so you can see what's happening step by step
Other changes I would suggest:
Use a subshell when changing the working directory temporarily. When it terminates, the working directory is restored automatically (retained by the parent), saving you the cd .. step, and errors.
Use set -e to cause the script to terminate on error. For expected errors, use || true explicitly.
Change grep .py to grep '\.py$', just for good measure.
To avoid Tilting Matchstick Syndrome, use something other than / as a sed substitute delimiter, e.g., sed 's://:/:g'. But sed could be avoided altogether with awk -F '/+' '{print $2}'.
Revised version:
#! /bin/sh
src_dir=lambdas
build_dir=bin
mkdir -p $build_dir/lambdas
rm -rf $build_dir/*.zip
cp -r $src_dir/* $build_dir/lambdas
#
# The sed is a bit complicated to be osx / linux cross compatible :
# ( .//run.sh vs ./run.sh
#
cd $build_dir/lambdas &&
for L in $(find . -exec grep -l '.*[^_].py' {} + |
sed 's/\/\//\//g' |
awk -F "/" '{print $2}' |
sort |
uniq |
grep -v .py)
do
if [ -f $i/requirements.txt ]
then
echo "Installing requirements"
pip install -r $i/requirements.txt -t $i/
fi
cd $L &&
zip -r $L.zip * > /dev/null &&
mv $L.zip ../../ &&
cd ../
done
cd ../
The find(1) manpage says its args are [path ...] [expression], where "expression" consists of "primaries" and "operands" (-flags). '.*[^-].py' doesn't look like any expression, so it's being interpreted as a path, and it's reporting that there is no file named '.*[^-].py' in the working directory.
Perhaps you meant:
find ./ -regex '.*[^-].py'
I am trying to upload my django project live and I am continuously getting this error:
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Errno 13] Permission denied: '/opt/bitnami/.tmp/simplejson-2.0.9-py2.7-linux-x86_64.egg-tmp/simplejson/tmpuYcIYB.$extract'
The Python egg cache directory is currently set to:
/opt/bitnami/.tmp
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
It is a permissions / ownership problem. You can solve it using these commands:
sudo su bitnami
sudo find /opt/bitnami/apps/django/ -type d -exec chmod 755 {} \;
sudo find /opt/bitnami/apps/django/ -type f -exec chmod 644 {} \;
sudo chown -R bitnami:daemon /opt/bitnami/apps/django/
You must also deploy your projects to the following path: /opt/bitnami/apps/django/django_projects/YOURPROJECT