The following command works:
$ pycco *.py
# generates literate-style documentation
# for all .py files in the current folder
And the following snippet in my tox.ini file works as expected:
[testenv:pycco]
deps =
pycco
commands =
pycco manage.py
# generates literate-style documentation
# for manage.py
But if I try to use a glob:
[testenv:pycco]
deps =
pycco
commands =
pycco *.py
...I get the following error:
File "/home/user/Documents/project/.tox/pycco/lib/python3.7/site-packages/pycco/main.py", line 79, in generate_documentation
code = open(source, "rb").read().decode(encoding)
FileNotFoundError: [Errno 2] No such file or directory: '*.py'
How can I pass *.py to pycco via tox?
The problem here is that pycco does not support glob expansions. What makes pycco *.py work is that before the execution happens the shell actually transforms *.py to actual files; and then passes that to the OS to run it.
When tox runs your command there's no shell involved, so whatever you write is as is passed on to the OS, so now pycco actually gets as argument *.py hence the error.
You can work around this by either explicitly listing the file paths or using the python interpreter to do the expansion:
python -c 'from glob import glob; import subprocess; subprocess.check_call(["pycco"] + glob("*.py"))'
Put the above command inside your tox commands and things will work now as python is now the shell doing the expansion of "*.py" to actual files list.
You cannot do this directly because pycco does not (currently) support glob expansions. Instead you can create a shell script execute_pycco.sh as follows:
#!/bin/sh
pycco *.py
Update tox.ini as follows:
[testenv:pycco]
deps =
pycco
commands =
./execute_pycco.sh
You will now execute your shell script in the "pycco" environment created by tox. This method also allows you to define more elaborate scripts:
#!/bin/sh
filelist=$( find . -name '*.py' | grep -v ".tox" )
# make a list of all .py files in all subfolders,
# except the .tox/ subfolder
pycco -ip $filelist
# generate literate-style documentation for all
# files in the list
Related
I have a folder called TEST with inside :
script.py
script.sh
The bash file is :
#!/bin/bash
# Run the python script
python script.py
If I run the bash file like this :
./TEST/script.sh
I have the following error :
python: can't open file 'script.py': [Errno 2] No such file or directory
How could I do, to tell my script.sh to look in the directory (which may change) and to allow me to run it for inside the TEST directory ?
Tricky, my python file run a sqlite database and I have the same problem when calling the script from outside the folder, it didn't look inside the folder to find the database!
Alternative
You are able to run the script directly by adding this line to the top of your python file:
#!/usr/bin/env python
and then making the file executable:
$ chmod +x script.py
With this, you can run the script directly with ./TEST/script.py
What you asked for specifically
This works to get the path of the script, and then pass that to python.
#!/bin/sh
SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )"
python "$SCRIPTPATH/script.py"
Also potentially useful:
You mentioned having this problem with accessing a sqlite DB in the same folder, if you are running this from a script to solve this problem, it will not work. I imagine this question may be of use to you for that problem: How do I get the path of a the Python script I am running in?
You could use $0 which is the name of the currently executing program, as invoked, combined with dirname which provides the directory component of a file path, to determine the path (absolute or relative) that the shell script was invoked under. Then, you can apply it to the python invocation.
This example worked for me:
$ t/t.sh
Hello, world!
$ cat t/t.sh
#!/bin/bash
python "$(dirname $0)/t.py"
Take it a step farther and change your current working directory which will also be inherited by python, thus helping it to find its database:
$ t/t.sh; cat t/t.sh ; cat t/t.py ; cat t/message.txt
hello, world!
#!/bin/bash
cd "$(dirname $0)"
python t.py
with(open('message.txt')) as msgf:
print(msgf.read())
hello, world!
From the shell script, you can always find your current directory: Getting the source directory of a Bash script from within. While the accepted answer to this question provide a very comprehensive and robust solution, your relatively simple case only really needs something like
#!/bin/bash
dir="$(dirname "${BASH_SOURCE[0]}")"
# Run the python script
python "$(dir)"/script.py
Another way to do it would be to change the directory from which you run the script:
#!/bin/bash
dir="$(dirname "${BASH_SOURCE[0]}")"
# Run the python script
(cd "$dir"; python script.py)
The parentheses ((...)) around cd and python create a subprocess, so that the directory does not change for the rest of your bash script. This may not be necessary if you don't do anything else in the bash portion, but is still useful to have if you ever decide to say source your script instead of running it as a subprocess.
If you do not change the directory in bash, you can do it in Python using a combination of sys.argv\[0\], os.path.dirname and os.chdir:
import sys
import os
...
os.chdir(os.path.dirname(sys.argv[0]))
I use rsync to move files from my home computer to a server. Here's the command that I use to transfer and update the directory of only files that contain a grep + glob. I execute this command from the toplevel/ directory in the directory structure I show below.
rsync -r --progress --include='**201609*/***' --exclude='*' -avzh files/ user#server.edu:/user/files
Here's what the file structure of the working directory on my home file looks like:
- toplevel
- items
- files
- 20160315
- 20160910
- dir1
- really_cool_file1
- 20160911
- dir2
This works fine, and the file structure on user#server.edu:/user/files is the same as on my home computer.
I wrote a python script to do this and it doesn't work. It also transfers over the files/20160315, which is not what I want.
#!/usr/bin/env python3
import os
from subprocess import run
os.chdir("toplevel")
command_run = ["rsync", "-r",
"--progress",
"--include='**201609*/***'",
"--exclude='*'",
"-avzh",
"files/", "user#server.edu:/user/files"]
run(command_run, shell=False, check=True)
What's going on here? I had the same problem when command_run was a string, and I passed it to subprocess.run() with shell=True.
Some of those quotes are removed by the shell before being passed to the called process. You need to do this yourself if you call the program with the default shell=False. This little script will tell you what your parameters need to look like
test.py
#!/usr/bin/env python3
import sys
print(sys.argv)
And then running with your command line
~/tmp $ ./test.py -r --progress --include='**201609*/***' --exclude='*' -avzh files/ user#server.edu:/user/files
['./test.py', '-r', '--progress', '--include=**201609*/***', '--exclude=*', '-avzh', 'files/', 'user#server.edu:/user/files']
~/tmp $
I am trying to do some scripting with IPython, but I am finding that it behaves very differently in a script to when I run an interactive shell.
For example, I can run the following interactively:
In [1]: %profile
default
In [2]: ls /
bin/ cdrom/ etc/ initrd.img# lib/ lib64/ media/ opt/ root/ sbin/ sys/ usr/ vmlinuz#
boot/ dev/ home/ initrd.img.old# lib32/ lost+found/ mnt/ proc/ run/ srv/ tmp/ var/ vmlinuz.old#
In [3]: mkdir tmpdir
In [4]: cd tmpdir
/home/alex/tmp/tmpdir
No problem.
However, none of these commands works when I run them in a script:
#!/usr/bin/ipython3
%profile
ls /
mkdir tmpdir
cd tmdir
I get an error:
$ ./tmp.py
File "/home/alex/tmp/tmp.ipython", line 3
%profile
^
SyntaxError: invalid syntax
I have tried running this by:
calling the file directly as above,
calling it explicitly with ipython: `ipython3 tmp.py'
passing the -i or --profile=sh arguments to ipython when calling it with ipython
changing the file extension to .ipython and .ipy
My question:
Why is it behaving differently in a script to the shell? How can I get IPython to run these commands in a script?
They are working due to IPython magic but they are shell commands and do not work in Python. To get them consider the subprocess library. Where you would have spaces in a shell command instead have comma-separated values in the list.
import subprocess
subprocess.check_call(['ls'])
subprocess.check_call(['ls', '-a'])
I have a simple bash script that allows cron to execute a series of Python scripts in a virtualenv. The script keeps raising No such file or directory errors.
~/nightly.sh works fine:
#!/bin/bash
source virt_env/myproject/bin/activate
cd virt_env/myproject/main
python script1.py
python script2.py
I want to keep everything in ~/virt_env/myproject/main/ to simplify deployment. I thought I could call bash virt_env/myproject/main/nightly.sh on this:
#!/bin/bash
MAINDIR=`dirname $0`
cd $MAINDIR
source ../bin/activate
python script1.py
python script2.py
but I get the No such file or directory. If I manually cd to ~/virt_env/myproject/main/, then I can run the main commands no problem. Clearly I'm missing something about how dirname and cd work in this context.
How can I point bash at the right place?
Solution
As proposed in the accepted answer, it's best to avoid calling cd from within the script and use an explicit path variable instead. Here's the working version of virt_env/myproject/main/nightly.sh:
#!/bin/bash
MAINDIR=`dirname $0`
echo "The main directory is" $MAINDIR
# Activate virtual environment
source $MAINDIR/../bin/activate
# Run Python scripts
python $MAINDIR/python1.py
python $MAINDIR/python2.py
Because the Python scripts are now called from an arbitrary path, I needed to update the python scripts to be smarter about path awareness as well.
This code fails because os.path.basename omits path information:
# works when called with "python python1.py"
# fails when called with "python $MAINDIR/python1.py"
CONFIG_FILE = os.path.basename(__file__)[:-3] + ".config"
f = open(CONFIG_FILE,"r")
Updating it to use os.path.abspath fixes the problem:
# works regardless of how it is called
CONFIG_FILE = os.path.abspath(__file__)[:-3] + ".config"
f = open(CONFIG_FILE,"r")
Perhaps it would simply be better to eliminate the 'cd' command. Invoke everything from a full path specification. In your example add $MAINDIR/ to the executables.
Your bash script can then be in any directory where the executables are reachable. You are not exposed to the problems of what happens when cd fails.
Example:
cd yourdir
rm -f yourglob # oops things got removed from where you started if yourdir did not exist.
Two things:
Are you sure you know what the dirname command is doing? It's going to remove the top-level directory as well as the leading slash on whatever you call it on. I would make absolutely sure that the output of dirname is exactly what you think it is.
For example, /home/user/ will output /home.
You're using ~, which references the $HOME variable in your environment. You didn't mention where the cron is listed, but make sure it's not being run as a different user. Root's ~ and your ~ will be two completely different directories.
That's all I can think of. I hope that helps!
Add echo $MAINDIR after
MAINDIR=`dirname $0`
cd $MAINDIR
So you can see, if the contents of MAINDIR is correct.
Also you can run sh with -x or add set -x to the beginning of the script to see what happens.
I have a python script let's name it script1.py. I can run it in the terminal this way:
python /path/script1.py
...
but I want to run like a command-line program:
arbitraryname
...
how can i do it ?
You use a shebang line at the start of your script:
#!/usr/bin/env python
make the file executable:
chmod +x arbitraryname
and put it in a directory on your PATH (can be a symlink):
cd ~/bin/
ln -s ~/some/path/to/myscript/arbitraryname
There are three parts:
Add a 'shebang' at the top of your script which tells how to execute your script
Give the script 'run' permissions.
Make the script in your PATH so you can run it from anywhere.
Adding a shebang
You need to add a shebang at the top of your script so the shell knows which interpreter to use when parsing your script. It is generally:
#!path/to/interpretter
To find the path to your python interpretter on your machine you can run the command:
which python
This will search your PATH to find the location of your python executable. It should come back with a absolute path which you can then use to form your shebang. Make sure your shebang is at the top of your python script:
#!/usr/bin/python
Run Permissions
You have to mark your script with run permissions so that your shell knows you want to actually execute it when you try to use it as a command. To do this you can run this command:
chmod +x myscript.py
Add the script to your path
The PATH environment variable is an ordered list of directories that your shell will search when looking for a command you are trying to run. So if you want your python script to be a command you can run from anywhere then it needs to be in your PATH. You can see the contents of your path running the command:
echo $PATH
This will print out a long line of text, where each directory is seperated by a semicolon. Whenever you are wondering where the actual location of an executable that you are running from your PATH, you can find it by running the command:
which <commandname>
Now you have two options: Add your script to a directory already in your PATH, or add a new directory to your PATH. I usually create a directory in my user home directory and then add it the PATH. To add things to your path you can run the command:
export PATH=/my/directory/with/pythonscript:$PATH
Now you should be able to run your python script as a command anywhere. BUT! if you close the shell window and open a new one, the new one won't remember the change you just made to your PATH. So if you want this change to be saved then you need to add that command at the bottom of your .bashrc or .bash_profile
Add the following line to the beginning script1.py
#!/usr/bin/env python
and then make the script executable:
$ chmod +x script1.py
If the script resides in a directory that appears in your PATH variable, you can simply type
$ script1.py
Otherwise, you'll need to provide the full path (either absolute or relative). This includes the current working directory, which should not be in your PATH.
$ ./script1.py
You need to use a hashbang. Add it to the first line of your python script.
#! <full path of python interpreter>
Then change the file permissions, and add the executing permission.
chmod +x <filename>
And finally execute it using
./<filename>
If its in the current directory,