I'm running into problems with the difference in environment between SConscripts and custom tools under site_tools.
I'm trying to find the absolute path of a string, which is fine in SConscript:
print File('#something').get_abspath()
>> /home/projects/something
Similar code in site_tools:
from SCons.Node.FS import File
print File('#something').get_abspath()
>> TypeError: __init__() takes exactly 4 arguments (2 given)
So clearly the File() constructed in SConscript is not a SCons.Node.FS.File.
So the questions are:
1) What is the File object in a SConscript, and how can I get one in a builder/site_tools?
2) How can I get the absolute path in site_tools? I need this because I'm constructing command-line arguments for tools which need absolute paths.
This looks like an XY problem to me. If you need the absolute path of single files or directories in your command lines, the construction of the full path should be left to SCons itself.
Please check out the MAN page (http://scons.org/doc/production/HTML/scons-man.html or "man scons") under the section "Variable Substitution". You're able to append further specifiers (like abspath in your case) to variables, and they get substituted before command execution.
So the command line string for your Builder action could look something like this:
$MYTOOL -i $SOURCE.abspath $TARGET.abspath
Related
I am trying to replicate another researcher's findings by using the Python file that he added as a supplement to his paper. It is the first time I am diving into Python, so the error might be extremely simple to fix, yet after two days I haven't still. For context, in the Readme file there's the following instruction:
"To run the script, make sure Python2 is installed. Put all files into one folder designated as “cf_dir”.
In the script I get an error at the following lines:
if __name__ == '__main__':
cf_dir, cf_file, cf_phys_file = sys.argv[1:4]
os.chdir(cf_dir)
cf = pd.read_csv(cf_file)
cf_phys = pd.read_csv(cf_phys_file)
ValueError: need more than 0 values to unpack
The "cf_file" and "cf_phys_file" are two major components of all files that are in the one folder named "cf_dir". The "cf_phys_file" relates only to two survey question's (Q22 and Q23), and the "cf_file" includes all other questions 1-21. Now it seems that the code is meant to retrieve those two files from the directory? Only for the "cf_phys_file" the columns 1:4 are needed. The current working directory is already set at the right location.
The path where I located "cf_dir" is as follows:
C:\Users\Marc-Marijn Ossel\Documents\RSM\Thesis\Data\Suitable for ML\Data en Artikelen\Per task Suitability for Machine Learning score readme\cf_dir
Alternative option in readme file,
In the readme file there's this option, but also here I cannot understand how to direct the path to the right location:
"Run the following command in an open terminal (substituting for file names
below): python cfProcessor_AEAPnP.py cf_dir cf_file cf_phys_file task_file jobTaskRatingFile
jobDataFile OESfile
This should generate the data and plots as necessary."
When I run that in "Command Prompt", I get the following error, and I am not sure how to set the working directory correctly.
- python: can't open file 'cfProcessor_AEAPnP.py': [Errno 2] No such file or directory
Thanks for the reading, and I hope there's someone who could help me!
Best regards & stay safe out there during Corona!!
Marc
cf_dir, cf_file, cf_phys_file = sys.argv[1:4]
means, the python file expects few arguments when called.
In order to run
python cfProcessor_AEAPnP.py cf_dir cf_file cf_phys_file task_file jobTaskRatingFile jobDataFile OESfile
the command prompt should be in that folder.
So, open command prompt and type
cd path_to_the_folder_where_ur_python_file_is_located
Now, you would have reached the path of the python file.
Also, make sure you give full path in double quotes for the arguments.
I am doing a script using SoX to merge multiple audio file together.
This command works in the terminal
sox &(ls *.mp3) out.mp3
but if I try using it inside a python script by calling subprocess.run() it doesn't
subprocess.run(['sox', '$(ls *.mp3)', 'out.mp3'])
> sox FAIL formats: can't open input file `$(ls *.mp3)': No such file or
> directory
I image that is because of the subshell operation, but I don't know how to pass it correctly.
I also tried, as some other post suggested, passing the argument shell=True but then it says
> sox FAIL sox: Not enough input filenames specified
I am in the same working directory and I also tried supplying the full path but doesn't work either.
I could just write a bash script and call it, but I would like to know how to deal in this scenario with Python.
you want to use shell=True to force subprocess to run your command through the shell interpreter and parse the wildcards/sub-commands. However this (depending on the platform) imposes that the argument is passed as string, not as list of parameters. A lot of constraints for a lazy & unsafe way of doing it.
Wait. You can do without shell=True using glob.glob:
subprocess.run(['sox'] + glob.glob('*.mp3') + ['out.mp3'])
Would be better to check if there actually are mp3 files in the current folder so:
input_files = glob.glob('*.mp3')
if input_files:
subprocess.run(['sox'] + input_files + ['out.mp3'])
else:
raise Exception("No mp3 files")
if you get the "No mp3 files" message, then check the current directory. It's always good to use a parameter for the input directory, and avoid relying on the current directory (glob.glob(os.path.join(input_directory,'*.mp3')))
This question already has answers here:
os.makedirs doesn't understand "~" in my path
(3 answers)
Closed 6 months ago.
I am trying to read a file in another directory. My current script is in path/to/dir. However, I want to read a file in ~/. I'm not sure how to do this.
I tried f = open("~/.file") but I am getting an error
IOError: [Errno 2] No such file or directory: '~/.file'
use os.path.expanduser:
import os
with open(os.path.expanduser('~/.file')) as f:
print f.read()
This should work
from os.path import expanduser
home = expanduser("~")
os.path.expanduser will help you
The short answer is: use os.path.expanduser, as m.wasowski shows:
f = open(os.path.expanduser("~/.file"))
But why do you have to do this?
Well, ~/.file doesn't actually mean what you think it does. It's just asking for a file named .file in a subdirectory named ~ in the current working directory.
Why does this work on the command line? Bcause shells don't just pass your arguments along as typed, they do all kinds of complicated processing.
This is why ./myscript.py *txt gives you ['./myscript.py', 'a.txt', 'b.txt', 'c.txt', 'd e f.txt'] in your argv—because the shell does glob expansion on *.txt by looking in the directory for everything that matches that pattern and turns it into a.txt b.txt c.txt "d e f.txt".
In the same way, it expands ~/.file into /Users/me/.file by looking at the appropriate environment variables and platform-specific defaults and so on. You can read the gory details for bash; things are similar with dash, tcsh, and most other shells (except Windows cmd and PowerShell).
Python talks to the filesystem directly, it doesn't go through the shell to do it. So, if you want to do the same things shells do, you have to do them explicitly: glob.glob, os.path.expanduser, os.path.expandvars, subprocess.check_output, etc.
The cool thing about doing it all in Python is that it works the same way everywhere. Even on Windows, where the shell doesn't know what ~/.file means, and the right answer may be some hideous thing like 'C:\\Documents and Settings\\me\\Documents' or '\\\\FileServer\\RoamingProfiles\\me', os.path.expanduser will give you the right hideous thing.
I am trying to make a python script that will open a directory, apply a perl script to every file in that directory and get its out put in either multiple text files or just one.
I currently have:
import shlex, subprocess
arg_str = "perl tilt.pl *.pdb > final.txt"
arg = shlex.split(arg_str)
import os
framespdb = os.listdir("prac_frames")
for frames in framespdb:
subprocess.Popen(arg, stdout=True)
I keep getting *.pdb not found. I am very new to all of this so any help trying to complete this script would help.
*.pdb not found means exactly that - there won't be a *.pdb in whatever directory you're running the script... and as I read the code - I don't see anything to imply it's within 'frames' when it runs the perl script.
you probably need os.chdir(path) before the Popen.
How do I "cd" in Python?
...using a python script to run somewhat dubious syscalls to perl may offend some people but everyone's done it.. aside from that I'd point out:
always specify full paths (this becomes a problem if you will later say, want to run your job automatically from cron or an environment that doesn't have your PATH).
i.e. 'which perl' - put that full path in.
./.pdb would be better but not as good as the fullpath/.pdb (which you could use instead of the os.chdir option).
subprocess.Popen(arg, stdout=True)
does not expand filename wildcards. To handle your *.pdb, use shell=True.
import subprocess
sample_file_directory = "..." # directory where file is
SCRIPT_DIR = "..." # directory where script is
p = subprocess.Popen([SCRIPT_DIR,sample_file_directory,'min','edgelen'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
min_side_len, err = p.communicate()
print len(err)
So i have this script that analyzes .ply files (3d file format) and gives me data about the file. I am trying to find out in my directory which files are corrupted and which aren't. So I am trying to use the subprocess library to run the script to find an arbitrary feature of the .ply files (in this case the minimum edge length) If err has anything in it, it means it couldn't retrieve the arbitrary feature and the file is corrupted. Here, I am only running it on one file. However, I keep getting an error.
p = subprocess.Popen([SCRIPT_DIR,sample_file_directory,'min','edgelen'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Can anyone help me figure out why I am getting this error? Both my data directory and script directory are valid. Also, sorry if I left anything out that's important. First time posting.
Is your SCRIPT_DIR variable the absolute path or a relative path valid in the location you're running the script? If it's the latter, it may not work, as the subprocess may start in a different environment that has a different working directory.
The problem lies here:
p = subprocess.Popen([SCRIPT_DIR,sample_file_directory,'min','edgelen'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
Notice this part:
Popen([SCRIPT_DIR,sample_file_directory,'min','edgelen']
Now, see, you already know that subprocess takes a list. But the problem is that you are passing the directory as part of the list. The list is formatted like this:
[command, arg1, arg2...]
Now, when you run that command, you are doing this:
[SCRIPT_DIR -> command
sample_file_directory -> arg1
'min' -> arg2
'edgelen' -> arg3]
See the problem? You are passing the script's directory as the command to run and the script's name as an argument. Import the os module and do this:
p = subprocess.Popen([os.path.join(SCRIPT_DIR,sample_file_directory),'min','edgelen'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
This does this:
[SCRIPT_DIR and sample_file_directory joined together -> command
'min' -> arg1
'edgelen' -> arg2]
os.path.join automatically adds the path separator.
subprocess.Popen() won't combine path components for you. From what I see, you're only providing a directory. You'll need to add the target executable's name to the path being given to Popen. Try the following:
p = subprocess.Popen([os.path.join(SCRIPT_DIR,SCRIPT_NAME),sample_file_directory,'min','edgelen'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
"One other thing it could potentially be is that you're trying to run a 32-bit binary on a 64-bit system without the 32-bit libraries installed. I actually ran into this issue earlier today. A fix is documented here. I don't know anything about your system, so this may be completely irrelevant advice, but I figured it wouldn't hurt to mention it." Dan Albert. He came up with the answer to the solution! Thanks so much Dan!