using subprocess over different files python - python

I've got a problem with a short script, it'd be great if you could have a look!
import os
import subprocess
root = "/Users/software/fmtomov1.0/remaker_lastplot/source_relocation/observed_arrivals_loc3d"
def loop_loc3d(file_in):
"""Loops loc3d over the source files"""
return subprocess.call (['loc3d'], shell=True)
def relocation ():
for subdir, dirs, files in os.walk(root):
for file in files:
file_in = open(os.path.join(subdir, file), 'r')
return loop_loc3d(file_in)
I think the script is quite easy to understand, it's very simple. However I'm not getting the result wanted. In a few word I just want 'loc3d' to operate over all the files contents present in the 'observed_arrivals_loc3d' directory, which means that I need to open all the files and that's what I've actually done. In fact, if I try to 'print files' after:
for subdir, dirs, files in os.walk(root)
I'll get the name of every file. Furthermore, if I try a 'print file_in' after
file_in = open(os.path.join(subdir, file), 'r')
I get something like this line for every file:
<open file '/Users/software/fmtomov1.0/remaker_lastplot/source_relocation/observed_arrivals_loc3d/EVENT2580', mode 'r' at 0x78fe38>
subprocess has been tested alone on only one file and it's working.
Overall I'm getting no errors but just -11 which means absolutely nothing to me. The output from loc3d should be completly different.
So does the code look fine to you? Is there anything I'm missing? Any suggestion?
Thanks for your help!


I assume you would call loc3d filename from the CLI. If so, then:
def loop_loc3d(filename):
"""Loops loc3d over the source files"""
return subprocess.call (['loc3d',filename])
def relocation():
for subdir, dirs, files in os.walk(root):
for file in files:
filename = os.path.join(subdir, file)
return loop_loc3d(filename)
In other words, don't open the file yourself, let loc3d do it.


Currently your relocation method will return after the first iteration (for the first file). You shouldn't need to return at all.
def loop_loc3d(filename):
"""Loops loc3d over the source files"""
return subprocess.call (['loc3d',filename])
def relocation ():
for subdir, dirs, files in os.walk(root):
for file in files:
filename = os.path.join(subdir, file)
loop_loc3d(filename)
This is only one of the issues. The other is concerning loc3d itself. Try providing the full path for loc3d.


-11 exit code might mean that the command killed by signal Segmentation fault.
It is a bug in loc3d. A well-behaved program should not produce 'Segmentation fault' on any user input.
Feed loc3d only files that it can understand. Print filenames or use subprocess.check_call() to find out which file it doesn't like:
#!/usr/bin/env python
import fnmatch
import os
import subprocess
def loc3d_files(root):
for dirpath, dirs, files in os.walk(root, topdown=True):
# skip hidden directories
dirs[:] = [d for d in dirs if not d.startswith('.')]
# process only known files
for file in fnmatch.filter(files, "*some?pattern[0-9][0-9].[ch]"):
yield os.path.join(dirpath, file)
for path in loc3d_files(root):
print path
subprocess.check_call(['loc3d', path]) # raise on any error


Just found out that loc3d, as unutbu said, relies on several variables and in the specific case one called 'observal_arrivals' that I have to create and delete every time from my directory. In Pythonic terms it means:
import os
import shutil
import subprocess
def loop_loc3d(file_in):
"""Loops loc3d over the source files"""
return subprocess.call(["loc3d"], shell=True)
path = "/Users/software/fmtomo/remaker_lastplot/source_relocation"
path2 = "/Users/Programming/working_directory/2test"
new_file_name = 'observed_arrivals'
def define_object_file ():
for filename in os.listdir("."):
file_in = os.rename (filename, new_file_name) # get the observal_arrivals file
file_in = shutil.copy ("/Users/simone/Programming/working_directory/2test/observed_arrivals", "/Users/software/fmtomo/remaker_lastplot/source_relocation")
os.chdir(path) # goes where loc3d is
loop_loc3d (file_in)
os.remove("/Users/software/fmtomo/remaker_lastplot/source_relocation/observed_arrivals")
os.remove ("/Users/Programming/working_directory/2test/observed_arrivals")
os.chdir(path2)
Now, this is working very well, so it should answer my question. I guess it's quite easy to understand, it's just copying, changing dir and that kind of stuff.

Related

Access denied from "shutil.rmtree('dir_path')", despite emptying dir_path's contents [duplicate]

How can I delete the contents of a local folder in Python?
The current project is for Windows, but I would like to see *nix also.

import os, shutil
folder = '/path/to/folder'
for filename in os.listdir(folder):
file_path = os.path.join(folder, filename)
try:
if os.path.isfile(file_path) or os.path.islink(file_path):
os.unlink(file_path)
elif os.path.isdir(file_path):
shutil.rmtree(file_path)
except Exception as e:
print('Failed to delete %s. Reason: %s' % (file_path, e))

You can simply do this:
import os
import glob
files = glob.glob('/YOUR/PATH/*')
for f in files:
os.remove(f)
You can of course use an other filter in you path, for example : /YOU/PATH/*.txt for removing all text files in a directory.

You can delete the folder itself, as well as all its contents, using shutil.rmtree:
import shutil
shutil.rmtree('/path/to/folder')
shutil.rmtree(path, ignore_errors=False, onerror=None)
Delete an entire directory tree; path must point to a directory (but not a symbolic link to a directory). If ignore_errors is true, errors resulting from failed removals will be ignored; if false or omitted, such errors are handled by calling a handler specified by onerror or, if that is omitted, they raise an exception.

Expanding on mhawke's answer this is what I've implemented. It removes all the content of a folder but not the folder itself. Tested on Linux with files, folders and symbolic links, should work on Windows as well.
import os
import shutil
for root, dirs, files in os.walk('/path/to/folder'):
for f in files:
os.unlink(os.path.join(root, f))
for d in dirs:
shutil.rmtree(os.path.join(root, d))

I'm surprised nobody has mentioned the awesome pathlib to do this job.
If you only want to remove files in a directory it can be a oneliner
from pathlib import Path
[f.unlink() for f in Path("/path/to/folder").glob("*") if f.is_file()]
To also recursively remove directories you can write something like this:
from pathlib import Path
from shutil import rmtree
for path in Path("/path/to/folder").glob("**/*"):
if path.is_file():
path.unlink()
elif path.is_dir():
rmtree(path)

Using rmtree and recreating the folder could work, but I have run into errors when deleting and immediately recreating folders on network drives.
The proposed solution using walk does not work as it uses rmtree to remove folders and then may attempt to use os.unlink on the files that were previously in those folders. This causes an error.
The posted glob solution will also attempt to delete non-empty folders, causing errors.
I suggest you use:
folder_path = '/path/to/folder'
for file_object in os.listdir(folder_path):
file_object_path = os.path.join(folder_path, file_object)
if os.path.isfile(file_object_path) or os.path.islink(file_object_path):
os.unlink(file_object_path)
else:
shutil.rmtree(file_object_path)

This:
removes all symbolic links
dead links
links to directories
links to files
removes subdirectories
does not remove the parent directory
Code:
for filename in os.listdir(dirpath):
filepath = os.path.join(dirpath, filename)
try:
shutil.rmtree(filepath)
except OSError:
os.remove(filepath)
As many other answers, this does not try to adjust permissions to enable removal of files/directories.

Using os.scandir and context manager protocol in Python 3.6+:
import os
import shutil
with os.scandir(target_dir) as entries:
for entry in entries:
if entry.is_dir() and not entry.is_symlink():
shutil.rmtree(entry.path)
else:
os.remove(entry.path)
Earlier versions of Python:
import os
import shutil
# Gather directory contents
contents = [os.path.join(target_dir, i) for i in os.listdir(target_dir)]
# Iterate and remove each item in the appropriate manner
[shutil.rmtree(i) if os.path.isdir(i) and not os.path.islink(i) else os.remove(i) for i in contents]

Notes: in case someone down voted my answer, I have something to explain here.
Everyone likes short 'n' simple answers. However, sometimes the reality is not so simple.
Back to my answer. I know shutil.rmtree() could be used to delete a directory tree. I've used it many times in my own projects. But you must realize that the directory itself will also be deleted by shutil.rmtree(). While this might be acceptable for some, it's not a valid answer for deleting the contents of a folder (without side effects).
I'll show you an example of the side effects. Suppose that you have a directory with customized owner and mode bits, where there are a lot of contents. Then you delete it with shutil.rmtree() and rebuild it with os.mkdir(). And you'll get an empty directory with default (inherited) owner and mode bits instead. While you might have the privilege to delete the contents and even the directory, you might not be able to set back the original owner and mode bits on the directory (e.g. you're not a superuser).
Finally, be patient and read the code. It's long and ugly (in sight), but proven to be reliable and efficient (in use).
Here's a long and ugly, but reliable and efficient solution.
It resolves a few problems which are not addressed by the other answerers:
It correctly handles symbolic links, including not calling shutil.rmtree() on a symbolic link (which will pass the os.path.isdir() test if it links to a directory; even the result of os.walk() contains symbolic linked directories as well).
It handles read-only files nicely.
Here's the code (the only useful function is clear_dir()):
import os
import stat
import shutil
# http://stackoverflow.com/questions/1889597/deleting-directory-in-python
def _remove_readonly(fn, path_, excinfo):
# Handle read-only files and directories
if fn is os.rmdir:
os.chmod(path_, stat.S_IWRITE)
os.rmdir(path_)
elif fn is os.remove:
os.lchmod(path_, stat.S_IWRITE)
os.remove(path_)
def force_remove_file_or_symlink(path_):
try:
os.remove(path_)
except OSError:
os.lchmod(path_, stat.S_IWRITE)
os.remove(path_)
# Code from shutil.rmtree()
def is_regular_dir(path_):
try:
mode = os.lstat(path_).st_mode
except os.error:
mode = 0
return stat.S_ISDIR(mode)
def clear_dir(path_):
if is_regular_dir(path_):
# Given path is a directory, clear its content
for name in os.listdir(path_):
fullpath = os.path.join(path_, name)
if is_regular_dir(fullpath):
shutil.rmtree(fullpath, onerror=_remove_readonly)
else:
force_remove_file_or_symlink(fullpath)
else:
# Given path is a file or a symlink.
# Raise an exception here to avoid accidentally clearing the content
# of a symbolic linked directory.
raise OSError("Cannot call clear_dir() on a symbolic link")

As a oneliner:
import os
# Python 2.7
map( os.unlink, (os.path.join( mydir,f) for f in os.listdir(mydir)) )
# Python 3+
list( map( os.unlink, (os.path.join( mydir,f) for f in os.listdir(mydir)) ) )
A more robust solution accounting for files and directories as well would be (2.7):
def rm(f):
if os.path.isdir(f): return os.rmdir(f)
if os.path.isfile(f): return os.unlink(f)
raise TypeError, 'must be either file or directory'
map( rm, (os.path.join( mydir,f) for f in os.listdir(mydir)) )

I used to solve the problem this way:
import shutil
import os
shutil.rmtree(dirpath)
os.mkdir(dirpath)

To delete all files inside a folder a I use:
import os
for i in os.listdir():
os.remove(i)

To delete all the files inside the directory as well as its sub-directories, without removing the folders themselves, simply do this:
import os
mypath = "my_folder" #Enter your path here
for root, dirs, files in os.walk(mypath, topdown=False):
for file in files:
os.remove(os.path.join(root, file))
# Add this block to remove folders
for dir in dirs:
os.rmdir(os.path.join(root, dir))
# Add this line to remove the root folder at the end
os.rmdir(mypath)

You might be better off using os.walk() for this.
os.listdir() doesn't distinguish files from directories and you will quickly get into trouble trying to unlink these. There is a good example of using os.walk() to recursively remove a directory here, and hints on how to adapt it to your circumstances.

If you are using a *nix system, why not leverage the system command?
import os
path = 'folder/to/clean'
os.system('rm -rf %s/*' % path)

I konw it's an old thread but I have found something interesting from the official site of python. Just for sharing another idea for removing of all contents in a directory. Because I have some problems of authorization when using shutil.rmtree() and I don't want to remove the directory and recreate it. The address original is http://docs.python.org/2/library/os.html#os.walk. Hope that could help someone.
def emptydir(top):
if(top == '/' or top == "\\"): return
else:
for root, dirs, files in os.walk(top, topdown=False):
for name in files:
os.remove(os.path.join(root, name))
for name in dirs:
os.rmdir(os.path.join(root, name))

I had to remove files from 3 separate folders inside a single parent directory:
directory
folderA
file1
folderB
file2
folderC
file3
This simple code did the trick for me: (I'm on Unix)
import os
import glob
folders = glob.glob('./path/to/parentdir/*')
for fo in folders:
file = glob.glob(f'{fo}/*')
for f in file:
os.remove(f)
Hope this helps.

Yet Another Solution:
import sh
sh.rm(sh.glob('/path/to/folder/*'))

Well, I think this code is working. It will not delete the folder and you can use this code to delete files having the particular extension.
import os
import glob
files = glob.glob(r'path/*')
for items in files:
os.remove(items)

Pretty intuitive way of doing it:
import shutil, os
def remove_folder_contents(path):
shutil.rmtree(path)
os.makedirs(path)
remove_folder_contents('/path/to/folder')

use this function
import os
import glob
def truncate(path):
files = glob.glob(path+'/*.*')
for f in files:
os.remove(f)
truncate('/my/path')

Use the method bellow to remove the contents of a directory, not the directory itself:
import os
import shutil
def remove_contents(path):
for c in os.listdir(path):
full_path = os.path.join(path, c)
if os.path.isfile(full_path):
os.remove(full_path)
else:
shutil.rmtree(full_path)

Answer for a limited, specific situation:
assuming you want to delete the files while maintainig the subfolders tree, you could use a recursive algorithm:
import os
def recursively_remove_files(f):
if os.path.isfile(f):
os.unlink(f)
elif os.path.isdir(f):
for fi in os.listdir(f):
recursively_remove_files(os.path.join(f, fi))
recursively_remove_files(my_directory)
Maybe slightly off-topic, but I think many would find it useful

I resolved the issue with rmtree makedirs by adding time.sleep() between:
if os.path.isdir(folder_location):
shutil.rmtree(folder_location)
time.sleep(.5)
os.makedirs(folder_location, 0o777)

the easiest way to delete all files in a folder/remove all files
import os
files = os.listdir(yourFilePath)
for f in files:
os.remove(yourFilePath + f)

This should do the trick just using the OS module to list and then remove!
import os
DIR = os.list('Folder')
for i in range(len(DIR)):
os.remove('Folder'+chr(92)+i)
Worked for me, any problems let me know!

Extract full Path and File Name

Attempting to write a function that walks a file system and returns the absolute path and filename for use in another function.
Example "/testdir/folderA/222/filename.ext".
Having tried multiple versions of this I cannot seem to get it to work properly.
filesCheck=[]
def findFiles(filepath):
files=[]
for root, dirs, files in os.walk(filepath):
for file in files:
currentFile = os.path.realpath(file)
print (currentFile)
if os.path.exists(currentFile):
files.append(currentFile)
return files
filesCheck = findFiles(/testdir)
This returns
"filename.ext" (only one).
Substitute in currentFile = os.path.join(root, file) for os.path.realpath(file) and it goes into a loop in the first directory. Tried os.path.join(dir, file) and it fails as one of my folders is named 222.
I have gone round in circles and get somewhat close but haven't been able to get it to work.
Running on Linux with Python 3.6

There's a several things wrong with your code.
There are multiple values are being assigned to the variable name files.
You're not adding the root directory to each filename os.walk() returns which can be done with os.path.join().
You're not passing a string to the findFiles() function.
If you fix those things there's no longer a need to call os.path.exists() because you can be sure it does.
Here's a working version:
import os
def findFiles(filepath):
found = []
for root, dirs, files in os.walk(filepath):
for file in files:
currentFile = os.path.realpath(os.path.join(root, file))
found.append(currentFile)
return found
filesCheck = findFiles('/testdir')
print(filesCheck)

Hi I think this is what you need. Perhaps you could give it a try :)
from os import walk
path = "C:/Users/SK/Desktop/New folder"
files = []
for (directoryPath, directoryNames, allFiles) in walk(path):
for file in allFiles:
files.append([file, f"{directoryPath}/{file}"])
print(files)
Output:
[ ['index.html', 'C:/Users/SK/Desktop/New folder/index.html'], ['test.py', 'C:/Users/SK/Desktop/New folder/test.py'] ]

Why doesn't python standalone file manipulation function work?

I have a file test.py which works good. see code:
import os
import shutil
import re
for root, dirs, files in os.walk("../config/"):
for file in files:
print os.path.join(root, file)
if file.endswith(".txt") and file.startswith("default_"):
file_name = os.path.basename(os.path.join(root, file))
file_name = re.sub(r'default_','',file_name)
shutil.copy(os.path.join(root, file),os.path.join(root,file_name))
but when I wrapped the code into a function and put it in another file config.py. And I called the function in another file as config.copy_default_files(), it doesn't work. So I put a raw_input() in the end of the function to see if the function is executed, and it did print 'miao', but it didn't print out the list of files. And no file is generated or copied.
I am so so confused.Can someone explain it to me please? Any help would be great appreciated. Let me know if you need more information on it. Manythanks!
import os
import shutil
import re
def copy_default_files(work_dir = "../config/"):
for root, dirs, files in os.walk(work_dir):
for file in files:
print os.path.join(root, file)
if file.endswith(".txt") and file.startswith("default_"):
file_name = os.path.basename(os.path.join(root, file))
file_name = re.sub(r'default_','',file_name)
shutil.copy(os.path.join(root, file),os.path.join(root,file_name))
raw_input('miao')
return 0

Defining the function is not enough. You also need to call it:
copy_default_files()
or
config.copy_default_files()
(depending on whether you're running config.py as a script or importing it as a module).

Get an arbitrary file from a directory tree

I'd like to get a path to an arbitrary text file (with .txt suffix) which is present somewhere in the directory tree. The file should not be hidden or in hidden directory.
I tried to write the code but it looks little cumbersome. How would you improve it to avoid useless steps?
def getSomeTextFile(rootDir):
"""Get the path to arbitrary text file under the rootDir"""
for root, dirs, files in os.walk(rootDir):
for f in files:
path = os.path.join(root, f)
ext = path.split(".")[-1]
if ext.lower() == "txt":
# it shouldn't be hidden or in hidden directory
if not "/." in path:
return path
return "" # there isn't any text file

Using os.walk (like in your example) is definitely a good start.
You can use fnmatch (link to the docs here) to simplify the rest of the code.
E.g:
...
if fnmatch.fnmatch(file, '*.txt'):
print file
...

I'd use fnmatch instead of string manipulations.
import os, os.path, fnmatch
def find_files(root, pattern, exclude_hidden=True):
""" Get the path to arbitrary .ext file under the root dir """
for dir, _, files in os.walk(root):
for f in fnmatch.filter(files, pattern):
path = os.path.join(dir, f)
if '/.' not in path or not exclude_hidden:
yield path
I've also rewritten the function to be more generic (and "pythonic"). To get just one pathname, call it like this:
first_txt = next(find_files(some_dir, '*.txt'))

Python program to traverse directories and read file information

I'm just getting started with Python but already have found it much more productive than Bash shell scripting.
I'm trying to write a Python script that will traverse every directory that branches from the directory I launch the script in, and for each file it encounters, load an instance of this class:
class FileInfo:
def __init__(self, filename, filepath):
self.filename = filename
self.filepath = filepath
The filepath attribute would be the full absolute path from root (/). Here's the pseudocode mockup for what I'd like the main program to do:
from (current directory):
for each file in this directory,
create an instance of FileInfo and load the file name and path
switch to a nested directory, or if there is none, back out of this directory
I've been reading about os.walk() and ok.path.walk(), but I'd like some advice about what the most straightforward way to implement this in Python would be. Thanks in advance.

I'd use os.walk doing the following:
def getInfos(currentDir):
infos = []
for root, dirs, files in os.walk(currentDir): # Walk directory tree
for f in files:
infos.append(FileInfo(f,root))
return infos

Try
info = []
for path, dirs, files in os.walk("."):
info.extend(FileInfo(filename, path) for filename in files)
or
info = [FileInfo(filename, path)
for path, dirs, files in os.walk(".")
for filename in files]
to get a list of one FileInfo instance per file.

Try it
import os
for item in os.walk(".", "*"):
print(item)

Categories