I need to write to a file (truncating) and the path it is on itself might not exist). For example, I want to write to /tmp/a/b/c/config, but /tmp/a itself might not exist. Then, open('/tmp/a/b/c/config', 'w') would not work, obviously, since it doesn't make the necessary directories. However, I can work with the following code:
import os
config_value = 'Foo=Bar' # Temporary placeholder
config_dir = '/tmp/a/b/c' # Temporary placeholder
config_file_path = os.path.join(config_dir, 'config')
if not os.path.exists(config_dir):
os.makedirs(config_dir)
with open(config_file_path, 'w') as f:
f.write(config_value)
Is there a more Pythonic way to do this? Both Python 2.x and Python 3.x would be nice to know (even though I use 2.x in my code, due to dependency reasons).
If you're repeating this pattern in multiple places, you could create your own Context Manager that extends open() and overloads __enter__():
import os
class OpenCreateDirs(open):
def __enter__(self, filename, *args, **kwargs):
file_dir = os.path.dirname(filename)
if not os.path.exists(file_dir):
os.makedirs(file_dir)
super(OpenCreateDirs, self).__enter__(filename, *args, **kwargs)
Then your code becomes:
import os
config_value = 'Foo=Bar' # Temporary placeholder
config_file_path = os.path.join('/tmp/a/b/c', 'config')
with OpenCreateDirs(config_file_path, 'w') as f:
f.write(config_value)
The first method to be called when you run with open(...) as f: is open.__enter__(). So by creating directories before calling super(...).__enter__(), you create the directories before attempting to open the file.
Related
I am stuck with a really weird problem.
I have a file called: "test.txt".
It is in the same directory with views.py. But I can't read it... FileNotFoundError.
But if I create read_file.py in the same directory with views.py and test.txt, it works absolutely fine.
What is wrong with views? Is this some sort of restriction by Django?
This code works on read_file, doesn't work on views.py:
fkey = open("test.txt", "rb")
key = fkey.read()
I think the problem may be relative vs absolute file handling. Using the following Python snippet, you can work out where the interpreter's current working directory is:
import os
print(os.getcwd())
You should notice it's not in the same directory as the views.py, as views.py is likely being invoked/called from another module. You may choose to change all your open calls to include the whole path to these files, or use an implementation like this:
import os
# This function can be used as a replacement for `open`
# which will allow you to access files from the file.
def open_relative(path, flag="r"):
# This builds the relative path by joining the current directory
# plus the current filename being executed.
relative_path = os.path.join(os.path.dirname(__file__), path)
return open(relative_path, flag) # return file handler
Then, this should work:
fkey = open_relative("test.txt", "rb")
key = fkey.read()
Hope this helps!
I made a class where I have a model in it. I also made a QT GUI which allows me to select the file (.py) of my class model in order to import it and use it.
What I'm seeking for is a way to know if I already import a modul (corresponding to the file I selected) and reload it if it has changed.
to import my module from a path, i use:
fileName = QFileDialog.getOpenFileName(self,"Open Data File" , "", "data files (*.py)")
if fileName[0]=='':
return
fileName = str(fileName[0])
abspath = os.path.dirname(os.path.abspath(__file__))
self.fileName = os.path.relpath(fileName,abspath)
(filepath, filename) = os.path.split(fileName)
sys.path.append(filepath)
(shortname, extension) = os.path.splitext(filename)
mod = __import__(shortname)
instead of mod = __import__(shortname) I need to make a test to know if I import or reload the module.
EDIT
if shortname not in sys.modules:
mod = __import__(shortname)
else:
importlib.reload(__import__(shortname))
I try the previous code. however I still have an issue. when I do importlib.reload(__import__(shortname)), I seem that when I modify the module in between the first import and the second one, I still load the first form of the class. I added self.A=0 in the class __init__ but I don't have the acces to it.
In Python, I am trying to create a tar with two empty directories in it, and then add a list of files to each empty directory within the tar. I have tried doing it this way below, but It does not work.
def ISIP_tar_files():
with tarfile.open("eeg_files.tar", "w") as f:
ep_dir = tarfile.TarInfo("Eplilepsy Reports")
not_ep_dir = tarfile.TarInfo("Non Epilepsy Reports")
ep_dir.type = not_ep_dir.type = tarfile.DIRTYPE
f.addfile(ep_dir)
f.addfile(not_ep_dir)
with ep_dir.open():
for name in ep_list:
f.tarfile.add(name)
I honestly did not believe it would work, but it was worth a try because I couldn't find any other solutions on Google. This is just one module of the code, and it does not include the main program or imports. ep_list is a list of files with paths, it looks similar to:
ep_list = [/data/foo/bar/file.txt, /data/foo/bar2/file2.txt, ...]
Any Sugegstions?
import tarfile
import os
ep_list = ['./foo/bar/file.txt', './foo/bar/file2.txt']
def ISIP_tar_files():
with tarfile.open("eeg_files.tar", "w") as f:
ep_dir = tarfile.TarInfo("Eplilepsy Reports")
not_ep_dir = tarfile.TarInfo("Non Epilepsy Reports")
ep_dir.type = not_ep_dir.type = tarfile.DIRTYPE
ep_dir.mode = not_ep_dir.mode = 0o777
f.addfile(ep_dir)
f.addfile(not_ep_dir)
for name in ep_list:
f.add(name, arcname="Eplilepsy Reports/" + os.path.basename(name), recursive=False)
The directory file permission mode should be made executable at least for the owner. Otherwise it cannot be extracted.
arcname is the alternative name for the file in the archive.
recursive means whether or not keep the original directories added recursively, its default value is True.
I'm trying to modify a test.tar.gz into test.tgz but it dosn't work. Here is the command:
temporalFolder= /home/albertserres/*.tar.gz
subprocess.call(["mv",temporalFolder,"*.tgz"])
It sends me error that the file doesn't exist. Why?
Also I just need to modify after the dot, not the entire name, because I'll probably doesn't know the file name, and if I do *.tgz it rename the file *.tgz and I want to keep the original name.
This should work:
import shutil
orig_file = '/home/albertserres/test.tar.gz'
new_file = orig_file.replace('tar.gz', 'tgz')
shutil.move(orig_file, new_file)
And if you want to do that for several files:
import shutil
import glob
for orig_file in glob.glob('/home/albertserres/*.tar.gz'):
new_file = orig_file.replace('tar.gz', 'tgz')
shutil.move(orig_file, new_file)
rename would probably be easier.
rename 's/\.tar\.gz/\.tgz/' *.tar.gz
In your case
params = "rename 's/\.tar\.gz/\.tgz/' /home/albertserres/*.tar.gz"
subprocess.call(params, shell=True)
To replace all .tar.gz file extensions with .tgz file extensions in a given directory (similar to #hitzg's answer):
#!/usr/bin/env python
from glob import glob
for filename in glob(b'/home/albertserres/*.tar.gz'):
new = bytearray(filename)
new[-len(b'tar.gz'):] = b'tgz'
os.rename(filename, new) # or os.replace() for portability
The code replaces tar.gz only at the end of the name. It raises an error if new is an existing directory otherwise it silently replaces the file on Unix.
So I am attempting to monkeypatch the shutil module in order to use a recent fix to their make_archive function that allows the creation fo large zip files.
I am proof of concepting something so figured a quick hack to get this issue out of the way would allow me to get on with what I want to do.
My code:
import shutil
import os
def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None):
zip_filename = base_name + ".zip"
archive_dir = os.path.dirname(base_name)
if not os.path.exists(archive_dir):
if logger is not None:
logger.info("creating %s", archive_dir)
if not dry_run:
os.makedirs(archive_dir)
# If zipfile module is not available, try spawning an external 'zip'
# command.
try:
import zipfile
except ImportError:
zipfile = None
if zipfile is None:
shutil._call_external_zip(base_dir, zip_filename, verbose, dry_run)
else:
if logger is not None:
logger.info("creating '%s' and adding '%s' to it",
zip_filename, base_dir)
if not dry_run:
zip = zipfile.ZipFile(zip_filename, "w",
compression=zipfile.ZIP_DEFLATED,
allowZip64=True) # this is the extra argument
for dirpath, dirnames, filenames in os.walk(base_dir):
for name in filenames:
path = os.path.normpath(os.path.join(dirpath, name))
if os.path.isfile(path):
zip.write(path, path)
if logger is not None:
logger.info("adding '%s'", path)
zip.close()
shutil._make_zipfile = _make_zipfile
# This function calls _make_zipfile when it runs
shutil.make_archive('blah', someargs)
So the issue is... it doesn't do anything. I am clearly doing something stupid, but for the life of me I can not see what it is. I am assuming there is something obvious that I have become blind to after looking at it for so long, so need some fresh eyes. I have tried following methods/checking against answers described in these:
Monkey-patch Python class
Python monkey patch private function
and What is a monkey patch?
plus some others. No joy
You'll have to update the _ARCHIVE_FORMATS mapping; it stores a reference to the function on import, so before you can patch it. shutil.make_archive() uses that mapping, and not the _make_zipfile function directly.
You can use the public shutil.register_archive_format() function to re-define the zip archiver:
shutil.register_archive_format('zip', _make_zipfile, description='ZIP file')
This replaces the existing callable registered for the zip format.