python: 3.6.8
code:
temp_file = tempfile.NamedTemporaryFile()
getui.download(path=f"{path}{file}", localPath=temp_file.name)
temp_file.seek(0)
import ipdb;ipdb.set_trace()
item = {"file": temp_file, "filename": file}
queue.put(item)
I cannot read anything from tempfile, but can read its content using open, as another file:
ipdb> temp_file.read(10)
b''
ipdb> temp_file.seek(0)
0
ipdb> temp_file.read(10)
b''
ipdb> f = open(temp_file.name)
ipdb> f.read(10)
'6C1D91DB-F'
ipdb>
Why is this happening?
You have not posted enough detail to be able to reproduce this, but the likelihood is that your getui.download is deleting and recreating the file, so it will be writing to a different inode than the one which Python has open (which no longer has any associated directory entry). When you reopen the file, you are now looking at the same inode as was written by getui.download.
To demonstrate by means of example what is likely to be happening, here is an example in which (in Linux) some basic file operations are performed to do a delete-and-recreate (using ctrl-Z to temporarily suspend the Python process while this is done):
>>> temp_file = tempfile.NamedTemporaryFile()
>>> temp_file.name
'/tmp/tmpo4j5k0ul'
>>> os.stat(temp_file.name).st_ino # <=== look at the inode number
42
>>> [[ctrl-Z pressed here]]
[1]+ Stopped python3
$ ls -li /tmp/tmpo4j5k0ul
42 -rw------- 1 myuser mygroup 0 Aug 14 10:33 /tmp/tmpo4j5k0ul
$ rm /tmp/tmpo4j5k0ul # <=== delete
$ echo hello > /tmp/tmpo4j5k0ul # <=== create new file
$ ls -li /tmp/tmpo4j5k0ul # <=== see the new inode number
41 -rw-rw-r-- 1 myuser mygroup 6 Aug 14 10:34 /tmp/tmpo4j5k0ul
$ fg # <=== return to the python session
python3
>>> os.fstat(temp_file.fileno()).st_ino # <=== recheck the inode number
42 # <=== still the old one
>>> temp_file.seek(0)
0
>>> temp_file.read()
b''
>>> f = open(temp_file.name) # <=== reopen from the filename
>>> os.fstat(f.fileno()).st_ino # <=== recheck the inode number
41 # <=== the new one this time
>>> f.read()
'hello\n'
Regarding how to fix this, you might find that your getui.download has an option to pass a file object rather than a file name, or at least to open to an existing file for writing rather than deleting and recreating it. Again, without exact details of where getui.download comes from, it is hard to give definite advice, but this will be the principle that you need to follow.
Probably something really stupid I am doing but can someone please assist. All I am trying to do is stat a file. Python will not make this happen, when I debug my python variables I can stat in the shell with it's output. Please see below:
[root#logmaster output]# cat /usr/local/nagios/libexec/check_logrip_log_not_stale.py
import os
import sys
import datetime
import time
# Nagios return values
nagiosRetValOk = 0
nagiosRetValWarn = 1
nagiosRetValCritical = 2
# Below is the filename I am after
#logrip-out-2016-03-19-1458386101
dateFormat = datetime.datetime.now().strftime("%Y-%m-%d")
logFormat = "/home/famnet/logs/output/logrip-out-%s-*" % dateFormat
print os.stat(logFormat)
Here is what happens when I run the basic script:
[root#logmaster output]# python /usr/local/nagios/libexec/check_logrip_log_not_stale.py
Traceback (most recent call last):
File "/usr/local/nagios/libexec/check_logrip_log_not_stale.py", line 36, in <module>
print os.stat(logFormat)
OSError: [Errno 2] No such file or directory: '/home/famnet/logs/output/logrip-out-2016-03-19-*'
Please forgive me if this is an easy waste of time for some experts.
Thanks,
However when I take the output of my print debug and run in the shell it works.
[root#logmaster output]# stat /home/famnet/logs/output/logrip-out-2016-03-19-*
File: `/home/famnet/logs/output/logrip-out-2016-03-19-1458386101'
Size: 42374797 Blocks: 82776 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 36590817 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 504/ famnet) Gid: ( 1100/ staff)
Access: 2016-03-19 07:15:01.725794193 -0400
Modify: 2016-03-19 07:44:09.847793116 -0400
Change: 2016-03-19 07:44:09.847793116 -0400
Expansion of wildcards is a feature of many common shells, such as bash in this case. It is not a feature of the system call underlying os.stat.
If you want to call os.stat against more than one file, you'll have to list them (using something like glob.glob) first, then call os.stat once per path. Something like this:
for full_path in glob.glob(logFormat):
print os.stat(full_path)
Observe as well that a path with a wildcard may expand to multiple concrete paths, which can work with the command-line STAT(1), but will certainly break os.stat which takes only a single path argument.
os.stat won't auto-expand the wildcard... try using glob
Is it possible to have a main Python process run as root, but then start a Python subprocess (multiprocessing) run as a different user? I also want that user's home directory and shell settings to apply just like if the Python interpreter had been started by that user.
Right now, when using os.setuid and gid, the permissions are correct, but the home directory is still incorrect as you'd expect:
>>> import os
>>> os.setgid(1000)
>>> os.setuid(1000)
>>> print os.getuid()
1000
>>> print os.getgid()
1000
>>> f = open('/etc/shadow')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 13] Permission denied: '/etc/shadow'
>>> os.system('echo $HOME')
/root
0
Any ideas?
Try:
os.environ['HOME'] = '/home/blah'
os.system("ls ~")
AFAIK, this is as good as you are likely to get, because installations with network home directories, or users with bash profiles that set HOME to wacky values aren't going to be replicable in pure Python, and could have side effects.
I'm using a crontab to run a maintenance script for my minecraft server. Most of the time it works fine, unless the crontab tries to use the restart script. If I run the restart script manually, there aren't any issues. Because I believe it's got to do with path names, I'm trying to make sure it's always doing any minecraft command FROM the minecraft directory. So I'm encasing the command in pushd/popd:
os.system("pushd /directory/path/here")
os.system("command to sent to minecraft")
os.system("popd")
Below is an interactive session taking minecraft out of the equation. A simple "ls" test. As you can see, it does not at all run the os.system command from the pushd directory, but instead from /etc/ which is the directory in which I was running python to illustrate my point.Clearly pushd isn't working via python, so I'm wondering how else I can achieve this. Thanks!
>>> def test():
... import os
... os.system("pushd /home/[path_goes_here]/minecraft")
... os.system("ls")
... os.system("popd")
...
>>> test()
~/minecraft /etc
DIR_COLORS cron.weekly gcrypt inputrc localtime mime.types ntp ppp rc3.d sasldb2 smrsh vsftpd.ftpusers
DIR_COLORS.xterm crontab gpm-root.conf iproute2 login.defs mke2fs.conf ntp.conf printcap rc4.d screenrc snmp vsftpd.tpsave
X11 csh.cshrc group issue logrotate.conf modprobe.d odbc.ini profile rc5.d scsi_id.config squirrelmail vz
adjtime csh.login group- issue.net logrotate.d motd odbcinst.ini profile.d rc6.d securetty ssh warnquota.conf
aliases cyrus.conf host.conf java lvm mtab openldap protocols redhat-release security stunnel webalizer.conf
alsa dbus-1 hosts jvm lynx-site.cfg multipath.conf opt quotagrpadmins resolv.conf selinux sudoers wgetrc
alternatives default hosts.allow jvm-commmon lynx.cfg my.cnf pam.d quotatab rndc.key sensors.conf sysconfig xinetd.conf
bashrc depmod.d hosts.deny jwhois.conf mail named.caching-nameserver.conf passwd rc rpc services sysctl.conf xinetd.d
blkid dev.d httpd krb5.conf mail.rc named.conf passwd- rc.d rpm sestatus.conf termcap yum
cron.d environment imapd.conf ld.so.cache mailcap named.rfc1912.zones pear.conf rc.local rsyslog.conf setuptool.d udev yum.conf
cron.daily exports imapd.conf.tpsave ld.so.conf mailman netplug php.d rc.sysinit rwtab shadow updatedb.conf yum.repos.d
cron.deny filesystems init.d ld.so.conf.d makedev.d netplug.d php.ini rc0.d rwtab.d shadow- vimrc
cron.hourly fonts initlog.conf libaudit.conf man.config nscd.conf pki rc1.d samba shells virc
cron.monthly fstab inittab libuser.conf maven nsswitch.conf postfix rc2.d sasl2 skel vsftpd
sh: line 0: popd: directory stack empty
===
(CentOS server with python 2.4)
In Python 2.5 and later, I think a better method would be using a context manager, like so:
import contextlib
import os
#contextlib.contextmanager
def pushd(new_dir):
previous_dir = os.getcwd()
os.chdir(new_dir)
try:
yield
finally:
os.chdir(previous_dir)
You can then use it like the following:
with pushd('somewhere'):
print os.getcwd() # "somewhere"
print os.getcwd() # "wherever you started"
By using a context manager you will be exception and return value safe: your code will always cd back to where it started from, even if you throw an exception or return from inside the context block.
You can also nest pushd calls in nested blocks, without having to rely on a global directory stack:
with pushd('somewhere'):
# do something
with pushd('another/place'):
# do something else
# do something back in "somewhere"
Each shell command runs in a separate process. It spawns a shell, executes the pushd command, and then the shell exits.
Just write the commands in the same shell script:
os.system("cd /directory/path/here; run the commands")
A nicer (perhaps) way is with the subprocess module:
from subprocess import Popen
Popen("run the commands", shell=True, cwd="/directory/path/here")
pushd and popd have some added functionality: they store previous working directories in a stack - in other words, you can pushd five times, do some stuff, and popd five times to end up where you started. You're not using that here, but it might be useful for others searching for the questions like this. This is how you can emulate it:
# initialise a directory stack
pushstack = list()
def pushdir(dirname):
global pushstack
pushstack.append(os.getcwd())
os.chdir(dirname)
def popdir():
global pushstack
os.chdir(pushstack.pop())
I don't think you can call pushd from within an os.system() call:
>>> import os
>>> ret = os.system("pushd /tmp")
sh: pushd: not found
Maybe just maybe your system actually provides a pushd binary that triggers a shell internal function (I think I've seen this on FreeBSD beforeFreeBSD has some tricks like this, but not for pushd), but the current working directory of a process cannot be influenced by other processes -- so your first system() starts a shell, runs a hypothetical pushd, starts a shell, runs ls, starts a shell, runs a hypothetical popd... none of which influence each other.
You can use os.chdir("/home/path/") to change path: http://docs.python.org/library/os.html#os-file-dir
No need to use pushd -- just use os.chdir:
>>> import os
>>> os.getcwd()
'/Users/me'
>>> os.chdir('..')
>>> os.getcwd()
'/Users'
>>> os.chdir('me')
>>> os.getcwd()
'/Users/me'
Or make a class to use with 'with'
import os
class pushd: # pylint: disable=invalid-name
__slots__ = ('_pushstack',)
def __init__(self, dirname):
self._pushstack = list()
self.pushd(dirname)
def __enter__(self):
return self
def __exit__(self, exec_type, exec_val, exc_tb) -> bool:
# skip all the intermediate directories, just go back to the original one.
if self._pushstack:
os.chdir(self._pushstack.pop(0)))
if exec_type:
return False
return True
def popd(self) -> None:
if len(self._pushstack):
os.chdir(self._pushstack.pop())
def pushd(self, dirname) -> None:
self._pushstack.append(os.getcwd())
os.chdir(dirname)
with pushd(dirname) as d:
... do stuff in that dirname
d.pushd("../..")
d.popd()
If you really need a stack, i.e. if you want to do several pushd and popd,
see naught101 above.
If not, simply do:
olddir = os.getcwd()
os.chdir('/directory/path/here')
os.system("command to sent to minecraft")
os.chdir(olddir)
If I call os.stat() on a broken symlink, python throws an OSError exception. This makes it useful for finding them. However, there are a few other reasons that os.stat() might throw a similar exception. Is there a more precise way of detecting broken symlinks with Python under Linux?
A common Python saying is that it's easier to ask forgiveness than permission. While I'm not a fan of this statement in real life, it does apply in a lot of cases. Usually you want to avoid code that chains two system calls on the same file, because you never know what will happen to the file in between your two calls in your code.
A typical mistake is to write something like:
if os.path.exists(path):
os.unlink(path)
The second call (os.unlink) may fail if something else deleted it after your if test, raise an Exception, and stop the rest of your function from executing. (You might think this doesn't happen in real life, but we just fished another bug like that out of our codebase last week - and it was the kind of bug that left a few programmers scratching their head and claiming 'Heisenbug' for the last few months)
So, in your particular case, I would probably do:
try:
os.stat(path)
except OSError, e:
if e.errno == errno.ENOENT:
print 'path %s does not exist or is a broken symlink' % path
else:
raise e
The annoyance here is that stat returns the same error code for a symlink that just isn't there and a broken symlink.
So, I guess you have no choice than to break the atomicity, and do something like
if not os.path.exists(os.readlink(path)):
print 'path %s is a broken symlink' % path
This is not atomic but it works.
os.path.islink(filename) and not os.path.exists(filename)
Indeed by RTFM
(reading the fantastic manual) we see
os.path.exists(path)
Return True if path refers to an existing path. Returns False for broken symbolic links.
It also says:
On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists.
So if you are worried about permissions, you should add other clauses.
os.lstat() may be helpful. If lstat() succeeds and stat() fails, then it's probably a broken link.
Can I mention testing for hardlinks without python? /bin/test has the FILE1 -ef FILE2 condition that is true when files share an inode.
Therefore, something like find . -type f -exec test \{} -ef /path/to/file \; -print works for hard link testing to a specific file.
Which brings me to reading man test and the mentions of -L and -h which both work on one file and return true if that file is a symbolic link, however that doesn't tell you if the target is missing.
I did find that head -0 FILE1 would return an exit code of 0 if the file can be opened and a 1 if it cannot, which in the case of a symbolic link to a regular file works as a test for whether it's target can be read.
os.path
You may try using realpath() to get what the symlink points to, then trying to determine if it's a valid file using is file.
(I'm not able to try that out at the moment, so you'll have to play around with it and see what you get)
I used this variant, When symlink is broken it will return false for the path.exists and true for path.islink, so combining this two facts we may use the following:
def kek(argum):
if path.exists("/root/" + argum) == False and path.islink("/root/" + argum) == True:
print("The path is a broken link, location: " + os.readlink("/root/" + argum))
else:
return "No broken links fond"
I'm not a python guy but it looks like os.readlink()? The logic I would use in perl is to use readlink() to find the target and the use stat() to test to see if the target exists.
Edit: I banged out some perl that demos readlink. I believe perl's stat and readlink and python's os.stat() and os.readlink()are both wrappers for the system calls, so this should translate reasonable well as proof of concept code:
wembley 0 /home/jj33/swap > cat p
my $f = shift;
while (my $l = readlink($f)) {
print "$f -> $l\n";
$f = $l;
}
if (!-e $f) {
print "$f doesn't exist\n";
}
wembley 0 /home/jj33/swap > ls -l | grep ^l
lrwxrwxrwx 1 jj33 users 17 Aug 21 14:30 link -> non-existant-file
lrwxrwxrwx 1 root users 31 Oct 10 2007 mm -> ../systems/mm/20071009-rewrite//
lrwxrwxrwx 1 jj33 users 2 Aug 21 14:34 mmm -> mm/
wembley 0 /home/jj33/swap > perl p mm
mm -> ../systems/mm/20071009-rewrite/
wembley 0 /home/jj33/swap > perl p mmm
mmm -> mm
mm -> ../systems/mm/20071009-rewrite/
wembley 0 /home/jj33/swap > perl p link
link -> non-existant-file
non-existant-file doesn't exist
wembley 0 /home/jj33/swap >
I had a similar problem: how to catch broken symlinks, even when they occur in some parent dir? I also wanted to log all of them (in an application dealing with a fairly large number of files), but without too many repeats.
Here is what I came up with, including unit tests.
fileutil.py:
import os
from functools import lru_cache
import logging
logger = logging.getLogger(__name__)
#lru_cache(maxsize=2000)
def check_broken_link(filename):
"""
Check for broken symlinks, either at the file level, or in the
hierarchy of parent dirs.
If it finds a broken link, an ERROR message is logged.
The function is cached, so that the same error messages are not repeated.
Args:
filename: file to check
Returns:
True if the file (or one of its parents) is a broken symlink.
False otherwise (i.e. either it exists or not, but no element
on its path is a broken link).
"""
if os.path.isfile(filename) or os.path.isdir(filename):
return False
if os.path.islink(filename):
# there is a symlink, but it is dead (pointing nowhere)
link = os.readlink(filename)
logger.error('broken symlink: {} -> {}'.format(filename, link))
return True
# ok, we have either:
# 1. a filename that simply doesn't exist (but the containing dir
does exist), or
# 2. a broken link in some parent dir
parent = os.path.dirname(filename)
if parent == filename:
# reached root
return False
return check_broken_link(parent)
Unit tests:
import logging
import shutil
import tempfile
import os
import unittest
from ..util import fileutil
class TestFile(unittest.TestCase):
def _mkdir(self, path, create=True):
d = os.path.join(self.test_dir, path)
if create:
os.makedirs(d, exist_ok=True)
return d
def _mkfile(self, path, create=True):
f = os.path.join(self.test_dir, path)
if create:
d = os.path.dirname(f)
os.makedirs(d, exist_ok=True)
with open(f, mode='w') as fp:
fp.write('hello')
return f
def _mklink(self, target, path):
f = os.path.join(self.test_dir, path)
d = os.path.dirname(f)
os.makedirs(d, exist_ok=True)
os.symlink(target, f)
return f
def setUp(self):
# reset the lru_cache of check_broken_link
fileutil.check_broken_link.cache_clear()
# create a temporary directory for our tests
self.test_dir = tempfile.mkdtemp()
# create a small tree of dirs, files, and symlinks
self._mkfile('a/b/c/foo.txt')
self._mklink('b', 'a/x')
self._mklink('b/c/foo.txt', 'a/f')
self._mklink('../..', 'a/b/c/y')
self._mklink('not_exist.txt', 'a/b/c/bad_link.txt')
bad_path = self._mkfile('a/XXX/c/foo.txt', create=False)
self._mklink(bad_path, 'a/b/c/bad_path.txt')
self._mklink('not_a_dir', 'a/bad_dir')
def tearDown(self):
# Remove the directory after the test
shutil.rmtree(self.test_dir)
def catch_check_broken_link(self, expected_errors, expected_result, path):
filename = self._mkfile(path, create=False)
with self.assertLogs(level='ERROR') as cm:
result = fileutil.check_broken_link(filename)
logging.critical('nothing') # trick: emit one extra message, so the with assertLogs block doesn't fail
error_logs = [r for r in cm.records if r.levelname is 'ERROR']
actual_errors = len(error_logs)
self.assertEqual(expected_result, result, msg=path)
self.assertEqual(expected_errors, actual_errors, msg=path)
def test_check_broken_link_exists(self):
self.catch_check_broken_link(0, False, 'a/b/c/foo.txt')
self.catch_check_broken_link(0, False, 'a/x/c/foo.txt')
self.catch_check_broken_link(0, False, 'a/f')
self.catch_check_broken_link(0, False, 'a/b/c/y/b/c/y/b/c/foo.txt')
def test_check_broken_link_notfound(self):
self.catch_check_broken_link(0, False, 'a/b/c/not_found.txt')
def test_check_broken_link_badlink(self):
self.catch_check_broken_link(1, True, 'a/b/c/bad_link.txt')
self.catch_check_broken_link(0, True, 'a/b/c/bad_link.txt')
def test_check_broken_link_badpath(self):
self.catch_check_broken_link(1, True, 'a/b/c/bad_path.txt')
self.catch_check_broken_link(0, True, 'a/b/c/bad_path.txt')
def test_check_broken_link_badparent(self):
self.catch_check_broken_link(1, True, 'a/bad_dir/c/foo.txt')
self.catch_check_broken_link(0, True, 'a/bad_dir/c/foo.txt')
# bad link, but shouldn't log a new error:
self.catch_check_broken_link(0, True, 'a/bad_dir/c')
# bad link, but shouldn't log a new error:
self.catch_check_broken_link(0, True, 'a/bad_dir')
if __name__ == '__main__':
unittest.main()
For Python 3, you can use the pathlib module. From its docs,
If the path points to a symlink, exists() returns whether the symlink points to an existing file or directory.
So this works too.
import pathlib
path = pathlib.Path("/path/to/somewhere")
if path.is_symlink() and not path.exists():
print(f"found dangling symlink at {path}")