We Keep Coding

How to skip files being used in another program? [duplicate]

I my application, i have below requests:
1. There has one thread will regularly record some logs in file. The log file will be rollovered in certain interval. for keeping the log files small.
2. There has another thread also will regularly to process these log files. ex: Move the log files to other place, parse the log's content to generate some log reports.
But, there has a condition is the second thread can not process the log file that's using to record the log. in code side, the pseudocode similars like below:
#code in second thread to process the log files
for logFile in os.listdir(logFolder):
if not file_is_open(logFile) or file_is_use(logFile):
ProcessLogFile(logFile) # move log file to other place, and generate log report....
So, how do i check is a file is already open or is used by other process?
I did some research in internet. And have some results:
try:
myfile = open(filename, "r+") # or "a+", whatever you need
except IOError:
print "Could not open file! Please close Excel!"
I tried this code, but it doesn't work, no matter i use "r+" or "a+" flag
try:
os.remove(filename) # try to remove it directly
except OSError as e:
if e.errno == errno.ENOENT: # file doesn't exist
break
This code can work, but it can not reach my request, since i don't want to delete the file to check if it is open.

An issue with trying to find out if a file is being used by another process is the possibility of a race condition. You could check a file, decide that it is not in use, then just before you open it another process (or thread) leaps in and grabs it (or even deletes it).
Ok, let's say you decide to live with that possibility and hope it does not occur. To check files in use by other processes is operating system dependant.
On Linux it is fairly easy, just iterate through the PIDs in /proc. Here is a generator that iterates over files in use for a specific PID:
def iterate_fds(pid):
dir = '/proc/'+str(pid)+'/fd'
if not os.access(dir,os.R_OK|os.X_OK): return
for fds in os.listdir(dir):
for fd in fds:
full_name = os.path.join(dir, fd)
try:
file = os.readlink(full_name)
if file == '/dev/null' or \
re.match(r'pipe:\[\d+\]',file) or \
re.match(r'socket:\[\d+\]',file):
file = None
except OSError as err:
if err.errno == 2:
file = None
else:
raise(err)
yield (fd,file)
On Windows it is not quite so straightforward, the APIs are not published. There is a sysinternals tool (handle.exe) that can be used, but I recommend the PyPi module psutil, which is portable (i.e., it runs on Linux as well, and probably on other OS):
import psutil
for proc in psutil.process_iter():
try:
# this returns the list of opened files by the current process
flist = proc.open_files()
if flist:
print(proc.pid,proc.name)
for nt in flist:
print("\t",nt.path)
# This catches a race condition where a process ends
# before we can examine its files
except psutil.NoSuchProcess as err:
print("****",err)

I like Daniel's answer, but for Windows users, I realized that it's safer and simpler to rename the file to the name it already has. That solves the problems brought up in the comments to his answer. Here's the code:
import os
f = 'C:/test.xlsx'
if os.path.exists(f):
try:
os.rename(f, f)
print 'Access on file "' + f +'" is available!'
except OSError as e:
print 'Access-error on file "' + f + '"! \n' + str(e)

You can check if a file has a handle on it using the next function (remember to pass the full path to that file):
import psutil
def has_handle(fpath):
for proc in psutil.process_iter():
try:
for item in proc.open_files():
if fpath == item.path:
return True
except Exception:
pass
return False

I know I'm late to the party but I also had this problem and I used the lsof command to solve it (which I think is new from the approaches mentioned above). With lsof we can basically check for the processes that are using this particular file.
Here is how I did it:
from subprocess import check_output,Popen, PIPE
try:
lsout=Popen(['lsof',filename],stdout=PIPE, shell=False)
check_output(["grep",filename], stdin=lsout.stdout, shell=False)
except:
#check_output will throw an exception here if it won't find any process using that file
just write your log processing code in the except part and you are good to go.

Instead on using os.remove() you may use the following workaround on Windows:
import os
file = "D:\\temp\\test.pdf"
if os.path.exists(file):
try:
os.rename(file,file+"_")
print "Access on file \"" + str(file) +"\" is available!"
os.rename(file+"_",file)
except OSError as e:
message = "Access-error on file \"" + str(file) + "\"!!! \n" + str(e)
print message

You can use inotify to watch for activity in file system. You can watch for file close events, indicating that a roll-over has happened. You should also add additional condition on file-size. Make sure you filter out file close events from the second thread.

A slightly more polished version of one of the answers from above.
from pathlib import Path
def is_file_in_use(file_path):
path = Path(file_path)
if not path.exists():
raise FileNotFoundError
try:
path.rename(path)
except PermissionError:
return True
else:
return False

On Windows, you can also directly retrieve the information by leveraging on the NTDLL/KERNEL32 Windows API. The following code returns a list of PIDs, in case the file is still opened/used by a process (including your own, if you have an open handle on the file):
import ctypes
from ctypes import wintypes
path = r"C:\temp\test.txt"
# -----------------------------------------------------------------------------
# generic strings and constants
# -----------------------------------------------------------------------------
ntdll = ctypes.WinDLL('ntdll')
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
NTSTATUS = wintypes.LONG
INVALID_HANDLE_VALUE = wintypes.HANDLE(-1).value
FILE_READ_ATTRIBUTES = 0x80
FILE_SHARE_READ = 1
OPEN_EXISTING = 3
FILE_FLAG_BACKUP_SEMANTICS = 0x02000000
FILE_INFORMATION_CLASS = wintypes.ULONG
FileProcessIdsUsingFileInformation = 47
LPSECURITY_ATTRIBUTES = wintypes.LPVOID
ULONG_PTR = wintypes.WPARAM
# -----------------------------------------------------------------------------
# create handle on concerned file with dwDesiredAccess == FILE_READ_ATTRIBUTES
# -----------------------------------------------------------------------------
kernel32.CreateFileW.restype = wintypes.HANDLE
kernel32.CreateFileW.argtypes = (
wintypes.LPCWSTR, # In lpFileName
wintypes.DWORD, # In dwDesiredAccess
wintypes.DWORD, # In dwShareMode
LPSECURITY_ATTRIBUTES, # In_opt lpSecurityAttributes
wintypes.DWORD, # In dwCreationDisposition
wintypes.DWORD, # In dwFlagsAndAttributes
wintypes.HANDLE) # In_opt hTemplateFile
hFile = kernel32.CreateFileW(
path, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, None, OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS, None)
if hFile == INVALID_HANDLE_VALUE:
raise ctypes.WinError(ctypes.get_last_error())
# -----------------------------------------------------------------------------
# prepare data types for system call
# -----------------------------------------------------------------------------
class IO_STATUS_BLOCK(ctypes.Structure):
class _STATUS(ctypes.Union):
_fields_ = (('Status', NTSTATUS),
('Pointer', wintypes.LPVOID))
_anonymous_ = '_Status',
_fields_ = (('_Status', _STATUS),
('Information', ULONG_PTR))
iosb = IO_STATUS_BLOCK()
class FILE_PROCESS_IDS_USING_FILE_INFORMATION(ctypes.Structure):
_fields_ = (('NumberOfProcessIdsInList', wintypes.LARGE_INTEGER),
('ProcessIdList', wintypes.LARGE_INTEGER * 64))
info = FILE_PROCESS_IDS_USING_FILE_INFORMATION()
PIO_STATUS_BLOCK = ctypes.POINTER(IO_STATUS_BLOCK)
ntdll.NtQueryInformationFile.restype = NTSTATUS
ntdll.NtQueryInformationFile.argtypes = (
wintypes.HANDLE, # In FileHandle
PIO_STATUS_BLOCK, # Out IoStatusBlock
wintypes.LPVOID, # Out FileInformation
wintypes.ULONG, # In Length
FILE_INFORMATION_CLASS) # In FileInformationClass
# -----------------------------------------------------------------------------
# system call to retrieve list of PIDs currently using the file
# -----------------------------------------------------------------------------
status = ntdll.NtQueryInformationFile(hFile, ctypes.byref(iosb),
ctypes.byref(info),
ctypes.sizeof(info),
FileProcessIdsUsingFileInformation)
pidList = info.ProcessIdList[0:info.NumberOfProcessIdsInList]
print(pidList)

I provided one solution. please see the following code.
def isFileinUsed(ifile):
widlcard = "/proc/*/fd/*"
lfds = glob.glob(widlcard)
for fds in lfds:
try:
file = os.readlink(fds)
if file == ifile:
return True
except OSError as err:
if err.errno == 2:
file = None
else:
raise(err)
return False
You can use this function to check if a file is in used.
Note:
This solution only can be used for Linux system.

Python 2.7.13 Livestreamer Error drving me crazy

I'm relatively new in python, but I'm trying to learn hard. For the last couple of days tried to solve an error on this script :
import requests
import subprocess
import json
import sys
import multiprocessing
import time
import random
channel_url = "gaming.youtube.com/game/"
processes = [5]
def get_channel():
# Reading the channel name - passed as an argument to this script
if len(sys.argv) >= 2:
global channel_url
channel_url += sys.argv[1]
else:
print "An error has occurred while trying to read arguments. Did you specify the channel?"
sys.exit(1)
def get_proxies():
# Reading the list of proxies
try:
lines = [line.rstrip("\n") for line in open("proxylist.txt")]
except IOError as e:
print "An error has occurred while trying to read the list of proxies: %s" % e.strerror
sys.exit(1)
return lines
def get_url():
# Getting the json with all data regarding the stream
try:
response = subprocess.Popen(
["livestreamer.exe", "--http-header", "Client-ID=ewvlchtxgqq88ru9gmfp1gmyt6h2b93",
channel_url, "-j"], stdout=subprocess.PIPE).communicate()[0]
except subprocess.CalledProcessError:
print "An error has occurred while trying to get the stream data. Is the channel online? Is the channel name correct?"
sys.exit(1)
except OSError:
print "An error has occurred while trying to use livestreamer package. Is it installed? Do you have Python in your PATH variable?"
# Decoding the url to the worst quality of the stream
try:
url = json.loads(response)['streams']['audio_only']['url']
except:
try:
url = json.loads(response)['streams']['worst']['url']
except (ValueError, KeyError):
print "An error has occurred while trying to get the stream data. Is the channel online? Is the channel name correct?"
sys.exit(1)
return url
def open_url(url, proxy):
# Sending HEAD requests
while True:
try:
with requests.Session() as s:
response = s.head(url, proxies=proxy)
print "Sent HEAD request with %s" % proxy["http"]
time.sleep(20)
except requests.exceptions.Timeout:
print " Timeout error for %s" % proxy["http"]
except requests.exceptions.ConnectionError:
print " Connection error for %s" % proxy["http"]
def prepare_processes():
global processes
proxies = get_proxies()
n = 0
if len(proxies) < 1:
print "An error has occurred while preparing the process: Not enough proxy servers. Need at least 1 to function."
sys.exit(1)
for proxy in proxies:
# Preparing the process and giving it its own proxy
processes.append(
multiprocessing.Process(
target=open_url, kwargs={
"url": get_url(), "proxy": {
"http": proxy}}))
print '.',
print ''
if __name__ == "__main__":
print "Obtaining the channel..."
get_channel()
print "Obtained the channel"
print "Preparing the processes..."
prepare_processes()
print "Prepared the processes"
print "Booting up the processes..."
# Timer multiplier
n = 8
# Starting up the processes
for process in processes:
time.sleep(random.randint(1, 5) * n)
process.daemon = True
process.start()
if n > 1:
n -= 1
# Running infinitely
while True:
time.sleep(1)
ERROR :
python test999.py UCbadKBJT1bE14AtfBnsw27g
Obtaining the channel...
Obtained the channel
Preparing the processes...
An error has occurred while trying to use livestreamer package. Is it installed? Do you have Python in your PATH variable?
Traceback (most recent call last):
File "test999.py", line 115, in <module>
prepare_processes()
File "test999.py", line 103, in prepare_processes
"url": get_url(), "proxy": {
File "test999.py", line 67, in get_url
url = json.loads(response)['streams']['worst']['url']
UnboundLocalError: local variable 'response' referenced before assignment
I had tried on windows, installed all modules (and updated them ) livestreamer, rtmdump, dlls and other needed binary files.
On linux : installed livestreamer, requests, json and all neede modules. Still can't resolve it. Please help !

response = subprocess.Popen() in try clause. When it fails, no response lkocal variable. You should assign response before try,and process None that subporcess.Popen() may return.

How to solve FTP Error 2 Not a Directory

I am using the below code for downloading files from a ftp server.But I am getting an error [Errno 2] No such file or directory:, but the file present in the server and I can able download it via terminal. Can anyone help me!!
import ftplib
import os
remotpath='folder/subfolder'
try:
ftpclient = ftplib.FTP('ftp.xxxx.com')
ftpclient.login('user', 'pass')
ftpclient.cwd(remotpath)
print "login succeessfull"
files = ftpclient.nlst()
for eachFile in files:
saveTo = os.path.join(remotpath,eachFile)
if (not os.path.exists(saveTo)):
try:
ftpclient.retrbinary('RETR ' + eachFile, open(saveTo, 'wb').write)
#logging.info('\tdownloaded ' + saveTo)
downloaded += 1
except BaseException as e:
print('\terror downloading inside first %s - %s' % (eachFile, e.__str__()))
except ftplib.error_perm:
print('\terror downloading inside second %s - %s' % (eachFile, ftplib.error_perm))
except Exception as e:
print e

Does the destination directory ./folder/subfolder exist?
If not you need to create it before downloading files. Either do so using your OS commands (mkdir), or in Python using os.makedirs() :
import os
try:
os.makedirs(remotpath)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
You can add it somewhere before the for loop.
On another issue, the order of you exception handlers means that all exceptions raised in the inner try block would be handled in the except BaseException statement. This means that ftplib.error_perm will be caught in that statement because BaseException is more general, and not in the ftplib.error_perm statement as you might expect.
You should reorder your except statements in order of increasing generality.

NameError while handling command line arguments

I was trying out a few python test scripts with sqlite3.
Here is the script that I wrote
#!/usr/bin/env python
from sqlite3 import dbapi2 as sqlite
from sys import argv,exit
db_name = "filenames.db"
def define_db():
try:
conn = sqlite.connect(db_name)
except IOError as e:
print "problem while creating/connecting the db:",e.args[0]
exit(1)
return conn
def write_db(conn,cursor,fni):
conn.execute("CREATE TABLE IF NOT EXISTS file (filenames TEXT UNIQUE)")
query = "INSERT OR REPLACE INTO file VALUES($filenames)"
cursor.execute(query,[fni])
cursor.close()
conn.commit()
conn.close()
print fni,"should now be in the db"
exit(0)
if __name__ == "__main__":
if len(argv) == 2:
etag = argv[1]
else:
print "no argument given - stopping now"
exit(1)
conn = define_db()
cursor = conn.cursor()
write_db(conn,cursor,fni)
I keep getting this error and was not able to solve it.
Traceback (most recent call last):
File "blah.py", line 37, in <module>
write_db(conn,cursor,fni)
NameError: name 'fni' is not defined
Any idea what the problem is.
At this moment I use python 2.7.3

The last line of your script refers to a name fni that is not defined.

You have not defined the variable "fni", but you are using it.

Static analysis tools like pyflakes or pylint can be useful to catch silly errors like this
If you wrote the bulk of the code in a function (so it doesn't assume blub is a global variable, which don't make pyflakes/pylint complain):
def main():
if len(argv) == 2:
blub = argv[1]
else:
print "no argument given - stopping now"
exit(1)
conn = define_db()
cursor = conn.cursor()
write_db(conn,cursor,fni)
if __name__ == "__main__":
main()
...then you would get a pair of errors, which points out exactly what the error is (you stored the argument in blub, but tried to access it with fni):
$ pip install pyflakes
$ pyflakes example.py
example.py:30: local variable 'blub' is assigned to but never used
example.py:37: undefined name 'fni'

Continue after exception

Help me figure out how the script will keep on running when WMI can't connect to host and go to next computer in list. Should I use continue after except?
import wmi
MachineList = ["Computer1","Computer2","Computer3"]
try:
for machines in MachineList:
c = wmi.WMI(machines) # <---- Go to next in the for loop when connection fail???
for os in c.Win32_OperatingSystem():
print os.Caption
except:
pass

import wmi
MachineList = ["Computer1","Computer2","Computer3"]
for machines in MachineList:
try:
c = wmi.WMI(machines) # <---- Go to next in the for loop when connection fail???
for os in c.Win32_OperatingSystem():
print os.Caption
except Exception: #Intended Exception should be mentioned here
print "Cannot Connect to {}".format(machines)
Generally speaking unless you are using exception for control flow, it should be caught as soon as plausible to prevent mixing with other exceptions. Also you should be specific what exception you want to catch rather than catching something generic.

for machine in MachineList:
try:
c = wmi.WMI(machine)
for os in c.Win32_OperatingSystem():
print os.caption
except Exception:
print "Failed on machine %s" % machine