I have a Shell Script, let's say run.sh, which reads a user input from keyboard and then does some specific tasks. For some technical reasons I'm migrating this script to Python, e.g run.py, in order to achieve the exact same goal.
In the run.sh file I ask the user a input, which is typically a file in the file system, so I gave the option of "tab-completing" it and I achieved it simply through the line:
read -e -p "Choose a file: " file
The -e flag does the job of tab-completing users input. For example, if user's current directory is project, which follows the structure:
project
-- src
-- shared
-- lib
-- imgs
-- image.png
-- include
-- README.txt
and the input file is image.png they could proceed as follow:
sh<tab>i<tab><tab>
the result would be shared/imgs/image.png.
Now how do I achieve it inside a Python script? You may believe there are tons of related questions but I haven't been able to reproduce this exact same result in run.py.
What I have tried so far:
1. Python's os module:
import os
os.system("read -e -p 'Choose a file:'")
Output: sh: 1: read: Illegal option -e
2. Python's subprocess module
import subprocess
subprocess.run(['read', '-e', '-p', 'Choose a file'])
Output:
Traceback (most recent call last):
File "run.py", line 26, in <module>
subprocess.run(['read', '-e', '-p', 'Choose a file'])
File "/usr/lib/python3.7/subprocess.py", line 453, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.7/subprocess.py", line 756, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1499, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'read': 'read'
3. Python's readline module
import readline
readline.parse_and_bind("tab:complete")
file = input("Choose a file: ")
This one almost seems to work, but there is one big issue: it completes only the files in user's current directory. If user hit s<tab> then src and shared show up, but if they hit sh<tab> the liband imgs directory do not show up.
I'd like some elegant and simple way to achieve this, but I am convinced this might be a little more difficult than expected. Are there any other approaches that can solve this problem?
Set sensible completion delimiters:
import readline
readline.set_completer_delims(' \t\n=')
readline.parse_and_bind("tab: complete")
option = input("Tab complete a file: ")
By default, readline will delimit based on any of the following:
>>> import readline
>>> readline.get_completer_delims()
' \t\n`~!##$%^&*()-=+[{]}\\|;:\'",<>/?'
Since / is part of this set, anything after a / will be completed independently of anything before it. This obviously makes no sense when you're trying to complete a file path.
Related
I am trying to use PyExifTool to examine the EXIF data of given images. I am using an almost identical copy of what can be found in the github support documents and another StackOverflow resource.
This is what my code looks like:
import exiftool
import os, errno
files = ["/Users/username/Pictures/testimage.jpg"]
with exiftool.ExifTool() as et:
metadata = et.get_tag('DateCreated', files)
print(metadata)
I then receive quite the error message. As I am very new to coding I am struggling to troubleshoot this response.
Traceback (most recent call last):
File "/Users/username/Documents/test.py", line 7, in <module>
with exiftool.ExifTool() as et:
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/exiftool.py", line 191, in __enter__
self.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/exiftool.py", line 170, in start
self._process = subprocess.Popen(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'exiftool'
Presumably, the key piece of information here is:
FileNotFoundError: [Errno 2] No such file or directory: 'exiftool'
Except that I don't really understand what I am being told? ExifTool is supposed to be a tool, I am not intentionally trying to access it as a file or directory.
I can be certain ExifTool is correctly installed because if in command line I run...
#bash:~
$ exiftool /Users/username/Pictures/testimage.jpg
...ExifTool returns all the Exif data for my test image including an in-tact 'Date Created' field. Incidentally, this also confirms that my path is correct.
Edit 1: Additional information as per the request of tripleee
(Excuse my ignorance if and when I give the wrong information)
- How do I run the script?
-> I just have that snippet of code in a .py document and then it runs in my idle shell?
- Running type -all exiftool in Bash
exiftool is /usr/local/bin/exiftool
exiftool is /usr/local/bin/exiftool
(yes it does give the same thing twice)
- Running print(os.environ['PATH']) in IDLE
/usr/bin:/bin:/usr/sbin:/sbin
Edit 2: Additional troubleshooting off the back of Tripleee's queries
- Running type -all exiftool in Bash gave two paths.
exiftool is /usr/local/bin/exiftool
exiftool is /usr/local/bin/exiftool
--> I used some code I found elsewhere to try and fix this.
PATH=$(printf "%s" "$PATH" | awk -v RS=':' '!a[$1]++ { if (NR > 1) printf RS; printf $1 }')
Now running type -all exiftool in Bash gives only...
exiftool is /usr/local/bin/exiftool
Until I open a new terminal window/tab, a which point the problem returns.
- How do I run the script?
Well now I open terminal and input IDLE3 so that it is opened from prompt, not via the icon on my hotbar.
This gives a new error...
Traceback (most recent call last):
File "/Users/username/Documents/test.py", line 9, in <module>
metadata = et.get_tag('Date Created', files)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/exiftool.py", line 325, in get_tag
return self.get_tag_batch(tag, [filename])[0]
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/exiftool.py", line 312, in get_tag_batch
data = self.get_tags_batch([tag], filenames)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/exiftool.py", line 291, in get_tags_batch
return self.execute_json(*params)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/exiftool.py", line 252, in execute_json
return json.loads(self.execute(b"-j", *params).decode("utf-8"))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/exiftool.py", line 108, in fsencode
return filename.encode(encoding, errors)
AttributeError: 'list' object has no attribute 'encode'
The key difference being it ends:
AttributeError: 'list' object has no attribute 'encode'
Edit 3: Additional troubleshooting off the back of Tripleee's queries
- Discovered I had added nano .bash_profile to the end of my .bash_profile.
--> This has been removed
- Running print(os.environ['PATH']) in IDLE but now from prompt using IDLE3
/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/fsl/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin~/.bash_profile
Let us consider Linux platform where I need to execute a program called smart.exe which uses input.dat file. Both the files are placed in the same directory with each file having the same file permission 777.
Now if I run the following command in the terminal window smart.exe is fully executed without any error.
$./smart.exe input.dat
On the other hand, if I use the following python script called my_script.py placed in the same directory, then I get an error.
my_script.py has the following code:
#!/usr/bin/python
import os, subprocess
exit_code = subprocess.call("./smart.exe input.dat", shell = False)
The error is as follows:
File "my_script.py", line 4, in <module>
exit_code = subprocess.call("./smart.exe input.dat", shell = False)
File "/usr/lib64/python2.6/subprocess.py", line 478, in call
p = Popen(*popenargs, **kwargs)
File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Can someone please tell me why this is happening. Please note that the smart.exe should take around 10 sec to fully complete. This may be a clue for the problem.
Please also advise if there is any other way to run smart.exe from my_script.py. Your solution is much appreciated!
You should decide if you want shell support or not.
If you want the shell to be used (which is not necessary here), you should use exit_code = subprocess.call("./smart.exe input.dat", shell=True). Then the shell interprets your command line.
If you don't want it (as you don't need it and want to avoid unnecessary complexity), you should do exit_code = subprocess.call(["./smart.exe", "input.dat"], shell=False).
(And there is no point naming your binarys .exe under Linux.)
I am trying to use this example script to test crontab in python:
from crontab import CronTab
tab = CronTab(user='www',fake_tab='True')
cmd = '/var/www/pjr-env/bin/python /var/www/PRJ/job.py'
cron_job = tab.new(cmd)
cron_job.minute().every(5)
#writes content to crontab
tab.write()
print tab.render()
It returns with an error 'fake_tab' not defined. If i remove this parameter and call the function
like this: CronTab(user='www'). I returns the following error :
Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
tab = CronTab(user='www')
File "C:\Python27\lib\site-packages\crontab.py", line 160, in __init__
self.read(tabfile)
File "C:\Python27\lib\site-packages\crontab.py", line 183, in read
p = sp.Popen(self._read_execute(), stdout=sp.PIPE)
File "C:\Python27\lib\subprocess.py", line 711, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 948, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Does any one know, what am I missing?
I think that Crontab is a Unix/Linux concept. Not sure if it can work under windows. This Page says "Windows support works for manual crontabs only". Not sure what he means by that though.
As the author of python-crontab I can report that the documentation has been updated. It's clear ineffective given the number of people puzzled over what manual means.
If you do this:
mem_cron = CronTab(tab="""
* * * * * command # comment
""")
You should have a memory only crontab. Same if you do a file as a crontab:
file_cron = CronTab(tabfile='filename.tab')
I'm always looking to improve the code and documentation, so please do email me.
The easiest way I found to make crontab find my job was:
In settings.py (I'm using django) I defined these variables:
CRONTAB_EXECUTABLE='C:/Users/myuser/myvirtualenv/Lib/site-packages/django_crontab/crontab.py'
CRONTAB_DJANGO_PROJECT_NAME='myproject'
CRONTAB_DJANGO_MANAGE_PATH='C:/Users/myuser/myvirtualenv/myproject/manage.py'
CRONTAB_PYTHON_EXECUTABLE='C:/Users/myuser/AppData/Local/Programs/Python/Python36-32/pythonw.exe'
Pay attention to the slash. It must be leaned to the right or it causes a syntax error.
By this way crontab will find the job or whatever you are trying to execute. In my case I was trying:
C:\Users\bsi\mlearning3\src>python manage.py crontab add
In a python script, I issue the command:
def copy_file(csv_file): #csv_file = "wpa-01.csv"
subprocess.call(["cp",csv_file,"tempfile.csv"])
I get the error:
cp: cannot stat 'wpa-01.csv' : No such file or directory
-tempfile.csv is a valid file, it is open
-I have tried adding quotes around wpa-01.csv, ie
subprocess.call(["cp","\"wpa-01.csv\"","tempfile.csv"])
-I have tried adding escape character in front of the '-'
-I have tried including the directory in front og the file name
-I am using gedit on a local Linux machine (so its not a dos2unix kind of solution), but the script is being ran on a remote Raspberry Pi
in every case I get the same error. I am at a loss for solutions. any suggestions?
***Here is the problem: "wpa-01.csv" is a 'live'/'dynamic' file. There is an active process that is updating that file in real time. I think that the file will have to be 'dead'/'static' in order to issue cp command? This is not ideal for my purposes. Is there a way to work around this like changing the mod or something? If not I suppose I can try to find an alternative solution.
print "wpa-01.csv" in os.listdir(".") #make sure file really does exist
subprocess.call(["cp","\"wpa-01.csv\"","tempfile.csv"],shell=True)
My guess is you need to set shell=True so that it uses your path to find cp executes in your shell ... if you don't use shell=True it wont use your path ...
Unfortunately all it is is a guess ...
Anyway, here is some supporting evidence:
>>> subprocess.call("copy tmp5.py tmp55.py")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python26\lib\subprocess.py", line 470, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python26\lib\subprocess.py", line 623, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 833, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
>>> subprocess.call("copy tmp5.py tmp55.py",shell=True)
1 file(s) copied.
0
I have a python cgi script that runs an application via subprocess over and over again (several thousand times). I keep getting the same error...
Traceback (most recent call last):
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 413, in <module>
webpage()
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 406, in main
displayOmpResult(form['odfFile'].value)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 342, in displayContainerDiv
makeSection(position,sAoiInput)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 360, in displayData
displayTable(i,j,lAmpAndVars,dOligoSet[key],position)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 247, in displayTable
p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1039, in _execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files
The function causing it is below.
def displayTable(sData):
# convert the data to the proper format
sFormattedData = convertToFormat(sData)
# write the formatted data to file
sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
sOutputFileLoc = sInputFile.replace('In_','Out_')
# run app, requires two files; an input and an output
# temp file to holds stdout stderr of subprocess
fh = tempfile.TemporaryFile(mode='w',dir=tempfile.gettempdir())
p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
p.communicate()
fh.close()
# open output file and print parsed data into a list of dictionaries
sOutput = open(sOutputFileLoc).read()
lOutputData = parseOutput(sOutput)
displayTableHeader(lOutputData)
displaySimpleTable(lOutputData)
As far as I can tell, I'm closing the files properly. When I run...
import resource
print resource.getrlimit(resource.RLIMIT_NOFILE)
I get...
(1024, 1024)
Do I have to increase this value? I read that subprocess opens several file descriptors. I tried adding "close_fds = True" and I tried using the with statement when creating my file but the result was the same. I suspect the problem may be with the application that I'm subprocessing, pDat, but this program was made by someone else. It requires two inputs; an input file and the location of where you want the output file written to. I suspect it may not be closing the output file that it creates. Aside from this, I can't see what I might be doing wrong. Any suggestions? Thanks.
EDIT:
I'm on ubuntu 10.04 running python 2.6.5 and apache 2.2.14
Instead of this...
sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
I should have done this...
iFileHandle,sInputFile = tempfile.mkstemp(prefix='In_')
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
os.close(iFileHandle)
The mkstemp function makes OS level handles to a file and I wasn't closing them. The solution is described in more detail here...
http://www.logilab.org/blogentry/17873
You want to add close_fds=True to the popen call (just in case).
Then, here:
# open output file and print parsed data into a list of dictionaries
sOutput = open(sOutputFileLoc).read()
lOutputData = parseOutput(sOutput)
...I might remember wrong, but unless you use the with syntax, I do not think that the output file descriptor has been closed.
UPDATE: the main problem is that you need to know which files are open. On Windows this would require something like Process Explorer. In Linux it's a bit simpler; you just have to invoke the CGI from command line, or be sure that there is only one instance of the CGI running, and fetch its pid with ps command.
Once you have the pid, run a ls -la on the content of the /proc/<PID>/fd directory. All open file descriptors will be there, with the name of the files they point to. Knowing that file so-and-so is opened 377 times, that goes a long way towards finding out where exactly that file is opened (but not closed).