I am trying to incorporate this sed command to remove the last comma in a son file.
sed -i -e '1h;1!H;$!d;${s/.*//;x};s/\(.*\),/\1 /' file.json"
when i run this in the command line, it works fine. When i try to run as a subprocess as so it doesn't work.
Popen("sed -e '1h;1!H;$!d;${s/.*//;x};s/\(.*\),/\1 /' file.json",shell=True).wait()
What am I doing wrong?
It doesn't work because when you write \1, python interprets that as \x01 and our regular expression doesn't work / is illegal.
That is already better:
check_call(["sed","-i","-e",r"1h;1!H;$!d;${s/.*//;x};s/\(.*\),/\1 /","file.json"])
because splitting as a real list and passing your regex as a raw string has a better chance to work. And check_call is what you need to just call a process, without caring about its output.
But I would do even better: since python is good at processing files, given your rather simple problem, I would create a fully portable version, no need for sed:
# read the file
with open("file.json") as f:
contents = f.read().rstrip().rstrip(",") # strip last newline/space then strip last comma
# write back the file
with open("file.json","w") as f:
f.write(contents)
In general, you might try the following solutions:
Pass the raw string, as was mentioned
Escape the '\' character.
This code also does what you need:
Popen("sed -e '1h;1!H;$!d;${s/.*//;x};s/\(.*\),/\\1 /' file.json", shell=True).wait()
or
try:
check_call(["sed", "-i", "-e", "1h;1!H;$!d;${s/.*//;x};s/\(.*\),/\\1 /", "file.json"])
except:
pass # or handle the error
Related
I am running the sed command inside python using os.system. Below is the code.
os.system("sed -i /solid/s/Visualization Toolkit generated SLA File/chestwall/g mesh1.stl")
The name to be changed has spaces in it. Also, in the end part i.e. mesh1.stl, the 1 need to be variable. How to do it?
Firstly, for this code, I am getting error as:
sed: -e expression #1, char 22: unterminated s command
I tried putting / at the end.
Second, I need the mesh1 to be a variable from previous line. Say, mesh1 as a and everytime, a changes. How to write like that?
Make sure that the sed statement/command is in either double or single quotes and then use "+" to concatenate strings before passing them to os.system
import os
var=1
os.system("sed -i 's/solid/s/Visualization Toolkit generated SLA File/chestwall/g' mesh" + var + ".stl")
The function os.system() is now considered to be superseded by
subprocess.call().
Would you please try the following:
import subprocess
a = 'mesh1'
cmd = ['sed', '-i', '/solid/s/Visualization Toolkit generated SLA File/chestwall/g', '{0}.stl'.format(a)]
subprocess.call(cmd)
You can pass the command as a list, not a string, and you can explicitly divide the arguments.
I am trying to execute the following command in python using plumbum:
sort -u -f -t$'\t' -k1,1 file1 > file2
However, I am having issues passing the -t$'\t' argument. Here is my code:
from plumbum.cmd import sort
separator = r"-t$'\t'"
print separator
cmd = (sort["-u", "-f", separator, "-k1,1", "file1"]) > "file2"
print cmd
print cmd()
I can see problems right away after print separator and print cmd() executes:
-t$'\t'
/usr/bin/sort -u -f "-t\$'\\t'" -k1,1 file1 > file2
The argument is wrapped in double quotes.
An extra \ before $ and \t is inserted.
How should I pass this argument to plumbum?
You may have stumbled into limitations of the command line escaping.
I could make it work using subprocess module, passing a real tabulation char litteraly:
import subprocess
p=subprocess.Popen(["sort","-u","-f","-t\t","-k1,1","file1",">","file2"],shell=True)
p.wait()
Also, full python short solution that does what you want:
with open("file1") as fr, open("file2","w") as fw:
fw.writelines(sorted(set(fr),key=lambda x : x.split("\t")[0]))
The full python solution doesn't work exactly the same way sort does when dealing with unicity. If 2 lines have the same first field but not the same second field, sort keeps one of them, whereas the set will keep both.
EDIT: unchecked but you just confirmed that it works: just tweak your plumbum code with:
separator = "-t\t"
could just work, although out of the 3 ones, I'd recommend the full python solution since it doesn't involve an external process and therefore is more pythonic and portable.
I'm a complete Python novice, but I need to create a small script as part of a larger project.
I'm trying to use a small Python script to input a variable into a line of code within a Unix file. I've been trying to use the subprocess.call() which enables me to execute the commands I need to won't allow me to place in the variable.
I have been trying the following:
import subprocess
var="some_variable"
subprocess.call(['sudo', 'sed', '-i', '-e' 's/line_to_replace.*/replacement_line' + var, /path/to/file.conf])
The most obvious failure here is that your code isn't putting a trailing / on the end of the replacement value to make it a valid sed command:
import subprocess
var="some_variable".replace('/', r'\/') # backslash-escape any sigil instances
subprocess.call(['sudo', 'sed', '-i',
'-e', 's/line_to_replace.*/replacement_line%s/' % (var,),
'/path/to/file.conf'])
That said, be aware that this won't work with POSIX-standard sed (which doesn't have -i), or BSD sed (for which the extension argument to sed -i is not optional). The POSIX-standard tools for editing files in-place are ed and ex; consider learning them.
Your immediate problem is a sed syntax error: You are missing the closing separator after the end of the replacement string (and the file name needs to be quoted). However, most simple Unix tools can easily be replaced with pure Python, often with immediate gains both in elegance and efficiency (avoiding an external process is frequently a significant performance improvement).
import fileinput
for line in fileinput('/path/to/file', inline=True):
if line.startswith('line_to_replace'):
line = var + '\n'
print line
If you genuinely need regular expression support, the change should be obvious. The sudo is harder to replace; perhaps you can simply run this from the prompt with sudo if obtaining write privileges to the file you want to manipulate is otherwise not trivial.
I'm trying to call 'sed' from Python and having troubles passing the command line via either subprocess.check_call() or os.system().
I'm on Windows 7, but using the 'sed' from Cygwin (it's in the path).
If I do this from the Cygwin shell, it works fine:
$ sed 's/ /\ /g' <"C:foobar" >"C:foobar.temp"
In Python, I've got the full pathname I'm working with in "name". I tried:
command = r"sed 's/ /\ /g' " + "<" '\"' + name + '\" >' '\"' + name + '.temp' + '\"'
subprocess.check_call(command, shell=True)
All the concatenation is there to make sure I have double quotes around the input and output filenames (in case there are blank spaces in the Windows file path).
I also tried it replacing the last line with:
os.system(command)
Either way, I get this error:
sed: -e expression #1, char 2: unterminated `s' command
'amp' is not recognized as an internal or external command,
operable program or batch file.
'nbsp' is not recognized as an internal or external command,
operable program or batch file.
Yet, as I said, it works OK from the console. What am I doing wrong?
The shell used by subprocess is probably not the shell you want. You can specify the shell with executable='path/to/executable'. Different shells have different quoting rules.
Even better might be to skip subprocess altogether, and write this as pure Python:
with open("c:foobar") as f_in:
with open("c:foobar.temp", "w") as f_out:
for line in f_in:
f_out.write(line.replace(' ', ' '))
I agree with Ned Batchelder's assessment, but think what you might want to consider using the following code because it likely does what you ultimately want to accomplish which can be done easily with the help of Python's fileinput module:
import fileinput
f = fileinput.input('C:foobar', inplace=1)
for line in f:
line = line.replace(' ', ' ')
print line,
f.close()
print 'done'
This will effectively update the given file in place as use of the keyword suggests. There's also an optional backup= keyword -- not used above -- which will save a copy of the original file if desired.
BTW, a word of caution about using something like C:foobar to specify the file name because on Windows it means a file of that name in whatever the current directory is on drive C:, which might not be what you want.
I think you'll find that, in Windows Python, it's not actually using the CygWin shell to run your command, it's instead using cmd.exe.
And, cmd doesn't play well with single quotes the way bash does.
You only have to do the following to confirm that:
c:\pax> echo hello >hello.txt
c:\pax> type "hello.txt"
hello
c:\pax> type 'hello.txt'
The system cannot find the file specified.
I think the best idea would be to use Python itself to process the file. The Python language is a cross-platform one which is meant to remove all those platform-specific inconsistencies, such as the one you've just found.
I'm trying to write an svn pre-commit hook in python. Part of this involves checking the diff file to see if there are any actual file changes (as opposed to just property changes).
I have a working grep command which I can execute fine on the shell
grep "^\(Added: \|Modified: \|Deleted: \)" diff filename | grep -v 'svn:'
However when I put it through subprocess.POpen it escapes all my backslashes, which knackers the regexp.
Executing command: ['grep', '"^\\Added: \\|Modified: \\|Deleted: \\)", ...]
How do I avoid this?
NB: I'm aware that I can pipe results between subprocesses and I can do the two greps that way. I need help getting the first one working first though :/
NB2: I also tried using filterdiff --clean instead and couldn't get it to work. Searching for Added, Modified or Deleted lines, removing those with 'svn:' in and checking I had some results seemed to work though.
Python code:
command = ['grep', '"^\(Added: \|Modified: \|Deleted: \)"', filename]
sys.stdout.write('Executing command: %s\n' % (command))
p = subprocess.Popen(command,
stdin = subprocess.PIPE
stdout = subprocess.PIPE
stderr = subprocess.STDOUT
shell = True)
data = p.stdout.read()
if len(data) == 0:
sys.stdout.write("Diff does not contain any file modifications./n")
exit(0)
You need to consider what you want grep to see in its command line arguments.
The first argument needs to be the literal string "^\(Added: \|Modified: \|Deleted: \)", so that means that it shouldn't include the double quotes but should include the backslashes.
The way to express this kind of string is to use Python raw strings:
command = ['grep', r'^\(Added: \|Modified: \|Deleted: \)', filename]
A good way to check what you're actually running is to replace grep by echo so you can at least see what you're passing to the command.