We Keep Coding

Why are Git log command outputs different when launched from Windows shell and from Python subprocess?

If I execute the command git log --no-walk --tags --reverse in the Windows shell, I get results such as:
commit e83dcf42f2a52183cf21642c7b8deb42961663f3 (tag: my_tag)
Author: an_author
Date: a_date
while if I execute the same Git command from a Python script:
subprocess.check_output(['git', 'log', '--no-walk', '--tags', '--reverse'], stderr=subprocess.PIPE).decode(sys.stdout.encoding)
I get the following result:
commit e83dcf42f2a52183cf21642c7b8deb42961663f3
Author: an_author
Date: a_date
As you can see, the tag information is missing and as you guess, I need it. Is there anything I can do to get the tag information from the Python subprocess.check_output?

When you want to parse git output, you should use the form of its commands intended for scripting. In this case it means the --pretty=format:1 option. The %D specifier provides the ref names pointing to the commit, and of course you need to explicitly specify all the fields you want.
You might also want to use the lower level git rev-list command instead of log; it has similar options, but is promised to have stable interface for script use.

Add option --decorate:
subprocess.check_output(['git', 'log', '--decorate', '--no-walk', '--tags', '--reverse'], stderr=subprocess.PIPE).decode(sys.stdout.encoding)

EoF / Ctrl+D command in Bash + Python

I'm trying to make some functions in python so that I can connect to a linux terminal and do stuff (like in this case, create a file). The code I have, works partially. The only thing that doesn't work is if you want to do something after you have entered the code. Like for instance you create the file and then want to navigate somewhere else (cd /tmp) for instance. Instead of doing the next command, it will just add to the file created.
def create_file(self, name, contents, location):
try:
log.info("Creating a file...")
self.device.execute("mkdir -p {}".format(location))
self.cd_path(location)
self.device.sendline("cat > {}".format(name))
self.device.sendline("{}".format(contents))
self.device.sendline("EOF") # send the CTRL + D command to save and exit I tried here with ^D as well
except:
log.info("Failed to create the file!")
The contents of the file is:
cat test.txt
#!/bin/bash
echo "Fail Method Requested"
exit 1
EOF
ls -d /tmp/asdasd
The order of commands executed is:
execute.create_file(test.txt, the_message, the_location)
execute.check_path("/tmp/adsasd") #this function just checks with ls -d if the directory exists.
I have tried with sendline the following combinations:
^D, EOF, <<EOF
I don't really understand how I could make this happen. I just want to create a file with a specific message. (When researching on how to do this with VI I got the same problem, but there the command I needed was the one for ESC)
If anyone could help with some input that would be great!!
Edit: As Rob mentioned below, sending the character "\x04" actually works. For anyone else having this issue, you can also consult this chart for other combinations if needed:
http://donsnotes.com/tech/charsets/ascii.html

You probably need to send the EOF character, which is typically CONTROL-D, not the three characters E, O, and F.
self.device.sendline("\x04")

http://wiki.bash-hackers.org/syntax/redirection#here_documents
Here docs allow you to use any file input termination string you like to represent end of file ( such as the literal EOF you're attempting to use now). Quoting that string tells the shell not to interpret expansions inside the heredoc content, ensuring that said content is treated as literal.
Using pipes.quote() here ensures that filenames with literal quotes, $s, spaces, or other surprising characters won't break your script. (Of course, you'll need to import pipes; on Python 3, by contrast, this has moved to shlex.quote()).
self.device.sendline("cat > {} <<'EOF'".format(pipes.quote(name)))
Then you can write the EOF as is, having told bash to interpret it as the end of file input.

command wrapped in os.system is ignored

(on os x 10.10.1)I am trying to use a paired-end merger (Casper) within a python script. i'm using os.system (don't want to use subprocess or pexpect modules). In my script here is the line that doesn't work:
os.system("casper %s %s -o %s"%(filein[0],filein[1],fileout))
#filein[0]: input file 1
#filein[1]: input file 2
#fileout: output prefix (default==casper)
Once my script is launched only the 2 first string parameters of this command are interpreted but not the third one, causing an output file with the default prefix name. Since my function is iterating through a lot of fastq files, they are all merged in a single "casper.fastq" file.
I tried to mess up with the part of the command that doesn't work (right after -o), putting meaningless string and still it is executed with no error and the default output, here is the "messed up line":
os.system("casper %s %s -ldkfnlqdskgfno %s"%(filein[0],filein[1],fileout))
Could anybody help in understanding what the heck is going on?

Print the command before execute it to check if your command wrapped correctly(like file name need to be quoted)
Execute your assumed output command directly to see if it is misinterpreted.

auto run python result in "EOF when reading a line "

I have add the line into my .vimrc
map <F4> :w !python<cr>
When I open gvim to edit an unname Python file, there are only two lines in it
x=input("which one do you like ? ")
print(x)
I press F4, and get EOF when reading a line, how to solve it?
When you add map <F4> :w<cr>:!python %<cr> or imap <F4> <Esc>:w <cr>:!python %<cr>, it can only make a named file run, if it is a no named file, the map will not work, how can I make the no named file run?

#benjifisher's answer is correct. The input (function) is the problem.
The :w !python pipes the program to python through stdin (Basically the same as
echo 'input("x")' | python
which also fails if run in the shell). However input() tries to read from stdin which it can't do because python read the program from stdin and stdin is still trying to read from the pipe. However the pipe is already at the end and won't ever contain new data. So input() just reads EOF.
To see that python is reading from stdin we look at :h :w_c which shows that the file is being passed to stdin of the cmd which in this case is python.
:w_c :write_c
:[range]w[rite] [++opt] !{cmd}
Execute {cmd} with [range] lines as standard input
(note the space in front of the '!'). {cmd} is
executed like with ":!{cmd}", any '!' is replaced with
the previous command :!.
If the buffer had contained something that wasn't reading from stdin your mapping would have worked.
For example if the unnamed buffer contains
print(42)
running the command :w !python in vim prints 42.
So the problem isn't that the mapping fails. The problem is that your program doesn't know how to get input. The solution is use either a named file or don't write interactive programs in vim. Use the python interpreter for that.

Since you have :w in your mapping, I am assuming you either want to run the script directly from the insert mode or you want to save it anyways before running it.
Multiple commands in vim require a separator like <bar> i.e. (|) or use <CR> (Carriage Return) between and after commands.
You can put both of the mappings below in your .vimrc and they should meet your requirement on hitting F4, whether you are in normal mode or insert mode.
If you are in normal mode, you are fine with map:
map <F4> :w<cr>:!python %<cr>
While for insert mode, you would need imap and an Esc to get out of insert mode:
imap <F4> <Esc>:w <cr>:!python %<cr>

I think the problem is the input() line. Python is looking for input and not finding any. All it finds (wherever it looks) is an EOF.

One way to do this without a temp file, would be to do something like this(with a small helper function):
function! GetContentsForPython()
let contents = join(getline(1,'$'), "\n")
let res = ''
for l in split(contents, '\n')
if len(l)
let res = res . l . ';'
endif
endfor
let res = '"' . escape(res, '"') . '"'
return res
endfunction
noremap <f4> :!python -c <c-r>=GetContentsForPython()<cr><cr>
This gets the contents of the current buffer, and replaces the newlines with semi colons so you can execute it with
python -c "print 'hello'"
There may be better ways accomplishing this, but this seems to work for me.

Get lines from stdout after timestamp

There is a huge log of errors/warnings/infos printed out on stdout. I am only interested in the lines logged after I start a specific action.
Other information: I am using Python to telnet to a shell environment. I execute the commands on shell and store the time the action is started. I then call a command to view the log which spits it on stdout. I expect to read in the greped lines after that timestamp back to Python. I also store the current time but not sure how to use that (maybe grep on a date range?)
I can redirect to a file and use find but the log is huge and I'd rather not read all of it.
I can grep -n to get line number and then read everything after but I'm not sure how to.
Concept regex to egrep on is something like: {a-timestamp}*
Any suggestions would be appreciated!

awk '/the-timestamp-I-have/,0' the-log-file
This will print the lines from the-log-file, starting at the first line that matches the-timestamp-I-have and continuing through the last line.
Ref:
http://www.catonmat.net/blog/awk-one-liners-explained-part-three/
http://www.catonmat.net/blog/ten-awk-tips-tricks-and-pitfalls/#awk_ranges