As programmers we read more than we write. I've started working at a company that uses a couple of "big" Python packages; packages or package-families that have a high KLOC. Case in point: Zope.
My problem is that I have trouble navigating this codebase fast/easily. My current strategy is
I start reading a module I need to change/understand
I hit an import which I need to know more of
I find out where the source code for that import is by placing a Python debug (pdb) statement after the imports and echoing the module, which tells me it's source file
I navigate to it, in shell or the Vim file explorer.
most of the time the module itself imports more modules and before I know it I've got 10KLOC "on my plate"
Alternatively:
I see a method/class I need to know more of
I do a search (ack-grep) for the definition of that method/class across the whole codebase (which can be a pain because the codebase is partly in ~/.buildout-eggs)
I find one or more pieces of code that define that method/class
I have to deduce which one of them is the one I need to read
This costs a lot of time, which is understandable for a big codebase. But I get the feeling that navigating a large and unknown Python codebase is a common enough problem.
So I'm looking for technical tools or strategic solutions for this problem.
...
I just can't imagine hardcore Python programmers using the strategies outlined above.
on Vim, I like NERDTree (a file browser) and taglist.vim (source code browser --> http://www.vim.org/scripts/script.php?script_id=273)
also in Vim, you can use CTRL-] to jump to a definition (:h CTRL-]):
download exuberant ctags http://ctags.sourceforge.net/
follow the install directions and put it somewhere on your PATH
from the 'root' directory of your source code, make a tags file from the shell: "ctags -R"
(make sure you have :set noautochdir, and make sure :pwd is the root directory from step 3)
go into Vim, cursor over some function or class name, hit CTRL-]
by default, if there's multiple matches for the tag, it shows you everywhere it was imported, and where it was declared
if the tag only has one match, it immediately jumps to it
...then use Ctrl+O and Ctrl+I to move back and forth from where you were
(repeat above steps for the source code of particular libraries you use, i usually keep a separate Vim window open to study stuff)
I use ipython's ?? command
You just need to figure out how to import the things you want to look for, then add ?? to the end of the module or class or function or method name to view their source code. And the command completion helps on figuring out long names as well.
Try red pill: https://github.com/klen/python-mode
Related
How can I jump to a function definition using Vim? For example with Visual Assist, I can type Alt+g under a function and it opens a context menu listing the files with definitions.
How can I do something like this in vim?
Use ctags. Generate a tags file, and tell vim where it is using the :tags command. Then you can just jump to the function definition using Ctrl-]
There are more tags tricks and tips in this question.
If everything is contained in one file, there's the command gd (as in 'goto definition'), which will take you to the first occurrence in the file of the word under the cursor, which is often the definition.
g* does a decent job without ctags being set up.
That is, type g,* (or just * - see below) to search for the word under the cursor (in this case, the function name). Then press n to go to the next (or Shift-n for previous) occurrence.
It doesn't jump directly to the definition, given that this command just searches for the word under the cursor, but if you don't want to deal with setting up ctags at the moment, you can at least save yourself from having to re-type the function name to search for its definition.
--Edit--
Although I've been using g* for a long time, I've recently discovered two shortcuts for these shortcuts!
(a) * will jump to the next occurrence of the word under the cursor. (No need to type the g, the 'goto' command in vi).
(b) # goes to the previous occurrence, in similar fashion.
N and n still work, but '#' is often very useful to start the search initially in the reverse direction, for example, when looking for the declaration of a variable under the cursor.
Use gd or gD while placing the cursor on any variable in your program.
gd will take you to the local declaration.
gD will take you to the global declaration.
more navigation options can be found in here.
Use cscope for cross referencing large project such as the linux kernel.
TL;DR:
You can do this using internal VIM functionality but a modern (and much easier) way is to use COC for intellisense-like completion and one or more language servers (LS) for jump-to-definition (and way way more). For even more functionality (but it's not needed for jump-to-definition) you can install one or more debuggers and get a full blown IDE experience.
Best second is to use native VIM's functionality called define-search but it was invented for C preprocessor's #define directive and for most other languages requires extra configuration, for some isn't possible at all (also you miss on other IDE features). Finally, a fallback to that is ctags.
Quick-start:
install vim-plug to manage your VIM plug-ins
add COC and (optionally) Vimspector at the top of ~/.vimrc:
call plug#begin()
Plug 'neoclide/coc.nvim', {'branch': 'release'}
Plug 'puremourning/vimspector'
call plug#end()
" key mappings example
nmap <silent> gd <Plug>(coc-definition)
nmap <silent> gD <Plug>(coc-implementation)
nmap <silent> gr <Plug>(coc-references)
" there's way more, see `:help coc-key-mappings#en'
call :source $MYVIMRC | PlugInstall to reload VIM config and download plug-ins
restart vim and call :CocInstall coc-marketplace to get easy access to COC extensions
call :CocList marketplace and search for language servers, e.g.:
type python to find coc-jedi,
type php to find coc-phpls, etc.
(optionally) see :h VimspectorInstall to install additional debuggers, e.g.:
:VimspectorInstall debugpy,
:VimspectorInstall vscode-php-debug, etc.
Full story:
Language server (LS) is a separate standalone application (one for each programming language) that runs in the background and analyses your whole project in real time exposing extra capabilities to your editor (any editor, not only vim). You get things like:
namespace aware tag completion
jump to definition
jump to next / previous error
find all references to an object
find all interface implementations
rename across a whole project
documentation on hover
snippets, code actions, formatting, linting and more...
Communication with language servers takes place via Language Server Protocol (LSP). Both nvim and vim8 (or higher) support LSP through plug-ins, the most popular being Conquer of Completion (COC).
List of actively developed language servers and their capabilities is available on Lang Server website. Not all of those are provided by COC extensions. If you want to use one of those you can either write a COC extension yourself or install LS manually and use the combo of following VIM plug-ins as alternative to COC:
LanguageClient - handles LSP
deoplete - triggers completion as you type
Communication with debuggers takes place via Debug Adapter Protocol (DAP). The most popular DAP plug-in for VIM is Vimspector.
Language Server Protocol (LSP) was created by Microsoft for Visual Studio Code and released as an open source project with a permissive MIT license (standardized by collaboration with Red Hat and Codenvy). Later on Microsoft released Debug Adapter Protocol (DAP) as well. Any language supported by VSCode is supported in VIM.
I personally recommend using COC + language servers provided by COC extensions + ALE for extra linting (but with LSP support disabled to avoid conflicts with COC) + Vimspector + debuggers provided by Vimspector (called "gadgets") + following VIM plug-ins:
call plug#begin()
Plug 'neoclide/coc.nvim'
Plug 'dense-analysis/ale'
Plug 'puremourning/vimspector'
Plug 'scrooloose/nerdtree'
Plug 'scrooloose/nerdcommenter'
Plug 'sheerun/vim-polyglot'
Plug 'yggdroot/indentline'
Plug 'tpope/vim-surround'
Plug 'kana/vim-textobj-user'
\| Plug 'glts/vim-textobj-comment'
Plug 'janko/vim-test'
Plug 'vim-scripts/vcscommand.vim'
Plug 'mhinz/vim-signify'
call plug#end()
You can google each to see what they do.
Native VIM jump to definition:
If you really don't want to use Language Server and still want a somewhat decent jump to definition with native VIM you should get familiar with :ij and :dj which stand for include-jump and definition-jump. These VIM commands let you jump to any file that's included by your project or jump to any defined symbol that's in any of the included files. For that to work, however, VIM has to know how lines that include files or define symbols look like in any given language. You can set it up per language in ~/.vim/ftplugin/$file_type.vim with set include=$regex and set define=$regex patterns as described in :h include-search, although, coming up with those patterns is a bit of an art and sometimes not possible at all, e.g. for languages where symbol definition or file import can span over multiple lines (e.g. Golang). If that's your case the usual fallback is ctags as described in other answers.
As Paul Tomblin mentioned you have to use ctags.
You could also consider using plugins to select appropriate one or to preview the definition of the function under cursor.
Without plugins you will have a headache trying to select one of the hundreds overloaded 'doAction' methods as built in ctags support doesn't take in account the context - just a name.
Also you can use cscope and its 'find global symbol' function. But your vim have to be compiled with +cscope support which isn't default one option of build.
If you know that the function is defined in the current file, you can use 'gD' keystrokes in a normal mode to jump to definition of the symbol under cursor.
Here is the most downloaded plugin for navigation
http://www.vim.org/scripts/script.php?script_id=273
Here is one I've written to select context while jump to tag
http://www.vim.org/scripts/script.php?script_id=2507
Another common technique is to place the function name in the first column. This allows the definition to be found with a simple search.
int
main(int argc, char *argv[])
{
...
}
The above function could then be found with /^main inside the file or with :grep -r '^main' *.c in a directory. As long as code is properly indented the only time the identifier will occur at the beginning of a line is at the function definition.
Of course, if you aren't using ctags from this point on you should be ashamed of yourself! However, I find this coding standard a helpful addition as well.
1- install exuberant ctags. If you're using osx, this article shows a little trick:
http://www.runtime-era.com/2012/05/exuberant-ctags-in-osx-107.html
2- If you only wish to include the ctags for the files in your directory only, run this command in your directory:
ctags -R
This will create a "tags" file for you.
3- If you're using Ruby and wish to include the ctags for your gems (this has been really helpful for me with RubyMotion and local gems that I have developed), do the following:
ctags --exclude=.git --exclude='*.log' -R * `bundle show --paths`
credit: https://coderwall.com/p/lv1qww
(Note that I omitted the -e option which generates tags for emacs instead of vim)
4- Add the following line to your ~/.vimrc
set autochdir
set tags+=./tags;
(Why the semi colon: http://vim.wikia.com/wiki/Single_tags_file_for_a_source_tree )
5- Go to the word you'd like to follow and hit ctrl + ] ; if you'd like to go back, use ctrl+o (source: https://stackoverflow.com/a/53929/226255)
To second Paul's response: yes, ctags (especially exuberant-ctags (http://ctags.sourceforge.net/)) is great. I have also added this to my vimrc, so I can use one tags file for an entire project:
set tags=tags;/
Install cscope. It works very much like ctags but more powerful. To go to definition, instead of Ctrl + ], do Ctrl + \ + g. Of course you may use both concurrently. But with a big project (say Linux kernel), cscope is miles ahead.
After generating ctags, you can also use the following in vim:
:tag <f_name>
Above will take you to function definition.
I'm working on a plug-in for Vim, and I'd like to test that it behaves correctly, under start-up, when users edit files e.t.c.
To do this, I'd like to start a terminal, and feed keys in to it.
I'm thinking of doing it all from a python script. Is there a way to do this?
In pseudo-python it might look something like this:
#start a terminal. Here konsole
konsole = os.system('konsole --width=200 --height=150')
#start vim in that terminal
konsole.feed_keys("vim\n")
#run the vim function to be tested
konsole.feed_keys(":let my_list = MyVimFunction()\n")
#save the return value to the file system
konsole.feed_keys(":writefile(my_list, '/tmp/result')\n")
#load result into python
with open('/tmp/result', 'r') as myfile:
data = myfile.read()
#validate the result
assertEqual('expect result', data)
I think you should verify the core functionality of your plugin inside Vim, using unit tests. There's a wide variety of Vim plugins, but most provide some additional mappings or commands, to be invoked by the user, and they usually leave behind some side effects in the buffer, or output, or opened windows. That can be verified from inside Vim. There are a various approaches for that, mine is the runVimTests test framework; the plugin page has links to several alternatives.
With the core functionality thus covered, there's little left to test "interactively". (I mean stuff like forgotten debug output, too long execution times, display mess-ups.) Since you're usually a heavy user of Vim and your plugin yourself, that mostly covers it.
Of course, if your plugin embeds itself tightly into Vim (like an "IDE for XXX"; though this is usually frowned upon), you may consider some external test driver. Maybe others will contribute pointers to some general-purpose, terminal-driven test frameworks. I'm almost sure such exist.
While I'm maintaining a plugin that permits to run unit tests on VimL functions and feed the quickfix window with the results, I use another couple of tools to check the state of the buffer after some actions, and even run the thing from travis -> vimrunner+rspec, and VimFlavour for installing the dependencies. (I vaguely remember a Python alternative inspired by vimrunner)
It mostly works well. Alas it uses the client-server feature and :redir (instead of the more recent execute() function). Even with the use of :silent, :redir catches noise which it returns to the client. Thus sometimes I fight tests that fail for very odd reasons. I also find myself inserting some pseudo-pauses to be sure that Vim has finished to interpret what I've feed it.
You'll find example of use in some of my plugins. See for instance lh-brackets or lh-cpp tests (.travis.yml file + .rspec/ directory + Rakefile + Gemfile + some helpers from vim-UT)
I'm new to eclipse. One thing I notice about eclipse+PyDev is that it give me warning if the indentation is not in multiple of 4 space. That is fine because I could use "reindent.py" to just reindent it. And I manage to setup it as the external tools. But the problem is, when I use it, (using Run->Externaltools->reindent) it would modify the code in the background, so after that, it would pop up saying that the source code has been modified (duh!) would you like to to reload the file? (duh again!) So, my question is, is there a way to pipe the current selection of the source code through reindent (it uses standard input/output when given no argument) and then replace the selection with reindent standard output automatically. Thanks!
Check in the preferences > general > workspace if you have 'refresh on access' and 'refresh with native hooks' both checked (that may solve your problem).
Another thing to check is in your external tool configuration: it has a 'refresh' tab, where you can choose to refresh the selected resource upon completion.
Another thing... if you have things just in a different indent (say 2 spaces), you can simply replace all 2 spaces by 4 spaces in the find instead of going through reindent...
When you run reindent.py as an external tool, you modify the whole file that contains the source code, not the current selection in your editor. But according to PEP 8 you should not mix tabs and spaces, so this might actually be what you want. In that case just click ok to reload the file and find that it worked.
In the settings you find some options under Window->Preferences->PyDev->Editor
There you can activate automatic replacement of a tabs with spaces when typing. Also you can choose that PyDev assumes tab-spacing for files that contain tabs.
You might want to read this:
How to integrate pep8.py in Eclipse?
When you activate Window->Preferences->PyDeV->Editor->Code Analysis->pep8.py you can generate a warning for every line, that conflicts with the convention.
I've been all over the web trying to find a way to get VIM to have code completion similar to PyDev. It doesn't seem like it is possible!
-I have tried to use the omnicompletion suggested at this link: http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/ .
-I have tried several addons to alleviate the problem, none work.
The "omnicomplete" functionality is NOT what I am looking for. It just takes all the words in the file you are working on and uses those to try and complete what I am doing. For example if I wrote:
import numpy
a_single_array = range(100)
np.a#[then I hit cntrl+n to code complete]
It would spit out "a_single_array" as a possible completion -- but that is absurd! That is not a valid completion for "numpy.a ..."
What is the issue here? All the addon would have to do is run a dir(work you want to find) from the folder you are in and then filter the output! This cannot be that difficult! (I suppose you would also have to read the file you are currently editing and filter that as well to take note of name changes... but that's pretty much it!)
Speaking of how easy it would be... if there isn't anything already made, I was thinking of writing the script myself! Any guides on how to do THAT?
No, the omni completion functionality is EXACTLY what you are looking for.
You are using <C-n> instead of <C-x><C-o>:
type <C-n> & <C-p> to complete with words from the buffer (after and before the cursor respectively)
type <C-x><C-o> to complete method/properties names
It's specifically explained in the article you linked:
In V7, VIM introduced omni completion – given it is configured to recognize Python (if not, this feature is only a plugin away) Ctrl+x Ctrl+o opens a drop down dialog like any other IDE – even the whole Pydoc gets to be displayed in a split window.
Ctrln is insert-completion.
Ctrlx Ctrlo is omni-completion.
I remap omnicompletion to CtrlSpace:
inoremap <C-Space> <C-x><C-o>
You could also try SuperTab.
I have no idea about the various completion options for Python in Vim. But if you want to roll your own you'd be well advised to study and modify one of the existing ones, like this:
http://www.vim.org/scripts/script.php?script_id=1542
Also, if all your omnicompletion is doing is listing words in current file then you don't have it set up properly for Python-specific completion. . . . Not sure how good the specialized Python completion systems get, but they certainly does compete based on Python units external to your current file. . . .
Has anyone managed successfully using cscope with Python code? I have VIM 7.2 and the latest version of cscope installed, however it doesn't get my code's tags correctly (always off by a couple of lines). I tried the pycscope script but its output isn't supported by the modern version of cscope.
Any ideas? Or an alternative for browsing Python code with VIM? (I'm specifically interested in the extra features cscope offers beyond the simple tags of ctags)
EDIT: I'm going to run through the process step by step:
Preparing the sources:
exhuberant ctags, has an option: -x
Alternatively, ctags can generate a cross reference file which lists,
in human readable form, information about the various source objects
found in a set of language files.
This is the key to the problem:
ctags -x $(ls **/*.py); # replace with find if no zsh
will give you your database of source objects in a known, format, described under
man ctags; # make sure you use exuberant ctags!
Gnu Global is not limited to only the "out of the box" type of files. Any regular file format will serve.
Also, you can use gtags-cscope, which comes with global as mentioned in section 3.7 of the manual, for a possible shortcut using gtags. You'll end up with an input of a ctags tabular file which Global/gtags can parse to get your objects, or you can use the source for pycscope together with your ctags file of known format to get an input for the vim cscope commands in
if_cscope.txt.
Either way it's quite doable.
Perhaps you'd prefer idutils?
Definintely possible since
z3c.recipe.tags
on pypi makes use of both ctags and idutils to create tag files for a buildout, which is a method I shall investigate in short while.
Of course, you could always use the greputils script below, it has support for idutils , we know idutils works with python, and if that fails, there is also something called vimentry from this year that also uses python, idutils and vim.
Reference links (not complete list):
gtags vimscript, uses Gnu global. updated 2008
greputils vimscript, contains support for the *id idutils, 2005
lid vimscript, Ancient, but this guy is pretty good, his tag and buffer howtos are amazing 2002
An updated version of pyscope, 2010
Hopefully this helps you with your problem, I certainly helped me. I would have been quite sad tonight with a maggoty pycscope.
This seems to work for me:
Change to the top directory of your python code. Create a file called cscope.files:
find . -name '*.py' > cscope.files
cscope -R
You may need to perform a cscope -b first if the cross references don't get built properly.
From a correspondence with the maintainer of cscope, this tool isn't designed to work with Python, and there are no plans to implement that compatibility. Whatever works now, apparently works by mistake, and there is no promise whatsoever that it will keep working.
It appears I've been using an out-of-date version of pycscope. The latest version 0.3 is supported by the cscope DB. The author of pycscope told me that he figured out the output format for the cscope DB from reading the source code of cscope. That format isn't documented, on purpose, but nevertheless it currently works with pycsope 0.3, which is the solution I'll be using.
I'm going to accept this answer since unfortunately no other answer provided help even after bounty was declared. No answers are upvoted, so I honestly have no idea where the bounty will go.
There is a wonderful Python-mode-klen plugin. If you have it and rope (python refactoring library) installed, then going to the definition of a particular term is as simple as <C-c>g or <C-c>rag (first is filetype mapping, second is a global one). There are much more useful features, some useless for me. All of them are disableable. Features from list of questions found at cscope-intro:
Where is this symbol used? <C-c>f. Rather confusing though, as results in quickfix list do show - instead of the actual lines (though they point to the correct location). Maybe it will be fixed.
Where is it defined?, What is this global symbol's definition?, Where is this function in the source files? <C-c>g
What is <...> global symbol's definition? <C-c>raj
Not very much, but I am not too experienced user of ropevim.
I got the same question you got, after browsing the internet, I found a way to fix this:
create a python script: cscope_scan.py
import os
codeRootDir = os.getcwd()
__revision__ = '0.1'
__author__ = 'lxd'
FILE_TYPE_LIST= ['py']
if __name__ == '__main__':
import os
f = open('cscope.files','w')
for root,dirs,files in os.walk(codeRootDir):
for file in files:
for file_type in FILE_TYPE_LIST:
if file.split('.')[-1] == file_type:
f.write('%s\n' %os.path.join(root,file))
f.close()
cmd = 'cscope -bk'
os.system(cmd)
excute this script under you code's root folder, this will generate the cscope.files and then excute cscope -b I don't know what happens to my computer, the last two lines aren't working well, but I think manually type a cscope -bk is acceptable:)
This hack also seems to force cscope to go through Python files:
cscope -Rb -s *
If you accept that cscope is apparently not designed to work with Python.
Superset any language any tool question: How to find all occurrences of a variable in Vim?