Has anyone managed successfully using cscope with Python code? I have VIM 7.2 and the latest version of cscope installed, however it doesn't get my code's tags correctly (always off by a couple of lines). I tried the pycscope script but its output isn't supported by the modern version of cscope.
Any ideas? Or an alternative for browsing Python code with VIM? (I'm specifically interested in the extra features cscope offers beyond the simple tags of ctags)
EDIT: I'm going to run through the process step by step:
Preparing the sources:
exhuberant ctags, has an option: -x
Alternatively, ctags can generate a cross reference file which lists,
in human readable form, information about the various source objects
found in a set of language files.
This is the key to the problem:
ctags -x $(ls **/*.py); # replace with find if no zsh
will give you your database of source objects in a known, format, described under
man ctags; # make sure you use exuberant ctags!
Gnu Global is not limited to only the "out of the box" type of files. Any regular file format will serve.
Also, you can use gtags-cscope, which comes with global as mentioned in section 3.7 of the manual, for a possible shortcut using gtags. You'll end up with an input of a ctags tabular file which Global/gtags can parse to get your objects, or you can use the source for pycscope together with your ctags file of known format to get an input for the vim cscope commands in
if_cscope.txt.
Either way it's quite doable.
Perhaps you'd prefer idutils?
Definintely possible since
z3c.recipe.tags
on pypi makes use of both ctags and idutils to create tag files for a buildout, which is a method I shall investigate in short while.
Of course, you could always use the greputils script below, it has support for idutils , we know idutils works with python, and if that fails, there is also something called vimentry from this year that also uses python, idutils and vim.
Reference links (not complete list):
gtags vimscript, uses Gnu global. updated 2008
greputils vimscript, contains support for the *id idutils, 2005
lid vimscript, Ancient, but this guy is pretty good, his tag and buffer howtos are amazing 2002
An updated version of pyscope, 2010
Hopefully this helps you with your problem, I certainly helped me. I would have been quite sad tonight with a maggoty pycscope.
This seems to work for me:
Change to the top directory of your python code. Create a file called cscope.files:
find . -name '*.py' > cscope.files
cscope -R
You may need to perform a cscope -b first if the cross references don't get built properly.
From a correspondence with the maintainer of cscope, this tool isn't designed to work with Python, and there are no plans to implement that compatibility. Whatever works now, apparently works by mistake, and there is no promise whatsoever that it will keep working.
It appears I've been using an out-of-date version of pycscope. The latest version 0.3 is supported by the cscope DB. The author of pycscope told me that he figured out the output format for the cscope DB from reading the source code of cscope. That format isn't documented, on purpose, but nevertheless it currently works with pycsope 0.3, which is the solution I'll be using.
I'm going to accept this answer since unfortunately no other answer provided help even after bounty was declared. No answers are upvoted, so I honestly have no idea where the bounty will go.
There is a wonderful Python-mode-klen plugin. If you have it and rope (python refactoring library) installed, then going to the definition of a particular term is as simple as <C-c>g or <C-c>rag (first is filetype mapping, second is a global one). There are much more useful features, some useless for me. All of them are disableable. Features from list of questions found at cscope-intro:
Where is this symbol used? <C-c>f. Rather confusing though, as results in quickfix list do show - instead of the actual lines (though they point to the correct location). Maybe it will be fixed.
Where is it defined?, What is this global symbol's definition?, Where is this function in the source files? <C-c>g
What is <...> global symbol's definition? <C-c>raj
Not very much, but I am not too experienced user of ropevim.
I got the same question you got, after browsing the internet, I found a way to fix this:
create a python script: cscope_scan.py
import os
codeRootDir = os.getcwd()
__revision__ = '0.1'
__author__ = 'lxd'
FILE_TYPE_LIST= ['py']
if __name__ == '__main__':
import os
f = open('cscope.files','w')
for root,dirs,files in os.walk(codeRootDir):
for file in files:
for file_type in FILE_TYPE_LIST:
if file.split('.')[-1] == file_type:
f.write('%s\n' %os.path.join(root,file))
f.close()
cmd = 'cscope -bk'
os.system(cmd)
excute this script under you code's root folder, this will generate the cscope.files and then excute cscope -b I don't know what happens to my computer, the last two lines aren't working well, but I think manually type a cscope -bk is acceptable:)
This hack also seems to force cscope to go through Python files:
cscope -Rb -s *
If you accept that cscope is apparently not designed to work with Python.
Superset any language any tool question: How to find all occurrences of a variable in Vim?
Related
As programmers we read more than we write. I've started working at a company that uses a couple of "big" Python packages; packages or package-families that have a high KLOC. Case in point: Zope.
My problem is that I have trouble navigating this codebase fast/easily. My current strategy is
I start reading a module I need to change/understand
I hit an import which I need to know more of
I find out where the source code for that import is by placing a Python debug (pdb) statement after the imports and echoing the module, which tells me it's source file
I navigate to it, in shell or the Vim file explorer.
most of the time the module itself imports more modules and before I know it I've got 10KLOC "on my plate"
Alternatively:
I see a method/class I need to know more of
I do a search (ack-grep) for the definition of that method/class across the whole codebase (which can be a pain because the codebase is partly in ~/.buildout-eggs)
I find one or more pieces of code that define that method/class
I have to deduce which one of them is the one I need to read
This costs a lot of time, which is understandable for a big codebase. But I get the feeling that navigating a large and unknown Python codebase is a common enough problem.
So I'm looking for technical tools or strategic solutions for this problem.
...
I just can't imagine hardcore Python programmers using the strategies outlined above.
on Vim, I like NERDTree (a file browser) and taglist.vim (source code browser --> http://www.vim.org/scripts/script.php?script_id=273)
also in Vim, you can use CTRL-] to jump to a definition (:h CTRL-]):
download exuberant ctags http://ctags.sourceforge.net/
follow the install directions and put it somewhere on your PATH
from the 'root' directory of your source code, make a tags file from the shell: "ctags -R"
(make sure you have :set noautochdir, and make sure :pwd is the root directory from step 3)
go into Vim, cursor over some function or class name, hit CTRL-]
by default, if there's multiple matches for the tag, it shows you everywhere it was imported, and where it was declared
if the tag only has one match, it immediately jumps to it
...then use Ctrl+O and Ctrl+I to move back and forth from where you were
(repeat above steps for the source code of particular libraries you use, i usually keep a separate Vim window open to study stuff)
I use ipython's ?? command
You just need to figure out how to import the things you want to look for, then add ?? to the end of the module or class or function or method name to view their source code. And the command completion helps on figuring out long names as well.
Try red pill: https://github.com/klen/python-mode
I've been all over the web trying to find a way to get VIM to have code completion similar to PyDev. It doesn't seem like it is possible!
-I have tried to use the omnicompletion suggested at this link: http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/ .
-I have tried several addons to alleviate the problem, none work.
The "omnicomplete" functionality is NOT what I am looking for. It just takes all the words in the file you are working on and uses those to try and complete what I am doing. For example if I wrote:
import numpy
a_single_array = range(100)
np.a#[then I hit cntrl+n to code complete]
It would spit out "a_single_array" as a possible completion -- but that is absurd! That is not a valid completion for "numpy.a ..."
What is the issue here? All the addon would have to do is run a dir(work you want to find) from the folder you are in and then filter the output! This cannot be that difficult! (I suppose you would also have to read the file you are currently editing and filter that as well to take note of name changes... but that's pretty much it!)
Speaking of how easy it would be... if there isn't anything already made, I was thinking of writing the script myself! Any guides on how to do THAT?
No, the omni completion functionality is EXACTLY what you are looking for.
You are using <C-n> instead of <C-x><C-o>:
type <C-n> & <C-p> to complete with words from the buffer (after and before the cursor respectively)
type <C-x><C-o> to complete method/properties names
It's specifically explained in the article you linked:
In V7, VIM introduced omni completion – given it is configured to recognize Python (if not, this feature is only a plugin away) Ctrl+x Ctrl+o opens a drop down dialog like any other IDE – even the whole Pydoc gets to be displayed in a split window.
Ctrln is insert-completion.
Ctrlx Ctrlo is omni-completion.
I remap omnicompletion to CtrlSpace:
inoremap <C-Space> <C-x><C-o>
You could also try SuperTab.
I have no idea about the various completion options for Python in Vim. But if you want to roll your own you'd be well advised to study and modify one of the existing ones, like this:
http://www.vim.org/scripts/script.php?script_id=1542
Also, if all your omnicompletion is doing is listing words in current file then you don't have it set up properly for Python-specific completion. . . . Not sure how good the specialized Python completion systems get, but they certainly does compete based on Python units external to your current file. . . .
I'm noticing that even for system modules, code completion doesn't work too well.
For example, if I have a simple file that does:
import re
p = re.compile(pattern)
m = p.search(line)
If I type p., I don't get completion for methods I'd expect to see (I don't see search() for example, but I do see others, such as func_closure(), func_code()).
If I type m., I don't get any completion what so ever (I'd expect .groups(), in this case).
This doesn't seem to affect all modules.. Has any one seen this behaviour and knows how to correct it?
I'm running Vim 7.2 on WinXP, with the latest pythoncomplete.vim from vim.org (0.9), running python 2.6.2.
Completion for this kind of things is tricky, because it would need to execute the actual code to work.
For example p.search() could return None or a MatchObject, depending on the data that is passed to it.
This is why omni-completion does not work here, and probably never will. It works for things that can be statically determined, for example a module's contents.
I never got the builtin omnicomplete to work for any languages. I had the most success with pysmell (which seems to have been updated slightly more recently on github than in the official repo). I still didn't find it to be reliable enough to use consistently but I can't remember exactly why.
I've resorted to building an extensive set of snipMate snippets for my primary libraries and using the default tab completion to supplement.
I need to be able to list the command line arguments (if any) passed to other running processes. I have the PIDs already of the running processes on the system, so basically I need to determine the arguments passed to process with given PID XXX.
I'm working on a core piece of a Python module for managing processes. The code is written as a Python extension in C and will be wrapped by a higher level Python library. The goal of this project is to avoid dependency on third party libs such as the pywin32 extensions, or on ugly hacks like calling 'ps' or taskkill on the command line, so I'm looking for a way to do this in C code.
I've Googled this around and found some brief suggestions of using CreateRemoteThread() to inject myself into the other process, then run GetCommandLine() but I was hoping someone might have some working code samples and/or better suggestions.
UPDATE: I've found full working demo code and a solution using NtQueryProcessInformation on CodeProject: http://www.codeproject.com/KB/threads/GetNtProcessInfo.aspx - It's not ideal since it's "unsupported" to cull the information directly from the NTDLL structures but I'll live with it. Thanks to all for the suggestions.
UPDATE 2: I managed through more Googling to dig up a C version that does not use C++ code, and is a little more direct/concisely pointed toward this problem. See http://wj32.wordpress.com/2009/01/24/howto-get-the-command-line-of-processes/ for details.
Thanks!
To answer my own question, I finally found a CodeProject solution that does exactly what I'm looking for:
http://www.codeproject.com/KB/threads/GetNtProcessInfo.aspx
As #Reuben already pointed out, you can use NtQueryProcessInformation to retrieve this information. Unfortuantely it's not a recommended approach, but given the only other solution seems to be to incur the overhead of a WMI query, I think we'll take this approach for now.
Note that this seems to not work if using code compiled from 32bit Windows on a 64bit Windows OS, but since our modules are compiled from source on the target that should be OK for our purposes. I'd rather use this existing code and should it break in Windows 7 or a later date, we can look again at using WMI. Thanks for the responses!
UPDATE: A more concise and C only (as opposed to C++) version of the same technique is illustrated here:
http://wj32.wordpress.com/2009/01/24/howto-get-the-command-line-of-processes/
The cached solution:
http://74.125.45.132/search?q=cache:-wPkE2PbsGwJ:windowsxp.mvps.org/listproc.htm+running+process+command+line&hl=es&ct=clnk&cd=1&gl=ar&client=firefox-a
in CMD
WMIC /OUTPUT:C:\ProcessList.txt PROCESS get Caption,Commandline,Processid
or
WMIC /OUTPUT:C:\ProcessList.txt path win32_process get Caption,Processid,Commandline
Also:
http://mail.python.org/pipermail/python-win32/2007-December/006498.html
http://tgolden.sc.sabren.com/python/wmi_cookbook.html#running_processes
seems to do the trick:
import wmi
c = wmi.WMI ()
for process in c.Win32_Process ():
print process.CommandLine
By using psutil ( https://github.com/giampaolo/psutil ):
>>> import psutil, os
>>> psutil.Process(os.getpid()).cmdline()
['C:\\Python26\\python.exe', '-O']
>>>
The WMI approach mentioned in another response is probably the most reliable way of doing this. Looking through MSDN, I spotted what looks like another possible approach; it's documented, but its not clear whether it's fully supported. In MSDN's language, it--
may be altered or unavailable in
future versions of Windows...
In any case, provided that your process has the right permissions, you should be able to call NtQueryProcessInformation with a ProcessInformationClass of ProcessBasicInformation. In the returned PROCESS_BASIC_INFORMATION structure, you should get back a pointer to the target process's process execution block (as field PebBaseAddress). The ProcessParameters field of the PEB will give you a pointer to an RTL_USER_PROCESS_PARAMETERS structure. The CommandLine field of that structure will be a UNICODE_STRING structure. (Be careful not too make too many assumptions about the string; there are no guarantees that it will be NULL-terminated, and it's not clear whether or not you'll need to strip off the name of the executed application from the beginning of the command line.)
I haven't tried this approach--and as I mentioned above, it seems a bit... iffy (read: non-portable)--but it might be worth a try. Best of luck...
If you aren't the parent of these processes, then this is not possible using documented functions :( Now, if you're the parent, you can do your CreateRemoteThread trick, but otherwise you will almost certainly get Access Denied unless your app has admin rights.
If I were to tag a bunch of images via XMP, in Python, what would be the best way? I've used Perl's Image::ExifTool and I am very much used to its reliability. I mean the thing never bricked on tens of thousands of images.
I found this, backed by some heavy-hitters like the European Space Agency, but it's clearly marked as unstable.
Now, assuming I am comfortable with C++, how easy is it to, say, use the Adobe XMP toolkit directly, in Python? Having never done this before, I am not sure what I'd sign up for.
Update: I tried some libraries out there and, including the fore mentioned toolkit, they are still pretty immature and have glaring problems. I resorted to actually writing an Perl-based server that accepts XML requests to read and write metadata, with the combat-tested Image::EXIF. The amount of code is actually very light, and definitely beats torturing yourself by trying to get the Python libraries to work. The server solution is language-agnostic, so it's a twofer.
Well, they website says that the python-xmp-toolkit uses Exempi, which is based on the Adobe XMP toolkit, via ctypes. What I'm trying to say is that you're not likely to create a better wrapping of the C++ code yourself. If it's unstable (i.e. buggy), it's most likely still cheaper for you to create patches than doing it yourself from scratch.
However, in your special situation, it depends on how much functionality you need. If you just need a single function, then wrapping the C++ code into a small C extension library or with Cython is feasible. When you need to have all functionality & flexibility, you have to create wrappers manually or using SWIG, basically repeating the work already done by other people.
I struggled for several hours with python-xmp-toolkit, and eventually gave up and just wrapped calls to ExifTool.
There is a Ruby library that wraps ExifTool as well (albeit, much better than what I created); I feel it'd be worth porting it to Python for a simple way of dealing with XMP.
For Python 3.x there's py3exiv2 which supports editing XMP metadata
With py3exiv2 you can read and write all standard metadata, create your own XMP namespace or extract the thumbnail embedded in image file.
One thing I like about py3exiv2 is that it's built on the (C++) exiv2 library which seems well-maintained
I did encounter a problem though when installing it on my system (Ubuntu 16.04). To get it working I first had to install the latest version of libexiv2-dev (sudo apt-get install libexiv2-dev), and only after this install py3exiv2 (sudo -H pip3 install py3exiv2)
Here's how I've used py3exiv2 to write a new tag:
import pyexiv2
metadata = pyexiv2.ImageMetadata("file_name.jpg")
metadata.read()
key = "Xmp.xmp.CustomTagKey"
value = "CustomTagValue"
metadata[key] = pyexiv2.XmpTag(key, value)
metadata.write()
(There's also a tutorial in the documentation)
For people finding this thread in the future, I would like to share my solution. I put a package up on the Python Package Index (PyPI) called imgtag. It lets you do basic XMP subject field tag editing using python-xmp-toolkit, but abstracts away all of the frustrating nonsense of actually using python-xmp-toolkit into one-line commands.
Install exempi for your platform, then run
python3 -m pip install imgtag
Now you can use it as such:
from imgtag import ImgTag
# Open image for tag editing
test = ImgTag(
filename="test.jpg", # The image file
force_case="lower", # Converts the case of all tags
# Can be `None`, `"lower"`, `"upper"`
# Default: None
strip=True, # Strips whitespace from the ends of all tags
# Default: True
no_duplicates=True # Removes all duplicate tags (case sensitive)
# Default: True
)
# Print existing tags
print("Current tags:")
for tag in test.get_tags():
print(" Tag:", tag)
# Add tags
test.add_tags(["sleepy", "happy"])
# Remove tags
test.remove_tags(["cute"])
# Set tags, removing all existing tags
test.set_tags(["dog", "good boy"])
# Save changes and close file
test.close()
# Re-open for tag editing
test.open()
# Remove all tags
test.clear_tags()
# Delete the ImgTag object, automatically saving and closing the file
del(test)
I haven't yet added methods for the other XMP fields like description, date, creator, etc. Maybe someday I will, but if you look at how the existing functions work in the source code, you can probably figure out how to add the method yourself. If you do add more methods, make a pull request please. :)
You can use ImageMagic convert, IIRC there's a Python module to it as well.