We Keep Coding

How to get programatic access to list of Pep8 Error codes

I'm having a tough time finding how how to get a list of the pep8 error codes and some table or function to lookup their messages or properties.
There is a list here:
https://gist.github.com/mjgreen/bda13692b696669cb3fcd5a0fb682958
I've tried pep8, pycodestyle, flake8, and autoflake8, but I'm not having luck. Whatever way exists to grab these doesn't exist the way I thought it would. (I was expecting a hard-coded message table). I'm probably just being dense. My thought is flake8 must have a way to get them, it prints them out.
How do I import flake8 and then just generate some dictionary that maps error codes to error messages / and or properties?

if you're looking particularly for "codes that could be emitted from flake8 plugins" given your current plugin set there isn't a way to know all of those
the way plugin registration works is flake8 is handed a prefix of the error codes that a plugin might emit -- the plugin is then free to emit anything in that range (and with any arbitrary string for message)
the plugin is also free to emit error codes outside of their prefix, but they are ignored (this may be come an error in the future, I haven't quite decided what the correct behaviour here is yet)
collecting those prefixes is relatively straightforward (though you'd probably also want to think about flake8:local-plugins as well):
for dist in importlib.metadata.distributions():
for ep in dist.entry_points or ():
if ep.group == 'flake8.extension':
print(ep.name)
note here that pycodestyle is a special case here (due to historical reasons)
pycodestyle (previously named pep8) - the tool which implements the "pep8" error codes also does not provide programmatic access to the codes and messages (they are inlined in the functions which produce the errors)
as such any such list of codes and messages is unofficial and likely outdated / incorrect
disclaimer: I am the current flake8 maintainer and I am one of the maintainers of pycodestyle

Inject code automatically to a python project and execute

I want to inject some code to all python modules in my project automatically.
I used ast.NodeTransformer and managed to change it quite easily, the problem is that I want to run the project.
An ast node is per module and I want to change all modules in the project and then run; and I have this example
The problem is that it applies to one node, viz. one file. I want to run this file, which imports and uses other files which I want to change too, so I'm not sure how to get it done.
I know I can use some ast-to-code module, like astor, but all are third party and I don't want to deal with bugs and unexpected issues.
Don't really know how to start, any suggestions?

I know I can use some ast-to-code module, like astor, but all are third party and I don't want to deal with bugs and unexpected issues.
From 3.9 onward there is ast.unparse, which practically does AST to source conversion after you transform it.

Navigating a big Python codebase faster

As programmers we read more than we write. I've started working at a company that uses a couple of "big" Python packages; packages or package-families that have a high KLOC. Case in point: Zope.
My problem is that I have trouble navigating this codebase fast/easily. My current strategy is
I start reading a module I need to change/understand
I hit an import which I need to know more of
I find out where the source code for that import is by placing a Python debug (pdb) statement after the imports and echoing the module, which tells me it's source file
I navigate to it, in shell or the Vim file explorer.
most of the time the module itself imports more modules and before I know it I've got 10KLOC "on my plate"
Alternatively:
I see a method/class I need to know more of
I do a search (ack-grep) for the definition of that method/class across the whole codebase (which can be a pain because the codebase is partly in ~/.buildout-eggs)
I find one or more pieces of code that define that method/class
I have to deduce which one of them is the one I need to read
This costs a lot of time, which is understandable for a big codebase. But I get the feeling that navigating a large and unknown Python codebase is a common enough problem.
So I'm looking for technical tools or strategic solutions for this problem.
...
I just can't imagine hardcore Python programmers using the strategies outlined above.

on Vim, I like NERDTree (a file browser) and taglist.vim (source code browser --> http://www.vim.org/scripts/script.php?script_id=273)
also in Vim, you can use CTRL-] to jump to a definition (:h CTRL-]):
download exuberant ctags http://ctags.sourceforge.net/
follow the install directions and put it somewhere on your PATH
from the 'root' directory of your source code, make a tags file from the shell: "ctags -R"
(make sure you have :set noautochdir, and make sure :pwd is the root directory from step 3)
go into Vim, cursor over some function or class name, hit CTRL-]
by default, if there's multiple matches for the tag, it shows you everywhere it was imported, and where it was declared
if the tag only has one match, it immediately jumps to it
...then use Ctrl+O and Ctrl+I to move back and forth from where you were
(repeat above steps for the source code of particular libraries you use, i usually keep a separate Vim window open to study stuff)

I use ipython's ?? command
You just need to figure out how to import the things you want to look for, then add ?? to the end of the module or class or function or method name to view their source code. And the command completion helps on figuring out long names as well.

Try red pill: https://github.com/klen/python-mode

Using cscope to browse Python code with VIM?

Has anyone managed successfully using cscope with Python code? I have VIM 7.2 and the latest version of cscope installed, however it doesn't get my code's tags correctly (always off by a couple of lines). I tried the pycscope script but its output isn't supported by the modern version of cscope.
Any ideas? Or an alternative for browsing Python code with VIM? (I'm specifically interested in the extra features cscope offers beyond the simple tags of ctags)

EDIT: I'm going to run through the process step by step:
Preparing the sources:
exhuberant ctags, has an option: -x
Alternatively, ctags can generate a cross reference file which lists,
in human readable form, information about the various source objects
found in a set of language files.
This is the key to the problem:
ctags -x $(ls **/*.py); # replace with find if no zsh
will give you your database of source objects in a known, format, described under
man ctags; # make sure you use exuberant ctags!
Gnu Global is not limited to only the "out of the box" type of files. Any regular file format will serve.
Also, you can use gtags-cscope, which comes with global as mentioned in section 3.7 of the manual, for a possible shortcut using gtags. You'll end up with an input of a ctags tabular file which Global/gtags can parse to get your objects, or you can use the source for pycscope together with your ctags file of known format to get an input for the vim cscope commands in
if_cscope.txt.
Either way it's quite doable.
Perhaps you'd prefer idutils?
Definintely possible since
z3c.recipe.tags
on pypi makes use of both ctags and idutils to create tag files for a buildout, which is a method I shall investigate in short while.
Of course, you could always use the greputils script below, it has support for idutils , we know idutils works with python, and if that fails, there is also something called vimentry from this year that also uses python, idutils and vim.
Reference links (not complete list):
gtags vimscript, uses Gnu global. updated 2008
greputils vimscript, contains support for the *id idutils, 2005
lid vimscript, Ancient, but this guy is pretty good, his tag and buffer howtos are amazing 2002
An updated version of pyscope, 2010
Hopefully this helps you with your problem, I certainly helped me. I would have been quite sad tonight with a maggoty pycscope.

This seems to work for me:
Change to the top directory of your python code. Create a file called cscope.files:
find . -name '*.py' > cscope.files
cscope -R
You may need to perform a cscope -b first if the cross references don't get built properly.

From a correspondence with the maintainer of cscope, this tool isn't designed to work with Python, and there are no plans to implement that compatibility. Whatever works now, apparently works by mistake, and there is no promise whatsoever that it will keep working.
It appears I've been using an out-of-date version of pycscope. The latest version 0.3 is supported by the cscope DB. The author of pycscope told me that he figured out the output format for the cscope DB from reading the source code of cscope. That format isn't documented, on purpose, but nevertheless it currently works with pycsope 0.3, which is the solution I'll be using.
I'm going to accept this answer since unfortunately no other answer provided help even after bounty was declared. No answers are upvoted, so I honestly have no idea where the bounty will go.

There is a wonderful Python-mode-klen plugin. If you have it and rope (python refactoring library) installed, then going to the definition of a particular term is as simple as <C-c>g or <C-c>rag (first is filetype mapping, second is a global one). There are much more useful features, some useless for me. All of them are disableable. Features from list of questions found at cscope-intro:
Where is this symbol used? <C-c>f. Rather confusing though, as results in quickfix list do show - instead of the actual lines (though they point to the correct location). Maybe it will be fixed.
Where is it defined?, What is this global symbol's definition?, Where is this function in the source files? <C-c>g
What is <...> global symbol's definition? <C-c>raj
Not very much, but I am not too experienced user of ropevim.

I got the same question you got, after browsing the internet, I found a way to fix this:
create a python script: cscope_scan.py
import os
codeRootDir = os.getcwd()
__revision__ = '0.1'
__author__ = 'lxd'
FILE_TYPE_LIST= ['py']
if __name__ == '__main__':
import os
f = open('cscope.files','w')
for root,dirs,files in os.walk(codeRootDir):
for file in files:
for file_type in FILE_TYPE_LIST:
if file.split('.')[-1] == file_type:
f.write('%s\n' %os.path.join(root,file))
f.close()
cmd = 'cscope -bk'
os.system(cmd)
excute this script under you code's root folder, this will generate the cscope.files and then excute cscope -b I don't know what happens to my computer, the last two lines aren't working well, but I think manually type a cscope -bk is acceptable:)

This hack also seems to force cscope to go through Python files:
cscope -Rb -s *
If you accept that cscope is apparently not designed to work with Python.
Superset any language any tool question: How to find all occurrences of a variable in Vim?

Vim Python omni-completion failing to work on system modules

I'm noticing that even for system modules, code completion doesn't work too well.
For example, if I have a simple file that does:
import re
p = re.compile(pattern)
m = p.search(line)
If I type p., I don't get completion for methods I'd expect to see (I don't see search() for example, but I do see others, such as func_closure(), func_code()).
If I type m., I don't get any completion what so ever (I'd expect .groups(), in this case).
This doesn't seem to affect all modules.. Has any one seen this behaviour and knows how to correct it?
I'm running Vim 7.2 on WinXP, with the latest pythoncomplete.vim from vim.org (0.9), running python 2.6.2.

Completion for this kind of things is tricky, because it would need to execute the actual code to work.
For example p.search() could return None or a MatchObject, depending on the data that is passed to it.
This is why omni-completion does not work here, and probably never will. It works for things that can be statically determined, for example a module's contents.

I never got the builtin omnicomplete to work for any languages. I had the most success with pysmell (which seems to have been updated slightly more recently on github than in the official repo). I still didn't find it to be reliable enough to use consistently but I can't remember exactly why.
I've resorted to building an extensive set of snipMate snippets for my primary libraries and using the default tab completion to supplement.