We Keep Coding

Inject code automatically to a python project and execute

I want to inject some code to all python modules in my project automatically.
I used ast.NodeTransformer and managed to change it quite easily, the problem is that I want to run the project.
An ast node is per module and I want to change all modules in the project and then run; and I have this example
The problem is that it applies to one node, viz. one file. I want to run this file, which imports and uses other files which I want to change too, so I'm not sure how to get it done.
I know I can use some ast-to-code module, like astor, but all are third party and I don't want to deal with bugs and unexpected issues.
Don't really know how to start, any suggestions?

I know I can use some ast-to-code module, like astor, but all are third party and I don't want to deal with bugs and unexpected issues.
From 3.9 onward there is ast.unparse, which practically does AST to source conversion after you transform it.

Sphinx: reference special methods

Sphinx allows linking to external documentation (such as the standard library docs) via intersphinx.
Is it possible to link to the definition of special methods like __del__(), without just making a regular link?

Ok, so in my case, I just needed to link to the object.__del__ method:
:py:meth:`__del__() <object.__del__>`
To do this generically:
Use python -m sphinx.ext.intersphinx https://docs.python.org/3/objects.inv to get the inventory of the python docs. You're gonna want to pipe the output through less or grep or save it to a file
Search through the results for the thing you're looking for. For special methods, it'll probably be object.__spam__
Look at the section the thing is under, and add it to your rst

VIM "go to definition" functionality for Python [duplicate]

How can I jump to a function definition using Vim? For example with Visual Assist, I can type Alt+g under a function and it opens a context menu listing the files with definitions.
How can I do something like this in vim?

Use ctags. Generate a tags file, and tell vim where it is using the :tags command. Then you can just jump to the function definition using Ctrl-]
There are more tags tricks and tips in this question.

If everything is contained in one file, there's the command gd (as in 'goto definition'), which will take you to the first occurrence in the file of the word under the cursor, which is often the definition.

g* does a decent job without ctags being set up.
That is, type g,* (or just * - see below) to search for the word under the cursor (in this case, the function name). Then press n to go to the next (or Shift-n for previous) occurrence.
It doesn't jump directly to the definition, given that this command just searches for the word under the cursor, but if you don't want to deal with setting up ctags at the moment, you can at least save yourself from having to re-type the function name to search for its definition.
--Edit--
Although I've been using g* for a long time, I've recently discovered two shortcuts for these shortcuts!
(a) * will jump to the next occurrence of the word under the cursor. (No need to type the g, the 'goto' command in vi).
(b) # goes to the previous occurrence, in similar fashion.
N and n still work, but '#' is often very useful to start the search initially in the reverse direction, for example, when looking for the declaration of a variable under the cursor.

Use gd or gD while placing the cursor on any variable in your program.
gd will take you to the local declaration.
gD will take you to the global declaration.
more navigation options can be found in here.
Use cscope for cross referencing large project such as the linux kernel.

TL;DR:
You can do this using internal VIM functionality but a modern (and much easier) way is to use COC for intellisense-like completion and one or more language servers (LS) for jump-to-definition (and way way more). For even more functionality (but it's not needed for jump-to-definition) you can install one or more debuggers and get a full blown IDE experience.
Best second is to use native VIM's functionality called define-search but it was invented for C preprocessor's #define directive and for most other languages requires extra configuration, for some isn't possible at all (also you miss on other IDE features). Finally, a fallback to that is ctags.
Quick-start:
install vim-plug to manage your VIM plug-ins
add COC and (optionally) Vimspector at the top of ~/.vimrc:
call plug#begin()
Plug 'neoclide/coc.nvim', {'branch': 'release'}
Plug 'puremourning/vimspector'
call plug#end()
" key mappings example
nmap <silent> gd <Plug>(coc-definition)
nmap <silent> gD <Plug>(coc-implementation)
nmap <silent> gr <Plug>(coc-references)
" there's way more, see `:help coc-key-mappings#en'
call :source $MYVIMRC | PlugInstall to reload VIM config and download plug-ins
restart vim and call :CocInstall coc-marketplace to get easy access to COC extensions
call :CocList marketplace and search for language servers, e.g.:
type python to find coc-jedi,
type php to find coc-phpls, etc.
(optionally) see :h VimspectorInstall to install additional debuggers, e.g.:
:VimspectorInstall debugpy,
:VimspectorInstall vscode-php-debug, etc.
Full story:
Language server (LS) is a separate standalone application (one for each programming language) that runs in the background and analyses your whole project in real time exposing extra capabilities to your editor (any editor, not only vim). You get things like:
namespace aware tag completion
jump to definition
jump to next / previous error
find all references to an object
find all interface implementations
rename across a whole project
documentation on hover
snippets, code actions, formatting, linting and more...
Communication with language servers takes place via Language Server Protocol (LSP). Both nvim and vim8 (or higher) support LSP through plug-ins, the most popular being Conquer of Completion (COC).
List of actively developed language servers and their capabilities is available on Lang Server website. Not all of those are provided by COC extensions. If you want to use one of those you can either write a COC extension yourself or install LS manually and use the combo of following VIM plug-ins as alternative to COC:
LanguageClient - handles LSP
deoplete - triggers completion as you type
Communication with debuggers takes place via Debug Adapter Protocol (DAP). The most popular DAP plug-in for VIM is Vimspector.
Language Server Protocol (LSP) was created by Microsoft for Visual Studio Code and released as an open source project with a permissive MIT license (standardized by collaboration with Red Hat and Codenvy). Later on Microsoft released Debug Adapter Protocol (DAP) as well. Any language supported by VSCode is supported in VIM.
I personally recommend using COC + language servers provided by COC extensions + ALE for extra linting (but with LSP support disabled to avoid conflicts with COC) + Vimspector + debuggers provided by Vimspector (called "gadgets") + following VIM plug-ins:
call plug#begin()
Plug 'neoclide/coc.nvim'
Plug 'dense-analysis/ale'
Plug 'puremourning/vimspector'
Plug 'scrooloose/nerdtree'
Plug 'scrooloose/nerdcommenter'
Plug 'sheerun/vim-polyglot'
Plug 'yggdroot/indentline'
Plug 'tpope/vim-surround'
Plug 'kana/vim-textobj-user'
\| Plug 'glts/vim-textobj-comment'
Plug 'janko/vim-test'
Plug 'vim-scripts/vcscommand.vim'
Plug 'mhinz/vim-signify'
call plug#end()
You can google each to see what they do.
Native VIM jump to definition:
If you really don't want to use Language Server and still want a somewhat decent jump to definition with native VIM you should get familiar with :ij and :dj which stand for include-jump and definition-jump. These VIM commands let you jump to any file that's included by your project or jump to any defined symbol that's in any of the included files. For that to work, however, VIM has to know how lines that include files or define symbols look like in any given language. You can set it up per language in ~/.vim/ftplugin/$file_type.vim with set include=$regex and set define=$regex patterns as described in :h include-search, although, coming up with those patterns is a bit of an art and sometimes not possible at all, e.g. for languages where symbol definition or file import can span over multiple lines (e.g. Golang). If that's your case the usual fallback is ctags as described in other answers.

As Paul Tomblin mentioned you have to use ctags.
You could also consider using plugins to select appropriate one or to preview the definition of the function under cursor.
Without plugins you will have a headache trying to select one of the hundreds overloaded 'doAction' methods as built in ctags support doesn't take in account the context - just a name.
Also you can use cscope and its 'find global symbol' function. But your vim have to be compiled with +cscope support which isn't default one option of build.
If you know that the function is defined in the current file, you can use 'gD' keystrokes in a normal mode to jump to definition of the symbol under cursor.
Here is the most downloaded plugin for navigation
http://www.vim.org/scripts/script.php?script_id=273
Here is one I've written to select context while jump to tag
http://www.vim.org/scripts/script.php?script_id=2507

Another common technique is to place the function name in the first column. This allows the definition to be found with a simple search.
int
main(int argc, char *argv[])
{
...
}
The above function could then be found with /^main inside the file or with :grep -r '^main' *.c in a directory. As long as code is properly indented the only time the identifier will occur at the beginning of a line is at the function definition.
Of course, if you aren't using ctags from this point on you should be ashamed of yourself! However, I find this coding standard a helpful addition as well.

1- install exuberant ctags. If you're using osx, this article shows a little trick:
http://www.runtime-era.com/2012/05/exuberant-ctags-in-osx-107.html
2- If you only wish to include the ctags for the files in your directory only, run this command in your directory:
ctags -R
This will create a "tags" file for you.
3- If you're using Ruby and wish to include the ctags for your gems (this has been really helpful for me with RubyMotion and local gems that I have developed), do the following:
ctags --exclude=.git --exclude='*.log' -R * `bundle show --paths`
credit: https://coderwall.com/p/lv1qww
(Note that I omitted the -e option which generates tags for emacs instead of vim)
4- Add the following line to your ~/.vimrc
set autochdir
set tags+=./tags;
(Why the semi colon: http://vim.wikia.com/wiki/Single_tags_file_for_a_source_tree )
5- Go to the word you'd like to follow and hit ctrl + ] ; if you'd like to go back, use ctrl+o (source: https://stackoverflow.com/a/53929/226255)

To second Paul's response: yes, ctags (especially exuberant-ctags (http://ctags.sourceforge.net/)) is great. I have also added this to my vimrc, so I can use one tags file for an entire project:
set tags=tags;/

Install cscope. It works very much like ctags but more powerful. To go to definition, instead of Ctrl + ], do Ctrl + \ + g. Of course you may use both concurrently. But with a big project (say Linux kernel), cscope is miles ahead.

After generating ctags, you can also use the following in vim:
:tag <f_name>
Above will take you to function definition.

Python API Compatibility Checker

In my current work environment, we produce a large number of Python packages for internal use (10s if not 100s). Each package has some dependencies, usually on a mixture of internal and external packages, and some of these dependencies are shared.
As we approach dependency hell, updating dependencies becomes a time consuming process. While we care about the functional changes a new version might introduce, of equal (if not more) importance are the API changes that break the code.
Although running unit/integration tests against newer versions of a dependency helps us to catch some issues, our coverage is not close enough to 100% to make this a robust strategy. Release notes and a change log help identify major changes at a high-level, but these rarely exist for internally developed tools or go into enough detail to understand the implications the new version has on the (public) API.
I am looking at otherways to automate this process.
I would like to be able to automatically compare two versions of a Python package and report the API differences between them. In particular this would include backwards incompatible changes such as removing functions/methods/classes/modules, adding positional arguments to a function/method/class and changing the number of items a function/method returns. As a developer, based on the report this generates I should have a greater understanding about the code level implications this version change will introduce, and so the time require to integrate it.
Elsewhere, we use the C++ abi-compliance-checker and are looking at the Java api-compliance-checker to help with this process. Is there a similar tool available for Python? I have found plenty of lint/analysis/refactor tools but nothing that provides this level of functionality. I understand that Python's dynamic typing will make a comprehensive report impossible.
If such a tool does not exist, are they any libraries that could help with implementing a solution? For example, my current approach would be to use an ast.NodeVisitor to traverse the package and build a tree where each node represents a module/class/method/function and then compare this tree to that of another version for the same package.
Edit: since posting the question I have found pysdiff which covers some of my requirements, but interested to see alternatives still.
Edit: also found Upstream-Tracker would is a good example of the sort of information I'd like to end up with.

What about using the AST module to parse the files?
import ast
with file("test.py") as f:
python_src = f.read()
node = ast.parse(python_src) # Note: doesn't compile the src
print ast.dump(node)
There's the walk method on the ast node (described http://docs.python.org/2/library/ast.html)
The astdump might work (available on pypi)
This out of date pretty printer
http://code.activestate.com/recipes/533146-ast-pretty-printer/
The documentation tool Sphinx also extracts the information you are looking for. Perhaps give that a look.
So walk the AST and build a tree with the information you want in it. Once you have a tree you can pickle it and diff later or convert the tree to a text representation in a
text file you can diff with difftools, or some external diff program.
The ast has parse() and compile() methods. Only thing is I'm not entirely sure how much information is available to you after parsing (as you don't want to compile()).

Perhaps you can start by using the inspect module
import inspect
import types
def genFunctions(module):
moduleDict = module.__dict__
for name in dir(module):
if name.startswith('_'):
continue
element = moduleDict[name]
if isinstance(element, types.FunctionType):
argSpec = inspect.getargspec(element)
argList = argSpec.args
print "{}.{}({})".format(module.__name__, name, ", ".join(argList))
That will give you a list of "public" (not starting with underscore) functions with their argument lists. You can add more stuff to print the kwargs, classes, etc.
Once you run that on all the packages/modules you care about, in both old and new versions, you'll have two lists like this:
myPackage.myModule.myFunction1(foo, bar)
myPackage.myModule.myFunction2(baz)
Then you can either just sort and diff them, or write some smarter tooling in Python to actually compare all the names, e.g. to permit additional optional arguments but reject new mandatory arguments.

Check out zope.interfaces (you can get it from PyPI). Then you can incorporate unit testing that modules support interfaces into your unit tests. May take a while to retro fit however - also it's not a silver bullet.

XMP image tagging and Python

If I were to tag a bunch of images via XMP, in Python, what would be the best way? I've used Perl's Image::ExifTool and I am very much used to its reliability. I mean the thing never bricked on tens of thousands of images.
I found this, backed by some heavy-hitters like the European Space Agency, but it's clearly marked as unstable.
Now, assuming I am comfortable with C++, how easy is it to, say, use the Adobe XMP toolkit directly, in Python? Having never done this before, I am not sure what I'd sign up for.
Update: I tried some libraries out there and, including the fore mentioned toolkit, they are still pretty immature and have glaring problems. I resorted to actually writing an Perl-based server that accepts XML requests to read and write metadata, with the combat-tested Image::EXIF. The amount of code is actually very light, and definitely beats torturing yourself by trying to get the Python libraries to work. The server solution is language-agnostic, so it's a twofer.

Well, they website says that the python-xmp-toolkit uses Exempi, which is based on the Adobe XMP toolkit, via ctypes. What I'm trying to say is that you're not likely to create a better wrapping of the C++ code yourself. If it's unstable (i.e. buggy), it's most likely still cheaper for you to create patches than doing it yourself from scratch.
However, in your special situation, it depends on how much functionality you need. If you just need a single function, then wrapping the C++ code into a small C extension library or with Cython is feasible. When you need to have all functionality & flexibility, you have to create wrappers manually or using SWIG, basically repeating the work already done by other people.

I struggled for several hours with python-xmp-toolkit, and eventually gave up and just wrapped calls to ExifTool.
There is a Ruby library that wraps ExifTool as well (albeit, much better than what I created); I feel it'd be worth porting it to Python for a simple way of dealing with XMP.

For Python 3.x there's py3exiv2 which supports editing XMP metadata
With py3exiv2 you can read and write all standard metadata, create your own XMP namespace or extract the thumbnail embedded in image file.
One thing I like about py3exiv2 is that it's built on the (C++) exiv2 library which seems well-maintained
I did encounter a problem though when installing it on my system (Ubuntu 16.04). To get it working I first had to install the latest version of libexiv2-dev (sudo apt-get install libexiv2-dev), and only after this install py3exiv2 (sudo -H pip3 install py3exiv2)
Here's how I've used py3exiv2 to write a new tag:
import pyexiv2
metadata = pyexiv2.ImageMetadata("file_name.jpg")
metadata.read()
key = "Xmp.xmp.CustomTagKey"
value = "CustomTagValue"
metadata[key] = pyexiv2.XmpTag(key, value)
metadata.write()
(There's also a tutorial in the documentation)

For people finding this thread in the future, I would like to share my solution. I put a package up on the Python Package Index (PyPI) called imgtag. It lets you do basic XMP subject field tag editing using python-xmp-toolkit, but abstracts away all of the frustrating nonsense of actually using python-xmp-toolkit into one-line commands.
Install exempi for your platform, then run
python3 -m pip install imgtag
Now you can use it as such:
from imgtag import ImgTag
# Open image for tag editing
test = ImgTag(
filename="test.jpg", # The image file
force_case="lower", # Converts the case of all tags
# Can be `None`, `"lower"`, `"upper"`
# Default: None
strip=True, # Strips whitespace from the ends of all tags
# Default: True
no_duplicates=True # Removes all duplicate tags (case sensitive)
# Default: True
)
# Print existing tags
print("Current tags:")
for tag in test.get_tags():
print(" Tag:", tag)
# Add tags
test.add_tags(["sleepy", "happy"])
# Remove tags
test.remove_tags(["cute"])
# Set tags, removing all existing tags
test.set_tags(["dog", "good boy"])
# Save changes and close file
test.close()
# Re-open for tag editing
test.open()
# Remove all tags
test.clear_tags()
# Delete the ImgTag object, automatically saving and closing the file
del(test)
I haven't yet added methods for the other XMP fields like description, date, creator, etc. Maybe someday I will, but if you look at how the existing functions work in the source code, you can probably figure out how to add the method yourself. If you do add more methods, make a pull request please. :)

You can use ImageMagic convert, IIRC there's a Python module to it as well.