A trace of the Neovim providers

2024-02-29

Introduction

A while back I was working a bit with the Neovim RPC, so this is an end to end review of the inner workings of RPC in Neovim. In particular I am focusing on what Neovim calls "hosts", the co-processes that execute code from various programming languages.

When Neovim forked Vim, authors decided to externalize certain features - namely the python/perl support. In Neovim these were moved to external processes, that communicate with the main Neovim process using the msgpack-rpc protocol. Communication happens via pipes between the two processes.

meta:

High level overview

At a high level Neovim provides the ability to invoke programming languages from Vimscript code, or the command input. For example if you type the following command:

you should see some output (provided python and pynvim are installed).

The same applies to other languages. In my case perl is not installed so this gets me an error:

Writing plugins in python/perl is just an extension of these commands. In Neovim this causes the python host process to be spawned, and code is executed by passing the python code to remote process and getting a response back.

You can detect these processes in your system by checking the list of processes. For example in my system

this shows that nvim has a child process (python3) which was invoked with arguments to execute the neovim host (part of the pynvim library).

For more documentation on how to configure these hosts check the Neovim docs (:h provider.txt). There are default providers for python/perl/nodejs/ruby.

The Neovim side

The entry point for remote processes in Neovim code is a vimscript plugin which can be found at runtime/autoload/remote/host.vim. In there you will find various calls executed when the vimscript code is loaded. Here is the one for python3

This triggers the registration of the python3 provider. The Register function is in the same file

The first argument is the name of the provider (python3), the second is the wildcard used to identify plugins (*) and the third is a callback function which is called when the provider needs to be started. Triggering of the callback happens from the remote#host#Require function, but only when the provider is needed.

For the callback function code for python check runtime/autoload/provider/python3.vim

but this is only a vimscript shim that calls into Lua code (besides vimscript Neovim also runs lua internally). We can follow this up by looking inside runtime/lua/vim/provider/python.lua

In here we can see something that resembles a python process invocation and a snippet of python code that runs something called start_host.

The two remaining pieces in this function are

The call to Poll is the one that actually spawns the python process (see provider.vim)

after starting the process, it invokes an rpc request to a function called poll(). The poll() RPC works as a ping to check if the remote process is alive. If all works, the function returns a channel id which identifies the open RPC channel.

However this callback is only really called when needed, from the remote#host#Require function (back in host.vim):

Require function is called from different places, e.g. RegistrationCommands

besides starting the host process, this function also handles individual plugin setup which we have not covered yet. Without going into further detail we can see there is an RPC call to a method called specs.

Another important location is the C function that calls providers in eval.c:

and here is the python3 entry point C (funcs.c):

This gives us an overview of how Neovim starts these processes and redirects calls to them. We should note here that this API is perfectly accessible to plugins, so you could write your provider with a bit of vimscript or lua to trigger registration.

For example you can call these as commands

The "host" side

Meanwhile the python host side is handled by the pynvim library. This is the function being called when python is spawned. This is the same function we saw being passed to the python process:

I wont list it all here, but after doing plugin initialization and nvim version checks it calls an event loop

which continues in api/nvim.py

and eventually calls the msgpack-rpc loop

I wont detail how the msgpack-rpc protocols work. It is general purpose RPC protocol that supports both synchronous request-response as well as notifications.

Most hosts seem to provide the following RPC methods

Here is the python code that registers them

poll is used as a ping function (it just returns the string ok). specs is a function called by Neovim to get a list of specification i.e. all the plugins and the functions/commands they provide.

Plugins

Each host program can provide individual plugins, loaded from files. The high level process is described in remote_plugin.txt.

For most remote hosts plugin setup works as follows

This means that the lifecycle of your host process can require it to be spawned once to search for plugins, but once rplugin.vim is in place it should only be started if needed.

I did not get far in understanding all the plugin semantics, but the best place to look seems to be the vimscript remote code. For example plugin registration, the format for rplugin.vim and the format for specs return value:

the rest is covered in the RegistrationCommands function (listed earlier). One aspect to node is that function returns code, which is then stored in rplugin.vim:

while the setup of the Neovim commands/functions is done inside autoload/remote/define.vim.

References

msgpack-rpc spec

https://github.com/neovim/pynvim

https://github.com/neovim/neovim

Proxied content from gemini://tilde.pink/~rrobin/2024-02-29-a-trace-of-the-neovim-providers.gmi (external content)

Gemini request details:

Original URL
gemini://tilde.pink/~rrobin/2024-02-29-a-trace-of-the-neovim-providers.gmi
Status code
Success
Meta
text/gemini;lang=en
Proxied by
kineto

Be advised that no attempt was made to verify the remote SSL certificate.