Developing a "recipe" extension
*******************************

The objective of this tutorial is to illustrate roles, directives and
domains. Once complete, we will be able to use this extension to
describe a recipe and reference that recipe from elsewhere in our
documentation.

Note:

  This tutorial is based on a guide first published on opensource.com
  and is provided here with the original author's permission.


Overview
========

We want the extension to add the following to Sphinx:

* A "recipe" *directive*, containing some content describing the
  recipe steps, along with a ":contains:" option highlighting the main
  ingredients of the recipe.

* A "ref" *role*, which provides a cross-reference to the recipe
  itself.

* A "recipe" *domain*, which allows us to tie together the above role
  and domain, along with things like indices.

For that, we will need to add the following elements to Sphinx:

* A new directive called "recipe"

* New indexes to allow us to reference ingredient and recipes

* A new domain called "recipe", which will contain the "recipe"
  directive and "ref" role


Prerequisites
=============

We need the same setup as in the previous extensions. This time, we
will be putting out extension in a file called "recipe.py".

Here is an example of the folder structure you might obtain:

   └── source
       ├── _ext
       │   └── recipe.py
       ├── conf.py
       └── index.rst


Writing the extension
=====================

Open "recipe.py" and paste the following code in it, all of which we
will explain in detail shortly:

   from collections import defaultdict

   from docutils.parsers.rst import directives

   from sphinx import addnodes
   from sphinx.directives import ObjectDescription
   from sphinx.domains import Domain
   from sphinx.domains import Index
   from sphinx.roles import XRefRole
   from sphinx.util.nodes import make_refnode


   class RecipeDirective(ObjectDescription):
       """A custom directive that describes a recipe."""

       has_content = True
       required_arguments = 1
       option_spec = {
           'contains': directives.unchanged_required,
       }

       def handle_signature(self, sig, signode):
           signode += addnodes.desc_name(text=sig)
           return sig

       def add_target_and_index(self, name_cls, sig, signode):
           signode['ids'].append('recipe' + '-' + sig)
           if 'noindex' not in self.options:
               ingredients = [
                   x.strip() for x in self.options.get('contains').split(',')]

               recipes = self.env.get_domain('recipe')
               recipes.add_recipe(sig, ingredients)


   class IngredientIndex(Index):
       """A custom index that creates an ingredient matrix."""

       name = 'ingredient'
       localname = 'Ingredient Index'
       shortname = 'Ingredient'

       def generate(self, docnames=None):
           content = defaultdict(list)

           recipes = {name: (dispname, typ, docname, anchor)
                      for name, dispname, typ, docname, anchor, _
                      in self.domain.get_objects()}
           recipe_ingredients = self.domain.data['recipe_ingredients']
           ingredient_recipes = defaultdict(list)

           # flip from recipe_ingredients to ingredient_recipes
           for recipe_name, ingredients in recipe_ingredients.items():
               for ingredient in ingredients:
                   ingredient_recipes[ingredient].append(recipe_name)

           # convert the mapping of ingredient to recipes to produce the expected
           # output, shown below, using the ingredient name as a key to group
           #
           # name, subtype, docname, anchor, extra, qualifier, description
           for ingredient, recipe_names in ingredient_recipes.items():
               for recipe_name in recipe_names:
                   dispname, typ, docname, anchor = recipes[recipe_name]
                   content[ingredient].append(
                       (dispname, 0, docname, anchor, docname, '', typ))

           # convert the dict to the sorted list of tuples expected
           content = sorted(content.items())

           return content, True


   class RecipeIndex(Index):
       """A custom index that creates an recipe matrix."""

       name = 'recipe'
       localname = 'Recipe Index'
       shortname = 'Recipe'

       def generate(self, docnames=None):
           content = defaultdict(list)

           # sort the list of recipes in alphabetical order
           recipes = self.domain.get_objects()
           recipes = sorted(recipes, key=lambda recipe: recipe[0])

           # generate the expected output, shown below, from the above using the
           # first letter of the recipe as a key to group thing
           #
           # name, subtype, docname, anchor, extra, qualifier, description
           for name, dispname, typ, docname, anchor, _ in recipes:
               content[dispname[0].lower()].append(
                   (dispname, 0, docname, anchor, docname, '', typ))

           # convert the dict to the sorted list of tuples expected
           content = sorted(content.items())

           return content, True


   class RecipeDomain(Domain):

       name = 'recipe'
       label = 'Recipe Sample'
       roles = {
           'ref': XRefRole()
       }
       directives = {
           'recipe': RecipeDirective,
       }
       indices = {
           RecipeIndex,
           IngredientIndex
       }
       initial_data = {
           'recipes': [],  # object list
           'recipe_ingredients': {},  # name -> object
       }

       def get_full_qualified_name(self, node):
           return '{}.{}'.format('recipe', node.arguments[0])

       def get_objects(self):
           for obj in self.data['recipes']:
               yield(obj)

       def resolve_xref(self, env, fromdocname, builder, typ, target, node,
                        contnode):
           match = [(docname, anchor)
                    for name, sig, typ, docname, anchor, prio
                    in self.get_objects() if sig == target]

           if len(match) > 0:
               todocname = match[0][0]
               targ = match[0][1]

               return make_refnode(builder, fromdocname, todocname, targ,
                                   contnode, targ)
           else:
               print('Awww, found nothing')
               return None

       def add_recipe(self, signature, ingredients):
           """Add a new recipe to the domain."""
           name = '{}.{}'.format('recipe', signature)
           anchor = 'recipe-{}'.format(signature)

           self.data['recipe_ingredients'][name] = ingredients
           # name, dispname, type, docname, anchor, priority
           self.data['recipes'].append(
               (name, signature, 'Recipe', self.env.docname, anchor, 0))


   def setup(app):
       app.add_domain(RecipeDomain)

       return {
           'version': '0.1',
           'parallel_read_safe': True,
           'parallel_write_safe': True,
       }

Let's look at each piece of this extension step-by-step to explain
what's going on.

-[ The directive class ]-

The first thing to examine is the "RecipeDirective" directive:

       required_arguments = 1
       option_spec = {
           'contains': directives.unchanged_required,
       }

       def handle_signature(self, sig, signode):
           signode += addnodes.desc_name(text=sig)
           return sig

       def add_target_and_index(self, name_cls, sig, signode):
           signode['ids'].append('recipe' + '-' + sig)
           if 'noindex' not in self.options:
               ingredients = [
                   x.strip() for x in self.options.get('contains').split(',')]

               recipes = self.env.get_domain('recipe')
               recipes.add_recipe(sig, ingredients)


   class IngredientIndex(Index):
       """A custom index that creates an ingredient matrix."""

Unlike Developing a "Hello world" extension and Developing a "TODO"
extension, this directive doesn't derive from
"docutils.parsers.rst.Directive" and doesn't define a "run" method.
Instead, it derives from "sphinx.directives.ObjectDescription" and
defines  "handle_signature" and "add_target_and_index" methods. This
is because "ObjectDescription" is a special-purpose directive that's
intended for describing things like classes, functions, or, in our
case, recipes. More specifically, "handle_signature" implements
parsing the signature of the directive and passes on the object's name
and type to its superclass, while "add_taget_and_index" adds a target
(to link to) and an entry to the index for this node.

We also see that this directive defines "has_content",
"required_arguments" and "option_spec". Unlike the "TodoDirective"
directive added in the previous tutorial, this directive takes a
single argument, the recipe name, and an option, "contains", in
addition to the nested reStructuredText in the body.

-[ The index classes ]-

Todo: Add brief overview of indices

       localname = 'Ingredient Index'
       shortname = 'Ingredient'

       def generate(self, docnames=None):
           content = defaultdict(list)

           recipes = {name: (dispname, typ, docname, anchor)
                      for name, dispname, typ, docname, anchor, _
                      in self.domain.get_objects()}
           recipe_ingredients = self.domain.data['recipe_ingredients']
           ingredient_recipes = defaultdict(list)

           # flip from recipe_ingredients to ingredient_recipes
           for recipe_name, ingredients in recipe_ingredients.items():
               for ingredient in ingredients:
                   ingredient_recipes[ingredient].append(recipe_name)

           # convert the mapping of ingredient to recipes to produce the expected
           # output, shown below, using the ingredient name as a key to group
           #
           # name, subtype, docname, anchor, extra, qualifier, description
           for ingredient, recipe_names in ingredient_recipes.items():
               for recipe_name in recipe_names:
                   dispname, typ, docname, anchor = recipes[recipe_name]
                   content[ingredient].append(
                       (dispname, 0, docname, anchor, docname, '', typ))

           # convert the dict to the sorted list of tuples expected
           content = sorted(content.items())

           return content, True


   class RecipeIndex(Index):
       """A custom index that creates an recipe matrix."""

       name = 'recipe'
       localname = 'Recipe Index'
       shortname = 'Recipe'

       def generate(self, docnames=None):
           content = defaultdict(list)

           # sort the list of recipes in alphabetical order
           recipes = self.domain.get_objects()
           recipes = sorted(recipes, key=lambda recipe: recipe[0])

           # generate the expected output, shown below, from the above using the
           # first letter of the recipe as a key to group thing
           #
           # name, subtype, docname, anchor, extra, qualifier, description
           for name, dispname, typ, docname, anchor, _ in recipes:
               content[dispname[0].lower()].append(
                   (dispname, 0, docname, anchor, docname, '', typ))

           # convert the dict to the sorted list of tuples expected
           content = sorted(content.items())

           return content, True


   class RecipeDomain(Domain):


Both "IngredientIndex" and "RecipeIndex" are derived from "Index".
They implement custom logic to generate a tuple of values that define
the index. Note that "RecipeIndex" is a simple index that has only one
entry. Extending it to cover more object types is not yet part of the
code.

Both indices use the method "Index.generate()" to do their work. This
method combines the information from our domain, sorts it, and returns
it in a list structure that will be accepted by Sphinx. This might
look complicated but all it really is is a list of tuples like
"('tomato', 'TomatoSoup', 'test', 'rec-TomatoSoup',...)". Refer to the
domain API guide for more information on this API.

-[ The domain ]-

A Sphinx domain is a specialized container that ties together roles,
directives, and indices, among other things. Let's look at the domain
we're creating here.

       roles = {
           'ref': XRefRole()
       }
       directives = {
           'recipe': RecipeDirective,
       }
       indices = {
           RecipeIndex,
           IngredientIndex
       }
       initial_data = {
           'recipes': [],  # object list
           'recipe_ingredients': {},  # name -> object
       }

       def get_full_qualified_name(self, node):
           return '{}.{}'.format('recipe', node.arguments[0])

       def get_objects(self):
           for obj in self.data['recipes']:
               yield(obj)

       def resolve_xref(self, env, fromdocname, builder, typ, target, node,
                        contnode):
           match = [(docname, anchor)
                    for name, sig, typ, docname, anchor, prio
                    in self.get_objects() if sig == target]

           if len(match) > 0:
               todocname = match[0][0]
               targ = match[0][1]

               return make_refnode(builder, fromdocname, todocname, targ,
                                   contnode, targ)
           else:
               print('Awww, found nothing')
               return None

       def add_recipe(self, signature, ingredients):
           """Add a new recipe to the domain."""
           name = '{}.{}'.format('recipe', signature)
           anchor = 'recipe-{}'.format(signature)

           self.data['recipe_ingredients'][name] = ingredients
           # name, dispname, type, docname, anchor, priority
           self.data['recipes'].append(
               (name, signature, 'Recipe', self.env.docname, anchor, 0))


   def setup(app):
       app.add_domain(RecipeDomain)

There are some interesting things to note about this "recipe" domain
and domains in general. Firstly, we actually register our directives,
roles and indices here, via the "directives", "roles" and "indices"
attributes, rather than via calls later on in "setup". We can also
note that we aren't actually defining a custom role and are instead
reusing the "sphinx.roles.XRefRole" role and defining the
"sphinx.domains.Domain.resolve_xref" method. This method takes two
arguments, "typ" and "target", which refer to the cross-reference type
and its target name. We'll use "target" to resolve our destination
from our domain's "recipes" because we currently have only one type of
node.

Moving on, we can see that we've defined "initial_data". The values
defined in "initial_data" will be copied to
"env.domaindata[domain_name]" as the initial data of the domain, and
domain instances can access it via "self.data". We see that we have
defined two items in "initial_data": "recipes" and
"recipe2ingredient". These contain a list of all objects defined (i.e.
all recipes) and a hash that maps a canonical ingredient name to the
list of objects. The way we name objects is common across our
extension and is defined in the "get_full_qualified_name" method. For
each object created, the canonical name is "recipe.<recipename>",
where "<recipename>" is the name the documentation writer gives the
object (a recipe). This enables the extension to use different object
types that share the same name. Having a canonical name and central
place for our objects is a huge advantage. Both our indices and our
cross-referencing code use this feature.

-[ The "setup" function ]-

As always, the "setup" function is a requirement and is used to hook
the various parts of our extension into Sphinx. Let's look at the
"setup" function for this extension.

           'version': '0.1',
           'parallel_read_safe': True,
           'parallel_write_safe': True,
       }

This looks a little different to what we're used to seeing. There are
no calls to "add_directive()" or even "add_role()". Instead, we have a
single call to "add_domain()" followed by some initialization of the
standard domain. This is because we had already registered our
directives, roles and indexes as part of the directive itself.


Using the extension
===================

You can now use the extension throughout your project. For example:

index.rst

   Joe's Recipes
   =============

   Below are a collection of my favourite recipes. I highly recommend the
   :recipe:ref:`TomatoSoup` recipe in particular!

   .. toctree::

      tomato-soup

tomato-soup.rst

   The recipe contains `tomato` and `cilantro`.

   .. recipe:recipe:: TomatoSoup
     :contains: tomato cilantro salt pepper

    This recipe is a tasty tomato soup, combine all ingredients
    and cook.

The important things to note are the use of the ":recipe:ref:" role to
cross-reference the recipe actually defined elsewhere (using the
":recipe:recipe:" directive.


Further reading
===============

For more information, refer to the docutils documentation and
Developing extensions for Sphinx.
