annotate en/ch13-mq-collab.xml @ 674:bebd5ff7fe7f

And there was much tidying, and the result was good.

<!-- vim: set filetype=docbkxml shiftwidth=2 autoindent expandtab tw=77 : -->

<chapter id="chap:mq-collab">
  <?dbhtml filename="advanced-uses-of-mercurial-queues.html"?>
  <title>Advanced uses of Mercurial Queues</title>

  <para>While it's easy to pick up straightforward uses of Mercurial
    Queues, use of a little discipline and some of MQ's less
    frequently used capabilities makes it possible to work in
    complicated development environments.</para>

  <para>In this chapter, I will use as an example a technique I have
    used to manage the development of an Infiniband device driver for
    the Linux kernel.  The driver in question is large (at least as
    drivers go), with 25,000 lines of code spread across 35 source
    files.  It is maintained by a small team of developers.</para>

  <para>While much of the material in this chapter is specific to
    Linux, the same principles apply to any code base for which you're
    not the primary owner, and upon which you need to do a lot of
    development.</para>

  <sect1>
    <title>The problem of many targets</title>

    <para>The Linux kernel changes rapidly, and has never been
      internally stable; developers frequently make drastic changes
      between releases. This means that a version of the driver that
      works well with a particular released version of the kernel will
      not even <emphasis>compile</emphasis> correctly against,
      typically, any other version.</para>

    <para>To maintain a driver, we have to keep a number of distinct
      versions of Linux in mind.</para>
    <itemizedlist>
      <listitem><para>One target is the main Linux kernel development
	  tree. Maintenance of the code is in this case partly shared
	  by other developers in the kernel community, who make
	  <quote>drive-by</quote> modifications to the driver as they
	  develop and refine kernel subsystems.</para>
      </listitem>
      <listitem><para>We also maintain a number of
	  <quote>backports</quote> to older versions of the Linux
	  kernel, to support the needs of customers who are running
	  older Linux distributions that do not incorporate our
	  drivers.  (To <emphasis>backport</emphasis> a piece of code
	  is to modify it to work in an older version of its target
	  environment than the version it was developed for.)</para>
      </listitem>
      <listitem><para>Finally, we make software releases on a schedule
	  that is necessarily not aligned with those used by Linux
	  distributors and kernel developers, so that we can deliver
	  new features to customers without forcing them to upgrade
	  their entire kernels or distributions.</para>
      </listitem></itemizedlist>

    <sect2>
      <title>Tempting approaches that don't work well</title>

      <para>There are two <quote>standard</quote> ways to maintain a
	piece of software that has to target many different
	environments.</para>

      <para>The first is to maintain a number of branches, each
	intended for a single target.  The trouble with this approach
	is that you must maintain iron discipline in the flow of
	changes between repositories. A new feature or bug fix must
	start life in a <quote>pristine</quote> repository, then
	percolate out to every backport repository.  Backport changes
	are more limited in the branches they should propagate to; a
	backport change that is applied to a branch where it doesn't
	belong will probably stop the driver from compiling.</para>

      <para>The second is to maintain a single source tree filled with
	conditional statements that turn chunks of code on or off
	depending on the intended target.  Because these
	<quote>ifdefs</quote> are not allowed in the Linux kernel
	tree, a manual or automatic process must be followed to strip
	them out and yield a clean tree.  A code base maintained in
	this fashion rapidly becomes a rat's nest of conditional
	blocks that are difficult to understand and maintain.</para>

      <para>Neither of these approaches is well suited to a situation
	where you don't <quote>own</quote> the canonical copy of a
	source tree.  In the case of a Linux driver that is
	distributed with the standard kernel, Linus's tree contains
	the copy of the code that will be treated by the world as
	canonical.  The upstream version of <quote>my</quote> driver
	can be modified by people I don't know, without me even
	finding out about it until after the changes show up in
	Linus's tree.</para>

      <para>These approaches have the added weakness of making it
	difficult to generate well-formed patches to submit
	upstream.</para>

      <para>In principle, Mercurial Queues seems like a good candidate
	to manage a development scenario such as the above.  While
	this is indeed the case, MQ contains a few added features that
	make the job more pleasant.</para>

    </sect2>
  </sect1>
  <sect1>
    <title>Conditionally applying patches with guards</title>

    <para>Perhaps the best way to maintain sanity with so many targets
      is to be able to choose specific patches to apply for a given
      situation.  MQ provides a feature called <quote>guards</quote>
      (which originates with quilt's <literal>guards</literal>
      command) that does just this.  To start off, let's create a
      simple repository for experimenting in.</para>

    &interaction.mq.guards.init;

    <para>This gives us a tiny repository that contains two patches
      that don't have any dependencies on each other, because they
      touch different files.</para>

    <para>The idea behind conditional application is that you can
      <quote>tag</quote> a patch with a <emphasis>guard</emphasis>,
      which is simply a text string of your choosing, then tell MQ to
      select specific guards to use when applying patches.  MQ will
      then either apply, or skip over, a guarded patch, depending on
      the guards that you have selected.</para>

    <para>A patch can have an arbitrary number of guards; each one is
      <emphasis>positive</emphasis> (<quote>apply this patch if this
	guard is selected</quote>) or <emphasis>negative</emphasis>
      (<quote>skip this patch if this guard is selected</quote>).  A
      patch with no guards is always applied.</para>

  </sect1>
  <sect1>
    <title>Controlling the guards on a patch</title>

    <para>The <command role="hg-ext-mq">qguard</command> command lets
      you determine which guards should apply to a patch, or display
      the guards that are already in effect. Without any arguments, it
      displays the guards on the current topmost patch.</para>

      &interaction.mq.guards.qguard;

    <para>To set a positive guard on a patch, prefix the name of the
      guard with a <quote><literal>+</literal></quote>.</para>

      &interaction.mq.guards.qguard.pos;

    <para>To set a negative guard
      on a patch, prefix the name of the guard with a
      <quote><literal>-</literal></quote>.</para>

    &interaction.mq.guards.qguard.neg;

    <note>
      <para>  The <command role="hg-ext-mq">qguard</command> command
	<emphasis>sets</emphasis> the guards on a patch; it doesn't
	<emphasis>modify</emphasis> them.  What this means is that if
	you run <command role="hg-cmd">hg qguard +a +b</command> on a
	patch, then <command role="hg-cmd">hg qguard +c</command> on
	the same patch, the <emphasis>only</emphasis> guard that will
	be set on it afterwards is <literal>+c</literal>.</para>
    </note>

    <para>Mercurial stores guards in the <filename
	role="special">series</filename> file; the form in which they
      are stored is easy both to understand and to edit by hand. (In
      other words, you don't have to use the <command
	role="hg-ext-mq">qguard</command> command if you don't want
      to; it's okay to simply edit the <filename
	role="special">series</filename> file.)</para>

    &interaction.mq.guards.series;

  </sect1>
  <sect1>
    <title>Selecting the guards to use</title>

    <para>The <command role="hg-ext-mq">qselect</command> command
      determines which guards are active at a given time.  The effect
      of this is to determine which patches MQ will apply the next
      time you run <command role="hg-ext-mq">qpush</command>.  It has
      no other effect; in particular, it doesn't do anything to
      patches that are already applied.</para>

    <para>With no arguments, the <command
	role="hg-ext-mq">qselect</command> command lists the guards
      currently in effect, one per line of output.  Each argument is
      treated as the name of a guard to apply.</para>

      &interaction.mq.guards.qselect.foo;

    <para>In case you're interested, the currently selected guards are
      stored in the <filename role="special">guards</filename> file.</para>

    &interaction.mq.guards.qselect.cat;

    <para>We can see the effect the selected guards have when we run
      <command role="hg-ext-mq">qpush</command>.</para>

    &interaction.mq.guards.qselect.qpush;

    <para>A guard cannot start with a
      <quote><literal>+</literal></quote> or
      <quote><literal>-</literal></quote> character.  The name of a
      guard must not contain white space, but most other characters
      are acceptable.  If you try to use a guard with an invalid name,
      MQ will complain:</para>

    &interaction.mq.guards.qselect.error;
      
    <para>Changing the selected guards changes the patches that are
      applied.</para>

    &interaction.mq.guards.qselect.quux;

    <para>You can see in the example below that negative guards take
      precedence over positive guards.</para>

    &interaction.mq.guards.qselect.foobar;

  </sect1>
  <sect1>
    <title>MQ's rules for applying patches</title>

    <para>The rules that MQ uses when deciding whether to apply a
      patch are as follows.</para>
    <itemizedlist>
      <listitem><para>A patch that has no guards is always
	  applied.</para>
      </listitem>
      <listitem><para>If the patch has any negative guard that matches
	  any currently selected guard, the patch is skipped.</para>
      </listitem>
      <listitem><para>If the patch has any positive guard that matches
	  any currently selected guard, the patch is applied.</para>
      </listitem>
      <listitem><para>If the patch has positive or negative guards,
	  but none matches any currently selected guard, the patch is
	  skipped.</para>
      </listitem></itemizedlist>

  </sect1>
  <sect1>
    <title>Trimming the work environment</title>

    <para>In working on the device driver I mentioned earlier, I don't
      apply the patches to a normal Linux kernel tree.  Instead, I use
      a repository that contains only a snapshot of the source files
      and headers that are relevant to Infiniband development.  This
      repository is 1% the size of a kernel repository, so it's easier
      to work with.</para>

    <para>I then choose a <quote>base</quote> version on top of which
      the patches are applied.  This is a snapshot of the Linux kernel
      tree as of a revision of my choosing.  When I take the snapshot,
      I record the changeset ID from the kernel repository in the
      commit message.  Since the snapshot preserves the
      <quote>shape</quote> and content of the relevant parts of the
      kernel tree, I can apply my patches on top of either my tiny
      repository or a normal kernel tree.</para>

    <para>Normally, the base tree atop which the patches apply should
      be a snapshot of a very recent upstream tree.  This best
      facilitates the development of patches that can easily be
      submitted upstream with few or no modifications.</para>

  </sect1>
  <sect1>
    <title>Dividing up the <filename role="special">series</filename>
      file</title>

    <para>I categorise the patches in the <filename
	role="special">series</filename> file into a number of logical
      groups.  Each section of like patches begins with a block of
      comments that describes the purpose of the patches that
      follow.</para>

    <para>The sequence of patch groups that I maintain follows.  The
      ordering of these groups is important; I'll describe why after I
      introduce the groups.</para>
    <itemizedlist>
      <listitem><para>The <quote>accepted</quote> group.  Patches that
	  the development team has submitted to the maintainer of the
	  Infiniband subsystem, and which he has accepted, but which
	  are not present in the snapshot that the tiny repository is
	  based on.  These are <quote>read only</quote> patches,
	  present only to transform the tree into a similar state as
	  it is in the upstream maintainer's repository.</para>
      </listitem>
      <listitem><para>The <quote>rework</quote> group.  Patches that I
	  have submitted, but that the upstream maintainer has
	  requested modifications to before he will accept
	  them.</para>
      </listitem>
      <listitem><para>The <quote>pending</quote> group.  Patches that
	  I have not yet submitted to the upstream maintainer, but
	  which we have finished working on. These will be <quote>read
	    only</quote> for a while.  If the upstream maintainer
	  accepts them upon submission, I'll move them to the end of
	  the <quote>accepted</quote> group.  If he requests that I
	  modify any, I'll move them to the beginning of the
	  <quote>rework</quote> group.</para>
      </listitem>
      <listitem><para>The <quote>in progress</quote> group.  Patches
	  that are actively being developed, and should not be
	  submitted anywhere yet.</para>
      </listitem>
      <listitem><para>The <quote>backport</quote> group.  Patches that
	  adapt the source tree to older versions of the kernel
	  tree.</para>
      </listitem>
      <listitem><para>The <quote>do not ship</quote> group.  Patches
	  that for some reason should never be submitted upstream.
	  For example, one such patch might change embedded driver
	  identification strings to make it easier to distinguish, in
	  the field, between an out-of-tree version of the driver and
	  a version shipped by a distribution vendor.</para>
      </listitem></itemizedlist>

    <para>Now to return to the reasons for ordering groups of patches
      in this way.  We would like the lowest patches in the stack to
      be as stable as possible, so that we will not need to rework
      higher patches due to changes in context.  Putting patches that
      will never be changed first in the <filename
	role="special">series</filename> file serves this
      purpose.</para>

    <para>We would also like the patches that we know we'll need to
      modify to be applied on top of a source tree that resembles the
      upstream tree as closely as possible.  This is why we keep
      accepted patches around for a while.</para>

    <para>The <quote>backport</quote> and <quote>do not ship</quote>
      patches float at the end of the <filename
	role="special">series</filename> file.  The backport patches
      must be applied on top of all other patches, and the <quote>do
	not ship</quote> patches might as well stay out of harm's
      way.</para>

  </sect1>
  <sect1>
    <title>Maintaining the patch series</title>

    <para>In my work, I use a number of guards to control which
      patches are to be applied.</para>

    <itemizedlist>
      <listitem><para><quote>Accepted</quote> patches are guarded with
	  <literal>accepted</literal>.  I enable this guard most of
	  the time.  When I'm applying the patches on top of a tree
	  where the patches are already present, I can turn this patch
	  off, and the patches that follow it will apply
	  cleanly.</para>
      </listitem>
      <listitem><para>Patches that are <quote>finished</quote>, but
	  not yet submitted, have no guards.  If I'm applying the
	  patch stack to a copy of the upstream tree, I don't need to
	  enable any guards in order to get a reasonably safe source
	  tree.</para>
      </listitem>
      <listitem><para>Those patches that need reworking before being
	  resubmitted are guarded with
	  <literal>rework</literal>.</para>
      </listitem>
      <listitem><para>For those patches that are still under
	  development, I use <literal>devel</literal>.</para>
      </listitem>
      <listitem><para>A backport patch may have several guards, one
	  for each version of the kernel to which it applies.  For
	  example, a patch that backports a piece of code to 2.6.9
	  will have a <literal>2.6.9</literal> guard.</para>
      </listitem></itemizedlist>
    <para>This variety of guards gives me considerable flexibility in
      determining what kind of source tree I want to end up with.  For
      most situations, the selection of appropriate guards is
      automated during the build process, but I can manually tune the
      guards to use for less common circumstances.</para>

    <sect2>
      <title>The art of writing backport patches</title>

      <para>Using MQ, writing a backport patch is a simple process.
	All such a patch has to do is modify a piece of code that uses
	a kernel feature not present in the older version of the
	kernel, so that the driver continues to work correctly under
	that older version.</para>

      <para>A useful goal when writing a good backport patch is to
	make your code look as if it was written for the older version
	of the kernel you're targeting.  The less obtrusive the patch,
	the easier it will be to understand and maintain.  If you're
	writing a collection of backport patches to avoid the
	<quote>rat's nest</quote> effect of lots of
	<literal>#ifdef</literal>s (hunks of source code that are only
	used conditionally) in your code, don't introduce
	version-dependent <literal>#ifdef</literal>s into the patches.
	Instead, write several patches, each of which makes
	unconditional changes, and control their application using
	guards.</para>

      <para>There are two reasons to divide backport patches into a
	distinct group, away from the <quote>regular</quote> patches
	whose effects they modify. The first is that intermingling the
	two makes it more difficult to use a tool like the <literal
	  role="hg-ext">patchbomb</literal> extension to automate the
	process of submitting the patches to an upstream maintainer.
	The second is that a backport patch could perturb the context
	in which a subsequent regular patch is applied, making it
	impossible to apply the regular patch cleanly
	<emphasis>without</emphasis> the earlier backport patch
	already being applied.</para>

    </sect2>
  </sect1>
  <sect1>
    <title>Useful tips for developing with MQ</title>

    <sect2>
      <title>Organising patches in directories</title>

      <para>If you're working on a substantial project with MQ, it's
	not difficult to accumulate a large number of patches.  For
	example, I have one patch repository that contains over 250
	patches.</para>

      <para>If you can group these patches into separate logical
	categories, you can if you like store them in different
	directories; MQ has no problems with patch names that contain
	path separators.</para>

    </sect2>
    <sect2 id="mq-collab:tips:interdiff">
      <title>Viewing the history of a patch</title>

      <para>If you're developing a set of patches over a long time,
	it's a good idea to maintain them in a repository, as
	discussed in section <xref linkend="sec:mq:repo"/>.  If you do
	so, you'll quickly
	discover that using the <command role="hg-cmd">hg
	  diff</command> command to look at the history of changes to
	a patch is unworkable.  This is in part because you're looking
	at the second derivative of the real code (a diff of a diff),
	but also because MQ adds noise to the process by modifying
	time stamps and directory names when it updates a
	patch.</para>

      <para>However, you can use the <literal
	  role="hg-ext">extdiff</literal> extension, which is bundled
	with Mercurial, to turn a diff of two versions of a patch into
	something readable.  To do this, you will need a third-party
	package called <literal role="package">patchutils</literal>
	<citation>web:patchutils</citation>.  This provides a command
	named <command>interdiff</command>, which shows the
	differences between two diffs as a diff.  Used on two versions
	of the same diff, it generates a diff that represents the diff
	from the first to the second version.</para>

      <para>You can enable the <literal
	  role="hg-ext">extdiff</literal> extension in the usual way,
	by adding a line to the <literal
	  role="rc-extensions">extensions</literal> section of your
	<filename role="special"> /.hgrc</filename>.</para>
      <programlisting>[extensions] extdiff =</programlisting>
      <para>The <command>interdiff</command> command expects to be
	passed the names of two files, but the <literal
	  role="hg-ext">extdiff</literal> extension passes the program
	it runs a pair of directories, each of which can contain an
	arbitrary number of files.  We thus need a small program that
	will run <command>interdiff</command> on each pair of files in
	these two directories.  This program is available as <filename
	  role="special">hg-interdiff</filename> in the <filename
	  class="directory">examples</filename> directory of the
	source code repository that accompanies this book. <!--
	&example.hg-interdiff; --></para>

      <para>With the <filename role="special">hg-interdiff</filename>
	program in your shell's search path, you can run it as
	follows, from inside an MQ patch directory:</para>
      <programlisting>hg extdiff -p hg-interdiff -r A:B
	my-change.patch</programlisting>
      <para>Since you'll probably want to use this long-winded command
	a lot, you can get <literal role="hg-ext">hgext</literal> to
	make it available as a normal Mercurial command, again by
	editing your <filename role="special">
	  /.hgrc</filename>.</para>
      <programlisting>[extdiff] cmd.interdiff =
	hg-interdiff</programlisting>
      <para>This directs <literal role="hg-ext">hgext</literal> to
	make an <literal>interdiff</literal> command available, so you
	can now shorten the previous invocation of <command
	  role="hg-ext-extdiff">extdiff</command> to something a
	little more wieldy.</para>
      <programlisting>hg interdiff -r A:B
	my-change.patch</programlisting>

      <note>
	<para>  The <command>interdiff</command> command works well
	  only if the underlying files against which versions of a
	  patch are generated remain the same.  If you create a patch,
	  modify the underlying files, and then regenerate the patch,
	  <command>interdiff</command> may not produce useful
	  output.</para>
      </note>

      <para>The <literal role="hg-ext">extdiff</literal> extension is
	useful for more than merely improving the presentation of MQ
	patches.  To read more about it, go to section <xref
	  linkend="sec:hgext:extdiff"/>.</para>

    </sect2>
  </sect1>
</chapter>

<!--
local variables: 
sgml-parent-document: ("00book.xml" "book" "chapter")
end:
-->