Update transport version docs (#133862)

This commit updates versioning docs to explain how to use the new transport version system.

Co-authored-by: Jack Conradson <osjdconrad@gmail.com>
Co-authored-by: Ryan Ernst <ryan@iernst.net>

docs/internal/Versioning.md

@@ -22,68 +22,126 @@ it could be any arbitrary string.
 
 ## Transport protocol
 
-The transport protocol is used to send binary data between Elasticsearch nodes;
-`TransportVersion` is the version number used for this protocol.
-This version number is negotiated between each pair of nodes in the cluster
-on first connection, and is set as the lower of the highest transport version
-understood by each node.
+The transport protocol is used to send binary data between Elasticsearch nodes; a
+`TransportVersion` encapsulates versioning of this protocol.
+This version is negotiated between each pair of nodes in the cluster
+on first connection, selecting the highest shared version.
 This version is then accessible through the `getTransportVersion` method
 on `StreamInput` and `StreamOutput`, so serialization code can read/write
 objects in a form that will be understood by the other node.
 
-Every change to the transport protocol is represented by a new transport version,
-higher than all previous transport versions, which then becomes the highest version
-recognized by that build of Elasticsearch. The version ids are stored
-as constants in the `TransportVersions` class.
-Each id has a standard pattern `M_NNN_S_PP`, where:
-* `M` is the major version
-* `NNN` is an incrementing id
-* `S` is used in subsidiary repos amending the default transport protocol
-* `PP` is used for patches and backports
-
-When you make a change to the serialization form of any object,
-you need to create a new sequential constant in `TransportVersions`,
-introduced in the same PR that adds the change, that increments
-the `NNN` component from the previous highest version,
-with other components  set to zero.
-For example, if the previous version number is `8_413_0_01`,
-the next version number should be `8_414_0_00`.
-
-Once you have defined your constant, you then need to use it
-in serialization code. If the transport version is at or above the new id,
-the modified protocol should be used:
-
-    str = in.readString();
-    bool = in.readBoolean();
-    if (in.getTransportVersion().onOrAfter(TransportVersions.NEW_CONSTANT)) {
-        num = in.readVInt();
-    }
+At a high level a `TransportVersion` contains one id per release branch it will
+be committed to. Each `TransportVersion` has a name selected when it is generated.
+In order to ensure consistency and robustness, all new `TransportVersion`s
+must first be created in the `main` branch and then backported to the relevant
+release branches.
+
+### Internal state files
+
+The Elasticsearch server jar contains resource files representing each
+transport version. These files are loaded at runtime to construct
+`TransportVersion` instances. Since each transport version has its own file
+they can be backported without conflict.
 
-If a transport version change needs to be reverted, a **new** version constant
-should be added representing the revert, and the version id checks
-adjusted appropriately to only use the modified protocol between the version id
-the change was added, and the new version id used for the revert (exclusive).
-The `between` method can be used for this.
+Additional resource files represent the latest transport version known on
+each release branch. If two transport versions are added at the same time,
+there will be a conflict in these internal state files, forcing one to be
+regenerated to resolve the conflict before merging to `main`.
 
-Once a transport change with a new version has been merged into main or a release branch,
-it **must not** be modified - this is so the meaning of that specific
-transport version does not change.
+All of these internal state files are managed by gradle tasks; they should
+not be edited directly.
 
 _Elastic developers_ - please see corresponding documentation for Serverless
 on creating transport versions for Serverless changes.
 
-### Collapsing transport versions
+### Creating transport versions locally
+
+To create a transport version, declare a reference anywhere in java code. For example:
+
+    private static final TransportVersion MY_NEW_TV = TransportVersion.fromName("my_new_tv");
+
+`fromName` takes an arbitrary String name. The String must be a String literal;
+it cannot be a reference to a String. It must match the regex `[_0-9a-zA-Z]+`.
+You can reference the same transport version name from multiple classes, but
+you must not use an existing transport version name after it as already been
+committed to `main`.
+
+Once you have declared your `TransportVersion` you can use it in serialization code.
+For example, in a constructor that takes `StreamInput in`:
+
+    if (in.getTransportVersion().supports(MY_NEW_TV)) {
+        // new serialization code
+    }
+
+Finally, in order to run Elasticsearch or run tests, the transport version ids
+must be generated. Run the following gradle task:
+
+    ./gradlew generateTransportVersion
+
+This will generate the internal state to support the new transport version. If
+you also intend to backport your code, include branches you will backport to:
+
+    ./gradlew generateTransportVersion --backport-branches=9.1,8.19
+
+### Updating transport versions
+
+You can modify a transport version before it is merged to `main`. This includes
+renaming the transport version, updating the branches it will be backported to,
+or even removing the transport version itself.
+
+The generation task is idempotent. It can be re-run at any time and will result
+in a valid internal state. For example, if you want to add an additional
+backport branch, re-run the generation task with all the target backport
+branches:
+
+    ./gradlew generateTransportVersion --backport-branches=9.1,9.0,8.19,8.18
+
+You can also let CI handle updating transport versions for you. As version
+labels are updated on your PR, the generation task is automatically run with
+the appropriate backport branches and any changes to the internal state files
+are committed to your branch.
+
+Transport versions can also have additional branches added after merging to
+`main`. When doing so, you must include all branches the transport version was
+added to in addition to new branch. For example, if you originally committed
+your transport version `my_tv` to `main` and `9.1`, and then realized you also
+needed to backport to `8.19` you would run (in `main`):
+
+    ./gradlew generateTransportVersion --name=my_tv --backport-branches=9.1,8.19
+
+In the above case CI will not know what transport version name to update, so you
+must run the generate task again as described. After merging the updated transport
+version it will need to be backported to all the applicable branches.
+
+### Resolving merge conflicts
 
-As each change adds a new constant, the list of constants in `TransportVersions`
-will keep growing. However, once there has been an official release of Elasticsearch,
-that includes that change, that specific transport version is no longer needed,
-apart from constants that happen to be used for release builds.
-As part of managing transport versions, consecutive transport versions can be
-periodically collapsed together into those that are only used for release builds.
-This task is normally performed by Core/Infra on a semi-regular basis,
-usually after each new minor release, to collapse the transport versions
-for the previous minor release. An example of such an operation can be found
-[here](https://github.com/elastic/elasticsearch/pull/104937).
+Transport versions are created sequentially. If two developers create a transport
+version at the same time, based on the same `main` commit, they will generate
+the same internal ids. The first of these two merged into `main` will "win", and
+the latter will have a merge conflict with `main`.
+
+In the event of a conflict, merge `main` into your branch. You will have
+conflict(s) with transport version internal state files. Run the following
+generate task to resolve the conflict(s):
+
+    ./gradlew generateTransportVersion --resolve-conflict
+
+This command will regenerate your transport version and stage the updated
+state files in git. You can then proceed with your merge as usual.
+
+### Reverting changes
+
+Transport versions cannot be removed, they can only be added. If the logic
+using a transport version needs to be reverted, it must be done with a
+new transport version.
+
+For example, if you have previously added a transport version named
+`original_tv` you could add `revert_tv` reversing the logic:
+
+    TransportVersion tv = in.getTransportVersion();
+    if (tv.supports(ORIGINAL_TV) && tv.supports(REVERT_TV) == false) {
+        // serialization code being reverted
+    }
 
 ### Minimum compatibility versions
 
@@ -114,54 +172,6 @@ is updated automatically as part of performing a release.
 In releases that do not have a release version number, that method becomes
 a no-op.
 
-### Managing patches and backports
-
-Backporting transport version changes to previous releases
-should only be done if absolutely necessary, as it is very easy to get wrong
-and break the release in a way that is very hard to recover from.
-
-If we consider the version number as an incrementing line, what we are doing is
-grafting a change that takes effect at a certain point in the line,
-to additionally take effect in a fixed window earlier in the line.
-
-To take an example, using indicative version numbers, when the latest
-transport version is 52, we decide we need to backport a change done in
-transport version 50 to transport version 45. We use the `P` version id component
-to create version 45.1 with the backported change.
-This change will apply for version ids 45.1 to 45.9 (should they exist in the future).
-
-The serialization code in the backport needs to use the backported protocol
-for all version numbers 45.1 to 45.9. The `TransportVersion.isPatchFrom` method
-can be used to easily determine if this is the case: `streamVersion.isPatchFrom(45.1)`.
-However, the `onOrAfter` also does what is needed on patch branches.
-
-The serialization code in version 53 then needs to additionally check
-version numbers 45.1-45.9 to use the backported protocol, also using the `isPatchFrom` method.
-
-As an example, [this transport change](https://github.com/elastic/elasticsearch/pull/107862)
-was backported from 8.15 to [8.14.0](https://github.com/elastic/elasticsearch/pull/108251)
-and [8.13.4](https://github.com/elastic/elasticsearch/pull/108250) at the same time
-(8.14 was a build candidate at the time).
-
-The 8.13 PR has:
-
-    if (transportVersion.onOrAfter(8.13_backport_id))
-
-The 8.14 PR has:
-
-    if (transportVersion.isPatchFrom(8.13_backport_id)
-        || transportVersion.onOrAfter(8.14_backport_id))
-
-The 8.15 PR has:
-
-    if (transportVersion.isPatchFrom(8.13_backport_id)
-        || transportVersion.isPatchFrom(8.14_backport_id)
-        || transportVersion.onOrAfter(8.15_transport_id))
-
-In particular, if you are backporting a change to a patch release,
-you also need to make sure that any subsequent released version on any branch
-also has that change, and knows about the patch backport ids and what they mean.
-
 ## Index version
 
 Index version is a single incrementing version number for the index data format,