|
|
@@ -170,6 +170,46 @@ shard.
|
|
|
|
|
|
== Completed
|
|
|
|
|
|
+[float]
|
|
|
+=== Divergence between primary and replica shard copies when documents deleted (STATUS: DONE, V6.3.0)
|
|
|
+
|
|
|
+Certain combinations of delays in performing activities related to the deletion
|
|
|
+of a document could result in the operations on that document being interpreted
|
|
|
+differently on different shard copies. This could lead to a divergence in the
|
|
|
+number of documents held in each copy.
|
|
|
+
|
|
|
+Deleting an unacknowledged document that was concurrently being inserted using
|
|
|
+an auto-generated ID was erroneously sensitive to the order in which those
|
|
|
+operations were processed on each shard copy. Thanks to the introduction of
|
|
|
+sequence numbers ({GIT}10708[#10708]) it is now possible to detect these
|
|
|
+out-of-order operations, and this issue was fixed in {GIT}28787[#28787].
|
|
|
+
|
|
|
+Re-creating a document a specific interval after it was deleted could result in
|
|
|
+that document's tombstone having being cleaned up on some, but not all, copies
|
|
|
+when processing the indexing operation that re-creates it. This resulted in
|
|
|
+varying behaviour across the shard copies. The problematic interval was set by
|
|
|
+the `index.gc_deletes` setting, which is 60 seconds by default. Again, sequence
|
|
|
+numbers ({GIT}10708[#10708]) gives us the machinery to detect these conflicting
|
|
|
+activities, and this issue was fixed in {GIT}28790[#28790].
|
|
|
+
|
|
|
+Under certain rare circumstances a replica might erroneously interpret a stale
|
|
|
+tombstone for a document as fresh, resulting in a concurrent indexing operation
|
|
|
+for that same document behaving differently on this replica than on the
|
|
|
+primary. This is fixed in {GIT}29619[#29619]. Triggering this issue required
|
|
|
+the following activities all to occur in a short time window, in a specific
|
|
|
+order on the primary and a different specific order on the replica:
|
|
|
+
|
|
|
+* a document is deleted twice
|
|
|
+* another document is indexed with the same ID as this first document
|
|
|
+* another document is indexed with a completely different, auto-generated, ID
|
|
|
+* two refreshes
|
|
|
+
|
|
|
+We found the first two of these issues by empirical testing, and then we built
|
|
|
+https://github.com/elastic/elasticsearch-formal-models/blob/master/ReplicaEngine/tla/ReplicaEngine.tla[a
|
|
|
+formal model of the replica's behaviour] using TLA+. Running the TLC model
|
|
|
+checker on this model found all three issues. We then applied the proposed
|
|
|
+fixes to the model and validated that the fixed design behaved as expected.
|
|
|
+
|
|
|
[float]
|
|
|
=== Port Jepsen tests dealing with loss of acknowledged writes to our testing framework (STATUS: DONE, V5.0.0)
|
|
|
|
David Turner