Return ingest byte stats even when 0-valued (#108796)

Change the ingest byte stats to always be returned
whether or not they have a value of 0. Add human readable
form of byte stats. Update docs to reflect changes.

docs/changelog/108796.yaml

@@ -0,0 +1,5 @@
+pr: 108796
+summary: Return ingest byte stats even when 0-valued
+area: Ingest Node
+type: enhancement
+issues: []

docs/reference/cluster/cluster-info.asciidoc

@@ -207,21 +207,33 @@ pipeline.
 (integer)
 Total number of failed operations for the ingest pipeline.
 
+`ingested_as_first_pipeline`::
+(<<byte-units,byte value>>)
+Total ingested size of all documents which were first processed by this pipeline.
+A document is not added to the stat value for this pipeline if it is a final pipeline after a default pipeline, a pipeline
+run after a reroute processor, or is within a pipeline processor.
+Instead, the document size is added to the stat value of the pipeline which initially ingested the document.
+
 `ingested_as_first_pipeline_in_bytes`::
-(Optional, integer)
-Total number of bytes of all documents ingested by the pipeline.
-This field is only present on pipelines which are the first to process a document.
-Thus, it is not present on pipelines which only serve as a final pipeline after a default pipeline, a pipeline run after
-a reroute processor, or pipelines in pipeline processors.
+(integer)
+Total ingested size, in bytes, of all documents which were first processed by this pipeline.
+A document is not added to the stat value for this pipeline if it is a final pipeline after a default pipeline, a pipeline
+run after a reroute processor, or is within a pipeline processor.
+Instead, the document size is added to the stat value of the pipeline which initially ingested the document.
+
+`produced_as_first_pipeline`::
+(<<byte-units,byte value>>)
+Total produced size of all documents which were first processed by this pipeline.
+A document is not added to the stat value for this pipeline if it is a final pipeline after a default pipeline, a pipeline
+run after a reroute processor, or is within a pipeline processor.
+Instead, the document size is added to the stat value of the pipeline which initially ingested the document.
 
 `produced_as_first_pipeline_in_bytes`::
-(Optional, integer)
-Total number of bytes of all documents produced by the pipeline.
-This field is only present on pipelines which are the first to process a document.
-Thus, it is not present on pipelines which only serve as a final pipeline after a default pipeline, a pipeline run after
-a reroute processor, or pipelines in pipeline processors.
-In situations where there are subsequent pipelines, the value represents the size of the document after all pipelines
-have run.
+(integer)
+Total produced size, in bytes, of all documents which were first processed by this pipeline.
+A document is not added to the stat value for this pipeline if it is a final pipeline after a default pipeline, a pipeline
+run after a reroute processor, or is within a pipeline processor.
+Instead, the document size is added to the stat value of the pipeline which initially ingested the document.
 
 `processors`::
 (array of objects)

docs/reference/cluster/nodes-stats.asciidoc

@@ -2643,21 +2643,33 @@ pipeline.
 (integer)
 Total number of failed operations for the ingest pipeline.
 
+`ingested_as_first_pipeline`::
+(<<byte-units,byte value>>)
+Total ingested size of all documents which were first processed by this pipeline.
+A document is not added to the stat value for this pipeline if it is a final pipeline after a default pipeline, a pipeline
+run after a reroute processor, or is within a pipeline processor.
+Instead, the document size is added to the stat value of the pipeline which initially ingested the document.
+
 `ingested_as_first_pipeline_in_bytes`::
-(Optional, integer)
-Total number of bytes of all documents ingested by the pipeline.
-This field is only present on pipelines which are the first to process a document.
-Thus, it is not present on pipelines which only serve as a final pipeline after a default pipeline, a pipeline run after
-a reroute processor, or pipelines in pipeline processors.
+(integer)
+Total ingested size, in bytes, of all documents which were first processed by this pipeline.
+A document is not added to the stat value for this pipeline if it is a final pipeline after a default pipeline, a pipeline
+run after a reroute processor, or is within a pipeline processor.
+Instead, the document size is added to the stat value of the pipeline which initially ingested the document.
+
+`produced_as_first_pipeline`::
+(<<byte-units,byte value>>)
+Total produced size of all documents which were first processed by this pipeline.
+A document is not added to the stat value for this pipeline if it is a final pipeline after a default pipeline, a pipeline
+run after a reroute processor, or is within a pipeline processor.
+Instead, the document size is added to the stat value of the pipeline which initially ingested the document.
 
 `produced_as_first_pipeline_in_bytes`::
-(Optional, integer)
-Total number of bytes of all documents produced by the pipeline.
-This field is only present on pipelines which are the first to process a document.
-Thus, it is not present on pipelines which only serve as a final pipeline after a default pipeline, a pipeline run after
-a reroute processor, or pipelines in pipeline processors.
-In situations where there are subsequent pipelines, the value represents the size of the document after all pipelines
-have run.
+(integer)
+Total produced size, in bytes, of all documents which were first processed by this pipeline.
+A document is not added to the stat value for this pipeline if it is a final pipeline after a default pipeline, a pipeline
+run after a reroute processor, or is within a pipeline processor.
+Instead, the document size is added to the stat value of the pipeline which initially ingested the document.
 
 `processors`::
 (array of objects)

modules/ingest-common/src/yamlRestTest/resources/rest-api-spec/test/ingest/15_info_ingest.yml

@@ -198,7 +198,6 @@ teardown:
   - do:
       bulk:
         refresh: true
-        index: index-foo
         body:
           - '{"index": { "_index": "an-index", "_id": 1 }}'
           - '{"some-field": 1 }'
@@ -214,8 +213,8 @@ teardown:
   - gt: { ingest.pipelines.pipeline-1.ingested_as_first_pipeline_in_bytes: 0 }
   - set: { ingest.pipelines.pipeline-1.ingested_as_first_pipeline_in_bytes: ingest_bytes }
   - gt: { ingest.pipelines.pipeline-1.produced_as_first_pipeline_in_bytes: $ingest_bytes }
-  - match: { ingest.pipelines.pipeline-2.ingested_as_first_pipeline_in_bytes: null }
-  - match: { ingest.pipelines.pipeline-2.produced_as_first_pipeline_in_bytes: null }
+  - match: { ingest.pipelines.pipeline-2.ingested_as_first_pipeline_in_bytes: 0 }
+  - match: { ingest.pipelines.pipeline-2.produced_as_first_pipeline_in_bytes: 0 }
 
 ---
 "Test that final pipeline has byte stats recorded in first pipeline":
@@ -251,7 +250,6 @@ teardown:
   - do:
       bulk:
         refresh: true
-        index: index-foo
         body:
           - '{"index": { "_index": "an-index", "_id": 1 }}'
           - '{"some-field": 1 }'
@@ -267,9 +265,8 @@ teardown:
   - gt: { ingest.pipelines.pipeline-1.ingested_as_first_pipeline_in_bytes: 0 }
   - set: { ingest.pipelines.pipeline-1.ingested_as_first_pipeline_in_bytes: ingest_bytes }
   - gt: { ingest.pipelines.pipeline-1.produced_as_first_pipeline_in_bytes: $ingest_bytes }
-  - match: { ingest.pipelines.pipeline-2.ingested_as_first_pipeline_in_bytes: null }
-  - match: { ingest.pipelines.pipeline-2.produced_as_first_pipeline_in_bytes: null }
-
+  - match: { ingest.pipelines.pipeline-2.ingested_as_first_pipeline_in_bytes: 0 }
+  - match: { ingest.pipelines.pipeline-2.produced_as_first_pipeline_in_bytes: 0 }
 
 ---
 "Test that reroute processor has byte stats recorded in first pipeline":
@@ -333,5 +330,38 @@ teardown:
   - gt: { ingest.pipelines.pipeline-1.ingested_as_first_pipeline_in_bytes: 0 }
   - set: { ingest.pipelines.pipeline-1.ingested_as_first_pipeline_in_bytes: ingest_bytes }
   - gt: { ingest.pipelines.pipeline-1.produced_as_first_pipeline_in_bytes: $ingest_bytes }
-  - match: { ingest.pipelines.pipeline-2.ingested_as_first_pipeline_in_bytes: null }
-  - match: { ingest.pipelines.pipeline-2.produced_as_first_pipeline_in_bytes: null }
+  - match: { ingest.pipelines.pipeline-2.ingested_as_first_pipeline_in_bytes: 0 }
+  - match: { ingest.pipelines.pipeline-2.produced_as_first_pipeline_in_bytes: 0 }
+
+---
+"Test human readable byte stat fields":
+  - do:
+      ingest.put_pipeline:
+        id: "pipeline-1"
+        body:  >
+          {
+            "processors": [
+              {
+                "set": {
+                  "field": "added-field",
+                  "value": true
+                }
+              }
+            ]
+          }
+  - do:
+      bulk:
+        refresh: true
+        body:
+          - '{"index": { "_index": "an-index", "_id": 1, "pipeline": "pipeline-1"}}'
+          - '{"some-field": 1 }'
+  - do:
+      cluster.info:
+        target: [ ingest ]
+        human: true
+
+  - match: { ingest.pipelines.pipeline-1.count: 1 }
+  - gt: { ingest.pipelines.pipeline-1.ingested_as_first_pipeline_in_bytes: 0 }
+  - gt: { ingest.pipelines.pipeline-1.produced_as_first_pipeline_in_bytes: 0 }
+  - is_true: ingest.pipelines.pipeline-1.ingested_as_first_pipeline
+  - is_true: ingest.pipelines.pipeline-1.produced_as_first_pipeline

server/src/main/java/org/elasticsearch/ingest/IngestStats.java

@@ -13,6 +13,7 @@ import org.elasticsearch.common.collect.Iterators;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.common.io.stream.Writeable;
+import org.elasticsearch.common.unit.ByteSizeValue;
 import org.elasticsearch.common.util.Maps;
 import org.elasticsearch.common.xcontent.ChunkedToXContent;
 import org.elasticsearch.core.TimeValue;
@@ -285,10 +286,16 @@ public record IngestStats(Stats totalStats, List<PipelineStat> pipelineStats, Ma
 
         @Override
         public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
-            if (bytesIngested > 0 || bytesProduced > 0) {
-                builder.field("ingested_as_first_pipeline_in_bytes", bytesIngested);
-                builder.field("produced_as_first_pipeline_in_bytes", bytesProduced);
-            }
+            builder.humanReadableField(
+                "ingested_as_first_pipeline_in_bytes",
+                "ingested_as_first_pipeline",
+                ByteSizeValue.ofBytes(bytesIngested)
+            );
+            builder.humanReadableField(
+                "produced_as_first_pipeline_in_bytes",
+                "produced_as_first_pipeline",
+                ByteSizeValue.ofBytes(bytesProduced)
+            );
             return builder;
         }