get-job-stats.asciidoc 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464
  1. [role="xpack"]
  2. [testenv="platinum"]
  3. [[ml-get-job-stats]]
  4. === Get {anomaly-job} statistics API
  5. ++++
  6. <titleabbrev>Get job statistics</titleabbrev>
  7. ++++
  8. Retrieves usage information for {anomaly-jobs}.
  9. [[ml-get-job-stats-request]]
  10. ==== {api-request-title}
  11. `GET _ml/anomaly_detectors/<job_id>/_stats`
  12. `GET _ml/anomaly_detectors/<job_id>,<job_id>/_stats` +
  13. `GET _ml/anomaly_detectors/_stats` +
  14. `GET _ml/anomaly_detectors/_all/_stats`
  15. [[ml-get-job-stats-prereqs]]
  16. ==== {api-prereq-title}
  17. * If the {es} {security-features} are enabled, you must have `monitor_ml`,
  18. `monitor`, `manage_ml`, or `manage` cluster privileges to use this API. See
  19. <<security-privileges>>.
  20. [[ml-get-job-stats-desc]]
  21. ==== {api-description-title}
  22. You can get statistics for multiple {anomaly-jobs} in a single API request by
  23. using a group name, a comma-separated list of jobs, or a wildcard expression.
  24. You can get statistics for all {anomaly-jobs} by using `_all`, by specifying `*`
  25. as the `<job_id>`, or by omitting the `<job_id>`.
  26. IMPORTANT: This API returns a maximum of 10,000 jobs.
  27. [[ml-get-job-stats-path-parms]]
  28. ==== {api-path-parms-title}
  29. `<job_id>`::
  30. (Optional, string)
  31. include::{docdir}/ml/ml-shared.asciidoc[tag=job-id-anomaly-detection-default]
  32. [[ml-get-job-stats-query-parms]]
  33. ==== {api-query-parms-title}
  34. `allow_no_jobs`::
  35. (Optional, boolean)
  36. include::{docdir}/ml/ml-shared.asciidoc[tag=allow-no-jobs]
  37. [role="child_attributes"]
  38. [[ml-get-job-stats-results]]
  39. ==== {api-response-body-title}
  40. The API returns the following information about the operational progress of a
  41. job:
  42. `assignment_explanation`::
  43. (string)
  44. include::{docdir}/ml/ml-shared.asciidoc[tag=assignment-explanation-anomaly-jobs]
  45. //Begin data_counts
  46. [[datacounts]]`data_counts`::
  47. (object) An object that describes the quantity of input to the job and any
  48. related error counts. The `data_count` values are cumulative for the lifetime of
  49. a job. If a model snapshot is reverted or old results are deleted, the job
  50. counts are not reset.
  51. +
  52. .Properties of `data_counts`
  53. [%collapsible%open]
  54. ====
  55. `bucket_count`:::
  56. (long)
  57. include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-count-anomaly-jobs]
  58. `earliest_record_timestamp`:::
  59. (date)
  60. include::{docdir}/ml/ml-shared.asciidoc[tag=earliest-record-timestamp]
  61. `empty_bucket_count`:::
  62. (long)
  63. include::{docdir}/ml/ml-shared.asciidoc[tag=empty-bucket-count]
  64. `input_bytes`:::
  65. (long)
  66. include::{docdir}/ml/ml-shared.asciidoc[tag=input-bytes]
  67. `input_field_count`:::
  68. (long)
  69. include::{docdir}/ml/ml-shared.asciidoc[tag=input-field-count]
  70. `input_record_count`:::
  71. (long)
  72. include::{docdir}/ml/ml-shared.asciidoc[tag=input-record-count]
  73. `invalid_date_count`:::
  74. (long)
  75. include::{docdir}/ml/ml-shared.asciidoc[tag=invalid-date-count]
  76. `job_id`:::
  77. (string)
  78. include::{docdir}/ml/ml-shared.asciidoc[tag=job-id-anomaly-detection]
  79. `last_data_time`:::
  80. (date)
  81. include::{docdir}/ml/ml-shared.asciidoc[tag=last-data-time]
  82. `latest_empty_bucket_timestamp`:::
  83. (date)
  84. include::{docdir}/ml/ml-shared.asciidoc[tag=latest-empty-bucket-timestamp]
  85. `latest_record_timestamp`:::
  86. (date)
  87. include::{docdir}/ml/ml-shared.asciidoc[tag=latest-record-timestamp]
  88. `latest_sparse_bucket_timestamp`:::
  89. (date)
  90. include::{docdir}/ml/ml-shared.asciidoc[tag=latest-sparse-record-timestamp]
  91. `missing_field_count`:::
  92. (long)
  93. include::{docdir}/ml/ml-shared.asciidoc[tag=missing-field-count]
  94. +
  95. The value of `processed_record_count` includes this count.
  96. `out_of_order_timestamp_count`:::
  97. (long)
  98. include::{docdir}/ml/ml-shared.asciidoc[tag=out-of-order-timestamp-count]
  99. `processed_field_count`:::
  100. include::{docdir}/ml/ml-shared.asciidoc[tag=processed-field-count]
  101. `processed_record_count`:::
  102. (long)
  103. include::{docdir}/ml/ml-shared.asciidoc[tag=processed-record-count]
  104. `sparse_bucket_count`:::
  105. (long)
  106. include::{docdir}/ml/ml-shared.asciidoc[tag=sparse-bucket-count]
  107. ====
  108. //End data_counts
  109. `deleting`::
  110. (boolean)
  111. Indicates that the process of deleting the job is in progress but not yet
  112. completed. It is only reported when `true`.
  113. //Begin forecasts_stats
  114. [[forecastsstats]]`forecasts_stats`::
  115. (object) An object that provides statistical information about forecasts
  116. belonging to this job. Some statistics are omitted if no forecasts have been
  117. made.
  118. +
  119. NOTE: Unless there is at least one forecast, `memory_bytes`, `records`,
  120. `processing_time_ms` and `status` properties are omitted.
  121. +
  122. .Properties of `forecasts_stats`
  123. [%collapsible%open]
  124. ====
  125. `forecasted_jobs`:::
  126. (long) A value of `0` indicates that forecasts do not exist for this job. A
  127. value of `1` indicates that at least one forecast exists.
  128. `memory_bytes`:::
  129. (object) The `avg`, `min`, `max` and `total` memory usage in bytes for forecasts
  130. related to this job. If there are no forecasts, this property is omitted.
  131. `records`:::
  132. (object) The `avg`, `min`, `max` and `total` number of `model_forecast` documents
  133. written for forecasts related to this job. If there are no forecasts, this
  134. property is omitted.
  135. `processing_time_ms`:::
  136. (object) The `avg`, `min`, `max` and `total` runtime in milliseconds for
  137. forecasts related to this job. If there are no forecasts, this property is omitted.
  138. `status`:::
  139. (object) The count of forecasts by their status. For example:
  140. {"finished" : 2, "started" : 1}. If there are no forecasts, this property is omitted.
  141. `total`:::
  142. (long)
  143. include::{docdir}/ml/ml-shared.asciidoc[tag=forecast-total]
  144. ====
  145. //End forecasts_stats
  146. `job_id`::
  147. (string)
  148. include::{docdir}/ml/ml-shared.asciidoc[tag=job-id-anomaly-detection]
  149. //Begin model_size_stats
  150. [[modelsizestats]]`model_size_stats`::
  151. (object) An object that provides information about the size and contents of the
  152. model.
  153. +
  154. .Properties of `model_size_stats`
  155. [%collapsible%open]
  156. ====
  157. `bucket_allocation_failures_count`:::
  158. (long)
  159. include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-allocation-failures-count]
  160. `categorized_doc_count`:::
  161. (long)
  162. include::{docdir}/ml/ml-shared.asciidoc[tag=categorized-doc-count]
  163. `categorization_status`:::
  164. (string)
  165. include::{docdir}/ml/ml-shared.asciidoc[tag=categorization-status]
  166. `dead_category_count`:::
  167. (long)
  168. include::{docdir}/ml/ml-shared.asciidoc[tag=dead-category-count]
  169. `frequent_category_count`:::
  170. (long)
  171. include::{docdir}/ml/ml-shared.asciidoc[tag=frequent-category-count]
  172. `job_id`:::
  173. (string)
  174. include::{docdir}/ml/ml-shared.asciidoc[tag=job-id-anomaly-detection]
  175. `log_time`:::
  176. (date) The timestamp of the `model_size_stats` according to server time.
  177. `memory_status`:::
  178. (string)
  179. include::{docdir}/ml/ml-shared.asciidoc[tag=model-memory-status]
  180. `model_bytes`:::
  181. (long)
  182. include::{docdir}/ml/ml-shared.asciidoc[tag=model-bytes]
  183. `model_bytes_exceeded`:::
  184. (long)
  185. include::{docdir}/ml/ml-shared.asciidoc[tag=model-bytes-exceeded]
  186. `model_bytes_memory_limit`:::
  187. (long)
  188. include::{docdir}/ml/ml-shared.asciidoc[tag=model-memory-limit-anomaly-jobs]
  189. `rare_category_count`:::
  190. (long)
  191. include::{docdir}/ml/ml-shared.asciidoc[tag=rare-category-count]
  192. `result_type`:::
  193. (string) For internal use. The type of result.
  194. `total_by_field_count`:::
  195. (long)
  196. include::{docdir}/ml/ml-shared.asciidoc[tag=total-by-field-count]
  197. `total_category_count`:::
  198. (long)
  199. include::{docdir}/ml/ml-shared.asciidoc[tag=total-category-count]
  200. `total_over_field_count`:::
  201. (long)
  202. include::{docdir}/ml/ml-shared.asciidoc[tag=total-over-field-count]
  203. `total_partition_field_count`:::
  204. (long)
  205. include::{docdir}/ml/ml-shared.asciidoc[tag=total-partition-field-count]
  206. `timestamp`:::
  207. (date)
  208. include::{docdir}/ml/ml-shared.asciidoc[tag=model-timestamp]
  209. ====
  210. //End model_size_stats
  211. //Begin node
  212. [[stats-node]]`node`::
  213. (object) Contains properties for the node that runs the job. This information is
  214. available only for open jobs.
  215. +
  216. .Properties of `node`
  217. [%collapsible%open]
  218. ====
  219. `attributes`:::
  220. (object) Lists node attributes. For example,
  221. `{"ml.machine_memory": "17179869184", "ml.max_open_jobs" : "20"}`.
  222. `ephemeral_id`:::
  223. (string)
  224. include::{docdir}/ml/ml-shared.asciidoc[tag=node-ephemeral-id]
  225. `id`:::
  226. (string)
  227. include::{docdir}/ml/ml-shared.asciidoc[tag=node-id]
  228. `name`:::
  229. (string) The node name.
  230. `transport_address`:::
  231. (string) The host and port where transport HTTP connections are accepted.
  232. ====
  233. //End node
  234. `open_time`::
  235. (string)
  236. include::{docdir}/ml/ml-shared.asciidoc[tag=open-time]
  237. `state`::
  238. (string)
  239. include::{docdir}/ml/ml-shared.asciidoc[tag=state-anomaly-job]
  240. //Begin timing_stats
  241. [[timingstats]]`timing_stats`::
  242. (object) An object that provides statistical information about timing aspect of
  243. this job.
  244. +
  245. .Properties of `timing_stats`
  246. [%collapsible%open]
  247. ====
  248. `average_bucket_processing_time_ms`:::
  249. (double) Average of all bucket processing times in milliseconds.
  250. `bucket_count`:::
  251. (long)
  252. include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-count]
  253. `exponential_average_bucket_processing_time_ms`:::
  254. (double)
  255. include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-time-exponential-average]
  256. `exponential_average_bucket_processing_time_per_hour_ms`:::
  257. (double)
  258. include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-time-exponential-average-hour]
  259. `job_id`:::
  260. (string)
  261. include::{docdir}/ml/ml-shared.asciidoc[tag=job-id-anomaly-detection]
  262. `maximum_bucket_processing_time_ms`:::
  263. (double)
  264. include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-time-maximum]
  265. `minimum_bucket_processing_time_ms`:::
  266. (double)
  267. include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-time-minimum]
  268. `total_bucket_processing_time_ms`:::
  269. (double)
  270. include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-time-total]
  271. ====
  272. //End timing_stats
  273. [[ml-get-job-stats-response-codes]]
  274. ==== {api-response-codes-title}
  275. `404` (Missing resources)::
  276. If `allow_no_jobs` is `false`, this code indicates that there are no
  277. resources that match the request or only partial matches for the request.
  278. [[ml-get-job-stats-example]]
  279. ==== {api-examples-title}
  280. [source,console]
  281. --------------------------------------------------
  282. GET _ml/anomaly_detectors/low_request_rate/_stats
  283. --------------------------------------------------
  284. // TEST[skip:Kibana sample data]
  285. The API returns the following results:
  286. [source,js]
  287. ----
  288. {
  289. "count" : 1,
  290. "jobs" : [
  291. {
  292. "job_id" : "low_request_rate",
  293. "data_counts" : {
  294. "job_id" : "low_request_rate",
  295. "processed_record_count" : 1216,
  296. "processed_field_count" : 1216,
  297. "input_bytes" : 51678,
  298. "input_field_count" : 1216,
  299. "invalid_date_count" : 0,
  300. "missing_field_count" : 0,
  301. "out_of_order_timestamp_count" : 0,
  302. "empty_bucket_count" : 242,
  303. "sparse_bucket_count" : 0,
  304. "bucket_count" : 1457,
  305. "earliest_record_timestamp" : 1575172659612,
  306. "latest_record_timestamp" : 1580417369440,
  307. "last_data_time" : 1576017595046,
  308. "latest_empty_bucket_timestamp" : 1580356800000,
  309. "input_record_count" : 1216
  310. },
  311. "model_size_stats" : {
  312. "job_id" : "low_request_rate",
  313. "result_type" : "model_size_stats",
  314. "model_bytes" : 41480,
  315. "model_bytes_exceeded" : 0,
  316. "model_bytes_memory_limit" : 10485760,
  317. "total_by_field_count" : 3,
  318. "total_over_field_count" : 0,
  319. "total_partition_field_count" : 2,
  320. "bucket_allocation_failures_count" : 0,
  321. "memory_status" : "ok",
  322. "categorized_doc_count" : 0,
  323. "total_category_count" : 0,
  324. "frequent_category_count" : 0,
  325. "rare_category_count" : 0,
  326. "dead_category_count" : 0,
  327. "categorization_status" : "ok",
  328. "log_time" : 1576017596000,
  329. "timestamp" : 1580410800000
  330. },
  331. "forecasts_stats" : {
  332. "total" : 1,
  333. "forecasted_jobs" : 1,
  334. "memory_bytes" : {
  335. "total" : 9179.0,
  336. "min" : 9179.0,
  337. "avg" : 9179.0,
  338. "max" : 9179.0
  339. },
  340. "records" : {
  341. "total" : 168.0,
  342. "min" : 168.0,
  343. "avg" : 168.0,
  344. "max" : 168.0
  345. },
  346. "processing_time_ms" : {
  347. "total" : 40.0,
  348. "min" : 40.0,
  349. "avg" : 40.0,
  350. "max" : 40.0
  351. },
  352. "status" : {
  353. "finished" : 1
  354. }
  355. },
  356. "state" : "opened",
  357. "node" : {
  358. "id" : "7bmMXyWCRs-TuPfGJJ_yMw",
  359. "name" : "node-0",
  360. "ephemeral_id" : "hoXMLZB0RWKfR9UPPUCxXX",
  361. "transport_address" : "127.0.0.1:9300",
  362. "attributes" : {
  363. "ml.machine_memory" : "17179869184",
  364. "xpack.installed" : "true",
  365. "ml.max_open_jobs" : "20"
  366. }
  367. },
  368. "assignment_explanation" : "",
  369. "open_time" : "13s",
  370. "timing_stats" : {
  371. "job_id" : "low_request_rate",
  372. "bucket_count" : 1457,
  373. "total_bucket_processing_time_ms" : 1094.000000000001,
  374. "minimum_bucket_processing_time_ms" : 0.0,
  375. "maximum_bucket_processing_time_ms" : 48.0,
  376. "average_bucket_processing_time_ms" : 0.75085792724777,
  377. "exponential_average_bucket_processing_time_ms" : 0.5571716855800993,
  378. "exponential_average_bucket_processing_time_per_hour_ms" : 15.0
  379. }
  380. }
  381. ]
  382. }
  383. ----