index_.asciidoc 21 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603
  1. [[docs-index_]]
  2. === Index API
  3. ++++
  4. <titleabbrev>Index</titleabbrev>
  5. ++++
  6. IMPORTANT: See <<removal-of-types>>.
  7. Adds a JSON document to the specified data stream or index and makes
  8. it searchable. If the target is an index and the document already exists,
  9. the request updates the document and increments its version.
  10. NOTE: You cannot use the index API to send update requests for existing
  11. documents to a data stream. See <<update-docs-in-a-data-stream-by-query>>
  12. and <<update-delete-docs-in-a-backing-index>>.
  13. [[docs-index-api-request]]
  14. ==== {api-request-title}
  15. `PUT /<target>/_doc/<_id>`
  16. `POST /<target>/_doc/`
  17. `PUT /<target>/_create/<_id>`
  18. `POST /<target>/_create/<_id>`
  19. IMPORTANT: You cannot add new documents to a data stream using the
  20. `PUT /<target>/_doc/<_id>` request format. To specify a document ID, use the
  21. `PUT /<target>/_create/<_id>` format instead. See
  22. <<add-documents-to-a-data-stream>>.
  23. [[docs-index-api-prereqs]]
  24. ==== {api-prereq-title}
  25. * If the {es} {security-features} are enabled, you must have the following
  26. <<privileges-list-indices,index privileges>> for the target data stream, index,
  27. or index alias:
  28. ** To add or overwrite a document using the `PUT /<target>/_doc/<_id>` request
  29. format, you must have the `create`, `index`, or `write` index privilege.
  30. ** To add a document using the `POST /<target>/_doc/`,
  31. `PUT /<target>/_create/<_id>`, or `POST /<target>/_create/<_id>` request
  32. formats, you must have the `create_doc`, `create`, `index`, or `write` index
  33. privilege.
  34. ** To automatically create a data stream or index with an index API request, you
  35. must have the `auto_configure`, `create_index`, or `manage` index privilege.
  36. * Automatic data stream creation requires a matching index template with data
  37. stream enabled. See <<set-up-a-data-stream>>.
  38. [[docs-index-api-path-params]]
  39. ==== {api-path-parms-title}
  40. `<target>`::
  41. (Required, string) Name of the data stream or index to target.
  42. +
  43. If the target doesn't exist and matches the name or wildcard (`*`) pattern of an
  44. <<create-index-template,index template with a `data_stream`
  45. definition>>, this request creates the data stream. See
  46. <<set-up-a-data-stream>>.
  47. +
  48. If the target doesn't exist and doesn't match a data stream template,
  49. this request creates the index.
  50. +
  51. You can check for existing targets using the resolve index API.
  52. `<_id>`::
  53. (Optional, string) Unique identifier for the document.
  54. +
  55. --
  56. This parameter is required for the following request formats:
  57. * `PUT /<target>/_doc/<_id>`
  58. * `PUT /<target>/_create/<_id>`
  59. * `POST /<target>/_create/<_id>`
  60. To automatically generate a document ID, use the `POST /<target>/_doc/` request
  61. format and omit this parameter.
  62. --
  63. [[docs-index-api-query-params]]
  64. ==== {api-query-parms-title}
  65. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=if_seq_no]
  66. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=if_primary_term]
  67. [[docs-index-api-op_type]]
  68. `op_type`::
  69. (Optional, enum) Set to `create` to only index the document
  70. if it does not already exist (_put if absent_). If a document with the specified
  71. `_id` already exists, the indexing operation will fail. Same as using the
  72. `<index>/_create` endpoint. Valid values: `index`, `create`.
  73. If document id is specified, it defaults to `index`. Otherwise, it defaults to `create`.
  74. +
  75. NOTE: If the request targets a data stream, an `op_type` of `create` is
  76. required. See <<add-documents-to-a-data-stream>>.
  77. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pipeline]
  78. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=refresh]
  79. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=routing]
  80. `timeout`::
  81. +
  82. --
  83. (Optional, <<time-units, time units>>)
  84. Period the request waits for the following operations:
  85. * <<index-creation,Automatic index creation>>
  86. * <<dynamic-mapping,Dynamic mapping>> updates
  87. * <<index-wait-for-active-shards,Waiting for active shards>>
  88. Defaults to `1m` (one minute). This guarantees {es} waits for at least the
  89. timeout before failing. The actual wait time could be longer, particularly when
  90. multiple waits occur.
  91. --
  92. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=doc-version]
  93. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=version_type]
  94. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards]
  95. include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=require-alias]
  96. [[docs-index-api-request-body]]
  97. ==== {api-request-body-title}
  98. `<field>`::
  99. (Required, string) Request body contains the JSON source for the document
  100. data.
  101. [[docs-index-api-response-body]]
  102. ==== {api-response-body-title}
  103. `_shards`::
  104. Provides information about the replication process of the index operation.
  105. `_shards.total`::
  106. Indicates how many shard copies (primary and replica shards) the index operation
  107. should be executed on.
  108. `_shards.successful`::
  109. Indicates the number of shard copies the index operation succeeded on.
  110. When the index operation is successful, `successful` is at least 1.
  111. +
  112. NOTE: Replica shards might not all be started when an indexing operation
  113. returns successfully--by default, only the primary is required. Set
  114. `wait_for_active_shards` to change this default behavior. See
  115. <<index-wait-for-active-shards>>.
  116. `_shards.failed`::
  117. An array that contains replication-related errors in the case an index operation
  118. failed on a replica shard. 0 indicates there were no failures.
  119. `_index`::
  120. The name of the index the document was added to.
  121. `_type`::
  122. The document type. {es} indices now support a single document type, `_doc`.
  123. `_id`::
  124. The unique identifier for the added document.
  125. `_version`::
  126. The document version. Incremented each time the document is updated.
  127. `_seq_no`::
  128. The sequence number assigned to the document for the indexing operation.
  129. Sequence numbers are used to ensure an older version of a document
  130. doesn’t overwrite a newer version. See <<optimistic-concurrency-control-index>>.
  131. `_primary_term`::
  132. The primary term assigned to the document for the indexing operation.
  133. See <<optimistic-concurrency-control-index>>.
  134. `result`::
  135. The result of the indexing operation, `created` or `updated`.
  136. [[docs-index-api-desc]]
  137. ==== {api-description-title}
  138. You can index a new JSON document with the `_doc` or `_create` resource. Using
  139. `_create` guarantees that the document is only indexed if it does not already
  140. exist. To update an existing document, you must use the `_doc` resource.
  141. [[index-creation]]
  142. ===== Automatically create data streams and indices
  143. If request's target doesn't exist and matches an
  144. <<create-index-template,index template with a `data_stream`
  145. definition>>, the index operation automatically creates the data stream. See
  146. <<set-up-a-data-stream>>.
  147. If the target doesn't exist and doesn't match a data stream template,
  148. the operation automatically creates the index and applies any matching
  149. <<index-templates,index templates>>.
  150. NOTE: {es} includes several built-in index templates. To avoid naming collisions
  151. with these templates, see <<avoid-index-pattern-collisions>>.
  152. If no mapping exists, the index operation
  153. creates a dynamic mapping. By default, new fields and objects are
  154. automatically added to the mapping if needed. For more information about field
  155. mapping, see <<mapping,mapping>> and the <<indices-put-mapping,update mapping>> API.
  156. Automatic index creation is controlled by the `action.auto_create_index`
  157. setting. This setting defaults to `true`, which allows any index to be created
  158. automatically. You can modify this setting to explicitly allow or block
  159. automatic creation of indices that match specified patterns, or set it to
  160. `false` to disable automatic index creation entirely. Specify a
  161. comma-separated list of patterns you want to allow, or prefix each pattern with
  162. `+` or `-` to indicate whether it should be allowed or blocked. When a list is
  163. specified, the default behaviour is to disallow.
  164. IMPORTANT: The `action.auto_create_index` setting only affects the automatic
  165. creation of indices. It does not affect the creation of data streams.
  166. [source,console]
  167. --------------------------------------------------
  168. PUT _cluster/settings
  169. {
  170. "persistent": {
  171. "action.auto_create_index": "my-index-000001,index10,-index1*,+ind*" <1>
  172. }
  173. }
  174. PUT _cluster/settings
  175. {
  176. "persistent": {
  177. "action.auto_create_index": "false" <2>
  178. }
  179. }
  180. PUT _cluster/settings
  181. {
  182. "persistent": {
  183. "action.auto_create_index": "true" <3>
  184. }
  185. }
  186. --------------------------------------------------
  187. <1> Allow auto-creation of indices called `my-index-000001` or `index10`, block the
  188. creation of indices that match the pattern `index1*`, and allow creation of
  189. any other indices that match the `ind*` pattern. Patterns are matched in
  190. the order specified.
  191. <2> Disable automatic index creation entirely.
  192. <3> Allow automatic creation of any index. This is the default.
  193. [discrete]
  194. [[operation-type]]
  195. ===== Put if absent
  196. You can force a create operation by using the `_create` resource or
  197. setting the `op_type` parameter to _create_. In this case,
  198. the index operation fails if a document with the specified ID
  199. already exists in the index.
  200. [discrete]
  201. [[create-document-ids-automatically]]
  202. ===== Create document IDs automatically
  203. When using the `POST /<target>/_doc/` request format, the `op_type` is
  204. automatically set to `create` and the index operation generates a unique ID for
  205. the document.
  206. [source,console]
  207. --------------------------------------------------
  208. POST my-index-000001/_doc/
  209. {
  210. "@timestamp": "2099-11-15T13:12:00",
  211. "message": "GET /search HTTP/1.1 200 1070000",
  212. "user": {
  213. "id": "kimchy"
  214. }
  215. }
  216. --------------------------------------------------
  217. The API returns the following result:
  218. [source,console-result]
  219. --------------------------------------------------
  220. {
  221. "_shards": {
  222. "total": 2,
  223. "failed": 0,
  224. "successful": 2
  225. },
  226. "_index": "my-index-000001",
  227. "_id": "W0tpsmIBdwcYyG50zbta",
  228. "_version": 1,
  229. "_seq_no": 0,
  230. "_primary_term": 1,
  231. "result": "created"
  232. }
  233. --------------------------------------------------
  234. // TESTRESPONSE[s/W0tpsmIBdwcYyG50zbta/$body._id/ s/"successful": 2/"successful": 1/]
  235. [discrete]
  236. [[optimistic-concurrency-control-index]]
  237. ===== Optimistic concurrency control
  238. Index operations can be made conditional and only be performed if the last
  239. modification to the document was assigned the sequence number and primary
  240. term specified by the `if_seq_no` and `if_primary_term` parameters. If a
  241. mismatch is detected, the operation will result in a `VersionConflictException`
  242. and a status code of 409. See <<optimistic-concurrency-control>> for more details.
  243. [discrete]
  244. [[index-routing]]
  245. ===== Routing
  246. By default, shard placement -- or `routing` -- is controlled by using a
  247. hash of the document's id value. For more explicit control, the value
  248. fed into the hash function used by the router can be directly specified
  249. on a per-operation basis using the `routing` parameter. For example:
  250. [source,console]
  251. --------------------------------------------------
  252. POST my-index-000001/_doc?routing=kimchy
  253. {
  254. "@timestamp": "2099-11-15T13:12:00",
  255. "message": "GET /search HTTP/1.1 200 1070000",
  256. "user": {
  257. "id": "kimchy"
  258. }
  259. }
  260. --------------------------------------------------
  261. In this example, the document is routed to a shard based on
  262. the `routing` parameter provided: "kimchy".
  263. When setting up explicit mapping, you can also use the `_routing` field
  264. to direct the index operation to extract the routing value from the
  265. document itself. This does come at the (very minimal) cost of an
  266. additional document parsing pass. If the `_routing` mapping is defined
  267. and set to be `required`, the index operation will fail if no routing
  268. value is provided or extracted.
  269. NOTE: Data streams do not support custom routing unless they were created with
  270. the <<put-index-template-api-request-body,`allow_custom_routing`>> setting
  271. enabled in the template.
  272. [discrete]
  273. [[index-distributed]]
  274. ===== Distributed
  275. The index operation is directed to the primary shard based on its route
  276. (see the Routing section above) and performed on the actual node
  277. containing this shard. After the primary shard completes the operation,
  278. if needed, the update is distributed to applicable replicas.
  279. [discrete]
  280. [[index-wait-for-active-shards]]
  281. ===== Active shards
  282. To improve the resiliency of writes to the system, indexing operations
  283. can be configured to wait for a certain number of active shard copies
  284. before proceeding with the operation. If the requisite number of active
  285. shard copies are not available, then the write operation must wait and
  286. retry, until either the requisite shard copies have started or a timeout
  287. occurs. By default, write operations only wait for the primary shards
  288. to be active before proceeding (i.e. `wait_for_active_shards=1`).
  289. This default can be overridden in the index settings dynamically
  290. by setting `index.write.wait_for_active_shards`. To alter this behavior
  291. per operation, the `wait_for_active_shards` request parameter can be used.
  292. Valid values are `all` or any positive integer up to the total number
  293. of configured copies per shard in the index (which is `number_of_replicas+1`).
  294. Specifying a negative value or a number greater than the number of
  295. shard copies will throw an error.
  296. For example, suppose we have a cluster of three nodes, `A`, `B`, and `C` and
  297. we create an index `index` with the number of replicas set to 3 (resulting in
  298. 4 shard copies, one more copy than there are nodes). If we
  299. attempt an indexing operation, by default the operation will only ensure
  300. the primary copy of each shard is available before proceeding. This means
  301. that even if `B` and `C` went down, and `A` hosted the primary shard copies,
  302. the indexing operation would still proceed with only one copy of the data.
  303. If `wait_for_active_shards` is set on the request to `3` (and all 3 nodes
  304. are up), then the indexing operation will require 3 active shard copies
  305. before proceeding, a requirement which should be met because there are 3
  306. active nodes in the cluster, each one holding a copy of the shard. However,
  307. if we set `wait_for_active_shards` to `all` (or to `4`, which is the same),
  308. the indexing operation will not proceed as we do not have all 4 copies of
  309. each shard active in the index. The operation will timeout
  310. unless a new node is brought up in the cluster to host the fourth copy of
  311. the shard.
  312. It is important to note that this setting greatly reduces the chances of
  313. the write operation not writing to the requisite number of shard copies,
  314. but it does not completely eliminate the possibility, because this check
  315. occurs before the write operation commences. Once the write operation
  316. is underway, it is still possible for replication to fail on any number of
  317. shard copies but still succeed on the primary. The `_shards` section of the
  318. write operation's response reveals the number of shard copies on which
  319. replication succeeded/failed.
  320. [source,js]
  321. --------------------------------------------------
  322. {
  323. "_shards": {
  324. "total": 2,
  325. "failed": 0,
  326. "successful": 2
  327. }
  328. }
  329. --------------------------------------------------
  330. // NOTCONSOLE
  331. [discrete]
  332. [[index-refresh]]
  333. ===== Refresh
  334. Control when the changes made by this request are visible to search. See
  335. <<docs-refresh,refresh>>.
  336. [discrete]
  337. [[index-noop]]
  338. ===== Noop updates
  339. When updating a document using the index API a new version of the document is
  340. always created even if the document hasn't changed. If this isn't acceptable
  341. use the `_update` API with `detect_noop` set to true. This option isn't
  342. available on the index API because the index API doesn't fetch the old source
  343. and isn't able to compare it against the new source.
  344. There isn't a hard and fast rule about when noop updates aren't acceptable.
  345. It's a combination of lots of factors like how frequently your data source
  346. sends updates that are actually noops and how many queries per second
  347. Elasticsearch runs on the shard receiving the updates.
  348. [discrete]
  349. [[timeout]]
  350. ===== Timeout
  351. The primary shard assigned to perform the index operation might not be
  352. available when the index operation is executed. Some reasons for this
  353. might be that the primary shard is currently recovering from a gateway
  354. or undergoing relocation. By default, the index operation will wait on
  355. the primary shard to become available for up to 1 minute before failing
  356. and responding with an error. The `timeout` parameter can be used to
  357. explicitly specify how long it waits. Here is an example of setting it
  358. to 5 minutes:
  359. [source,console]
  360. --------------------------------------------------
  361. PUT my-index-000001/_doc/1?timeout=5m
  362. {
  363. "@timestamp": "2099-11-15T13:12:00",
  364. "message": "GET /search HTTP/1.1 200 1070000",
  365. "user": {
  366. "id": "kimchy"
  367. }
  368. }
  369. --------------------------------------------------
  370. [discrete]
  371. [[index-versioning]]
  372. ===== Versioning
  373. Each indexed document is given a version number. By default,
  374. internal versioning is used that starts at 1 and increments
  375. with each update, deletes included. Optionally, the version number can be
  376. set to an external value (for example, if maintained in a
  377. database). To enable this functionality, `version_type` should be set to
  378. `external`. The value provided must be a numeric, long value greater than or equal to 0,
  379. and less than around 9.2e+18.
  380. When using the external version type, the system checks to see if
  381. the version number passed to the index request is greater than the
  382. version of the currently stored document. If true, the document will be
  383. indexed and the new version number used. If the value provided is less
  384. than or equal to the stored document's version number, a version
  385. conflict will occur and the index operation will fail. For example:
  386. [source,console]
  387. --------------------------------------------------
  388. PUT my-index-000001/_doc/1?version=2&version_type=external
  389. {
  390. "user": {
  391. "id": "elkbee"
  392. }
  393. }
  394. --------------------------------------------------
  395. // TEST[continued]
  396. NOTE: Versioning is completely real time, and is not affected by the
  397. near real time aspects of search operations. If no version is provided,
  398. then the operation is executed without any version checks.
  399. In the previous example, the operation will succeed since the supplied
  400. version of 2 is higher than
  401. the current document version of 1. If the document was already updated
  402. and its version was set to 2 or higher, the indexing command will fail
  403. and result in a conflict (409 http status code).
  404. A nice side effect is that there is no need to maintain strict ordering
  405. of async indexing operations executed as a result of changes to a source
  406. database, as long as version numbers from the source database are used.
  407. Even the simple case of updating the Elasticsearch index using data from
  408. a database is simplified if external versioning is used, as only the
  409. latest version will be used if the index operations arrive out of order for
  410. whatever reason.
  411. [discrete]
  412. [[index-version-types]]
  413. ===== Version types
  414. In addition to the `external` version type, Elasticsearch
  415. also supports other types for specific use cases:
  416. [[_version_types]]
  417. `external` or `external_gt`:: Only index the document if the given version is strictly higher
  418. than the version of the stored document *or* if there is no existing document. The given
  419. version will be used as the new version and will be stored with the new document. The supplied
  420. version must be a non-negative long number.
  421. `external_gte`:: Only index the document if the given version is *equal* or higher
  422. than the version of the stored document. If there is no existing document
  423. the operation will succeed as well. The given version will be used as the new version
  424. and will be stored with the new document. The supplied version must be a non-negative long number.
  425. NOTE: The `external_gte` version type is meant for special use cases and
  426. should be used with care. If used incorrectly, it can result in loss of data.
  427. There is another option, `force`, which is deprecated because it can cause
  428. primary and replica shards to diverge.
  429. [[docs-index-api-example]]
  430. ==== {api-examples-title}
  431. Insert a JSON document into the `my-index-000001` index with an `_id` of 1:
  432. [source,console]
  433. --------------------------------------------------
  434. PUT my-index-000001/_doc/1
  435. {
  436. "@timestamp": "2099-11-15T13:12:00",
  437. "message": "GET /search HTTP/1.1 200 1070000",
  438. "user": {
  439. "id": "kimchy"
  440. }
  441. }
  442. --------------------------------------------------
  443. The API returns the following result:
  444. [source,console-result]
  445. --------------------------------------------------
  446. {
  447. "_shards": {
  448. "total": 2,
  449. "failed": 0,
  450. "successful": 2
  451. },
  452. "_index": "my-index-000001",
  453. "_id": "1",
  454. "_version": 1,
  455. "_seq_no": 0,
  456. "_primary_term": 1,
  457. "result": "created"
  458. }
  459. --------------------------------------------------
  460. // TESTRESPONSE[s/"successful": 2/"successful": 1/]
  461. Use the `_create` resource to index a document into the `my-index-000001` index if
  462. no document with that ID exists:
  463. [source,console]
  464. --------------------------------------------------
  465. PUT my-index-000001/_create/1
  466. {
  467. "@timestamp": "2099-11-15T13:12:00",
  468. "message": "GET /search HTTP/1.1 200 1070000",
  469. "user": {
  470. "id": "kimchy"
  471. }
  472. }
  473. --------------------------------------------------
  474. Set the `op_type` parameter to _create_ to index a document into the `my-index-000001`
  475. index if no document with that ID exists:
  476. [source,console]
  477. --------------------------------------------------
  478. PUT my-index-000001/_doc/1?op_type=create
  479. {
  480. "@timestamp": "2099-11-15T13:12:00",
  481. "message": "GET /search HTTP/1.1 200 1070000",
  482. "user": {
  483. "id": "kimchy"
  484. }
  485. }
  486. --------------------------------------------------