index_.asciidoc 18 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513
  1. [[docs-index_]]
  2. === Index API
  3. ++++
  4. <titleabbrev>Index</titleabbrev>
  5. ++++
  6. IMPORTANT: See <<removal-of-types>>.
  7. Adds a JSON document to the specified index and makes
  8. it searchable. If the document already exists,
  9. updates the document and increments its version.
  10. [[docs-index-api-request]]
  11. ==== {api-request-title}
  12. `PUT /<index>/_doc/<_id>`
  13. `POST /<index>/_doc/`
  14. `PUT /<index>/_create/<_id>`
  15. `POST /<index>/_create/<_id>`
  16. [[docs-index-api-path-params]]
  17. ==== {api-path-parms-title}
  18. `<index>`::
  19. (Required, string) Name of the target index. By default, the index is created
  20. automatically if it doesn't exist. For more information, see <<index-creation>>.
  21. `<_id>`::
  22. (Optional, string) Unique identifier for the document. Required if you are
  23. using a PUT request. Omit to automatically generate an ID when using a
  24. POST request.
  25. [[docs-index-api-query-params]]
  26. ==== {api-query-parms-title}
  27. include::{docdir}/rest-api/common-parms.asciidoc[tag=doc-seq-no]
  28. include::{docdir}/rest-api/common-parms.asciidoc[tag=doc-primary-term]
  29. `op_type`::
  30. (Optional, enum) Set to `create` to only index the document
  31. if it does not already exist (_put if absent_). If a document with the specified
  32. `_id` already exists, the indexing operation will fail. Same as using the
  33. `<index>/_create` endpoint. Valid values: `index`, `create`. Default: `index`.
  34. include::{docdir}/rest-api/common-parms.asciidoc[tag=doc-pipeline]
  35. include::{docdir}/rest-api/common-parms.asciidoc[tag=doc-refresh]
  36. include::{docdir}/rest-api/common-parms.asciidoc[tag=doc-routing]
  37. include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
  38. include::{docdir}/rest-api/common-parms.asciidoc[tag=doc-version]
  39. include::{docdir}/rest-api/common-parms.asciidoc[tag=doc-version-type]
  40. include::{docdir}/rest-api/common-parms.asciidoc[tag=doc-wait-for-active-shards]
  41. [[docs-index-api-request-body]]
  42. ==== {api-request-body-title}
  43. `<field>`::
  44. (Required, string) Request body contains the JSON source for the document
  45. data.
  46. [[docs-index-api-response-body]]
  47. ==== {api-response-body-title}
  48. `_shards`::
  49. Provides information about the replication process of the index operation.
  50. `_shards.total`::
  51. Indicates how many shard copies (primary and replica shards) the index operation
  52. should be executed on.
  53. `_shards.successful`::
  54. Indicates the number of shard copies the index operation succeeded on.
  55. When the index operation is successful, `successful` is at least 1.
  56. +
  57. NOTE: Replica shards might not all be started when an indexing operation
  58. returns successfully--by default, only the primary is required. Set
  59. `wait_for_active_shards` to change this default behavior. See
  60. <<index-wait-for-active-shards>>.
  61. `_shards.failed`::
  62. An array that contains replication-related errors in the case an index operation
  63. failed on a replica shard. 0 indicates there were no failures.
  64. `_index`::
  65. The name of the index the document was added to.
  66. `_type`::
  67. The document type. {es} indices now support a single document type, `_doc`.
  68. `_id`::
  69. The unique identifier for the added document.
  70. `_version`::
  71. The document version. Incremented each time the document is updated.
  72. `_seq_no`::
  73. The sequence number assigned to the document for the indexing operation.
  74. Sequence numbers are used to ensure an older version of a document
  75. doesn’t overwrite a newer version. See <<optimistic-concurrency-control-index>>.
  76. `_primary_term`::
  77. The primary term assigned to the document for the indexing operation.
  78. See <<optimistic-concurrency-control-index>>.
  79. `result`::
  80. The result of the indexing operation, `created` or `updated`.
  81. [[docs-index-api-desc]]
  82. ==== {api-description-title}
  83. You can index a new JSON document with the `_doc` or `_create` resource. Using
  84. `_create` guarantees that the document is only indexed if it does not already
  85. exist. To update an existing document, you must use the `_doc` resource.
  86. [[index-creation]]
  87. ===== Create indices automatically
  88. If the specified index does not already exist, by default the index operation
  89. automatically creates it and applies any configured
  90. <<indices-templates,index templates>>. If no mapping exists, the index opration
  91. creates a dynamic mapping. By default, new fields and objects are
  92. automatically added to the mapping if needed. For more information about field
  93. mapping, see <<mapping,mapping>> and the <<indices-put-mapping,put mapping>> API.
  94. Automatic index creation is controlled by the `action.auto_create_index`
  95. setting. This setting defaults to `true`, which allows any index to be created
  96. automatically. You can modify this setting to explicitly allow or block
  97. automatic creation of indices that match specified patterns, or set it to
  98. `false` to disable automatic index creation entirely. Specify a
  99. comma-separated list of patterns you want to allow, or prefix each pattern with
  100. `+` or `-` to indicate whether it should be allowed or blocked.
  101. [source,js]
  102. --------------------------------------------------
  103. PUT _cluster/settings
  104. {
  105. "persistent": {
  106. "action.auto_create_index": "twitter,index10,-index1*,+ind*" <1>
  107. }
  108. }
  109. PUT _cluster/settings
  110. {
  111. "persistent": {
  112. "action.auto_create_index": "false" <2>
  113. }
  114. }
  115. PUT _cluster/settings
  116. {
  117. "persistent": {
  118. "action.auto_create_index": "true" <3>
  119. }
  120. }
  121. --------------------------------------------------
  122. // CONSOLE
  123. <1> Allow auto-creation of indices called `twitter` or `index10`, block the
  124. creation of indices that match the pattern `index1*`, and allow creation of
  125. any other indices that match the `ind*` pattern. Patterns are matched in
  126. the order specified.
  127. <2> Disable automatic index creation entirely.
  128. <3> Allow automatic creation of any index. This is the default.
  129. [float]
  130. [[operation-type]]
  131. ===== Put if absent
  132. You can force a create operation by using the `_create` resource or
  133. setting the `op_type` parameter to _create_. In this case,
  134. the index operation fails if a document with the specified ID
  135. already exists in the index.
  136. [float]
  137. ===== Create document IDs automatically
  138. If you don't specify a document ID when using POST, the `op_type` is
  139. automatically set to `create` and the index operation generates a unique ID
  140. for the document.
  141. [source,js]
  142. --------------------------------------------------
  143. POST twitter/_doc/
  144. {
  145. "user" : "kimchy",
  146. "post_date" : "2009-11-15T14:12:12",
  147. "message" : "trying out Elasticsearch"
  148. }
  149. --------------------------------------------------
  150. // CONSOLE
  151. The API returns the following result:
  152. [source,js]
  153. --------------------------------------------------
  154. {
  155. "_shards" : {
  156. "total" : 2,
  157. "failed" : 0,
  158. "successful" : 2
  159. },
  160. "_index" : "twitter",
  161. "_type" : "_doc",
  162. "_id" : "W0tpsmIBdwcYyG50zbta",
  163. "_version" : 1,
  164. "_seq_no" : 0,
  165. "_primary_term" : 1,
  166. "result": "created"
  167. }
  168. --------------------------------------------------
  169. // TESTRESPONSE[s/W0tpsmIBdwcYyG50zbta/$body._id/ s/"successful" : 2/"successful" : 1/]
  170. [float]
  171. [[optimistic-concurrency-control-index]]
  172. ===== Optimistic concurrency control
  173. Index operations can be made conditional and only be performed if the last
  174. modification to the document was assigned the sequence number and primary
  175. term specified by the `if_seq_no` and `if_primary_term` parameters. If a
  176. mismatch is detected, the operation will result in a `VersionConflictException`
  177. and a status code of 409. See <<optimistic-concurrency-control>> for more details.
  178. [float]
  179. [[index-routing]]
  180. ===== Routing
  181. By default, shard placement ? or `routing` ? is controlled by using a
  182. hash of the document's id value. For more explicit control, the value
  183. fed into the hash function used by the router can be directly specified
  184. on a per-operation basis using the `routing` parameter. For example:
  185. [source,js]
  186. --------------------------------------------------
  187. POST twitter/_doc?routing=kimchy
  188. {
  189. "user" : "kimchy",
  190. "post_date" : "2009-11-15T14:12:12",
  191. "message" : "trying out Elasticsearch"
  192. }
  193. --------------------------------------------------
  194. // CONSOLE
  195. In this example, the document is routed to a shard based on
  196. the `routing` parameter provided: "kimchy".
  197. When setting up explicit mapping, you can also use the `_routing` field
  198. to direct the index operation to extract the routing value from the
  199. document itself. This does come at the (very minimal) cost of an
  200. additional document parsing pass. If the `_routing` mapping is defined
  201. and set to be `required`, the index operation will fail if no routing
  202. value is provided or extracted.
  203. [float]
  204. [[index-distributed]]
  205. ===== Distributed
  206. The index operation is directed to the primary shard based on its route
  207. (see the Routing section above) and performed on the actual node
  208. containing this shard. After the primary shard completes the operation,
  209. if needed, the update is distributed to applicable replicas.
  210. [float]
  211. [[index-wait-for-active-shards]]
  212. ===== Active shards
  213. To improve the resiliency of writes to the system, indexing operations
  214. can be configured to wait for a certain number of active shard copies
  215. before proceeding with the operation. If the requisite number of active
  216. shard copies are not available, then the write operation must wait and
  217. retry, until either the requisite shard copies have started or a timeout
  218. occurs. By default, write operations only wait for the primary shards
  219. to be active before proceeding (i.e. `wait_for_active_shards=1`).
  220. This default can be overridden in the index settings dynamically
  221. by setting `index.write.wait_for_active_shards`. To alter this behavior
  222. per operation, the `wait_for_active_shards` request parameter can be used.
  223. Valid values are `all` or any positive integer up to the total number
  224. of configured copies per shard in the index (which is `number_of_replicas+1`).
  225. Specifying a negative value or a number greater than the number of
  226. shard copies will throw an error.
  227. For example, suppose we have a cluster of three nodes, `A`, `B`, and `C` and
  228. we create an index `index` with the number of replicas set to 3 (resulting in
  229. 4 shard copies, one more copy than there are nodes). If we
  230. attempt an indexing operation, by default the operation will only ensure
  231. the primary copy of each shard is available before proceeding. This means
  232. that even if `B` and `C` went down, and `A` hosted the primary shard copies,
  233. the indexing operation would still proceed with only one copy of the data.
  234. If `wait_for_active_shards` is set on the request to `3` (and all 3 nodes
  235. are up), then the indexing operation will require 3 active shard copies
  236. before proceeding, a requirement which should be met because there are 3
  237. active nodes in the cluster, each one holding a copy of the shard. However,
  238. if we set `wait_for_active_shards` to `all` (or to `4`, which is the same),
  239. the indexing operation will not proceed as we do not have all 4 copies of
  240. each shard active in the index. The operation will timeout
  241. unless a new node is brought up in the cluster to host the fourth copy of
  242. the shard.
  243. It is important to note that this setting greatly reduces the chances of
  244. the write operation not writing to the requisite number of shard copies,
  245. but it does not completely eliminate the possibility, because this check
  246. occurs before the write operation commences. Once the write operation
  247. is underway, it is still possible for replication to fail on any number of
  248. shard copies but still succeed on the primary. The `_shards` section of the
  249. write operation's response reveals the number of shard copies on which
  250. replication succeeded/failed.
  251. [source,js]
  252. --------------------------------------------------
  253. {
  254. "_shards" : {
  255. "total" : 2,
  256. "failed" : 0,
  257. "successful" : 2
  258. }
  259. }
  260. --------------------------------------------------
  261. // NOTCONSOLE
  262. [float]
  263. [[index-refresh]]
  264. ===== Refresh
  265. Control when the changes made by this request are visible to search. See
  266. <<docs-refresh,refresh>>.
  267. [float]
  268. [[index-noop]]
  269. ===== Noop updates
  270. When updating a document using the index API a new version of the document is
  271. always created even if the document hasn't changed. If this isn't acceptable
  272. use the `_update` API with `detect_noop` set to true. This option isn't
  273. available on the index API because the index API doesn't fetch the old source
  274. and isn't able to compare it against the new source.
  275. There isn't a hard and fast rule about when noop updates aren't acceptable.
  276. It's a combination of lots of factors like how frequently your data source
  277. sends updates that are actually noops and how many queries per second
  278. Elasticsearch runs on the shard receiving the updates.
  279. [float]
  280. [[timeout]]
  281. ===== Timeout
  282. The primary shard assigned to perform the index operation might not be
  283. available when the index operation is executed. Some reasons for this
  284. might be that the primary shard is currently recovering from a gateway
  285. or undergoing relocation. By default, the index operation will wait on
  286. the primary shard to become available for up to 1 minute before failing
  287. and responding with an error. The `timeout` parameter can be used to
  288. explicitly specify how long it waits. Here is an example of setting it
  289. to 5 minutes:
  290. [source,js]
  291. --------------------------------------------------
  292. PUT twitter/_doc/1?timeout=5m
  293. {
  294. "user" : "kimchy",
  295. "post_date" : "2009-11-15T14:12:12",
  296. "message" : "trying out Elasticsearch"
  297. }
  298. --------------------------------------------------
  299. // CONSOLE
  300. [float]
  301. [[index-versioning]]
  302. ===== Versioning
  303. Each indexed document is given a version number. By default,
  304. internal versioning is used that starts at 1 and increments
  305. with each update, deletes included. Optionally, the version number can be
  306. set to an external value (for example, if maintained in a
  307. database). To enable this functionality, `version_type` should be set to
  308. `external`. The value provided must be a numeric, long value greater than or equal to 0,
  309. and less than around 9.2e+18.
  310. When using the external version type, the system checks to see if
  311. the version number passed to the index request is greater than the
  312. version of the currently stored document. If true, the document will be
  313. indexed and the new version number used. If the value provided is less
  314. than or equal to the stored document's version number, a version
  315. conflict will occur and the index operation will fail. For example:
  316. [source,js]
  317. --------------------------------------------------
  318. PUT twitter/_doc/1?version=2&version_type=external
  319. {
  320. "message" : "elasticsearch now has versioning support, double cool!"
  321. }
  322. --------------------------------------------------
  323. // CONSOLE
  324. // TEST[continued]
  325. NOTE: Versioning is completely real time, and is not affected by the
  326. near real time aspects of search operations. If no version is provided,
  327. then the operation is executed without any version checks.
  328. In the previous example, the operation will succeed since the supplied
  329. version of 2 is higher than
  330. the current document version of 1. If the document was already updated
  331. and its version was set to 2 or higher, the indexing command will fail
  332. and result in a conflict (409 http status code).
  333. A nice side effect is that there is no need to maintain strict ordering
  334. of async indexing operations executed as a result of changes to a source
  335. database, as long as version numbers from the source database are used.
  336. Even the simple case of updating the Elasticsearch index using data from
  337. a database is simplified if external versioning is used, as only the
  338. latest version will be used if the index operations arrive out of order for
  339. whatever reason.
  340. [float]
  341. [[index-version-types]]
  342. ===== Version types
  343. In addition to the `external` version type, Elasticsearch
  344. also supports other types for specific use cases:
  345. `internal`:: Only index the document if the given version is identical to the version
  346. of the stored document.
  347. `external` or `external_gt`:: Only index the document if the given version is strictly higher
  348. than the version of the stored document *or* if there is no existing document. The given
  349. version will be used as the new version and will be stored with the new document. The supplied
  350. version must be a non-negative long number.
  351. `external_gte`:: Only index the document if the given version is *equal* or higher
  352. than the version of the stored document. If there is no existing document
  353. the operation will succeed as well. The given version will be used as the new version
  354. and will be stored with the new document. The supplied version must be a non-negative long number.
  355. NOTE: The `external_gte` version type is meant for special use cases and
  356. should be used with care. If used incorrectly, it can result in loss of data.
  357. There is another option, `force`, which is deprecated because it can cause
  358. primary and replica shards to diverge.
  359. [[docs-index-api-example]]
  360. ==== {api-examples-title}
  361. Insert a JSON document into the `twitter` index with an `_id` of 1:
  362. [source,js]
  363. --------------------------------------------------
  364. PUT twitter/_doc/1
  365. {
  366. "user" : "kimchy",
  367. "post_date" : "2009-11-15T14:12:12",
  368. "message" : "trying out Elasticsearch"
  369. }
  370. --------------------------------------------------
  371. // CONSOLE
  372. The API returns the following result:
  373. [source,js]
  374. --------------------------------------------------
  375. {
  376. "_shards" : {
  377. "total" : 2,
  378. "failed" : 0,
  379. "successful" : 2
  380. },
  381. "_index" : "twitter",
  382. "_type" : "_doc",
  383. "_id" : "1",
  384. "_version" : 1,
  385. "_seq_no" : 0,
  386. "_primary_term" : 1,
  387. "result" : "created"
  388. }
  389. --------------------------------------------------
  390. // TESTRESPONSE[s/"successful" : 2/"successful" : 1/]
  391. Use the `_create` resource to index a document into the `twitter` index if
  392. no document with that ID exists:
  393. [source,js]
  394. --------------------------------------------------
  395. PUT twitter/_create/1
  396. {
  397. "user" : "kimchy",
  398. "post_date" : "2009-11-15T14:12:12",
  399. "message" : "trying out Elasticsearch"
  400. }
  401. --------------------------------------------------
  402. // CONSOLE
  403. Set the `op_type` parameter to _create_ to index a document into the `twitter`
  404. index if no document with that ID exists:
  405. [source,js]
  406. --------------------------------------------------
  407. PUT twitter/_doc/1?op_type=create
  408. {
  409. "user" : "kimchy",
  410. "post_date" : "2009-11-15T14:12:12",
  411. "message" : "trying out Elasticsearch"
  412. }
  413. --------------------------------------------------
  414. // CONSOLE