rolling_upgrade.asciidoc 6.0 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175
  1. [[rolling-upgrades]]
  2. === Rolling upgrades
  3. A rolling upgrade allows the Elasticsearch cluster to be upgraded one node at
  4. a time, with no downtime for end users. Running multiple versions of
  5. Elasticsearch in the same cluster for any length of time beyond that required
  6. for an upgrade is not supported, as shards will not be replicated from the
  7. more recent version to the older version.
  8. Consult this <<setup-upgrade,table>> to verify that rolling upgrades are
  9. supported for your version of Elasticsearch.
  10. To perform a rolling upgrade:
  11. ==== Step 1: Disable shard allocation
  12. When you shut down a node, the allocation process will wait for one minute
  13. before starting to replicate the shards that were on that node to other nodes
  14. in the cluster, causing a lot of wasted I/O. This can be avoided by disabling
  15. allocation before shutting down a node:
  16. [source,js]
  17. --------------------------------------------------
  18. PUT _cluster/settings
  19. {
  20. "transient": {
  21. "cluster.routing.allocation.enable": "none"
  22. }
  23. }
  24. --------------------------------------------------
  25. // CONSOLE
  26. // TEST[skip:indexes don't assign]
  27. ==== Step 2: Stop non-essential indexing and perform a synced flush (Optional)
  28. You may happily continue indexing during the upgrade. However, shard recovery
  29. will be much faster if you temporarily stop non-essential indexing and issue a
  30. <<indices-synced-flush, synced-flush>> request:
  31. [source,js]
  32. --------------------------------------------------
  33. POST _flush/synced
  34. --------------------------------------------------
  35. // CONSOLE
  36. A synced flush request is a ``best effort'' operation. It will fail if there
  37. are any pending indexing operations, but it is safe to reissue the request
  38. multiple times if necessary.
  39. [[upgrade-node]]
  40. ==== Step 3: Stop and upgrade a single node
  41. Shut down one of the nodes in the cluster *before* starting the upgrade.
  42. [TIP]
  43. ================================================
  44. When using the zip or tarball packages, the `config`, `data`, `logs` and
  45. `plugins` directories are placed within the Elasticsearch home directory by
  46. default.
  47. It is a good idea to place these directories in a different location so that
  48. there is no chance of deleting them when upgrading Elasticsearch. These
  49. custom paths can be <<path-settings,configured>> with the `path.conf`,
  50. `path.logs`, and `path.data` settings.
  51. The <<deb,Debian>> and <<rpm,RPM>> packages place these directories in the
  52. appropriate place for each operating system.
  53. ================================================
  54. To upgrade using a <<deb,Debian>> or <<rpm,RPM>> package:
  55. * Use `rpm` or `dpkg` to install the new package. All files should be
  56. placed in their proper locations, and config files should not be
  57. overwritten.
  58. To upgrade using a zip or compressed tarball:
  59. * Extract the zip or tarball to a new directory, to be sure that you don't
  60. overwrite the `config` or `data` directories.
  61. * Either copy the files in the `config` directory from your old installation
  62. to your new installation, or use the `-E path.conf=` option on the command
  63. line to point to an external config directory.
  64. * Either copy the files in the `data` directory from your old installation
  65. to your new installation, or configure the location of the data directory
  66. in the `config/elasticsearch.yml` file, with the `path.data` setting.
  67. ==== Step 4: Upgrade any plugins
  68. Elasticsearch plugins must be upgraded when upgrading a node. Use the
  69. `elasticsearch-plugin` script to install the correct version of any plugins
  70. that you need.
  71. ==== Step 5: Start the upgraded node
  72. Start the now upgraded node and confirm that it joins the cluster by checking
  73. the log file or by checking the output of this request:
  74. [source,sh]
  75. --------------------------------------------------
  76. GET _cat/nodes
  77. --------------------------------------------------
  78. // CONSOLE
  79. ==== Step 6: Reenable shard allocation
  80. Once the node has joined the cluster, reenable shard allocation to start using
  81. the node:
  82. [source,js]
  83. --------------------------------------------------
  84. PUT _cluster/settings
  85. {
  86. "transient": {
  87. "cluster.routing.allocation.enable": "all"
  88. }
  89. }
  90. --------------------------------------------------
  91. // CONSOLE
  92. ==== Step 7: Wait for the node to recover
  93. You should wait for the cluster to finish shard allocation before upgrading
  94. the next node. You can check on progress with the <<cat-health,`_cat/health`>>
  95. request:
  96. [source,sh]
  97. --------------------------------------------------
  98. GET _cat/health
  99. --------------------------------------------------
  100. // CONSOLE
  101. Wait for the `status` column to move from `yellow` to `green`. Status `green`
  102. means that all primary and replica shards have been allocated.
  103. [IMPORTANT]
  104. ====================================================
  105. During a rolling upgrade, primary shards assigned to a node with the higher
  106. version will never have their replicas assigned to a node with the lower
  107. version, because the newer version may have a different data format which is
  108. not understood by the older version.
  109. If it is not possible to assign the replica shards to another node with the
  110. higher version -- e.g. if there is only one node with the higher version in
  111. the cluster -- then the replica shards will remain unassigned and the
  112. cluster health will remain status `yellow`.
  113. In this case, check that there are no initializing or relocating shards (the
  114. `init` and `relo` columns) before proceding.
  115. As soon as another node is upgraded, the replicas should be assigned and the
  116. cluster health will reach status `green`.
  117. ====================================================
  118. Shards that have not been <<indices-synced-flush,sync-flushed>> may take some time to
  119. recover. The recovery status of individual shards can be monitored with the
  120. <<cat-recovery,`_cat/recovery`>> request:
  121. [source,sh]
  122. --------------------------------------------------
  123. GET _cat/recovery
  124. --------------------------------------------------
  125. // CONSOLE
  126. If you stopped indexing, then it is safe to resume indexing as soon as
  127. recovery has completed.
  128. ==== Step 8: Repeat
  129. When the cluster is stable and the node has recovered, repeat the above steps
  130. for all remaining nodes.