elasticsearch/docs/reference/snapshot-restore/register-repository.asciidoc

[[snapshots-register-repository]]
== Register a snapshot repository
++++
<titleabbrev>Register a repository</titleabbrev>
++++

This guide shows you how to register a snapshot repository. A snapshot
repository is an off-cluster storage location for your snapshots. You must
register a repository before you can take or restore snapshots.

In this guide, you’ll learn how to:

* Register a snapshot repository
* Verify that a repository is functional
* Clean up a repository to remove unneeded files

[discrete]
[[snapshot-repo-prereqs]]
=== Prerequisites

// tag::kib-snapshot-prereqs[]
* To use {kib}'s **Snapshot and Restore** feature, you must have the following
permissions:

** <<privileges-list-cluster,Cluster privileges>>: `monitor`, `manage_slm`,
`cluster:admin/snapshot`, and `cluster:admin/repository`

** <<privileges-list-indices,Index privilege>>: `all` on the `monitor` index
// end::kib-snapshot-prereqs[]

include::apis/put-repo-api.asciidoc[tag=put-repo-api-prereqs]

[discrete]
[[snapshot-repo-considerations]]
=== Considerations

When registering a snapshot repository, keep the following in mind:

* Each snapshot repository is separate and independent. {es} doesn't share
data between repositories.

* {blank}
+
--
// tag::multi-cluster-repo[]
If you register the same snapshot repository with multiple clusters, only one
cluster should have write access to the repository. On other clusters, register
the repository as read-only.

This prevents multiple clusters from writing to the repository at the same time
and corrupting the repository’s contents. It also prevents {es} from caching the
repository's contents, which means that changes made by other clusters will
become visible straight away.
// end::multi-cluster-repo[]
--

* Use a different snapshot repository for each major version of {es}. Mixing
snapshots from different major versions can corrupt a repository’s contents.

[discrete]
[[manage-snapshot-repos]]
=== Manage snapshot repositories

You can register and manage snapshot repositories in two ways:

* {kib}'s **Snapshot and Restore** feature
* {es}'s <<snapshot-restore-repo-apis,snapshot repository management APIs>>

To manage repositories in {kib}, go to the main menu and click **Stack
Management** > **Snapshot and Restore** > **Repositories**. To register a
snapshot repository, click **Register repository**.

You can also register a repository using the <<put-snapshot-repo-api,Create
snapshot repository API>>.

[discrete]
[[snapshot-repo-types]]
=== Snapshot repository types

Supported snapshot repository types vary based on your deployment type:

* <<ess-repo-types>>
* <<self-managed-repo-types>>

[discrete]
[[ess-repo-types]]
==== {ess} repository types

{ess-trial}[{ess} deployments] automatically register the
{cloud}/ec-snapshot-restore.html[`found-snapshots`] repository. {ess} uses this
repository and the `cloud-snapshot-policy` to take periodic snapshots of your
cluster. You can also use the `found-snapshots` repository for your own
<<automate-snapshots-slm,{slm-init} policies>> or to store searchable snapshots.

The `found-snapshots` repository is specific to each deployment. However, you
can restore snapshots from another deployment's `found-snapshots` repository if
the deployments are under the same account and in the same region. See
{cloud}/ec_share_a_repository_across_clusters.html[Share a repository across
clusters].

{ess} deployments also support the following repository types:

* {cloud}/ec-azure-snapshotting.html[Azure]
* {cloud}/ec-gcs-snapshotting.html[Google Cloud Storage]
* {cloud}/ec-aws-custom-repository.html[AWS S3]
* <<snapshots-source-only-repository,Source-only>>

[discrete]
[[self-managed-repo-types]]
==== Self-managed repository types

If you run {es} on your own hardware, you can use the following built-in
snapshot repository types:

* <<repository-azure,Azure>>
* <<repository-gcs,Google Cloud Storage>>
* <<repository-s3,AWS S3>>
* <<snapshots-filesystem-repository,Shared file system>>
* <<snapshots-read-only-repository,Read-only URL>>
* <<snapshots-source-only-repository,Source-only>>

[[snapshots-repository-plugins]]
Other repository types are available through official plugins:

* {plugins}/repository-hdfs.html[Hadoop Distributed File System (HDFS)]

You can also use alternative implementations of these repository types, such as
MinIO, as long as they're compatible. To verify a repository's compatibility,
see <<snapshots-repository-verification>>.

[discrete]
[[snapshots-repository-verification]]
=== Verify a repository

When you register a snapshot repository, {es} automatically verifies that the
repository is available and functional on all master and data nodes.

To disable this verification, set the <<put-snapshot-repo-api,create snapshot
repository API>>'s `verify` query parameter to `false`. You can't disable
repository verification in {kib}.

[source,console]
----
PUT _snapshot/my_unverified_backup?verify=false
{
  "type": "fs",
  "settings": {
    "location": "my_unverified_backup_location"
  }
}
----
// TEST[setup:setup-repository]
// TEST[s/my_unverified_backup_location/my_repository/]

If wanted, you can manually run the repository verification check. To verify a
repository in {kib}, go to the **Repositories** list page and click the name of
a repository. Then click **Verify repository**. You can also use the
<<verify-snapshot-repo-api,verify snapshot repository API>>.

[source,console]
----
POST _snapshot/my_unverified_backup/_verify
----
// TEST[continued]
// TEST[s/my_unverified_backup_location/my_repository/]

If successful, the request returns a list of nodes used to verify the
repository. If verification fails, the request returns an error.

You can test a repository more thoroughly using the
<<repo-analysis-api,repository analysis API>>.

[discrete]
[[snapshots-repository-cleanup]]
=== Clean up a repository

Repositories can over time accumulate data that is not referenced by any existing snapshot. This is a result of the data safety guarantees
the snapshot functionality provides in failure scenarios during snapshot creation and the decentralized nature of the snapshot creation
process. This unreferenced data does in no way negatively impact the performance or safety of a snapshot repository but leads to higher
than necessary storage use. To remove this unreferenced data, you can run a cleanup operation on the repository. This will
trigger a complete accounting of the repository's contents and delete any unreferenced data.

To run the repository cleanup operation in {kib}, go to the **Repositories**
list page and click the name of a repository. Then click **Clean up
repository**.

You can also use the <<clean-up-snapshot-repo-api,clean up snapshot repository
API>>.

[source,console]
----
POST _snapshot/my_repository/_cleanup
----
// TEST[setup:setup-snapshots]

The API returns:

[source,console-result]
----
{
  "results": {
    "deleted_bytes": 20,
    "deleted_blobs": 5
  }
}
----
// TESTRESPONSE[s/"deleted_bytes": 20/"deleted_bytes": $body.results.deleted_bytes/]
// TESTRESPONSE[s/"deleted_blobs": 5/"deleted_blobs": $body.results.deleted_bytes/]

Depending on the concrete repository implementation the numbers shown for bytes free as well as the number of blobs removed will either
be an approximation or an exact result. Any non-zero value for the number of blobs removed implies that unreferenced blobs were found and
subsequently cleaned up.

Please note that most of the cleanup operations executed by this endpoint are automatically executed when deleting any snapshot from a
repository. If you regularly delete snapshots, you will in most cases not get any or only minor space savings from using this functionality
and should lower your frequency of invoking it accordingly.

[discrete]
[[snapshots-repository-backup]]
=== Back up a repository

You may wish to make an independent backup of your repository, for instance so
that you have an archive copy of its contents that you can use to recreate the
repository in its current state at a later date.

You must ensure that {es} does not write to the repository while you are taking
the backup of its contents. You can do this by unregistering it, or registering
it with `readonly: true`, on all your clusters. If {es} writes any data to the
repository during the backup then the contents of the backup may not be
consistent and it may not be possible to recover any data from it in future.

Alternatively, if your repository supports it, you may take an atomic snapshot
of the underlying filesystem and then take a backup of this filesystem
snapshot. It is very important that the filesystem snapshot is taken
atomically.

WARNING: You cannot use filesystem snapshots of individual nodes as a backup
mechanism. You must use the {es} snapshot and restore feature to copy the
cluster contents to a separate repository. Then, if desired, you can take a
filesystem snapshot of this repository.

When restoring a repository from a backup, you must not register the repository
with {es} until the repository contents are fully restored. If you alter the
contents of a repository while it is registered with {es} then the repository
may become unreadable or may silently lose some of its contents.

include::repository-azure.asciidoc[]
include::repository-gcs.asciidoc[]
include::repository-s3.asciidoc[]
include::repository-shared-file-system.asciidoc[]
include::repository-read-only-url.asciidoc[]
include::repository-source-only.asciidoc[]