elasticsearch/docs/plugins/repository-gcs.asciidoc

[[repository-gcs]]
=== Google Cloud Storage Repository Plugin

The GCS repository plugin adds support for using the https://cloud.google.com/storage/[Google Cloud Storage]
service as a repository for {ref}/modules-snapshots.html[Snapshot/Restore].

:plugin_name: repository-gcs
include::install_remove.asciidoc[]

[[repository-gcs-usage]]
==== Getting started

The plugin uses the https://cloud.google.com/storage/docs/json_api/[Google Cloud Storage JSON API] (v1)
to connect to the Storage service. If this is the first time you use Google Cloud Storage, you first
need to connect to the https://console.cloud.google.com/[Google Cloud Platform Console] and create a new
project. Once your project is created, you must enable the Cloud Storage Service for your project.

[[repository-gcs-creating-bucket]]
===== Creating a Bucket

Google Cloud Storage service uses the concept of https://cloud.google.com/storage/docs/key-terms[Bucket]
as a container for all the data. Buckets are usually created using the
https://console.cloud.google.com/[Google Cloud Platform Console]. The plugin will not automatically
create buckets.

To create a new bucket:

1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console]
2. Select your project
3. Got to the https://console.cloud.google.com/storage/browser[Storage Browser]
4. Click the "Create Bucket" button
5. Enter a the name of the new bucket
6. Select a storage class
7. Select a location
8. Click the "Create" button

The bucket should now be created.

[[repository-gcs-service-authentication]]
===== Service Authentication

The plugin supports two authentication modes:

* The built-in <<repository-gcs-using-compute-engine, Compute Engine authentication>>. This mode is
recommended if your Elasticsearch node is running on a Compute Engine virtual machine.

* Specifying <<repository-gcs-using-service-account, Service Account>> credentials.

[[repository-gcs-using-compute-engine]]
===== Using Compute Engine
When running on Compute Engine, the plugin use Google's built-in authentication mechanism to
authenticate on the Storage service. Compute Engine virtual machines are usually associated to a
default service account. This service account can be found in the VM instance details in the
https://console.cloud.google.com/compute/[Compute Engine console].

This is the default authentication mode and requires no configuration.

NOTE: The Compute Engine VM must be allowed to use the Storage service. This can be done only at VM
creation time, when "Storage" access can be configured to "Read/Write" permission. Check your
instance details at the section "Cloud API access scopes".

[[repository-gcs-using-service-account]]
===== Using a Service Account
If your Elasticsearch node is not running on Compute Engine, or if you don't want to use Google's
built-in authentication mechanism, you can authenticate on the Storage service using a
https://cloud.google.com/iam/docs/overview#service_account[Service Account] file.

To create a service account file:

1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console]
2. Select your project
3. Got to the https://console.cloud.google.com/permissions[Permission] tab
4. Select the https://console.cloud.google.com/permissions/serviceaccounts[Service Accounts] tab
5. Click on "Create service account"
6. Once created, select the new service account and download a JSON key file

A service account file looks like this:

[source,js]
----
{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "service-account-for-your-repository@your-project-id.iam.gserviceaccount.com",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "..."
}
----
// NOTCONSOLE

This file must be stored in the {ref}/secure-settings.html[elasticsearch keystore], under a setting name
of the form `gcs.client.NAME.credentials_file`, where `NAME` is the name of the client configuration.
The default client name is `default`, but a different client name can be specified in repository
settings using `client`.

For example, if specifying the credentials file in the keystore under
`gcs.client.my_alternate_client.credentials_file`, you can configure a repository to use these
credentials like this:

[source,js]
----
PUT _snapshot/my_gcs_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_bucket",
    "client": "my_alternate_client"
  }
}
----
// CONSOLE
// TEST[skip:we don't have gcs setup while testing this]

[[repository-gcs-client]]
==== Client Settings

The client used to connect to Google Cloud Storage has a number of settings available.
Client setting names are of the form `gcs.client.CLIENT_NAME.SETTING_NAME` and specified
inside `elasticsearch.yml`. The default client name looked up by a `gcs` repository is
called `default`, but can be customized with the repository setting `client`.

For example:

[source,js]
----
PUT _snapshot/my_gcs_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_bucket",
    "client": "my_alternate_client"
  }
}
----
// CONSOLE
// TEST[skip:we don't have gcs setup while testing this]

Some settings are sensitive and must be stored in the
{ref}/secure-settings.html[elasticsearch keystore]. This is the case for the service account file:

[source,sh]
----
bin/elasticsearch-keystore add-file gcs.client.default.credentials_file
----

The following are the available client settings. Those that must be stored in the keystore
are marked as `Secure`.

`credentials_file`::

    The service account file that is used to authenticate to the Google Cloud Storage service. (Secure)

`endpoint`::

    The Google Cloud Storage service endpoint to connect to. This will be automatically
    determined by the Google Cloud Storage client but can be specified explicitly.

`connect_timeout`::

    The timeout to establish a connection to the Google Cloud Storage service. The value should
    specify the unit. For example, a value of `5s` specifies a 5 second timeout. The value of `-1`
    corresponds to an infinite timeout. The default value is 20 seconds.

`read_timeout`::

    The timeout to read data from an established connection. The value should
    specify the unit. For example, a value of `5s` specifies a 5 second timeout. The value of `-1`
    corresponds to an infinite timeout. The default value is 20 seconds.

`application_name`::

    Name used by the client when it uses the Google Cloud Storage service. Setting
    a custom name can be useful to authenticate your cluster when requests
    statistics are logged in the Google Cloud Platform. Default to `repository-gcs`

[[repository-gcs-repository]]
==== Repository Settings

The `gcs` repository type supports a number of settings to customize how data
is stored in Google Cloud Storage.

These can be specified when creating the repository. For example:

[source,js]
----
PUT _snapshot/my_gcs_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_other_bucket",
    "base_path": "dev"
  }
}
----
// CONSOLE
// TEST[skip:we don't have gcs set up while testing this]

The following settings are supported:

`bucket`::

    The name of the bucket to be used for snapshots. (Mandatory)

`client`::

    The name of the client to use to connect to Google Cloud Storage.
    Defaults to `default`.

`base_path`::

    Specifies the path within bucket to repository data. Defaults to
    the root of the bucket.

`chunk_size`::

    Big files can be broken down into chunks during snapshotting if needed.
    The chunk size can be specified in bytes or by using size value notation,
    i.e. `1g`, `10m`, `5k`. Defaults to `100m`.

`compress`::

    When set to `true` metadata files are stored in compressed format. This
    setting doesn't affect index files that are already compressed by default.
    Defaults to `false`.

`application_name`::

    deprecated[7.0.0, This setting is now defined in the <<repository-gcs-client, client settings>>]

[[repository-gcs-bucket-permission]]
===== Recommended Bucket Permission

The service account used to access the bucket must have the "Writer" access to the bucket:

1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console]
2. Select your project
3. Got to the https://console.cloud.google.com/storage/browser[Storage Browser]
4. Select the bucket and "Edit bucket permission"
5. The service account must be configured as a "User" with "Writer" access