elasticsearch/docs/plugins/repository-gcs.asciidoc

[[repository-gcs]]
=== Google Cloud Storage Repository Plugin

The GCS repository plugin adds support for using the https://cloud.google.com/storage/[Google Cloud Storage]
service as a repository for {ref}/modules-snapshots.html[Snapshot/Restore].

:plugin_name: repository-gcs
include::install_remove.asciidoc[]

[[repository-gcs-usage]]
==== Getting started

The plugin uses the https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-clients/google-cloud-storage[Google Cloud Java Client for Storage]
to connect to the Storage service. If you are using
https://cloud.google.com/storage/[Google Cloud Storage] for the first time, you
must connect to the https://console.cloud.google.com/[Google Cloud Platform Console]
and create a new project. After your project is created, you must enable the
Cloud Storage Service for your project.

[[repository-gcs-creating-bucket]]
===== Creating a Bucket

The Google Cloud Storage service uses the concept of a
https://cloud.google.com/storage/docs/key-terms[bucket] as a container for all
the data. Buckets are usually created using the
https://console.cloud.google.com/[Google Cloud Platform Console]. The plugin
does not automatically create buckets.

To create a new bucket:

1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console].
2. Select your project.
3. Go to the https://console.cloud.google.com/storage/browser[Storage Browser].
4. Click the *Create Bucket* button.
5. Enter the name of the new bucket.
6. Select a storage class.
7. Select a location.
8. Click the *Create* button.

For more detailed instructions, see the
https://cloud.google.com/storage/docs/quickstart-console#create_a_bucket[Google Cloud documentation].

[[repository-gcs-service-authentication]]
===== Service Authentication

The plugin must authenticate the requests it makes to the Google Cloud Storage
service. It is common for Google client libraries to employ a strategy named https://cloud.google.com/docs/authentication/production#providing_credentials_to_your_application[application default credentials].
However, that strategy is **not** supported for use with Elasticsearch. The
plugin operates under the Elasticsearch process, which runs with the security
manager enabled. The security manager obstructs the "automatic" credential discovery.
Therefore, you must configure <<repository-gcs-using-service-account,service account>>
credentials even if you are using an environment that does not normally require
this configuration (such as Compute Engine, Kubernetes Engine or App Engine).

[[repository-gcs-using-service-account]]
===== Using a Service Account
You have to obtain and provide https://cloud.google.com/iam/docs/overview#service_account[service account credentials]
manually.

For detailed information about generating JSON service account files, see the  https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts[Google Cloud documentation].
Note that the PKCS12 format is not supported by this plugin.

Here is a summary of the steps:

1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console].
2. Select your project.
3. Go to the https://console.cloud.google.com/permissions[Permission] tab.
4. Select the https://console.cloud.google.com/permissions/serviceaccounts[Service Accounts] tab.
5. Click *Create service account*.
6. After the account is created, select it and download a JSON key file.

A JSON service account file looks like this:

[source,js]
----
{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "service-account-for-your-repository@your-project-id.iam.gserviceaccount.com",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/your-bucket@your-project-id.iam.gserviceaccount.com"
}
----
// NOTCONSOLE

To provide this file to the plugin, it must be stored in the {ref}/secure-settings.html[Elasticsearch keystore].  You must
add a `file` setting with the name `gcs.client.NAME.credentials_file` using the `add-file` subcommand.
 `NAME` is the name of the client configuration for the repository. The implicit client
name is `default`, but a different client name can be specified in the
repository settings with the `client` key.

NOTE: Passing the file path via the GOOGLE_APPLICATION_CREDENTIALS environment
variable is **not** supported.

For example, if you added a `gcs.client.my_alternate_client.credentials_file`
setting in the keystore, you can configure a repository to use those credentials
like this:

[source,console]
----
PUT _snapshot/my_gcs_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_bucket",
    "client": "my_alternate_client"
  }
}
----
// TEST[skip:we don't have gcs setup while testing this]

The `credentials_file` settings are {ref}/secure-settings.html#reloadable-secure-settings[reloadable].
After you reload the settings, the internal `gcs` clients, which are used to
transfer the snapshot contents, utilize the latest settings from the keystore.

NOTE: Snapshot or restore jobs that are in progress are not preempted by a *reload*
of the client's `credentials_file` settings. They complete using the client as
it was built when the operation started.

[[repository-gcs-client]]
==== Client Settings

The client used to connect to Google Cloud Storage has a number of settings available.
Client setting names are of the form `gcs.client.CLIENT_NAME.SETTING_NAME` and are specified
inside `elasticsearch.yml`. The default client name looked up by a `gcs` repository is
called `default`, but can be customized with the repository setting `client`.

For example:

[source,console]
----
PUT _snapshot/my_gcs_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_bucket",
    "client": "my_alternate_client"
  }
}
----
// TEST[skip:we don't have gcs setup while testing this]

Some settings are sensitive and must be stored in the
{ref}/secure-settings.html[Elasticsearch keystore]. This is the case for the service account file:

[source,sh]
----
bin/elasticsearch-keystore add-file gcs.client.default.credentials_file /path/service-account.json
----

The following are the available client settings. Those that must be stored in the keystore
are marked as `Secure`.

`credentials_file` ({ref}/secure-settings.html[Secure], {ref}/secure-settings.html#reloadable-secure-settings[reloadable])::

    The service account file that is used to authenticate to the Google Cloud Storage service.

`endpoint`::

    The Google Cloud Storage service endpoint to connect to. This will be automatically
    determined by the Google Cloud Storage client but can be specified explicitly.

`connect_timeout`::

    The timeout to establish a connection to the Google Cloud Storage service. The value should
    specify the unit. For example, a value of `5s` specifies a 5 second timeout. The value of `-1`
    corresponds to an infinite timeout. The default value is 20 seconds.

`read_timeout`::

    The timeout to read data from an established connection. The value should
    specify the unit. For example, a value of `5s` specifies a 5 second timeout. The value of `-1`
    corresponds to an infinite timeout. The default value is 20 seconds.

`application_name`::

    Name used by the client when it uses the Google Cloud Storage service. Setting
    a custom name can be useful to authenticate your cluster when requests
    statistics are logged in the Google Cloud Platform. Default to `repository-gcs`

`project_id`::

    The Google Cloud project id. This will be automatically inferred from the credentials file but
    can be specified explicitly. For example, it can be used to switch between projects when the
    same credentials are usable for both the production and the development projects.

[[repository-gcs-repository]]
==== Repository Settings

The `gcs` repository type supports a number of settings to customize how data
is stored in Google Cloud Storage.

These can be specified when creating the repository. For example:

[source,console]
----
PUT _snapshot/my_gcs_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_other_bucket",
    "base_path": "dev"
  }
}
----
// TEST[skip:we don't have gcs set up while testing this]

The following settings are supported:

`bucket`::

    The name of the bucket to be used for snapshots. (Mandatory)

`client`::

    The name of the client to use to connect to Google Cloud Storage.
    Defaults to `default`.

`base_path`::

    Specifies the path within bucket to repository data. Defaults to
    the root of the bucket.

`chunk_size`::

    Big files can be broken down into multiple smaller blobs in the blob store during snapshotting.
    It is not recommended to change this value from its default unless there is an explicit reason for limiting the
    size of blobs in the repository. Setting a value lower than the default can result in an increased number of API
    calls to the Google Cloud Storage Service during snapshot create as well as restore operations compared to using
    the default value and thus make both operations slower as well as more costly.
    Specify the chunk size as a value and unit, for example:
    `10MB`, `5KB`, `500B`. Defaults to the maximum size of a blob in the Google Cloud Storage Service which is `5TB`.

`compress`::

    When set to `true` metadata files are stored in compressed format. This
    setting doesn't affect index files that are already compressed by default.
    Defaults to `true`.

include::repository-shared-settings.asciidoc[]

`application_name`::

    deprecated:[6.3.0, "This setting is now defined in the <<repository-gcs-client, client settings>>."]
    Name used by the client when it uses the Google Cloud Storage service.

[[repository-gcs-bucket-permission]]
===== Recommended Bucket Permission

The service account used to access the bucket must have the "Writer" access to the bucket:

1. Connect to the https://console.cloud.google.com/[Google Cloud Platform Console].
2. Select your project.
3. Go to the https://console.cloud.google.com/storage/browser[Storage Browser].
4. Select the bucket and "Edit bucket permission".
5. The service account must be configured as a "User" with "Writer" access.