elasticsearch/docs/plugins/repository-hdfs.asciidoc

[[repository-hdfs]]
=== Hadoop HDFS repository plugin

The HDFS repository plugin adds support for using HDFS File System as a repository for
{ref}/snapshot-restore.html[Snapshot/Restore].

:plugin_name: repository-hdfs
include::install_remove.asciidoc[]

[[repository-hdfs-usage]]
==== Getting started with HDFS

The HDFS snapshot/restore plugin is built against the latest Apache Hadoop 2.x (currently 2.7.1). If the distro you are using is not protocol
compatible with Apache Hadoop, consider replacing the Hadoop libraries inside the plugin folder with your own (you might have to adjust the security permissions required).

Even if Hadoop is already installed on the Elasticsearch nodes, for security reasons, the required libraries need to be placed under the plugin folder. Note that in most cases, if the distro is compatible, one simply needs to configure the repository with the appropriate Hadoop configuration files (see below).

Windows Users::
Using Apache Hadoop on Windows is problematic and thus it is not recommended. For those _really_ wanting to use it, make sure you place the elusive `winutils.exe` under the
plugin folder and point `HADOOP_HOME` variable to it; this should minimize the amount of permissions Hadoop requires (though one would still have to add some more).

[[repository-hdfs-config]]
==== Configuration properties

Once installed, define the configuration for the `hdfs` repository through the
{ref}/snapshot-restore.html[REST API]:

[source,console]
----
PUT _snapshot/my_hdfs_repository
{
  "type": "hdfs",
  "settings": {
    "uri": "hdfs://namenode:8020/",
    "path": "elasticsearch/repositories/my_hdfs_repository",
    "conf.dfs.client.read.shortcircuit": "true"
  }
}
----
// TEST[skip:we don't have hdfs set up while testing this]

The following settings are supported:

[horizontal]
`uri`::

    The uri address for hdfs. ex: "hdfs://<host>:<port>/". (Required)

`path`::

    The file path within the filesystem where data is stored/loaded. ex: "path/to/file". (Required)

`load_defaults`::

    Whether to load the default Hadoop configuration or not. (Enabled by default)

`conf.<key>`::

    Inlined configuration parameter to be added to Hadoop configuration. (Optional)
    Only client oriented properties from the hadoop https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml[core] and https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml[hdfs] configuration files will be recognized by the plugin.

`compress`::

    Whether to compress the metadata or not. (Enabled by default)

include::repository-shared-settings.asciidoc[]

`chunk_size`::

    Override the chunk size. (Disabled by default)

`security.principal`::

    Kerberos principal to use when connecting to a secured HDFS cluster.
    If you are using a service principal for your elasticsearch node, you may
    use the `_HOST` pattern in the principal name and the plugin will replace
    the pattern with the hostname of the node at runtime (see
    link:repository-hdfs-security-runtime[Creating the Secure Repository]).

`replication_factor`::

    The replication factor for all new HDFS files created by this repository.
    Must be greater or equal to `dfs.replication.min` and less or equal to `dfs.replication.max` HDFS option.
    Defaults to using HDFS cluster setting.

[[repository-hdfs-availability]]
[discrete]
===== A note on HDFS availability
When you initialize a repository, its settings are persisted in the cluster state. When a node comes online, it will
attempt to initialize all repositories for which it has settings. If your cluster has an HDFS repository configured, then
all nodes in the cluster must be able to reach HDFS when starting. If not, then the node will fail to initialize the
repository at start up and the repository will be unusable. If this happens, you will need to remove and re-add the
repository or restart the offending node.

[[repository-hdfs-security]]
==== Hadoop security

The HDFS repository plugin integrates seamlessly with Hadoop's authentication model. The following authentication
methods are supported by the plugin:

[horizontal]
`simple`::

    Also means "no security" and is enabled by default. Uses information from underlying operating system account
    running Elasticsearch to inform Hadoop of the name of the current user. Hadoop makes no attempts to verify this
    information.

`kerberos`::

    Authenticates to Hadoop through the usage of a Kerberos principal and keytab. Interfacing with HDFS clusters
    secured with Kerberos requires a few additional steps to enable (See <<repository-hdfs-security-keytabs>> and
    <<repository-hdfs-security-runtime>> for more info)

[[repository-hdfs-security-keytabs]]
[discrete]
===== Principals and keytabs
Before attempting to connect to a secured HDFS cluster, provision the Kerberos principals and keytabs that the
Elasticsearch nodes will use for authenticating to Kerberos. For maximum security and to avoid tripping up the Kerberos
replay protection, you should create a service principal per node, following the pattern of
`elasticsearch/hostname@REALM`.

WARNING: In some cases, if the same principal is authenticating from multiple clients at once, services may reject
authentication for those principals under the assumption that they could be replay attacks. If you are running the
plugin in production with multiple nodes you should be using a unique service principal for each node.

On each Elasticsearch node, place the appropriate keytab file in the node's configuration location under the
`repository-hdfs` directory using the name `krb5.keytab`:

[source, bash]
----
$> cd elasticsearch/config
$> ls
elasticsearch.yml  jvm.options        log4j2.properties  repository-hdfs/   scripts/
$> cd repository-hdfs
$> ls
krb5.keytab
----
// TEST[skip:this is for demonstration purposes only

NOTE: Make sure you have the correct keytabs! If you are using a service principal per node (like
`elasticsearch/hostname@REALM`) then each node will need its own unique keytab file for the principal assigned to that
host!

// Setup at runtime (principal name)
[[repository-hdfs-security-runtime]]
[discrete]
===== Creating the secure repository
Once your keytab files are in place and your cluster is started, creating a secured HDFS repository is simple. Just
add the name of the principal that you will be authenticating as in the repository settings under the
`security.principal` option:

[source,console]
----
PUT _snapshot/my_hdfs_repository
{
  "type": "hdfs",
  "settings": {
    "uri": "hdfs://namenode:8020/",
    "path": "/user/elasticsearch/repositories/my_hdfs_repository",
    "security.principal": "elasticsearch@REALM"
  }
}
----
// TEST[skip:we don't have hdfs set up while testing this]

If you are using different service principals for each node, you can use the `_HOST` pattern in your principal
name. Elasticsearch will automatically replace the pattern with the hostname of the node at runtime:

[source,console]
----
PUT _snapshot/my_hdfs_repository
{
  "type": "hdfs",
  "settings": {
    "uri": "hdfs://namenode:8020/",
    "path": "/user/elasticsearch/repositories/my_hdfs_repository",
    "security.principal": "elasticsearch/_HOST@REALM"
  }
}
----
// TEST[skip:we don't have hdfs set up while testing this]

[[repository-hdfs-security-authorization]]
[discrete]
===== Authorization
Once Elasticsearch is connected and authenticated to HDFS, HDFS will infer a username to use for
authorizing file access for the client. By default, it picks this username from the primary part of
the kerberos principal used to authenticate to the service. For example, in the case of a principal
like `elasticsearch@REALM` or `elasticsearch/hostname@REALM` then the username that HDFS
extracts for file access checks will be `elasticsearch`.

NOTE: The repository plugin makes no assumptions of what Elasticsearch's principal name is. The main fragment of the
Kerberos principal is not required to be `elasticsearch`. If you have a principal or service name that works better
for you or your organization then feel free to use it instead!