Browse Source

GCS Repository: Add secure storage of credentials (#24697)

This commit adds gcs credential settings to the elasticsearch keystore.
The setting name follows the same pattern as the s3 client settings,
beginning with `gcs.client.`, followed by the client name, and then the
setting name, in this case, `credentials_file`. Using the legacy service
file setting is also deprecated.
Ryan Ernst 8 years ago
parent
commit
d74760c306

+ 17 - 29
docs/plugins/repository-gcs.asciidoc

@@ -41,34 +41,19 @@ The bucket should now be created.
 
 
 The plugin supports two authentication modes:
 The plugin supports two authentication modes:
 
 
-* the built-in <<repository-gcs-using-compute-engine, Compute Engine authentication>>. This mode is
+* The built-in <<repository-gcs-using-compute-engine, Compute Engine authentication>>. This mode is
 recommended if your elasticsearch node is running on a Compute Engine virtual machine.
 recommended if your elasticsearch node is running on a Compute Engine virtual machine.
 
 
-* the <<repository-gcs-using-service-account, Service Account>> authentication mode.
+* Specifying <<repository-gcs-using-service-account, Service Account>> credentials.
 
 
 [[repository-gcs-using-compute-engine]]
 [[repository-gcs-using-compute-engine]]
 ===== Using Compute Engine
 ===== Using Compute Engine
-When running on Compute Engine, the plugin use the Google's built-in authentication mechanism to
+When running on Compute Engine, the plugin use Google's built-in authentication mechanism to
 authenticate on the Storage service. Compute Engine virtual machines are usually associated to a
 authenticate on the Storage service. Compute Engine virtual machines are usually associated to a
 default service account. This service account can be found in the VM instance details in the
 default service account. This service account can be found in the VM instance details in the
 https://console.cloud.google.com/compute/[Compute Engine console].
 https://console.cloud.google.com/compute/[Compute Engine console].
 
 
-To indicate that a repository should use the built-in authentication,
-the repository `service_account` setting must be set to `_default_`:
-
-[source,js]
-----
-PUT _snapshot/my_gcs_repository_on_compute_engine
-{
-  "type": "gcs",
-  "settings": {
-    "bucket": "my_bucket",
-    "service_account": "_default_"
-  }
-}
-----
-// CONSOLE
-// TEST[skip:we don't have gcs setup while testing this]
+This is the default authentication mode and requires no configuration.
 
 
 NOTE: The Compute Engine VM must be allowed to use the Storage service. This can be done only at VM
 NOTE: The Compute Engine VM must be allowed to use the Storage service. This can be done only at VM
 creation time, when "Storage" access can be configured to "Read/Write" permission. Check your
 creation time, when "Storage" access can be configured to "Read/Write" permission. Check your
@@ -76,7 +61,7 @@ instance details at the section "Cloud API access scopes".
 
 
 [[repository-gcs-using-service-account]]
 [[repository-gcs-using-service-account]]
 ===== Using a Service Account
 ===== Using a Service Account
-If your elasticsearch node is not running on Compute Engine, or if you don't want to use Google
+If your elasticsearch node is not running on Compute Engine, or if you don't want to use Google's
 built-in authentication mechanism, you can authenticate on the Storage service using a
 built-in authentication mechanism, you can authenticate on the Storage service using a
 https://cloud.google.com/iam/docs/overview#service_account[Service Account] file.
 https://cloud.google.com/iam/docs/overview#service_account[Service Account] file.
 
 
@@ -107,10 +92,14 @@ A service account file looks like this:
 ----
 ----
 // NOTCONSOLE
 // NOTCONSOLE
 
 
-This file must be copied in the `config` directory of the elasticsearch installation and on
-every node of the cluster.
+This file must be stored in the <<secure-settings, elasticsearch keystore>>, under a setting name
+of the form `gcs.client.NAME.credentials_file`, where `NAME` is the name of the client congiguration.
+The default client name is `default`, but a different client name can be specified in repository
+settings using `client`.
 
 
-To indicate that a repository should use a service account file:
+For example, if specifying the credentials file in the keystore under
+`gcs.client.my_alternate_client.credentials_file`, you can configure a repository to use these
+credentials like this:
 
 
 [source,js]
 [source,js]
 ----
 ----
@@ -119,7 +108,7 @@ PUT _snapshot/my_gcs_repository
   "type": "gcs",
   "type": "gcs",
   "settings": {
   "settings": {
     "bucket": "my_bucket",
     "bucket": "my_bucket",
-    "service_account": "service_account.json"
+    "client": "my_alternate_client"
   }
   }
 }
 }
 ----
 ----
@@ -150,8 +139,7 @@ PUT _snapshot/my_gcs_repository
 {
 {
   "type": "gcs",
   "type": "gcs",
   "settings": {
   "settings": {
-    "bucket": "my_bucket",
-    "service_account": "service_account.json"
+    "bucket": "my_bucket"
   }
   }
 }
 }
 ----
 ----
@@ -164,10 +152,10 @@ The following settings are supported:
 
 
     The name of the bucket to be used for snapshots. (Mandatory)
     The name of the bucket to be used for snapshots. (Mandatory)
 
 
-`service_account`::
+`client`::
 
 
-    The service account to use. It can be a relative path to a service account JSON file
-    or the value `_default_` that indicate to use built-in Compute Engine service account.
+    The client congfiguration to use. This controls which credentials are used to connect
+    to Compute Engine.
 
 
 `base_path`::
 `base_path`::
 
 

+ 16 - 1
plugins/repository-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStoragePlugin.java

@@ -22,10 +22,12 @@ package org.elasticsearch.repositories.gcs;
 import java.security.AccessController;
 import java.security.AccessController;
 import java.security.PrivilegedAction;
 import java.security.PrivilegedAction;
 import java.util.Collections;
 import java.util.Collections;
+import java.util.List;
 import java.util.Map;
 import java.util.Map;
 
 
 import com.google.api.client.auth.oauth2.TokenRequest;
 import com.google.api.client.auth.oauth2.TokenRequest;
 import com.google.api.client.auth.oauth2.TokenResponse;
 import com.google.api.client.auth.oauth2.TokenResponse;
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
 import com.google.api.client.googleapis.json.GoogleJsonError;
 import com.google.api.client.googleapis.json.GoogleJsonError;
 import com.google.api.client.http.GenericUrl;
 import com.google.api.client.http.GenericUrl;
 import com.google.api.client.http.HttpHeaders;
 import com.google.api.client.http.HttpHeaders;
@@ -39,6 +41,8 @@ import com.google.api.services.storage.model.Bucket;
 import com.google.api.services.storage.model.Objects;
 import com.google.api.services.storage.model.Objects;
 import com.google.api.services.storage.model.StorageObject;
 import com.google.api.services.storage.model.StorageObject;
 import org.elasticsearch.SpecialPermission;
 import org.elasticsearch.SpecialPermission;
+import org.elasticsearch.common.settings.Setting;
+import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.NamedXContentRegistry;
 import org.elasticsearch.common.xcontent.NamedXContentRegistry;
 import org.elasticsearch.env.Environment;
 import org.elasticsearch.env.Environment;
 import org.elasticsearch.plugins.Plugin;
 import org.elasticsearch.plugins.Plugin;
@@ -108,9 +112,15 @@ public class GoogleCloudStoragePlugin extends Plugin implements RepositoryPlugin
         });
         });
     }
     }
 
 
+    private final Map<String, GoogleCredential> credentials;
+
+    public GoogleCloudStoragePlugin(Settings settings) {
+        credentials = GoogleCloudStorageService.loadClientCredentials(settings);
+    }
+
     // overridable for tests
     // overridable for tests
     protected GoogleCloudStorageService createStorageService(Environment environment) {
     protected GoogleCloudStorageService createStorageService(Environment environment) {
-        return new GoogleCloudStorageService.InternalGoogleCloudStorageService(environment);
+        return new GoogleCloudStorageService.InternalGoogleCloudStorageService(environment, credentials);
     }
     }
 
 
     @Override
     @Override
@@ -118,4 +128,9 @@ public class GoogleCloudStoragePlugin extends Plugin implements RepositoryPlugin
         return Collections.singletonMap(GoogleCloudStorageRepository.TYPE,
         return Collections.singletonMap(GoogleCloudStorageRepository.TYPE,
             (metadata) -> new GoogleCloudStorageRepository(metadata, env, namedXContentRegistry, createStorageService(env)));
             (metadata) -> new GoogleCloudStorageRepository(metadata, env, namedXContentRegistry, createStorageService(env)));
     }
     }
+
+    @Override
+    public List<Setting<?>> getSettings() {
+        return Collections.singletonList(GoogleCloudStorageService.CREDENTIALS_FILE_SETTING);
+    }
 }
 }

+ 15 - 13
plugins/repository-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageRepository.java

@@ -48,25 +48,26 @@ class GoogleCloudStorageRepository extends BlobStoreRepository {
     static final ByteSizeValue MIN_CHUNK_SIZE = new ByteSizeValue(1, ByteSizeUnit.BYTES);
     static final ByteSizeValue MIN_CHUNK_SIZE = new ByteSizeValue(1, ByteSizeUnit.BYTES);
     static final ByteSizeValue MAX_CHUNK_SIZE = new ByteSizeValue(100, ByteSizeUnit.MB);
     static final ByteSizeValue MAX_CHUNK_SIZE = new ByteSizeValue(100, ByteSizeUnit.MB);
 
 
-    public static final String TYPE = "gcs";
+    static final String TYPE = "gcs";
 
 
-    public static final TimeValue NO_TIMEOUT = timeValueMillis(-1);
+    static final TimeValue NO_TIMEOUT = timeValueMillis(-1);
 
 
-    public static final Setting<String> BUCKET =
+    static final Setting<String> BUCKET =
             simpleString("bucket", Property.NodeScope, Property.Dynamic);
             simpleString("bucket", Property.NodeScope, Property.Dynamic);
-    public static final Setting<String> BASE_PATH =
+    static final Setting<String> BASE_PATH =
             simpleString("base_path", Property.NodeScope, Property.Dynamic);
             simpleString("base_path", Property.NodeScope, Property.Dynamic);
-    public static final Setting<Boolean> COMPRESS =
+    static final Setting<Boolean> COMPRESS =
             boolSetting("compress", false, Property.NodeScope, Property.Dynamic);
             boolSetting("compress", false, Property.NodeScope, Property.Dynamic);
-    public static final Setting<ByteSizeValue> CHUNK_SIZE =
+    static final Setting<ByteSizeValue> CHUNK_SIZE =
             byteSizeSetting("chunk_size", MAX_CHUNK_SIZE, MIN_CHUNK_SIZE, MAX_CHUNK_SIZE, Property.NodeScope, Property.Dynamic);
             byteSizeSetting("chunk_size", MAX_CHUNK_SIZE, MIN_CHUNK_SIZE, MAX_CHUNK_SIZE, Property.NodeScope, Property.Dynamic);
-    public static final Setting<String> APPLICATION_NAME =
+    static final Setting<String> APPLICATION_NAME =
             new Setting<>("application_name", GoogleCloudStoragePlugin.NAME, Function.identity(), Property.NodeScope, Property.Dynamic);
             new Setting<>("application_name", GoogleCloudStoragePlugin.NAME, Function.identity(), Property.NodeScope, Property.Dynamic);
-    public static final Setting<String> SERVICE_ACCOUNT =
-            simpleString("service_account", Property.NodeScope, Property.Dynamic);
-    public static final Setting<TimeValue> HTTP_READ_TIMEOUT =
+    static final Setting<String> SERVICE_ACCOUNT =
+            new Setting<>("service_account", "_default_", Function.identity(), Property.NodeScope, Property.Dynamic, Property.Deprecated);
+    static final Setting<String> CLIENT_NAME = new Setting<>("client", "default", Function.identity());
+    static final Setting<TimeValue> HTTP_READ_TIMEOUT =
             timeSetting("http.read_timeout", NO_TIMEOUT, Property.NodeScope, Property.Dynamic);
             timeSetting("http.read_timeout", NO_TIMEOUT, Property.NodeScope, Property.Dynamic);
-    public static final Setting<TimeValue> HTTP_CONNECT_TIMEOUT =
+    static final Setting<TimeValue> HTTP_CONNECT_TIMEOUT =
             timeSetting("http.connect_timeout", NO_TIMEOUT, Property.NodeScope, Property.Dynamic);
             timeSetting("http.connect_timeout", NO_TIMEOUT, Property.NodeScope, Property.Dynamic);
 
 
     private final ByteSizeValue chunkSize;
     private final ByteSizeValue chunkSize;
@@ -81,7 +82,8 @@ class GoogleCloudStorageRepository extends BlobStoreRepository {
 
 
         String bucket = getSetting(BUCKET, metadata);
         String bucket = getSetting(BUCKET, metadata);
         String application = getSetting(APPLICATION_NAME, metadata);
         String application = getSetting(APPLICATION_NAME, metadata);
-        String serviceAccount = getSetting(SERVICE_ACCOUNT, metadata);
+        String serviceAccount = SERVICE_ACCOUNT.get(metadata.settings());
+        String clientName = CLIENT_NAME.get(metadata.settings());
 
 
         String basePath = BASE_PATH.get(metadata.settings());
         String basePath = BASE_PATH.get(metadata.settings());
         if (Strings.hasLength(basePath)) {
         if (Strings.hasLength(basePath)) {
@@ -113,7 +115,7 @@ class GoogleCloudStorageRepository extends BlobStoreRepository {
         logger.debug("using bucket [{}], base_path [{}], chunk_size [{}], compress [{}], application [{}]",
         logger.debug("using bucket [{}], base_path [{}], chunk_size [{}], compress [{}], application [{}]",
                 bucket, basePath, chunkSize, compress, application);
                 bucket, basePath, chunkSize, compress, application);
 
 
-        Storage client = storageService.createClient(serviceAccount, application, connectTimeout, readTimeout);
+        Storage client = storageService.createClient(serviceAccount, clientName, application, connectTimeout, readTimeout);
         this.blobStore = new GoogleCloudStorageBlobStore(settings, bucket, client);
         this.blobStore = new GoogleCloudStorageBlobStore(settings, bucket, client);
     }
     }
 
 

+ 65 - 29
plugins/repository-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageService.java

@@ -35,28 +35,43 @@ import com.google.api.services.storage.StorageScopes;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.common.component.AbstractComponent;
 import org.elasticsearch.common.component.AbstractComponent;
 import org.elasticsearch.common.inject.Inject;
 import org.elasticsearch.common.inject.Inject;
+import org.elasticsearch.common.settings.SecureSetting;
+import org.elasticsearch.common.settings.SecureString;
+import org.elasticsearch.common.settings.Setting;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.unit.TimeValue;
 import org.elasticsearch.common.unit.TimeValue;
 import org.elasticsearch.env.Environment;
 import org.elasticsearch.env.Environment;
 
 
 import java.io.IOException;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.InputStream;
+import java.io.UncheckedIOException;
 import java.nio.file.Files;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Path;
 import java.util.Collections;
 import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
 
 
 interface GoogleCloudStorageService {
 interface GoogleCloudStorageService {
 
 
+    String SETTINGS_PREFIX = "gcs.client.";
+
+    /** A json credentials file loaded from secure settings. */
+    Setting.AffixSetting<InputStream> CREDENTIALS_FILE_SETTING = Setting.affixKeySetting(SETTINGS_PREFIX, "credentials_file",
+        key -> SecureSetting.secureFile(key, null));
+
     /**
     /**
      * Creates a client that can be used to manage Google Cloud Storage objects.
      * Creates a client that can be used to manage Google Cloud Storage objects.
      *
      *
      * @param serviceAccount path to service account file
      * @param serviceAccount path to service account file
+     * @param clientName     name of client settings to use from secure settings
      * @param application    name of the application
      * @param application    name of the application
      * @param connectTimeout connection timeout for HTTP requests
      * @param connectTimeout connection timeout for HTTP requests
      * @param readTimeout    read timeout for HTTP requests
      * @param readTimeout    read timeout for HTTP requests
      * @return a Client instance that can be used to manage objects
      * @return a Client instance that can be used to manage objects
      */
      */
-    Storage createClient(String serviceAccount, String application, TimeValue connectTimeout, TimeValue readTimeout) throws Exception;
+    Storage createClient(String serviceAccount, String clientName, String application,
+                         TimeValue connectTimeout, TimeValue readTimeout) throws Exception;
 
 
     /**
     /**
      * Default implementation
      * Default implementation
@@ -67,58 +82,60 @@ interface GoogleCloudStorageService {
 
 
         private final Environment environment;
         private final Environment environment;
 
 
-        InternalGoogleCloudStorageService(Environment environment) {
+        /** Credentials identified by client name. */
+        private final Map<String, GoogleCredential> credentials;
+
+        InternalGoogleCloudStorageService(Environment environment, Map<String, GoogleCredential> credentials) {
             super(environment.settings());
             super(environment.settings());
             this.environment = environment;
             this.environment = environment;
+            this.credentials = credentials;
         }
         }
 
 
         @Override
         @Override
-        public Storage createClient(String serviceAccount, String application, TimeValue connectTimeout, TimeValue readTimeout)
-                throws Exception {
+        public Storage createClient(String serviceAccountFile, String clientName, String application,
+                                    TimeValue connectTimeout, TimeValue readTimeout) throws Exception {
             try {
             try {
-                GoogleCredential credentials = (DEFAULT.equalsIgnoreCase(serviceAccount)) ? loadDefault() : loadCredentials(serviceAccount);
+                GoogleCredential credential = getCredential(serviceAccountFile, clientName);
                 NetHttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
                 NetHttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
 
 
                 Storage.Builder storage = new Storage.Builder(httpTransport, JacksonFactory.getDefaultInstance(),
                 Storage.Builder storage = new Storage.Builder(httpTransport, JacksonFactory.getDefaultInstance(),
-                        new DefaultHttpRequestInitializer(credentials, connectTimeout, readTimeout));
+                        new DefaultHttpRequestInitializer(credential, connectTimeout, readTimeout));
                 storage.setApplicationName(application);
                 storage.setApplicationName(application);
 
 
                 logger.debug("initializing client with service account [{}/{}]",
                 logger.debug("initializing client with service account [{}/{}]",
-                        credentials.getServiceAccountId(), credentials.getServiceAccountUser());
+                        credential.getServiceAccountId(), credential.getServiceAccountUser());
                 return storage.build();
                 return storage.build();
             } catch (IOException e) {
             } catch (IOException e) {
                 throw new ElasticsearchException("Error when loading Google Cloud Storage credentials file", e);
                 throw new ElasticsearchException("Error when loading Google Cloud Storage credentials file", e);
             }
             }
         }
         }
 
 
-        /**
-         * HTTP request initializer that loads credentials from the service account file
-         * and manages authentication for HTTP requests
-         */
-        private GoogleCredential loadCredentials(String serviceAccount) throws IOException {
-            if (serviceAccount == null) {
-                throw new ElasticsearchException("Cannot load Google Cloud Storage service account file from a null path");
-            }
-
-            Path account = environment.configFile().resolve(serviceAccount);
-            if (Files.exists(account) == false) {
-                throw new ElasticsearchException("Unable to find service account file [" + serviceAccount
+        // pkg private for tests
+        GoogleCredential getCredential(String serviceAccountFile, String clientName) throws IOException {
+            if (DEFAULT.equalsIgnoreCase(serviceAccountFile) == false) {
+                deprecationLogger.deprecated("Using GCS service account file from disk is deprecated. " +
+                    "Move the file into the elasticsearch keystore.");
+                Path account = environment.configFile().resolve(serviceAccountFile);
+                if (Files.exists(account) == false) {
+                    throw new IllegalArgumentException("Unable to find service account file [" + serviceAccountFile
                         + "] defined for repository");
                         + "] defined for repository");
-            }
+                }
 
 
-            try (InputStream is = Files.newInputStream(account)) {
-                GoogleCredential credential = GoogleCredential.fromStream(is);
-                if (credential.createScopedRequired()) {
-                    credential = credential.createScoped(Collections.singleton(StorageScopes.DEVSTORAGE_FULL_CONTROL));
+                try (InputStream is = Files.newInputStream(account)) {
+                    GoogleCredential credential = GoogleCredential.fromStream(is);
+                    if (credential.createScopedRequired()) {
+                        credential = credential.createScoped(Collections.singleton(StorageScopes.DEVSTORAGE_FULL_CONTROL));
+                    }
+                    return credential;
                 }
                 }
-                return credential;
+            } else if (credentials.containsKey(clientName)) {
+                return credentials.get(clientName);
             }
             }
+            return getDefaultCredential();
         }
         }
 
 
-        /**
-         * HTTP request initializer that loads default credentials when running on Compute Engine
-         */
-        private GoogleCredential loadDefault() throws IOException {
+        // pkg private for tests
+        GoogleCredential getDefaultCredential() throws IOException {
             return GoogleCredential.getApplicationDefault();
             return GoogleCredential.getApplicationDefault();
         }
         }
 
 
@@ -172,4 +189,23 @@ interface GoogleCloudStorageService {
             }
             }
         }
         }
     }
     }
+
+    /** Load all secure credentials from the settings. */
+    static Map<String, GoogleCredential> loadClientCredentials(Settings settings) {
+        Set<String> clientNames = settings.getGroups(SETTINGS_PREFIX).keySet();
+        Map<String, GoogleCredential> credentials = new HashMap<>();
+        for (String clientName : clientNames) {
+            Setting<InputStream> concreteSetting = CREDENTIALS_FILE_SETTING.getConcreteSettingForNamespace(clientName);
+            try (InputStream credStream = concreteSetting.get(settings)) {
+                GoogleCredential credential = GoogleCredential.fromStream(credStream);
+                if (credential.createScopedRequired()) {
+                    credential = credential.createScoped(Collections.singleton(StorageScopes.DEVSTORAGE_FULL_CONTROL));
+                }
+                credentials.put(clientName, credential);
+            } catch (IOException e) {
+                throw new UncheckedIOException(e);
+            }
+        }
+        return credentials;
+    }
 }
 }

+ 5 - 1
plugins/repository-gcs/src/test/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageBlobStoreRepositoryTests.java

@@ -67,6 +67,9 @@ public class GoogleCloudStorageBlobStoreRepositoryTests extends ESBlobStoreRepos
     }
     }
 
 
     public static class MockGoogleCloudStoragePlugin extends GoogleCloudStoragePlugin {
     public static class MockGoogleCloudStoragePlugin extends GoogleCloudStoragePlugin {
+        public MockGoogleCloudStoragePlugin() {
+            super(Settings.EMPTY);
+        }
         @Override
         @Override
         protected GoogleCloudStorageService createStorageService(Environment environment) {
         protected GoogleCloudStorageService createStorageService(Environment environment) {
             return new MockGoogleCloudStorageService();
             return new MockGoogleCloudStorageService();
@@ -75,7 +78,8 @@ public class GoogleCloudStorageBlobStoreRepositoryTests extends ESBlobStoreRepos
 
 
     public static class MockGoogleCloudStorageService implements GoogleCloudStorageService {
     public static class MockGoogleCloudStorageService implements GoogleCloudStorageService {
         @Override
         @Override
-        public Storage createClient(String serviceAccount, String application, TimeValue connectTimeout, TimeValue readTimeout) throws
+        public Storage createClient(String serviceAccount, String accountName, String application,
+                                    TimeValue connectTimeout, TimeValue readTimeout) throws
                 Exception {
                 Exception {
             return storage.get();
             return storage.get();
         }
         }

+ 85 - 0
plugins/repository-gcs/src/test/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageServiceTests.java

@@ -0,0 +1,85 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.repositories.gcs;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Map;
+
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.env.Environment;
+import org.elasticsearch.repositories.gcs.GoogleCloudStorageService.InternalGoogleCloudStorageService;
+import org.elasticsearch.test.ESTestCase;
+
+import static org.hamcrest.Matchers.containsString;
+
+public class GoogleCloudStorageServiceTests extends ESTestCase {
+
+    private InputStream getDummyCredentialStream() throws IOException {
+        return GoogleCloudStorageServiceTests.class.getResourceAsStream("/dummy-account.json");
+    }
+
+    public void testDefaultCredential() throws Exception {
+        Environment env = new Environment(Settings.builder().put("path.home", createTempDir()).build());
+        GoogleCredential cred = GoogleCredential.fromStream(getDummyCredentialStream());
+        InternalGoogleCloudStorageService service = new InternalGoogleCloudStorageService(env, Collections.emptyMap()) {
+            @Override
+            GoogleCredential getDefaultCredential() throws IOException {
+                return cred;
+            }
+        };
+        assertSame(cred, service.getCredential("_default_", "default"));
+    }
+
+    public void testFileCredentialBackcompat() throws Exception {
+        Path home = createTempDir();
+        Path config = home.resolve("config");
+        Files.createDirectories(config);
+        Settings settings = Settings.builder()
+            .put("path.home", home).build();
+        Environment env = new Environment(settings);
+        Files.copy(getDummyCredentialStream(), config.resolve("test-cred.json"));
+        InternalGoogleCloudStorageService service = new InternalGoogleCloudStorageService(env, Collections.emptyMap());
+        GoogleCredential cred = service.getCredential("test-cred.json", "default");
+        assertEquals("some-project-name@appspot.gserviceaccount.com", cred.getServiceAccountId());
+        assertWarnings("Using GCS service account file from disk is deprecated. Move the file into the elasticsearch keystore.");
+    }
+
+    public void testFileCredentialMissing() throws Exception {
+        Environment env = new Environment(Settings.builder().put("path.home", createTempDir()).build());
+        InternalGoogleCloudStorageService service = new InternalGoogleCloudStorageService(env, Collections.emptyMap());
+        IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () ->
+            service.getCredential("test-cred.json", "default"));
+        assertThat(e.getMessage(), containsString("Unable to find service account file"));
+        assertWarnings("Using GCS service account file from disk is deprecated. Move the file into the elasticsearch keystore.");
+    }
+
+    public void testClientCredential() throws Exception {
+        GoogleCredential cred = GoogleCredential.fromStream(getDummyCredentialStream());
+        Map<String, GoogleCredential> credentials = Collections.singletonMap("clientname", cred);
+        Environment env = new Environment(Settings.builder().put("path.home", createTempDir()).build());
+        InternalGoogleCloudStorageService service = new InternalGoogleCloudStorageService(env, credentials);
+        assertSame(cred, service.getCredential("_default_", "clientname"));
+    }
+}

+ 12 - 0
plugins/repository-gcs/src/test/resources/dummy-account.json

@@ -0,0 +1,12 @@
+{
+  "type": "service_account",
+  "project_id": "some-project-name",
+  "private_key_id": "c7cefcb7c72a2880ecce49cb9d1095de5a61aff0",
+  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDa+7r0RE1YykXC\n+d+DXlN3Dg3aL1YOfYuhy5PF/Vi0FFQHyXuPtAvkVHZD2NxMDZq2DxTu3AVLh1UE\nt2hMrWjDQDuArPl8FezpyYQwde04Qlx1YpQ1xUjTaFWd0hrOZEfsxY00h3ilxR3G\nJsofR3PZBKYI11VGruNemCgjiJg5hcoJDxLXUgcfpKJaiPeHutczCeZ1RANQwQF1\n/dPXhqbtWiaS/iu5so64P54TsrVX5DcXmbGr6hQAReIcI6cjA8QhSu6QBtdvEPhv\n27uTuSu4XRtTh3djVGFzFV9pamGZeGELkTiHVSDI8IkQ32s8yuP5Zys/4bFJk7nn\niqJpe0/DAgMBAAECggEAEexQnPWKLx4/H3o8JRBvXGs2DwmYzY7RAukaqzXVMMgJ\nKKoBBv4Biyquk1cIkOD8LLKHUBWKCWiGOOCaFMyMqo5zUFDYCqPwxCHOQ/ki9VvZ\nHXJ4Fv6Su1rqxwQPVZ03ldWFfSspYMgFa9Z47J54iOasgES/og1mZrOldWMUsoBu\nCKf0fH+vIsxWPwmRtyxKCMwqenqdc22nGGLhmpm8tuw1eQp6XtTXagqkPtAVMMga\nmgC0EGqhZA/IklGW1JuGWELjXVMgS/tLIPq+hYsmY14y6Ie032YoSMWkz6Z5p7i0\n/JwCzVZNO1mD0MwVj7nDmokXOpoyM7Qcbx8r1E4Q4QKBgQDxqAZ6D+A671mCNU0J\n6Qzc3cOZq7MBj4y7M/2qPXHC7i/DdbmnM7PPPriaBBch2nX7jZRlRmVDBsmrC4OG\n3m5+HAx7YPVbefwe6h5ki5O9wg1pLcgYY9uvgLSlD85lVZKAzO7QK2W5zfM19kPD\nSckIa+U7DKFbwKhtCsxcP6ARJwKBgQDn+zAPHepGP2Zf78pOLlVXcQqVA1uOu+bW\nrG4G+lPrytB0C4QdwWdBV3Lcqmnw/ae5PkQBs0dCbtWG8+MT8gA6k5kleflaZrAY\ngdUJIUP6J7ocWYxVTfqGFyFF1n5VT8/jbVucaT7izBZfZvlGyf7Vz7ewQzgWQWlK\nCQ0qstV2BQKBgQCajAQAYlDcQCC1dlMbqHDye91RVQ65S84MF1b+Xid4LA5d6dde\nyGERhKJY1Y7ZtrZHt6cVEe1G7XtiKY3nXi+59URCT6L66svEFaR0VxOYgxdCkeXr\nO0nPNvfQrIgqJIz6VJXSij6XktAdTa7OoUyxVxeWKSC05kSQ4BwMTyCWdwKBgQCW\noqlmZ4qE6w5TJaY8diG8kg7JDFEbsjAHHhikN1DfP+d0MzYrDDc8WsifOZlpf4y1\n4RTP9dZD8Sx+YUgG35H+d3FuwHGGnj+i6kunjg5SFhHn7s4NZoFTKRnV+541T4oy\nqARg4IaRRu0QLhGYQfpUZHlm339AFGGGTbJbE51A8QKBgQDTEN5O+3bRG3Fa1J6z\nU9PMrjjs6l8xhXFso10YEYG5KRnfhzCFujyWNiLE6WrlUL8invVBaCxsZr51GDgA\nhyEEdm4kXCRrv4JyhOvIuGxNcAIiQK/e91UQEM6u1t6hUI1rE7ZOyJQzBxj9hFlV\n7OvhBlHXQUtAOdq0XLHr9GzdSA==\n-----END PRIVATE KEY-----\n",
+  "client_email": "some-project-name@appspot.gserviceaccount.com",
+  "client_id": "123456789101112130594",
+  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+  "token_uri": "https://accounts.google.com/o/oauth2/token",
+  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/some-project-name%40appspot.gserviceaccount.com"
+}