|
|
@@ -6,19 +6,22 @@
|
|
|
|
|
|
preview::[]
|
|
|
|
|
|
+Updates the draft `filtering` configuration of a connector and marks the draft validation state as `edited`. The filtering configuration can be activated once validated by the Elastic connector service.
|
|
|
|
|
|
-Updates the `filtering` configuration of a connector. Learn more about filtering in the {enterprise-search-ref}/sync-rules.html[sync rules] documentation.
|
|
|
+The filtering property is used to configure sync rules (both basic and advanced) for a connector. Learn more in the {enterprise-search-ref}/sync-rules.html[sync rules documentation].
|
|
|
|
|
|
[[update-connector-filtering-api-request]]
|
|
|
==== {api-request-title}
|
|
|
|
|
|
`PUT _connector/<connector_id>/_filtering`
|
|
|
+`PUT _connector/<connector_id>/_filtering/_activate`
|
|
|
|
|
|
[[update-connector-filtering-api-prereq]]
|
|
|
==== {api-prereq-title}
|
|
|
|
|
|
* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
|
|
|
* The `connector_id` parameter should reference an existing connector.
|
|
|
+* To activate filtering rules, the `draft.validation.state` must be `valid`.
|
|
|
|
|
|
[[update-connector-filtering-api-path-params]]
|
|
|
==== {api-path-parms-title}
|
|
|
@@ -30,65 +33,42 @@ Updates the `filtering` configuration of a connector. Learn more about filtering
|
|
|
[[update-connector-filtering-api-request-body]]
|
|
|
==== {api-request-body-title}
|
|
|
|
|
|
-`filtering`::
|
|
|
-(Required, array) The filtering configuration for the connector. This configuration determines the set of rules applied for filtering data during syncs.
|
|
|
-
|
|
|
-Each entry in the `filtering` array represents a set of filtering rules for a specific data domain and includes the following attributes:
|
|
|
-
|
|
|
-- `domain` (Required, string) +
|
|
|
-Specifies the data domain to which these filtering rules apply.
|
|
|
-
|
|
|
-- `active` (Required, object) +
|
|
|
-Contains the set of rules that are actively used for sync jobs. The `active` object includes:
|
|
|
-
|
|
|
- * `rules` (Required, array of objects) +
|
|
|
- An array of individual filtering rule objects, each with the following sub-attributes:
|
|
|
- ** `id` (Required, string) +
|
|
|
- A unique identifier for the rule.
|
|
|
- ** `policy` (Required, string) +
|
|
|
- Specifies the policy, such as "include" or "exclude".
|
|
|
- ** `field` (Required, string) +
|
|
|
- The field in the document to which this rule applies.
|
|
|
- ** `rule` (Required, string) +
|
|
|
- The type of rule, such as "regex", "starts_with", "ends_with", "contains", "equals", "<", ">", etc.
|
|
|
- ** `value` (Required, string) +
|
|
|
- The value to be used in conjunction with the rule for matching the contents of the document's field.
|
|
|
- ** `order` (Required, number) +
|
|
|
- The order in which the rules are applied. The first rule to match has its policy applied.
|
|
|
- ** `created_at` (Required, datetime) +
|
|
|
- The timestamp when the rule was added.
|
|
|
- ** `updated_at` (Required, datetime) +
|
|
|
- The timestamp when the rule was last edited.
|
|
|
-
|
|
|
- * `advanced_snippet` (Required, object) +
|
|
|
- Used for {enterprise-search-ref}/sync-rules.html#sync-rules-advanced[advanced filtering] at query time, with the following sub-attributes:
|
|
|
- ** `value` (Required, object) +
|
|
|
- A JSON object passed directly to the connector for advanced filtering.
|
|
|
- ** `created_at` (Required, datetime) +
|
|
|
- The timestamp when this JSON object was created.
|
|
|
- ** `updated_at` (Required, datetime) +
|
|
|
- The timestamp when this JSON object was last edited.
|
|
|
-
|
|
|
- * `validation` (Required, object) +
|
|
|
- Provides validation status for the rules, including:
|
|
|
- ** `state` (Required, string) +
|
|
|
- Indicates the validation state: "edited", "valid", or "invalid".
|
|
|
- ** `errors` (Required, object) +
|
|
|
- Contains details about any validation errors, with sub-attributes:
|
|
|
- *** `ids` (Required, string) +
|
|
|
- The ID(s) of any rules deemed invalid.
|
|
|
- *** `messages` (Required, string) +
|
|
|
- Messages explaining what is invalid about the rules.
|
|
|
-
|
|
|
-- `draft` (Required, object) +
|
|
|
-An object identical in structure to the `active` object, but used for drafting and editing filtering rules before they become active.
|
|
|
+`rules`::
|
|
|
+(Optional, array of objects)
|
|
|
+An array of {enterprise-search-ref}/sync-rules.html#sync-rules-basic[basic sync rules], each with the following sub-attributes:
|
|
|
+* `id` (Required, string) +
|
|
|
+A unique identifier for the rule.
|
|
|
+* `policy` (Required, string) +
|
|
|
+Specifies the policy, such as `include` or `exclude`.
|
|
|
+* `field` (Required, string) +
|
|
|
+The field in the document to which this rule applies.
|
|
|
+* `rule` (Required, string) +
|
|
|
+The type of rule, such as `regex`, `starts_with`, `ends_with`, `contains`, `equals`, `<`, `>`, etc.
|
|
|
+* `value` (Required, string) +
|
|
|
+The value to be used in conjunction with the rule for matching the contents of the document's field.
|
|
|
+* `order` (Required, number) +
|
|
|
+The order in which the rules are applied. The first rule to match has its policy applied.
|
|
|
+* `created_at` (Optional, datetime) +
|
|
|
+The timestamp when the rule was added. Defaults to `now` UTC timestamp.
|
|
|
+* `updated_at` (Optional, datetime) +
|
|
|
+The timestamp when the rule was last edited. Defaults to `now` UTC timestamp.
|
|
|
+
|
|
|
+`advanced_snippet`::
|
|
|
+(Optional, object)
|
|
|
+Used for {enterprise-search-ref}/sync-rules.html#sync-rules-advanced[advanced filtering] at query time, with the following sub-attributes:
|
|
|
+* `value` (Required, object or array) +
|
|
|
+A JSON object/array passed directly to the connector for advanced filtering.
|
|
|
+* `created_at` (Optional, datetime) +
|
|
|
+The timestamp when this JSON object was created. Defaults to `now` UTC timestamp.
|
|
|
+* `updated_at` (Optional, datetime) +
|
|
|
+The timestamp when this JSON object was last edited. Defaults to `now` UTC timestamp.
|
|
|
|
|
|
|
|
|
[[update-connector-filtering-api-response-codes]]
|
|
|
==== {api-response-codes-title}
|
|
|
|
|
|
`200`::
|
|
|
-Connector `filtering` field was successfully updated.
|
|
|
+Connector draft filtering was successfully updated.
|
|
|
|
|
|
`400`::
|
|
|
The `connector_id` was not provided or the request payload was malformed.
|
|
|
@@ -99,80 +79,56 @@ No connector matching `connector_id` could be found.
|
|
|
[[update-connector-filtering-api-example]]
|
|
|
==== {api-examples-title}
|
|
|
|
|
|
-The following example updates the `filtering` property for the connector with ID `my-connector`:
|
|
|
+The following example updates the draft {enterprise-search-ref}/sync-rules.html#sync-rules-basic[basic sync rules] for a Google Drive connector with ID `my-g-drive-connector`. All Google Drive files with `.txt` extension will be skipped:
|
|
|
|
|
|
////
|
|
|
[source, console]
|
|
|
--------------------------------------------------
|
|
|
-PUT _connector/my-connector
|
|
|
+PUT _connector/my-g-drive-connector
|
|
|
{
|
|
|
"index_name": "search-google-drive",
|
|
|
"name": "My Connector",
|
|
|
"service_type": "google_drive"
|
|
|
}
|
|
|
+
|
|
|
+PUT _connector/my-sql-connector
|
|
|
+{
|
|
|
+ "index_name": "search-sql",
|
|
|
+ "name": "My SQL Connector",
|
|
|
+ "service_type": "google_drive"
|
|
|
+}
|
|
|
+
|
|
|
--------------------------------------------------
|
|
|
// TESTSETUP
|
|
|
|
|
|
[source,console]
|
|
|
--------------------------------------------------
|
|
|
-DELETE _connector/my-connector
|
|
|
+DELETE _connector/my-g-drive-connector
|
|
|
+DELETE _connector/my-sql-connector
|
|
|
--------------------------------------------------
|
|
|
// TEARDOWN
|
|
|
////
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
|
-PUT _connector/my-connector/_filtering
|
|
|
+PUT _connector/my-g-drive-connector/_filtering
|
|
|
{
|
|
|
- "filtering": [
|
|
|
+ "rules": [
|
|
|
+ {
|
|
|
+ "field": "file_extension",
|
|
|
+ "id": "exclude-txt-files",
|
|
|
+ "order": 0,
|
|
|
+ "policy": "exclude",
|
|
|
+ "rule": "equals",
|
|
|
+ "value": "txt"
|
|
|
+ },
|
|
|
{
|
|
|
- "active": {
|
|
|
- "advanced_snippet": {
|
|
|
- "created_at": "2023-11-09T15:13:08.231Z",
|
|
|
- "updated_at": "2023-11-09T15:13:08.231Z",
|
|
|
- "value": {}
|
|
|
- },
|
|
|
- "rules": [
|
|
|
- {
|
|
|
- "created_at": "2023-11-09T15:13:08.231Z",
|
|
|
- "field": "_",
|
|
|
- "id": "DEFAULT",
|
|
|
- "order": 0,
|
|
|
- "policy": "include",
|
|
|
- "rule": "regex",
|
|
|
- "updated_at": "2023-11-09T15:13:08.231Z",
|
|
|
- "value": ".*"
|
|
|
- }
|
|
|
- ],
|
|
|
- "validation": {
|
|
|
- "errors": [],
|
|
|
- "state": "valid"
|
|
|
- }
|
|
|
- },
|
|
|
- "domain": "DEFAULT",
|
|
|
- "draft": {
|
|
|
- "advanced_snippet": {
|
|
|
- "created_at": "2023-11-09T15:13:08.231Z",
|
|
|
- "updated_at": "2023-11-09T15:13:08.231Z",
|
|
|
- "value": {}
|
|
|
- },
|
|
|
- "rules": [
|
|
|
- {
|
|
|
- "created_at": "2023-11-09T15:13:08.231Z",
|
|
|
- "field": "_",
|
|
|
- "id": "DEFAULT",
|
|
|
- "order": 0,
|
|
|
- "policy": "include",
|
|
|
- "rule": "regex",
|
|
|
- "updated_at": "2023-11-09T15:13:08.231Z",
|
|
|
- "value": ".*"
|
|
|
- }
|
|
|
- ],
|
|
|
- "validation": {
|
|
|
- "errors": [],
|
|
|
- "state": "valid"
|
|
|
- }
|
|
|
- }
|
|
|
+ "field": "_",
|
|
|
+ "id": "DEFAULT",
|
|
|
+ "order": 1,
|
|
|
+ "policy": "include",
|
|
|
+ "rule": "regex",
|
|
|
+ "value": ".*"
|
|
|
}
|
|
|
]
|
|
|
}
|
|
|
@@ -184,3 +140,64 @@ PUT _connector/my-connector/_filtering
|
|
|
"result": "updated"
|
|
|
}
|
|
|
----
|
|
|
+
|
|
|
+The following example updates the draft advanced sync rules for a MySQL connector with id `my-sql-connector`. Advanced sync rules are specific to each connector type. Refer to the references for connectors that support {enterprise-search-ref}/sync-rules.html#sync-rules-advanced[advanced sync rules] for syntax and examples.
|
|
|
+
|
|
|
+[source,console]
|
|
|
+----
|
|
|
+PUT _connector/my-sql-connector/_filtering
|
|
|
+{
|
|
|
+ "advanced_snippet": {
|
|
|
+ "value": [{
|
|
|
+ "tables": [
|
|
|
+ "users",
|
|
|
+ "orders"
|
|
|
+ ],
|
|
|
+ "query": "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id"
|
|
|
+ }]
|
|
|
+ }
|
|
|
+}
|
|
|
+----
|
|
|
+
|
|
|
+[source,console-result]
|
|
|
+----
|
|
|
+{
|
|
|
+ "result": "updated"
|
|
|
+}
|
|
|
+----
|
|
|
+
|
|
|
+
|
|
|
+////
|
|
|
+[source, console]
|
|
|
+--------------------------------------------------
|
|
|
+PUT _connector/my-sql-connector/_filtering/_validation
|
|
|
+{
|
|
|
+ "validation": {
|
|
|
+ "state": "valid",
|
|
|
+ "errors": []
|
|
|
+ }
|
|
|
+}
|
|
|
+--------------------------------------------------
|
|
|
+// TEST[continued]
|
|
|
+////
|
|
|
+
|
|
|
+
|
|
|
+Note, you can also update draft `rules` and `advanced_snippet` in a single request.
|
|
|
+
|
|
|
+Once the draft is updated, its validation state is set to `edited`. The connector service will then validate the rules and report the validation state as either `invalid` or `valid`. If the state is `valid`, the draft filtering can be activated with:
|
|
|
+
|
|
|
+
|
|
|
+[source,console]
|
|
|
+----
|
|
|
+PUT _connector/my-sql-connector/_filtering/_activate
|
|
|
+----
|
|
|
+// TEST[continued]
|
|
|
+
|
|
|
+[source,console-result]
|
|
|
+----
|
|
|
+{
|
|
|
+ "result": "updated"
|
|
|
+}
|
|
|
+----
|
|
|
+
|
|
|
+Once filtering rules are activated, they will be applied to all subsequent full or incremental syncs.
|
Jedr Blaszyk