ecommerce-tutorial.asciidoc 7.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270
  1. [role="xpack"]
  2. [testenv="basic"]
  3. [[ecommerce-transforms]]
  4. === Tutorial: Transforming the eCommerce sample data
  5. <<transforms,{transforms-cap}>> enable you to retrieve information
  6. from an {es} index, transform it, and store it in another index. Let's use the
  7. {kibana-ref}/add-sample-data.html[{kib} sample data] to demonstrate how you can
  8. pivot and summarize your data with {transforms}.
  9. . If the {es} {security-features} are enabled, obtain a user ID with sufficient
  10. privileges to complete these steps.
  11. +
  12. --
  13. You need `manage_transform` cluster privileges to preview and create
  14. {transforms}. Members of the built-in `transform_admin` role have these
  15. privileges.
  16. You also need `read` and `view_index_metadata` index privileges on the source
  17. index and `read`, `create_index`, and `index` privileges on the destination
  18. index.
  19. For more information, see <<security-privileges>> and <<built-in-roles>>.
  20. --
  21. . Choose your _source index_.
  22. +
  23. --
  24. In this example, we'll use the eCommerce orders sample data. If you're not
  25. already familiar with the `kibana_sample_data_ecommerce` index, use the
  26. *Revenue* dashboard in {kib} to explore the data. Consider what insights you
  27. might want to derive from this eCommerce data.
  28. --
  29. . Play with various options for grouping and aggregating the data.
  30. +
  31. --
  32. _Pivoting_ your data involves using at least one field to group it and applying
  33. at least one aggregation. You can preview what the transformed data will look
  34. like, so go ahead and play with it!
  35. For example, you might want to group the data by product ID and calculate the
  36. total number of sales for each product and its average price. Alternatively, you
  37. might want to look at the behavior of individual customers and calculate how
  38. much each customer spent in total and how many different categories of products
  39. they purchased. Or you might want to take the currencies or geographies into
  40. consideration. What are the most interesting ways you can transform and
  41. interpret this data?
  42. Go to *Management* > *Elasticsearch* > *Transforms* in {kib} and use the
  43. wizard to create a {transform}:
  44. [role="screenshot"]
  45. image::images/ecommerce-pivot1.jpg["Creating a simple {transform} in {kib}"]
  46. In this case, we grouped the data by customer ID and calculated the sum of
  47. products each customer purchased.
  48. Let's add some more aggregations to learn more about our customers' orders. For
  49. example, let's calculate the total sum of their purchases, the maximum number of
  50. products that they purchased in a single order, and their total number of orders.
  51. We'll accomplish this by using the
  52. <<search-aggregations-metrics-sum-aggregation,`sum` aggregation>> on the
  53. `taxless_total_price` field, the
  54. <<search-aggregations-metrics-max-aggregation,`max` aggregation>> on the
  55. `total_quantity` field, and the
  56. <<search-aggregations-metrics-cardinality-aggregation,`cardinality` aggregation>>
  57. on the `order_id` field:
  58. [role="screenshot"]
  59. image::images/ecommerce-pivot2.jpg["Adding multiple aggregations to a {transform} in {kib}"]
  60. TIP: If you're interested in a subset of the data, you can optionally include a
  61. <<request-body-search-query,query>> element. In this
  62. example, we've filtered the data so that we're only looking at orders with a
  63. `currency` of `EUR`. Alternatively, we could group the data by that field too.
  64. If you want to use more complex queries, you can create your {dataframe} from a
  65. {kibana-ref}/save-open-search.html[saved search].
  66. If you prefer, you can use the
  67. <<preview-transform,preview {transforms} API>>.
  68. .API example
  69. [%collapsible]
  70. ====
  71. [source,console]
  72. --------------------------------------------------
  73. POST _transform/_preview
  74. {
  75. "source": {
  76. "index": "kibana_sample_data_ecommerce",
  77. "query": {
  78. "bool": {
  79. "filter": {
  80. "term": {"currency": "EUR"}
  81. }
  82. }
  83. }
  84. },
  85. "pivot": {
  86. "group_by": {
  87. "customer_id": {
  88. "terms": {
  89. "field": "customer_id"
  90. }
  91. }
  92. },
  93. "aggregations": {
  94. "total_quantity.sum": {
  95. "sum": {
  96. "field": "total_quantity"
  97. }
  98. },
  99. "taxless_total_price.sum": {
  100. "sum": {
  101. "field": "taxless_total_price"
  102. }
  103. },
  104. "total_quantity.max": {
  105. "max": {
  106. "field": "total_quantity"
  107. }
  108. },
  109. "order_id.cardinality": {
  110. "cardinality": {
  111. "field": "order_id"
  112. }
  113. }
  114. }
  115. }
  116. }
  117. --------------------------------------------------
  118. // TEST[skip:set up sample data]
  119. ====
  120. --
  121. . When you are satisfied with what you see in the preview, create the
  122. {transform}.
  123. +
  124. --
  125. .. Supply a job ID and the name of the target (or _destination_) index. If the
  126. target index does not exist, it will be created automatically.
  127. .. Decide whether you want the {transform} to run once or continuously.
  128. --
  129. +
  130. --
  131. Since this sample data index is unchanging, let's use the default behavior and
  132. just run the {transform} once.
  133. If you want to try it out, however, go ahead and click on *Continuous mode*.
  134. You must choose a field that the {transform} can use to check which
  135. entities have changed. In general, it's a good idea to use the ingest timestamp
  136. field. In this example, however, you can use the `order_date` field.
  137. If you prefer, you can use the
  138. <<put-transform,create {transforms} API>>.
  139. .API example
  140. [%collapsible]
  141. ====
  142. [source,console]
  143. --------------------------------------------------
  144. PUT _transform/ecommerce-customer-transform
  145. {
  146. "source": {
  147. "index": [
  148. "kibana_sample_data_ecommerce"
  149. ],
  150. "query": {
  151. "bool": {
  152. "filter": {
  153. "term": {
  154. "currency": "EUR"
  155. }
  156. }
  157. }
  158. }
  159. },
  160. "pivot": {
  161. "group_by": {
  162. "customer_id": {
  163. "terms": {
  164. "field": "customer_id"
  165. }
  166. }
  167. },
  168. "aggregations": {
  169. "total_quantity.sum": {
  170. "sum": {
  171. "field": "total_quantity"
  172. }
  173. },
  174. "taxless_total_price.sum": {
  175. "sum": {
  176. "field": "taxless_total_price"
  177. }
  178. },
  179. "total_quantity.max": {
  180. "max": {
  181. "field": "total_quantity"
  182. }
  183. },
  184. "order_id.cardinality": {
  185. "cardinality": {
  186. "field": "order_id"
  187. }
  188. }
  189. }
  190. },
  191. "dest": {
  192. "index": "ecommerce-customers"
  193. }
  194. }
  195. --------------------------------------------------
  196. // TEST[skip:setup kibana sample data]
  197. ====
  198. --
  199. . Start the {transform}.
  200. +
  201. --
  202. TIP: Even though resource utilization is automatically adjusted based on the
  203. cluster load, a {transform} increases search and indexing load on your
  204. cluster while it runs. If you're experiencing an excessive load, however, you
  205. can stop it.
  206. You can start, stop, and manage {transforms} in {kib}:
  207. [role="screenshot"]
  208. image::images/manage-transforms.jpg["Managing {transforms} in {kib}"]
  209. Alternatively, you can use the
  210. <<start-transform,start {transforms}>> and
  211. <<stop-transform,stop {transforms}>> APIs.
  212. .API example
  213. [%collapsible]
  214. ====
  215. [source,console]
  216. --------------------------------------------------
  217. POST _transform/ecommerce-customer-transform/_start
  218. --------------------------------------------------
  219. // TEST[skip:setup kibana sample data]
  220. ====
  221. TIP: If you chose a batch {transform}, it is a single operation that has a
  222. single checkpoint. You cannot restart it when it's complete. {ctransforms-cap}
  223. differ in that they continually increment and process checkpoints as new source
  224. data is ingested.
  225. --
  226. . Explore the data in your new index.
  227. +
  228. --
  229. For example, use the *Discover* application in {kib}:
  230. [role="screenshot"]
  231. image::images/ecommerce-results.jpg["Exploring the new index in {kib}"]
  232. --
  233. TIP: If you do not want to keep the {transform}, you can delete it in
  234. {kib} or use the
  235. <<delete-transform,delete {transform} API>>. When
  236. you delete a {transform}, its destination index and {kib} index
  237. patterns remain.