Elasticsearch：使用 ILM 示例运行降采样 (downsampling)

news2024/11/23 12:37:10

如果你对降采样还不是很熟的话，请阅读之前的文章 “Elasticsearch：对时间序列数据流进行降采样（downsampling)”。这是一个简化的示例，可让你快速了解降采样如何作为 ILM 策略的一部分来减少一组采样指标的存储大小。 该示例使用典型的 Kubernetes 集群监控数据。 要使用 ILM 测试降采样，请执行以下步骤：

  • 检查先决条件。
  • 创建索引生命周期策略。
  • 创建索引模板。
  • 摄取时间序列数据。
  • 查看结果。

以下展示是使用最新的 Elastic Stack 8.11 来进行的。

先决条件

  • 请参阅时间序列数据流先决条件。
    • 集群权限：manage_ilm 和 manage_index_templates。
    • 索引权限：你创建或转换的任何 TSDS 的 create_doc 和 create_index。 要滚动 TSDS，你必须具有 manage 权限。
  • 在运行此示例之前，你可能需要尝试手动运行降采样示例。

创建索引生命周期策略

为你的时间序列数据创建 ILM 策略。 虽然不是必需的，但建议使用 ILM 策略来自动管理时间序列数据流索引。

要启用降采样，请添加 Downsample 操作并将 fixed_interval 设置为要聚合原始时间序列数据的降采样间隔。

本例中为热阶段配置了 ILM 策略。 降采样发生在索引滚动并且索引时间序列结束时间（index.time_series.end_time）已过之后，因为在此之前源索引仍有望接收主要写入。 在索引结束时间过去之前，索引生命周期管理不会继续执行任何期望索引不再接收写入的操作。 在继续等待结束时间之前的索引生命周期管理操作包括： - 删除 (Delete) - 降采样 (Downsample) - 强制合并 (Force merge) - 只读 (Read only) - 可搜索快照 (Searchable snapshot) - 收缩 (Shrink)

PUT _ilm/policy/datastream_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_age": "5m"
          },
          "downsample": {
  	        "fixed_interval": "1h"
  	      }
        }
      }
    }
  }
}

创建索引模板

这将为基本数据流创建索引模板。 设置时间序列数据流中详细描述了索引模板的可用参数。

为了简单起见，在时间序列映射中，所有 time_series_metric 参数都设置为 gauge 类型，但也可以使 counter 指标类型。 time_series_metric 值确定降采样期间使用的统计表示的类型。

索引模板包含一组静态时间序列维度：主机 (host)、命名空间 (namespace)、节点 (node) 和 Pod。 时间序列维度不会因降采样过程而改变。

PUT _index_template/datastream_template
{
  "index_patterns": [
    "datastream*"
  ],
  "data_stream": {},
  "template": {
    "settings": {
      "index": {
        "mode": "time_series",
        "number_of_replicas": 0,
        "number_of_shards": 2
      },
      "index.lifecycle.name": "datastream_policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "kubernetes": {
          "properties": {
            "container": {
              "properties": {
                "cpu": {
                  "properties": {
                    "usage": {
                      "properties": {
                        "core": {
                          "properties": {
                            "ns": {
                              "type": "long"
                            }
                          }
                        },
                        "limit": {
                          "properties": {
                            "pct": {
                              "type": "float"
                            }
                          }
                        },
                        "nanocores": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        },
                        "node": {
                          "properties": {
                            "pct": {
                              "type": "float"
                            }
                          }
                        }
                      }
                    }
                  }
                },
                "memory": {
                  "properties": {
                    "available": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        }
                      }
                    },
                    "majorpagefaults": {
                      "type": "long"
                    },
                    "pagefaults": {
                      "type": "long",
                      "time_series_metric": "gauge"
                    },
                    "rss": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        }
                      }
                    },
                    "usage": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        },
                        "limit": {
                          "properties": {
                            "pct": {
                              "type": "float"
                            }
                          }
                        },
                        "node": {
                          "properties": {
                            "pct": {
                              "type": "float"
                            }
                          }
                        }
                      }
                    },
                    "workingset": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        }
                      }
                    }
                  }
                },
                "name": {
                  "type": "keyword"
                },
                "start_time": {
                  "type": "date"
                }
              }
            },
            "host": {
              "type": "keyword",
              "time_series_dimension": true
            },
            "namespace": {
              "type": "keyword",
              "time_series_dimension": true
            },
            "node": {
              "type": "keyword",
              "time_series_dimension": true
            },
            "pod": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        }
      }
    }
  }
}

摄取时间序列数据

使用 bulk API 请求自动创建 TSDS 并为一组 10 个文档编制索引。

重要提示：在运行此批量请求之前，你需要将时间戳更新为当前时间后三到五个小时内。 也就是说，搜索 2022-06-21T15 并替换为你当前的日期，并将小时调整为你当前的时间加三个小时。

PUT /datastream/_bulk?refresh
{"create": {}}
{"@timestamp":"2022-06-21T15:49:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":91153,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":463314616},"usage":{"bytes":307007078,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":585236},"rss":{"bytes":102728},"pagefaults":120901,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:45:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":124501,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":982546514},"usage":{"bytes":360035574,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1339884},"rss":{"bytes":381174},"pagefaults":178473,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":38907,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":862723768},"usage":{"bytes":379572388,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":431227},"rss":{"bytes":386580},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":86706,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":103266017,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1724908},"rss":{"bytes":105431},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":150069,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":639054643},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1786511},"rss":{"bytes":189235},"pagefaults":138172,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:42:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":82260,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":854735585},"usage":{"bytes":309798052,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":924058},"rss":{"bytes":110838},"pagefaults":259073,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:42:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":153404,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":279586406},"usage":{"bytes":214904955,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1047265},"rss":{"bytes":91914},"pagefaults":302252,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:40:20Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":125613,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":822782853},"usage":{"bytes":100475044,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2109932},"rss":{"bytes":278446},"pagefaults":74843,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:40:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":100046,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":362826547,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1986724},"rss":{"bytes":402801},"pagefaults":296495,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:38:30Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":40018,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":1062428344},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2294743},"rss":{"bytes":340623},"pagefaults":224530,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}

针对我的情况：

PUT /datastream/_bulk?refresh
{"create": {}}
{"@timestamp":"2023-11-30T06:49:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":91153,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":463314616},"usage":{"bytes":307007078,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":585236},"rss":{"bytes":102728},"pagefaults":120901,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:45:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":124501,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":982546514},"usage":{"bytes":360035574,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1339884},"rss":{"bytes":381174},"pagefaults":178473,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":38907,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":862723768},"usage":{"bytes":379572388,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":431227},"rss":{"bytes":386580},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":86706,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":103266017,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1724908},"rss":{"bytes":105431},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":150069,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":639054643},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1786511},"rss":{"bytes":189235},"pagefaults":138172,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:42:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":82260,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":854735585},"usage":{"bytes":309798052,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":924058},"rss":{"bytes":110838},"pagefaults":259073,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:42:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":153404,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":279586406},"usage":{"bytes":214904955,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1047265},"rss":{"bytes":91914},"pagefaults":302252,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:40:20Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":125613,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":822782853},"usage":{"bytes":100475044,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2109932},"rss":{"bytes":278446},"pagefaults":74843,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:40:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":100046,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":362826547,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1986724},"rss":{"bytes":402801},"pagefaults":296495,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:38:30Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":40018,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":1062428344},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2294743},"rss":{"bytes":340623},"pagefaults":224530,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}

查看结果

现在你已创建文档并将其添加到数据流中，请检查以确认新索引的当前状态。

GET _data_stream

如果尚未应用 ILM 策略，你的结果将如下所示。 请注意原始索引名称：.ds-datastream-<timestamp>-000001。

{
  "data_streams": [
    {
      "name": "datastream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-datastream-2023.11.30-000001",
          "index_uuid": "muM9y4_ORDK1wLY-t_UtnA",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 1,
      "status": "GREEN",
      "template": "datastream_template",
      "ilm_policy": "datastream_policy",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T03:21:48.000Z",
            "end": "2023-11-30T07:27:32.000Z"
          }
        ]
      }
    },
    {
      "name": "kibana_sample_data_logs",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-kibana_sample_data_logs-2023.11.21-000001",
          "index_uuid": "JkbH5-8uTvyeZL6icuHN1A",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        }
      ],
      "generation": 1,
      "status": "YELLOW",
      "template": "kibana_sample_data_logs",
      "next_generation_managed_by": "Unmanaged",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false
    },
    {
      "name": "my-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-my-data-stream-2023.11.30-000001-downsample",
          "index_uuid": "KXtWAQCBRlW0sWYCVUu7Fw",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        },
        {
          "index_name": ".ds-my-data-stream-2023.11.30-000002",
          "index_uuid": "yBDLixmpRDahmS4G5_uxQw",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        }
      ],
      "generation": 4,
      "status": "GREEN",
      "template": "my-data-stream-template",
      "next_generation_managed_by": "Unmanaged",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T02:24:20.000Z",
            "end": "2023-11-30T08:32:32.000Z"
          }
        ]
      }
    }
  ]
}

接下来，运行搜索查询：

GET datastream/_search

经过一段时间后，它会变成如下的结果：

{
  "data_streams": [
    {
      "name": "datastream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": "downsample-1h-.ds-datastream-2023.11.30-000001",
          "index_uuid": "KQMxcaJATf24XAesWs6Xbw",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-datastream-2023.11.30-000002",
          "index_uuid": "yKLevDQkQuS-8yqJQb8gWw",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "status": "GREEN",
      "template": "datastream_template",
      "ilm_policy": "datastream_policy",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T03:21:48.000Z",
            "end": "2023-11-30T10:45:44.000Z"
          }
        ]
      }
    },
    {
      "name": "kibana_sample_data_logs",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-kibana_sample_data_logs-2023.11.21-000001",
          "index_uuid": "JkbH5-8uTvyeZL6icuHN1A",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        }
      ],
      "generation": 1,
      "status": "YELLOW",
      "template": "kibana_sample_data_logs",
      "next_generation_managed_by": "Unmanaged",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false
    },
    {
      "name": "my-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-my-data-stream-2023.11.30-000001-downsample",
          "index_uuid": "KXtWAQCBRlW0sWYCVUu7Fw",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        },
        {
          "index_name": ".ds-my-data-stream-2023.11.30-000002",
          "index_uuid": "yBDLixmpRDahmS4G5_uxQw",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        }
      ],
      "generation": 4,
      "status": "GREEN",
      "template": "my-data-stream-template",
      "next_generation_managed_by": "Unmanaged",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T02:24:20.000Z",
            "end": "2023-11-30T10:45:44.000Z"
          }
        ]
      }
    }
  ]
}

默认情况下，索引生命周期管理每十分钟检查一次是否符合策略标准。 等待大约十分钟（也许可以冲泡一杯咖啡或茶☕），然后重新运行 GET _data_stream 请求。

GET _data_stream

ILM 策略生效后，原始 .ds-datastream-2023.11.30-000001 索引将替换为新的降采样索引，在本例中为 downsample-1h-.ds-datastream-2023.11.30-000001。

{
  "data_streams": [
    {
      "name": "datastream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": "downsample-1h-.ds-datastream-2023.11.30-000001",
          "index_uuid": "KQMxcaJATf24XAesWs6Xbw",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-datastream-2023.11.30-000002",
          "index_uuid": "yKLevDQkQuS-8yqJQb8gWw",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "status": "GREEN",
      "template": "datastream_template",
      "ilm_policy": "datastream_policy",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T03:21:48.000Z",
            "end": "2023-11-30T10:45:44.000Z"
          }
        ]
      }
    },

在数据流上运行搜索查询（请注意，在查询降采样索引时，需要注意一些细微差别）。

GET datastream/_search

新的降采样索引仅包含一个文档，其中包括基于原始采样指标的 min、max、sum 和 value_count 统计信息。

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 4,
    "successful": 4,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "downsample-1h-.ds-datastream-2023.11.30-000001",
        "_id": "0eL0wC_4-45SnTNFAAABjB7QCwA",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-11-30T06:00:00.000Z",
          "_doc_count": 10,
          "kubernetes": {
            "container": {
              "cpu": {
                "usage": {
                  "core": {
                    "ns": 12828317850
                  },
                  "limit": {
                    "pct": 0.0000277905
                  },
                  "nanocores": {
                    "min": 38907,
                    "max": 153404,
                    "sum": 992677,
                    "value_count": 10
                  },
                  "node": {
                    "pct": 0.0000277905
                  }
                }
              },
              "memory": {
                "available": {
                  "bytes": {
                    "min": 279586406,
                    "max": 1062428344,
                    "sum": 7101494721,
                    "value_count": 10
                  }
                },
                "majorpagefaults": 0,
                "pagefaults": {
                  "min": 74843,
                  "max": 302252,
                  "sum": 2061071,
                  "value_count": 10
                },
                "rss": {
                  "bytes": {
                    "min": 91914,
                    "max": 402801,
                    "sum": 2389770,
                    "value_count": 10
                  }
                },
                "usage": {
                  "bytes": {
                    "min": 100475044,
                    "max": 379572388,
                    "sum": 2668170609,
                    "value_count": 10
                  },
                  "limit": {
                    "pct": 0.00009923134
                  },
                  "node": {
                    "pct": 0.017700378
                  }
                },
                "workingset": {
                  "bytes": {
                    "min": 431227,
                    "max": 2294743,
                    "sum": 14230488,
                    "value_count": 10
                  }
                }
              },
              "name": "container-name-44",
              "start_time": "2021-03-30T07:59:06.000Z"
            },
            "host": "gke-apps-0",
            "namespace": "namespace26",
            "node": "gke-apps-0-0",
            "pod": "gke-apps-0-0-0"
          }
        }
      }
    ]
  }
}

使用数据流统计 API 获取数据流的统计信息，包括存储大小。

GET /_data_stream/datastream/_stats?human=true

此示例演示了降采样如何作为 ILM 策略的一部分来工作，以减少指标数据的存储大小，因为它变得不那么最新且查询频率较低。

你还可以尝试我们的手动运行降采样示例，了解降采样如何在 ILM 策略之外工作。

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/1273895.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

LeetCode | 二叉树的最大深度

LeetCode | 二叉树的最大深度

LeetCode | 二叉树的最大深度 OJ链接 这里需要注意的一点是每次有返回值&#xff0c;需要定义变量来保存上一次的值最后取最高的一方加1 int maxDepth(struct TreeNode* root) {if(root NULL)return NULL;int left maxDepth(root->left);int right maxDepth(root->r…
阅读更多...
C语言中一些有关字符串的常见函数的使用及模拟实现（1）

C语言中一些有关字符串的常见函数的使用及模拟实现（1）

在编程的过程中&#xff0c;我们经常要处理字符和字符串&#xff0c;为了⽅便操作字符和字符串&#xff0c;C语⾔标准库中提供了 ⼀系列库函数&#xff0c;接下来我们就学习⼀下这些函数。 文章目录 strlen函数的使用及模拟实现strcmp函数的使用及模拟实现strcpy函数的使用及代…
阅读更多...
《C++PrimerPlus》第9章 内存模型和名称空间

《C++PrimerPlus》第9章 内存模型和名称空间

9.1 单独编译 Visual Studio中新建头文件和源代码 通过解决方案资源管理器&#xff0c;如图所示&#xff1a; 分成三部分的程序&#xff08;直角坐标转换为极坐标&#xff09; 头文件coordin.h #ifndef __COORDIN_H__ // 如果没有被定义过 #define __COORDIN_H__struct pola…
阅读更多...
阅读文献总结2023

阅读文献总结2023

阅读文献基于卷积神经网络多源融合的网络安全态势感知模型 阅读文献 基于卷积神经网络多源融合的网络安全态势感知模型 题目基于卷积神经网络多源融合的网络安全态势感知模型文章信息&#xff1a;年份2023发文单位山西财经大学收录刊会计算机科学 &#xff08;北大核心&#…
阅读更多...
和鲸科技与国科环宇建立战略合作伙伴关系，以软硬件一体化解决方案促进科技创新

和鲸科技与国科环宇建立战略合作伙伴关系，以软硬件一体化解决方案促进科技创新

近日&#xff0c;在国科环宇土星云算力服务器产品发布会暨合作伙伴年度会上&#xff0c;和鲸科技与国科环宇正式完成战略伙伴签约仪式&#xff0c;宣布达成战略合作伙伴关系。未来&#xff0c;双方将深化合作&#xff0c;充分发挥在产品和市场方面的互补优势&#xff0c;为企事…
阅读更多...
el-table实现动态表头

el-table实现动态表头

1.1el-table渲染 <el-tableref"refreshTable":data"tableData"highlight-current-row><el-table-columnfixedwidth"170px"label"测点"align"center"prop"测站名称"/><el-table-column label"…
阅读更多...
echarts实现3D柱状图

echarts实现3D柱状图

效果如图 let setData function(data, constData, showData) {data.filter(function(item) {if (item) {constData.push(1);showData.push(item);} else {constData.push(0);showData.push({value: 1,itemStyle: {normal: {borderColor: "rgba(0,0,0,0)",borderWidt…
阅读更多...
中信建投在金融电于化期刊发布 DataOps 实践

中信建投在金融电于化期刊发布 DataOps 实践

文 ‖ 中信建投证券股份有限公司 马丽霞 高宇航 李可 许哲 李海伟 近年来&#xff0c;数据的分析和应用对各行各工业的业务模式和竞争形态进行重塑&#xff0c;而积极应对挑战和顺应时代变化是各个市场参与者的必选项。作为资本市场数字化转型的领航者&#xff0c;中信建投证券…
阅读更多...
react-flip-move结合array-move实现前端列表置顶效果

react-flip-move结合array-move实现前端列表置顶效果

你有没有遇到这样的需求&#xff1f;点击左侧列表项&#xff0c;则像聊天会话窗口一样将被点击的列表项置顶。 如果只是单纯的置顶的话&#xff0c;直接使用array-move就可以实现了&#xff0c;但置顶效果多少有点突兀~ 先上代码&#xff0c;直接使用array-move的情况&#xf…
阅读更多...
11-30 SpringBoot2

11-30 SpringBoot2

热部署 开发过程中,修改代码,不需要重启,自动更新 项目上线,一定要关闭 SpringBoot热部署的实现&#xff1f;&#xff1f; ideal默认阻止class类更新 2&#xff0e;需要手动构建项目&#xff0c;可以使用快捷键激活此功能ctrl F9 / build project 自动构建项目 允许程序运行…
阅读更多...
rabbitMQ镜像队列的使用

rabbitMQ镜像队列的使用

在rabbitMQ集群中&#xff0c;默认发送消息时&#xff0c;队列默认时在一个节点上存在的。 我们以node01 node02 node03三节点集群为例&#xff0c;在node01声明队列发送消息后&#xff0c;发现&#xff1a; 测试队列只在节点node01上出现。 我们手动停止node01后&#xff0c…
阅读更多...
Vue大屏自适应终极解决方案

Vue大屏自适应终极解决方案

v-scale-screenv-scale-screen是一个大屏自适应组件&#xff0c;在实际业务中&#xff0c;我们常用图表来做数据统计&#xff0c;数据展示&#xff0c;数据可视化等比较直观的方式来达到一目了然的数据查看&#xff0c;但在大屏开发过程中&#xff0c;常会因为适配不同屏幕而感…
阅读更多...
Redis7--基础篇4（Redis事务）

Redis7--基础篇4（Redis事务）

Redis事务是什么 可以一次执行多个命令&#xff0c;本质是一组命令的集合&#xff0c;一个事务中的所有命令都会序列化&#xff0c;按顺序串行&#xff0c;而不会被其他命令插入。 其作用就是在一个队列中&#xff0c;一次性、顺序、排他的执行一系列命令。 Redis事务 VS 数据…
阅读更多...
华为云cce负载配置时间同步

华为云cce负载配置时间同步

华为云cce将负载配置好之后&#xff0c;发现里面的时间与真实时间不同步&#xff0c;差了12小时&#xff0c;怎么办&#xff1f; 这时候就需要配置时间同步了。 华为云cce里面通过配置数据存储的路径来解决这个问题的&#xff0c;配置后&#xff0c;需要重启负载。 新建负载…
阅读更多...
Java中的synchronized关键字

Java中的synchronized关键字

目录 1、synchronized是什么 2、synchronized的用法 synchronized可以用在方法或者代码块上&#xff0c;分别称为同步方法和同步代码块。 用法理解 3、synchronized的实现原理 ⭐synchronized锁的对比 4、synchronized的优缺点 ⭐扩展&#xff1a;synchronized 和 vola…
阅读更多...
JAVA全栈开发 day15_集合（Set接口、增强For循环、Map体系）

JAVA全栈开发 day15_集合（Set接口、增强For循环、Map体系）

一、增加for遍历集合 语法&#xff1a; for(数据类型 变量名: 数组名或集合){​ }//集合遍历 &#xff0c;推荐使用增加for 1.静态导入 注意事项&#xff1a; 方法必须是静态注意不要和本类的方法同名&#xff0c;如果同名&#xff0c;记得加前缀&#xff0c;由此可…
阅读更多...
vue中.sync修饰符与$emit(update：xxx)双向数据绑定

vue中.sync修饰符与$emit(update：xxx)双向数据绑定

文章目录 一、单向数据流二、props父子传值2.1、父组件2.2、子组件2.3、优缺点2.3.1、优点2.3.2、缺点 三、.sync修饰符双向绑定3.1、父组件3.2、子组件3.3、优缺点3.3.1、优点3.3.2、缺点 3.4、[文档](https://v2.cn.vuejs.org/v2/guide/components-custom-events.html#sync-%…
阅读更多...
【快速见刊|投稿优惠】2024年机电一体与自动化技术国际学术会议(IACMAT 2024)

【快速见刊|投稿优惠】2024年机电一体与自动化技术国际学术会议(IACMAT 2024)

2024年机电一体与自动化技术国际学术会议(IACMAT 2024) 2024 International Academic Conference on Mechatronics and Automation Technology(IACMAT 2024) 一【会议简介】 2024年机电一体与自动化技术国际学术会议(IACMAT 2024)即将召开&#xff0c;它以“机电一体&#xff0…
阅读更多...
利用Linux中的iptables进行网络代理配置

利用Linux中的iptables进行网络代理配置

作为资深爬虫技术员&#xff0c;爬虫需要代理IP池介入这是众所周知的。今天我将用我毕生所学&#xff0c;谈谈linux中使用iptables工具来进行网络配置&#xff0c;并通过linux系统创建属于自己的ip库池&#xff0c;如有错误望各位大佬指正。 我们知道&#xff0c;在Linux中&am…
阅读更多...
Windows10中在Visual Studio2017中VC++项目安装使用GoogleTest库

Windows10中在Visual Studio2017中VC++项目安装使用GoogleTest库

Windows10中在Visual Studio2017中VC项目安装使用GoogleTest库 在Windows10中VC程序中可以不用自己手动下载GoogleTest源代码&#xff0c;可以直接通过【项目】-> 【管理 NuGet 程序包】-> 【浏览】-> 搜索 googletest&#xff0c; 找到Microsoft.googletest.v140.wi…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023