Elasticsearch 存储水位线测试验证

Elasticsearch 存储水位线测试验证

环境准备

# 加载环境exportKUBECONFIG=/bpx/.145-admin.conf# 获取ES密码exportES_PASS=$(kubectl get secret es-0c1a4a65-es-elastic-user-nqfusion-admin-ojsonpath='{.data.elastic}'|base64-d)echo$ES_PASS# 检查集群状态kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS"http://localhost:9200/_cluster/health?pretty"

一、水位线说明

默认水位线

注意事项：本地存储场景

使用csi-localpv/hostPath时：

• ES看到的磁盘容量= 底层物理磁盘总容量（如200G）
• PVC限制= K8s分配的软限制（如20Gi）
• 水位线计算基于物理磁盘，而非PVC限制

PVC: 20Gi ──────────────────────────┐ 实际限制 物理磁盘: 200G ──────────────────────────────────────────────┐ ES用于水位线计算

二、查看水位线设置

方式1：查看当前有效值（推荐）

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS"http://localhost:9200/_cluster/settings?flat_settings=true"

输出示例：

{"persistent":{},"transient":{"cluster.routing.allocation.disk.watermark.flood_stage":"95%","cluster.routing.allocation.disk.watermark.high":"90%","cluster.routing.allocation.disk.watermark.low":"85%"}}

• persistent: 永久设置（重启保留）
• transient: 临时设置（重启失效）
• 两者为空表示使用默认值

方式2：查看所有默认值

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS"http://localhost:9200/_cluster/settings?include_defaults=true&flat_settings=true"|\grep-E"cluster.routing.allocation.disk.watermark"

方式3：查看磁盘使用情况

# 查看各节点磁盘使用率kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS"http://localhost:9200/_cat/allocation?v&h=name,disk.used_percent,disk.total"# 查看详细分配信息kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS"http://localhost:9200/_cat/allocation?v"

三、修改水位线

3.1 永久修改（重启保留）

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT"http://localhost:9200/_cluster/settings"\-H"Content-Type: application/json"\-d'{ "persistent": { "cluster.routing.allocation.disk.watermark.low": "80%", "cluster.routing.allocation.disk.watermark.high": "85%", "cluster.routing.allocation.disk.watermark.flood_stage": "90%" } }'

3.2 临时修改（重启失效）

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT"http://localhost:9200/_cluster/settings"\-H"Content-Type: application/json"\-d'{ "transient": { "cluster.routing.allocation.disk.watermark.low": "80%", "cluster.routing.allocation.disk.watermark.high": "85%", "cluster.routing.allocation.disk.watermark.flood_stage": "90%" } }'

3.3 恢复默认值

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT"http://localhost:9200/_cluster/settings"\-H"Content-Type: application/json"\-d'{ "transient": { "cluster.routing.allocation.disk.watermark.low": null, "cluster.routing.allocation.disk.watermark.high": null, "cluster.routing.allocation.disk.watermark.flood_stage": null } }'

3.4 支持的格式

# 百分比"low":"85%"# 绝对值（剩余空间）"low":"500gb""low":"15gb"# 组合（两者都满足才触发）"low":"85%, 500gb"

四、只读索引解锁

4.1 查看只读索引

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS"http://localhost:9200/_all/_settings?pretty"|\grep-B2"read_only_allow_delete"

输出示例：

"blocks":{"read_only_allow_delete":"true"}

4.2 解锁单个索引

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT\"http://localhost:9200/<index_name>/_settings"\-H"Content-Type: application/json"\-d'{"index.blocks.read_only_allow_delete": null}'

用途：只解锁指定的索引，其他索引不受影响

4.3 批量解锁所有索引

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT\"http://localhost:9200/_all/_settings"\-H"Content-Type: application/json"\-d'{"index.blocks.read_only_allow_delete": null}'

用途：一次性解锁所有索引（注意：确保磁盘空间已释放）

4.4 单个解锁 vs 批量解锁

五、完整测试验证流程

5.1 准备测试索引

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT"http://localhost:9200/test_watermark"\-H"Content-Type: application/json"\-d'{"settings": {"number_of_shards": 1, "number_of_replicas": 0}}'

5.2 降低水位线触发只读

# 设置 flood_stage 为 30%（低于当前使用率）kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT"http://localhost:9200/_cluster/settings"\-H"Content-Type: application/json"\-d'{ "transient": { "cluster.routing.allocation.disk.watermark.low": "25%", "cluster.routing.allocation.disk.watermark.high": "28%", "cluster.routing.allocation.disk.watermark.flood_stage": "30%" } }'

5.3 验证只读状态

# 等待ES检测（约30秒）sleep30# 检查集群块kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS"http://localhost:9200/_cluster/state?pretty"|grep-A3"blocks"# 尝试写入（应该失败）kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPOST"http://localhost:9200/test_watermark/_doc"\-H"Content-Type: application/json"\-d'{"message": "test"}'

预期错误：

{"error":{"type":"cluster_block_exception","reason":"index blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark...]"},"status":429}

5.4 解锁索引并验证

# 批量解锁所有索引kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT"http://localhost:9200/_all/_settings"\-H"Content-Type: application/json"\-d'{"index.blocks.read_only_allow_delete": null}'# 验证写入成功kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPOST"http://localhost:9200/test_watermark/_doc"\-H"Content-Type: application/json"\-d'{"message": "test after unlock"}'

5.5 恢复默认水位线

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XPUT"http://localhost:9200/_cluster/settings"\-H"Content-Type: application/json"\-d'{ "transient": { "cluster.routing.allocation.disk.watermark.low": null, "cluster.routing.allocation.disk.watermark.high": null, "cluster.routing.allocation.disk.watermark.flood_stage": null } }'

5.6 清理测试数据

kubectlexec-nqfusion-admin es-0c1a4a65-es-data-0-celasticsearch --\curl-uelastic:$ES_PASS-XDELETE"http://localhost:9200/test_watermark*"

六、常见问题

Q1: 水位线设置报错 “high disk watermark more than flood stage”

原因：flood_stage 必须大于 high，high 必须大于 low

解决：同时设置三个值，确保 low < high < flood_stage

# 错误示例"flood_stage":"30%"# high默认是90%，报错# 正确示例"low":"25%","high":"28%","flood_stage":"30%"

Q2: 磁盘空间已释放，索引仍然只读

原因：ES不会自动解除只读锁，需要手动解锁

解决：执行解锁命令

curl-XPUT"_all/_settings"-d'{"index.blocks.read_only_allow_delete": null}'

Q3: PVC显示20G，ES认为磁盘充足

原因：使用本地存储时，ES看到的是物理磁盘总容量

建议：监控PVC使用量，不要仅依赖ES水位线

七、验证记录

测试环境信息

• ES 集群: es-0c1a4a65
• 集群状态: green (3节点)
• 测试索引: test_watermark
• 初始磁盘使用率: 33%-39%