elk Elasticsearch 管理维护操作记录 (持续更新) — Jevic

Jevic

elk Elasticsearch 管理维护操作记录 (持续更新)

2017/07/21 ELK

集群信息

curl es-90:9200/_cluster/settings?pretty
....

cat 信息

curl es-90:9200/_cat/indices  // 索引
curl es-90:9200/_cat/shards   // 分片
curl es-90:9200/_cat/nodes?pretty  // 节点
.....

node 接口信息

curl es-90:9200/_nodes/stats/process?filter_path=**.max_file_descriptors

数据均衡

开启: all 关闭: node

开启均衡

curl -XPUT es-90:9200/_cluster/settings?pretty -d '
{
    "transient" : {
        "cluster.routing.allocation.enable" : "all"
    }
}'

关闭均衡

curl -XPUT es-90:9200/_cluster/settings?pretty -d '
{
    "transient" : {
        "cluster.routing.allocation.enable" : "none"
    }
}'

冷热数据分离

设置zone 标签，标签值自定义即可

此处为了方便理解配置为 热:hot 冷:stale

一般建议hot节点,磁盘使用ssd 提高吞吐量及效率，stale 则为普通SATA盘只用来存放历史数据

例如:索引名称基于日期来命名那么可将当前写入以及使用频率较高的最近时间的索引标记为hot，提高写入读取效率。而对应过期的冷门数据标记为stale

热节点分配

curl -XPUT $URL/${Inx}-${date}/_settings
'{ "index.routing.allocation.include.zone": "hot"}'

冷节点分配

curl -XPUT $URL/${Inx}-${date_ops}/_settings -d 
'{ "index.routing.allocation.include.zone": "stale"}'

索引优化分段

将冷门数据 设置此参数提高搜索速度

2.x

curl -XPOST node-es:9200/test-2017.07.05/_optimize?max_num_segments=1&wait_for_merge=false

5.x

curl -XPOST node-es:9200/test-2017.07.05/_forcemerge?max_num_segments=1

数据搜索

GET index/_search
GET index/doc/_search
GET index/doc/_search?pretty
GET index/doc/_search|jq
curl node-es:9200/index/doc/_search |python -mjson.tool

搜索超时设置

curl -XPUT es-90:9200/_cluster/settings -d '
{
    "transient" : {
  "search.default_search_timeout" : "60s"
  }
}'

数据刷新时间

设置为 -1 为不刷新 通常在导出数据时，建议设置为-1,默认设置为60s即可

PUT test/_settings
 { 
    "index" : { 
        "refresh_interval" : "60s" 
    } 
}

更新文档数据

修改某个索引指定文档的目标字段值

POST nginx-log/logs/AVxiwZcu_Z9aU351KXOL/_update
{
  "doc": {
    "value" : 732439
  }
}

删除数据

删除索引

curl -XDELETE node-es:9200/index

删除索引指定的文档

2.x

curl -XDELETE node-es:9200/test-index/_query -d '{
"query": { 
    "match": {
      "message": "some message"
    }
  }
}'

5.x

  • 推荐使用:
curl -XPOST _delete_by_query?scroll_size=2000&wait_for_completion=false

curl -XPOST node-es:9200/test-index/_delete_by_query -d '
{
  "query": { 
    "match": {
      "message": "some message"
    }
  }
}'

任务查询 _tasks

官网API文档

es 2.x

GET /tasks
GET _cluster/pending_tasks
GET /_tasks?actions=*/list*
GET /_tasks?actions=*reindex,*by_query
返回的结果:
{
.....
      "tasks": {
        "w8S8uYR9ThelFe7u5blX-Q:466799605": {
          "node": "w8S8uYR9ThelFe7u5blX-Q",
          "id": 466799605,
          "type": "transport",
          "action": "indices:data/write/delete/by_query",
          "start_time_in_millis": 1499846410168,
          "running_time_in_nanos": 527881720186
.....
}

es 5.x

GET _cat/tasks?v&time
GET _tasks?detailed=true&actions=*/delete/byquery
GET _tasks?pretty&detailed&actions=*reindex,*byquery
返回结果:
{
.....
      "tasks": {
        "RiuComp3TOWhz0y804Ym7A:55597577": {
          "node": "RiuComp3TOWhz0y804Ym7A",
          "id": 55597577,
          "type": "transport",
          "action": "indices:data/write/delete/byquery",
          "status": {
            "total": 47683364,
            "updated": 0,
            "created": 0,
            "deleted": 15294000,
            "batches": 7648,
            "version_conflicts": 0,
            "noops": 0,
            .....
      }
    }
  }
}

取消任务

POST _tasks/RiuComp3TOWhz0y804Ym7A:55597577/_cancel

模板

模板建议使用kopf [es 2.x] 或者cerebro[es 5.x] 插件进行管理

添加模板

PUT  /_template/test
{
  
  "template":"test-*",
  "settings" : {
    "index.refresh_interval" : "60s"
  }
}

查询模板

GET _template/test

调整数据查询窗口大小

默认查询为1W条 根据查询需求适当的调整此参数，不建议过大

GET index-test/_settings

PUT index-test/_settings
{ 
  "index" : { 
    "max_result_window" : 500000 
  } 
}

磁盘配额

此设置覆盖配置文件参数

PUT _cluster/settings
{
    "transient" : {
        "cluster.routing.allocation.disk.watermark.low" : "90%",
        "cluster.routing.allocation.disk.watermark.high" : "10gb"
    }
}

指定索引分片分配到某个特定主机

官网文档

PUT test/_settings
{
  "index.routing.allocation.include._ip": "192.168.2.*"
}

数据迁移导入导出

yum install epel-release
yum install nodejs
yum install nodejs npm
npm install elasticdump
cd node_modules/elasticdump/bin
  • 批量执行
#!/bin/bash
InputES="http://es.jevic.cn"
OutputES="http://test.es.jevic.cn"

indexs="test-20181010"
for index in $indexs;do
/root/node_modules/elasticdump/bin/elasticdump \
--input=${InputES}/$index \
--output=${OutputES}/$index \
--type=data
done

另外 –type=mapping 意思是把原始索引的mapping结构迁移给目标索引(2.x,5.x)

sql 插件

sql 2.x


curl http://es-node/_sql -d "SELECT count(*) FROM index-test/type where timestamp=1484048580"

sql 5.x

http://es-node/_sql?sql=SELECT count(*) FROM index-test where @timestamp>=149925060000 and @timestamp<1499254200000

参考链接

Elasticsearch Reference 官网

Elasticsearch: 权威指南 中文

转载请注明出处，本文采用 CC4.0 协议授权

Search

    Post Directory