绑定完请刷新页面
取消
刷新

分享好友

×
取消 复制
Ingest Attachment Processor Plugin 基本用法
2022-01-30 16:03:10

前言

elasticsearch5.x 新增一个比较重要的特性 IngestNode。
之前如果需要对数据进行加工,都是在索引之前进行处理,比如logstash可以对日志进行结构化和转换,现在直接在es就可以处理了。
目前es提供了一些常用的诸如convert、grok之类的处理器,在使用的时候,先定义一个pipeline管道,里面设置文档的加工逻辑,在建索引的时候指定pipeline名称,那么这个索引就会按照预先定义好的pipeline来处理了。

Ingest Attachment Processor Plugin

处理文档附件,替换之前的 mapper attachment plugin。
默认存储附件内容必须base64编码的数据,不想base64转换,可以使用CBOR(没有试验)
官网说明:

The source field must be a base64 encoded binary. 
If you do not want to incur the overhead of converting back and forth between base64, 
you can use the CBOR format instead of JSON and specify the field as a bytes array instead of a string representation. 
The processor will skip the base64 decoding then.

安装

./bin/elasticsearch-plugin install ingest-attachment

卸载

./bin/elasticsearch-plugin remove ingest-attachment

用管道处理单个附件示例(Using the Attachment Processor in a Pipeline)

1.创建管道single_attachment

PUT _ingest/pipeline/single_attachment
{
  "description" : "Extract single attachment information",
  "processors" : [
    {
      "attachment" : {
        "field": "data",
        "indexed_chars" : -1,
        "ignore_missing" : true
      }
    }
  ]
}

2.创建index

PUT /index1
{
    "mappings" : {
        "type1" : {
            "properties" : {
                "id": {
                    "type": "keyword"
                },
                "filename": {
                    "type": "text",
                    "analyzer": "english"
                },
                "data":{
                    "type": "text",
                    "analyzer": "english"
                }
            }
        }
    }
}

3.索引数据

PUT index1/type1/1?pipeline=single_attachment&refresh=true&pretty=1
{
    "id": "1",
    "filename": "1.txt",
    "data" : "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
PUT index1/type1/2?pipeline=single_attachment&refresh=true&pretty=1
{
  "id": "2",
  "subject": "2.txt",
  "data": "dGVzdGluZyBteSBmaXJzdCBlbmNvZGVkIHRleHQ="
}

4.查看结果

GET index1/type1/1
GET index1/type1/2
POST index1/type1/_search?pretty=true
{
  "query": {
    "match": {
      "attachment.content_type": "text plain"
    }
  }
}
POST index1/type1/_search?pretty=true
{
  "query": {
    "match": {
      "attachment.content": "testing"
    }
  },
  "highlight": {
    "fields": {
      "attachment.content": {}
    }
  }
}

返回结果

"hits": [
    {
        "_index": "index1",
        "_type": "type1",
        "_id": "2",
        "_score": 0.2824934,
        "_source": {
            "data": "dGVzdGluZyBteSBmaXJzdCBlbmNvZGVkIHRleHQ=",
            "attachment": {
                "content_type": "text/plain; charset=ISO-8859-1",
                "language": "et",
                "content": "testing my first encoded text",
                "content_length": 30
            },
            "subject": "2.txt",
            "id": "2"
        },
        "highlight": {
            "attachment.content": [
                "<em>testing</em> my first encoded text"
            ]
        }
    }
]

用管道处理多个附件示例(Using the Attachment Processor with arrays)、

1.创建管道multi_attachment

PUT _ingest/pipeline/multi_attachment
{
  "description" : "Extract attachment information from arrays",
  "processors" : [
    {
      "foreach": {
        "field": "attachments",
        "processor": {
          "attachment": {
            "target_field": "_ingest._value.attachment",
            "field": "_ingest._value.data",
            "indexed_chars" : -1,
            "ignore_missing" : true
          }
        }
      }
    }
  ]
}

2.创建index

PUT /index2
{
    "mappings" : {
        "type2" : {
            "properties" : {
                "id": { 
                    "type": "keyword"
                },
                "subject": { 
                    "type": "text",
                    "analyzer": "ik_max_word"
                },
                "attachments": {
                    "properties":{
                         "filename" : {"type": "text","analyzer": "english"},
                         "data":{"type": "text","analyzer": "english"}
                    }
                }
            }
        }
    }
}

3.索引数据

PUT index2/type2/1?pipeline=attachment&refresh=true&pretty=1
{
  "id": "1",
  "subject": "Elasticsearch: The Definitive Guide007",
  "attachments" : [
    {
      "filename" : "a.txt",
      "data" : "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
    },
    {
      "filename" : "b.txt",
      "data" : "dGVzdGluZyBteSBmaXJzdCBlbmNvZGVkIHRleHQ="
    }
  ]
}
PUT index2/type2/2?pipeline=attachment&refresh=true&pretty=1
{
  "id": "2",
  "subject": "Using the Attachment Processor with arrays",
  "attachments" : [
    {
      "filename" : "test1.txt",
      "data" : "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo="
    },
    {
      "filename" : "test2.txt",
      "data" : "VGhpcyBpcyBhIHRlc3QK"
    }
  ]
}

4.查看结果

POST index2/type2/_search?pretty=true
{
  "query": {
    "match": {
      "attachments.attachment.content": "test"
    }
  },
  "highlight": {
    "fields": {
      "attachments.attachment.content": {}
    }
  }
}
返回结果
"hits": [
    {
        "_index": "index2",
        "_type": "type2",
        "_id": "2",
        "_score": 0.27233246,
        "_source": {
            "attachments": [
                {
                    "filename": "test1.txt",
                    "data": "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo=",
                    "attachment": {
                        "content_type": "text/plain; charset=ISO-8859-1",
                        "language": "en",
                        "content": "this is just some text",
                        "content_length": 24
                    }
                }
                ,
                {
                    "filename": "test2.txt",
                    "data": "VGhpcyBpcyBhIHRlc3QK",
                    "attachment": {
                        "content_type": "text/plain; charset=ISO-8859-1",
                        "language": "en",
                        "content": "This is a test",
                        "content_length": 16
                    }
                }
            ],
            "id": "2",
            "subject": "Using the Attachment Processor with arrays"
        },
        "highlight": {
            "attachments.attachment.content": [
                "This is a <em>test</em>"
            ]
        }
    }
]


作者:codeMan_6616
链接:https://www.jianshu.com/p/774e5ed120ba
来源:简书
著作权归作者所有。商业转载请联系作者获得授权,非商业转载请注明出处。
分享好友

分享这个小栈给你的朋友们,一起进步吧。

Elasticsearch
创建时间:2020-05-22 14:49:51
ElasticSearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口。Elasticsearch是用Java开发的,并作为Apache许可条款下的开放源码发布,是当前流行的企业级搜索引擎。设计用于云计算中,能够达到实时搜索,稳定,可靠,快速,安装使用方便。我们建立一个网站或应用程序,并要添加搜索功能,但是想要完成搜索工作的创建是非常困难的。我们希望搜索解决方案要运行速度快,我们希望能有一个零配置和一个完全免费的搜索模式,我们希望能够简单地使用JSON通过HTTP来索引数据,我们希望我们的搜索服务器始终可用,我们希望能够从一台开始并扩展到数百台,我们要实时搜索,我们要简单的多租户,我们希望建立一个云的解决方案。因此我们利用Elasticsearch来解决所有这些问题及可能出现的更多其它问题。
展开
订阅须知

• 所有用户可根据关注领域订阅专区或所有专区

• 付费订阅:虚拟交易,一经交易不退款;若特殊情况,可3日内客服咨询

• 专区发布评论属默认订阅所评论专区(除付费小栈外)

技术专家

查看更多
  • 栈栈
    专家
戳我,来吐槽~