15、ElasticSearch 实战： ES 聚合计算
聚合框架用于收集搜索查询选择的所有数据。该框架由许多构建块组成，有助于构建复杂的数据摘要
下面的JSON 对象使用聚合函数的一般请求正文格式
"aggregations" : {
"<aggregation_name>" : {
"<aggregation_type>" : {
<aggregation_body>
}
[,"meta" : { [<meta_data_body>] } ]?
[,"aggregations" : { [<sub_aggregation>]+ } ]?
}
}
Elasticsearch 提供了大量的聚合函数，它们都有各自不同的目的
矩阵聚合 ( Metrics )
这些聚合函数可以根据聚合文档的字段值计算度量值，而且有时可以从脚本生成一些值
数字矩阵既可以是单值，也可以是平均聚合或多值统计等
平均数聚合 ( avg )
该聚合函数用于计算文档中出现的任何数字字段的平均值
例如
POST http://localhost:9200/user_admin/_search?pretty
请求正文
{
"aggs":{
"avg_money":{"avg":{"field":"money"}}
}
}
返回响应结果
{
"took" : 160,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "雅少",
"description" : "虚怀若谷",
"street" : "四川大学",
"city" : "Chengdu",
"state" : "Sichuan",
"zip" : "610044",
"location" : [
104.094537,
30.640174
],
"money" : 68023,
"tags" : [
"Python",
"HTML"
],
"vitality" : "7.8"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "站长",
"description" : "DDKK.COM 弟弟快看，程序员编程资料站 ，教程 ",
"street" : "东四十条",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100007",
"location" : [
116.432727,
39.937732
],
"money" : 5201814,
"tags" : [
"PHP",
"Python"
],
"vitality" : "9.0"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"nickname" : "歌者",
"description" : "程序设计也是设计，研发新菜也是研发",
"street" : "五道口",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100083",
"location" : [
116.346346,
39.999333
],
"money" : 71128,
"tags" : [
"Java",
"Scala"
],
"vitality" : "6.9"
}
}
]
},
"aggregations" : {
"avg_money" : {
"value" : 1780321.6666666667
}
}
}
如果一个或多个聚合文档中不存在此值，默认情况下它们会被忽略
我们可以在聚合中添加缺失字段来设置缺失字段的默认值
{
"aggs":{
"avg_money":{
"avg":{
"field":"money"
"missing":0
}
}
}
}
基数聚合 ( cardinality )
基数聚合 ( cardinality ) 用于计算特定字段的不同值的计数
例如
POST http://localhost:9200/user*/_search?pretty
请求正文
{
"aggs":{
"distinct_nickname_count":{"cardinality":{"field":"nickname"}}
}
}
响应内容
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [nickname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "user",
"node" : "4zwAMlTzRCaioBeOE9PaNw",
"reason" : {
"type" : "illegal_argument_exception",
"reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [nickname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
}
],
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [nickname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [nickname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
}
},
"status" : 400
}
很明显响应出错了，提示 Fielddata 要单独加载
好吧，那我们先运行下面的请求来修改下
PUT http://localhost:9200/user*/_mapping/user
请求正文
{
"properties": {
"nickname": {
"type": "text",
"fielddata": true
}
}
}
响应内容
{"acknowledged":true}
然后重新发起刚刚报错的请求，响应如下
{
"took" : 186,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 5,
"max_score" : 1.0,
"hits" : [
{
"_index" : "user",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "枫晚",
"description" : "停车坐爰枫林晚",
"street" : "苏州大学",
"city" : "Suzhou",
"state" : "Jiangsu",
"zip" : "215006",
"location" : [
120.65426,
31.30797
],
"money" : 10235,
"tags" : [
"Java",
"Android"
],
"vitality" : "3.5"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "雅少",
"description" : "虚怀若谷",
"street" : "四川大学",
"city" : "Chengdu",
"state" : "Sichuan",
"zip" : "610044",
"location" : [
104.094537,
30.640174
],
"money" : 68023,
"tags" : [
"Python",
"HTML"
],
"vitality" : "7.8"
}
},
{
"_index" : "user",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "question",
"description" : "问题少年也是少年",
"street" : "张江高科技园区",
"city" : "Shanghai",
"state" : "Shanghai",
"zip" : "201204",
"location" : [
121.60632,
31.199305
],
"money" : 13648,
"tags" : [
"VUE",
"HTML"
],
"vitality" : "8.8"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "站长",
"description" : "DDKK.COM 弟弟快看，程序员编程资料站 ，教程 ",
"street" : "东四十条",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100007",
"location" : [
116.432727,
39.937732
],
"money" : 5201814,
"tags" : [
"PHP",
"Python"
],
"vitality" : "9.0"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"nickname" : "歌者",
"description" : "程序设计也是设计，研发新菜也是研发",
"street" : "五道口",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100083",
"location" : [
116.346346,
39.999333
],
"money" : 71128,
"tags" : [
"Java",
"Scala"
],
"vitality" : "6.9"
}
}
]
},
"aggregations" : {
"distinct_nickname_count" : {
"value" : 9
}
}
}
扩展统计聚合 ( extended_stats )
此聚合用于生成有关聚合文档中特定数字字段的所有统计信息
例如
POST http://localhost:9200/user_admin/user/_search?pretty
请求正文
{
"aggs" : {
"money_stats" : { "extended_stats" : { "field" : "money" } }
}
}
响应内容
{
"took" : 12,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "雅少",
"description" : "虚怀若谷",
"street" : "四川大学",
"city" : "Chengdu",
"state" : "Sichuan",
"zip" : "610044",
"location" : [
104.094537,
30.640174
],
"money" : 68023,
"tags" : [
"Python",
"HTML"
],
"vitality" : "7.8"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "站长",
"description" : "DDKK.COM 弟弟快看，程序员编程资料站 ，教程 ",
"street" : "东四十条",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100007",
"location" : [
116.432727,
39.937732
],
"money" : 5201814,
"tags" : [
"PHP",
"Python"
],
"vitality" : "9.0"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"nickname" : "歌者",
"description" : "程序设计也是设计，研发新菜也是研发",
"street" : "五道口",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100083",
"location" : [
116.346346,
39.999333
],
"money" : 71128,
"tags" : [
"Java",
"Scala"
],
"vitality" : "6.9"
}
}
]
},
"aggregations" : {
"money_stats" : {
"count" : 3,
"min" : 68023.0,
"max" : 5201814.0,
"avg" : 1780321.6666666667,
"sum" : 5340965.0,
"sum_of_squares" : 2.7068555211509E13,
"variance" : 5.853306500366889E12,
"std_deviation" : 2419360.762756743,
"std_deviation_bounds" : {
"upper" : 6619043.192180153,
"lower" : -3058399.858846819
}
}
}
}
最大值聚合 ( max )
最大值聚合用于查找聚合文档中特定数字字段的最大值
例如
POST http://localhost:9200/user*/_search
请求正文
{
"aggs" : {
"max_money" : { "max" : { "field" : "money" } }
}
}
响应内容
{
"took" : 22,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "雅少",
"description" : "虚怀若谷",
"street" : "四川大学",
"city" : "Chengdu",
"state" : "Sichuan",
"zip" : "610044",
"location" : [
104.094537,
30.640174
],
"money" : 68023,
"tags" : [
"Python",
"HTML"
],
"vitality" : "7.8"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "站长",
"description" : "DDKK.COM 弟弟快看，程序员编程资料站 ，教程 ",
"street" : "东四十条",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100007",
"location" : [
116.432727,
39.937732
],
"money" : 5201814,
"tags" : [
"PHP",
"Python"
],
"vitality" : "9.0"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"nickname" : "歌者",
"description" : "程序设计也是设计，研发新菜也是研发",
"street" : "五道口",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100083",
"location" : [
116.346346,
39.999333
],
"money" : 71128,
"tags" : [
"Java",
"Scala"
],
"vitality" : "6.9"
}
}
]
},
"aggregations" : {
"max_money" : {
"value" : 5201814.0
}
}
}
最小值聚合 ( min )
最小值聚合用于查找聚合文档中特定数字字段的最小值
例如
POST http://localhost:9200/user*/_search?pretty
请求正文
{
"aggs" : {
"min_money" : { "min" : { "field" : "money" } }
}
}
响应内容
{
"took" : 20,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 5,
"max_score" : 1.0,
"hits" : [
{
"_index" : "user",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "枫晚",
"description" : "停车坐爰枫林晚",
"street" : "苏州大学",
"city" : "Suzhou",
"state" : "Jiangsu",
"zip" : "215006",
"location" : [
120.65426,
31.30797
],
"money" : 10235,
"tags" : [
"Java",
"Android"
],
"vitality" : "3.5"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "雅少",
"description" : "虚怀若谷",
"street" : "四川大学",
"city" : "Chengdu",
"state" : "Sichuan",
"zip" : "610044",
"location" : [
104.094537,
30.640174
],
"money" : 68023,
"tags" : [
"Python",
"HTML"
],
"vitality" : "7.8"
}
},
{
"_index" : "user",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "question",
"description" : "问题少年也是少年",
"street" : "张江高科技园区",
"city" : "Shanghai",
"state" : "Shanghai",
"zip" : "201204",
"location" : [
121.60632,
31.199305
],
"money" : 13648,
"tags" : [
"VUE",
"HTML"
],
"vitality" : "8.8"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "站长",
"description" : "DDKK.COM 弟弟快看，程序员编程资料站 ，教程 ",
"street" : "东四十条",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100007",
"location" : [
116.432727,
39.937732
],
"money" : 5201814,
"tags" : [
"PHP",
"Python"
],
"vitality" : "9.0"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"nickname" : "歌者",
"description" : "程序设计也是设计，研发新菜也是研发",
"street" : "五道口",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100083",
"location" : [
116.346346,
39.999333
],
"money" : 71128,
"tags" : [
"Java",
"Scala"
],
"vitality" : "6.9"
}
}
]
},
"aggregations" : {
"min_money" : {
"value" : 10235.0
}
}
}
求和聚合 ( sum )
求和聚合 ( sum ) 用于计算聚合文档中特定数字字段的总和
例如
POST http://localhost:9200/user*/_search?pretty
请求正文
{
"aggs" : {
"total_money" : { "sum" : { "field" : "money" } }
}
}
返回响应
{
"took" : 10,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 5,
"max_score" : 1.0,
"hits" : [
{
"_index" : "user",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "枫晚",
"description" : "停车坐爰枫林晚",
"street" : "苏州大学",
"city" : "Suzhou",
"state" : "Jiangsu",
"zip" : "215006",
"location" : [
120.65426,
31.30797
],
"money" : 10235,
"tags" : [
"Java",
"Android"
],
"vitality" : "3.5"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"nickname" : "雅少",
"description" : "虚怀若谷",
"street" : "四川大学",
"city" : "Chengdu",
"state" : "Sichuan",
"zip" : "610044",
"location" : [
104.094537,
30.640174
],
"money" : 68023,
"tags" : [
"Python",
"HTML"
],
"vitality" : "7.8"
}
},
{
"_index" : "user",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "question",
"description" : "问题少年也是少年",
"street" : "张江高科技园区",
"city" : "Shanghai",
"state" : "Shanghai",
"zip" : "201204",
"location" : [
121.60632,
31.199305
],
"money" : 13648,
"tags" : [
"VUE",
"HTML"
],
"vitality" : "8.8"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"nickname" : "站长",
"description" : "DDKK.COM 弟弟快看，程序员编程资料站 ，教程 ",
"street" : "东四十条",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100007",
"location" : [
116.432727,
39.937732
],
"money" : 5201814,
"tags" : [
"PHP",
"Python"
],
"vitality" : "9.0"
}
},
{
"_index" : "user_admin",
"_type" : "user",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"nickname" : "歌者",
"description" : "程序设计也是设计，研发新菜也是研发",
"street" : "五道口",
"city" : "Beijing",
"state" : "Beijing",
"zip" : "100083",
"location" : [
116.346346,
39.999333
],
"money" : 71128,
"tags" : [
"Java",
"Scala"
],
"vitality" : "6.9"
}
}
]
},
"aggregations" : {
"total_money" : {
"value" : 5364848.0
}
}
}
此外，还存在一些其它聚合函数用于计算地理位置，如地理边界聚合和地理质心聚合
批量聚合 ( Bucket )
这些聚合包含了许多具有统一标准的不同类型的桶聚合，它们用于确定文档是否应该属于某个桶。
下面我们将会罗列这些桶聚合
子聚合
批量聚合会生成一组文档，这些文档将映射到父桶中
参数type 用于定义父索引
例如，假如我们有一个品牌及其不同的模型，然后模型类型将包含以下 _parent 字段
{
"model" : {
"_parent" : {
"type" : "brand"
}
}
}
还有很多其它的特殊的批量集合，在某些特定的情况下很好用，我们罗列如下
1、 DateHistogram聚合；
2、 DateRange聚合；
3、 Filter聚合；
4、 Filters聚合；
5、 GeoDistance聚合；
6、 GeoHashgrid聚合；
7、 Global聚合；
8、 Histogram聚合；
9、 IPv4Range聚合；
10、 Missing聚合；
11、 Nested聚合；
12、 Range聚合；
13、 Reversenested聚合；
14、 Sampler聚合；
15、 SignificantTerms聚合；
16、 Terms聚合；
聚合元数据
可以在请求时使用 meta 参数添加关于聚合的一些数据，然后就可以在响应时获取到这些数据
POST http://localhost:9200/user*/report/_search?pretty
请求正文
{
"aggs" : {
"min_money" : {
"avg" : { "field" : "money" } ,
"meta" :{"dsc" :"Lowest Moneys"}
}
}
}
响应内容
{
"took" : 30,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"min_money" : {
"meta" : {
"dsc" : "Lowest Moneys"
},
"value" : null
}
}
}