架构师系列-搜索引擎ElasticSearch（四）- 高级查询

ES查询

matchAll

脚本方式

该方式可以通过kabana、curl、elasticsearch-head（纯前端）去操作

# 默认情况下，es一次展示10条数据,通过from和size来控制分页
# 查询结果详解
GET goods/_search
{
    "query": {
        "match_all": {}
    },
    "from": 0,
    "size": 100
}

# 默认为10条
GET goods

Java API方式

/**
     * 查询所有
     *  1. matchAll
     *  2. 将查询结果封装为Goods对象，装载到List中
     *  3. 分页。默认显示10条
     */
    @Test
    public void testMatchAll() throws IOException {
        //2. 构建查询请求对象，指定查询的索引名称
        SearchRequest searchRequest = new SearchRequest("goods");
        //4. 创建查询条件构建器SearchSourceBuilder
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();

        //6. 查询条件
        QueryBuilder query = QueryBuilders.matchAllQuery();//查询所有文档
        //5. 指定查询条件
        sourceBuilder.query(query);

        //3. 添加查询条件构建器 SearchSourceBuilder
        searchRequest.source(sourceBuilder);

        // 8 . 添加分页信息
        sourceBuilder.from(0);
        sourceBuilder.size(100);

        //1. 查询,获取查询结果
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

        //7. 获取命中对象 SearchHits
        SearchHits searchHits = searchResponse.getHits();
        //7.1 获取总记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);


        List<Goods> goodsList = new ArrayList<>();
        //7.2 获取Hits数据  数组
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            //获取json字符串格式的数据
            String sourceAsString = hit.getSourceAsString();
            //转为java对象
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);

            goodsList.add(goods);

        }


        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }

termQuery

term查询和字段类型有关系，首先回顾一下ElasticSearch两个数据类型
ElasticSearch两个数据类型
1、text：会分词，不支持聚合
2、keyword：不会分词，将全部内容作为一个词条，支持聚合

脚本方式

term查询：不会对查询条件进行分词。

GET goods/_search
{
    "query": {
        "term": {
            "title": {
                "value": "华为"
            }
        }
    }
}

term查询，查询text类型字段时，只有其中的单词相匹配都会查到，text字段会对数据进行分词
例如：查询title 为“华为”的，title type 为text

"title": {
    "type": "text",
    "analyzer": "ik_smart"
}

查询categoryName 字段时，categoryName字段为keyword ,keyword：不会分词，将全部内容作为一个词条,即完全匹配，才能查询出结果.

GET goods/_search
{
    "query": {
        "term": {
            "categoryName": {
                "value": "华为手机"
            }
        }
    }
}

Java API

/**
     * termQuery:词条查询
     */
    @Test
    public void testTermQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        QueryBuilder query = QueryBuilders.termQuery("title","华为");//term词条查询
        sourceBulider.query(query);
        searchRequest.source(sourceBulider);

        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);
        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            goodsList.add(goods);
        }
        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }

matchQuery

match查询：
•会对查询条件进行分词。
•然后将分词后的查询条件和词条进行等值匹配
•默认取并集（OR）

# match查询
GET goods/_search
{
    "query": {
        "match": {
            "title": "华为手机"
        }
    },
    "size": 500
}

match 的默认搜索（or 并集）
例如：华为手机，会分词为 “华为”，“手机” 只要出现其中一个词条都会搜索到
match的 and（交集）搜索
例如：例如：华为手机，会分词为 “华为”，“手机” 但要求“华为”，和“手机”同时出现在词条中

总结：
term query会去倒排索引中寻找确切的term，它并不知道分词器的存在。这种查询适合keyword
、numeric、date
match query知道分词器的存在。并且理解是如何被分词的

java api

 /**
     * matchQuery:词条分词查询
     */
    @Test
    public void testMatchQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        MatchQueryBuilder query = QueryBuilders.matchQuery("title", "华为手机");
        query.operator(Operator.AND);//求并集
        sourceBulider.query(query);

        searchRequest.source(sourceBulider);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);
        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            goodsList.add(goods);
        }
        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }

模糊查询-脚本

wildcard查询

wildcard查询：会对查询条件进行分词。还可以使用通配符 ?（任意单个字符）和 * （0个或多个字符）


*华*" 包含华字的
"华*" 华字后边多个字符
"华?" 华字后边多个字符
"*华"或"?华" 会引发全表（全索引）扫描 注意效率问题

# wildcard 查询。查询条件分词，模糊查询
GET goods/_search
{
    "query": {
        "wildcard": {
            "title": {
                "value": "华*"
            }
        }
    }
}

正则查询

\W：匹配包括下划线的任何单词字符，等价于 [A-Z a-z 0-9_] 开头的反斜杠是转义符
+号多次出现
(.)*为任意字符
正则查询取决于正则表达式的效率

GET goods/_search
{
    "query": {
        "regexp": {
            "title": "\\w+(.)*"
        }    
    }
}

前缀查询

#前缀查询 对keyword类型支持比较好
GET goods/_search
{
    "query": {
        "prefix": {
            "brandName": {
                "value": "三"
            }
        }
    }
}

模糊查询- Java Api

 /**
     * 模糊查询:WildcardQuery
     */
    @Test
    public void testWildcardQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        WildcardQueryBuilder query = QueryBuilders.wildcardQuery("title", "华*");
        sourceBulider.query(query);
        searchRequest.source(sourceBulider);

        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);
        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            goodsList.add(goods);
        }
        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }




    /**
     * 模糊查询:regexpQuery
     */
    @Test
    public void testRegexpQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        RegexpQueryBuilder query = QueryBuilders.regexpQuery("title", "\\w+(.)*");
        sourceBulider.query(query);
        searchRequest.source(sourceBulider);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);

        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            goodsList.add(goods);
        }
        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }



    /**
     * 模糊查询:perfixQuery
     */
    @Test
    public void testPrefixQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        PrefixQueryBuilder query = QueryBuilders.prefixQuery("brandName", "三");
        sourceBulider.query(query);
        searchRequest.source(sourceBulider);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);
        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            goodsList.add(goods);
        }
        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }

范围和排序

# 范围查询
GET goods/_search
{
    "query": {
        "range": {
            "price": {
                "gte": 2000,
                "lte": 3000
            }
        }
    },
    "sort": [
        {
            "price": {
                "order": "desc"
            }
        }
    ]
}

/**
     * 1. 范围查询：rangeQuery
     * 2. 排序
     */
    @Test
    public void testRangeQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        //范围查询
        RangeQueryBuilder query = QueryBuilders.rangeQuery("price");
        //指定下限
        query.gte(2000);
        //指定上限
        query.lte(3000);
        sourceBulider.query(query);
        //排序
        sourceBulider.sort("price", SortOrder.DESC);
        searchRequest.source(sourceBulider);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);

        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            goodsList.add(goods);
        }
        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }

queryString查询

queryString 多条件查询
•会对查询条件进行分词。
•然后将分词后的查询条件和词条进行等值匹配
•默认取并集（OR）
•可以指定多个查询字段
query_string：识别query中的连接符（or 、and）

# queryString
GET goods/_search
{
    "query": {
        "query_string": {
            "fields": ["title","categoryName","brandName"],
            "query": "华为 AND 手机"
        }
    }
}

#simple_query_string：不识别query中的连接符（or 、and），查询时会将 “华为”、"and"、“手机”分别进行查询

GET goods/_search
{
    "query": {
        "simple_query_string": {
            "fields": ["title","categoryName","brandName"],
            "query": "华为 AND 手机"
        }
    }
}

# simple_query_string：有default_operator连接符的脚本

GET goods/_search
{
    "query": {
        "simple_query_string": {
            "fields": ["title","brandName","categoryName"],
            "query": "华为手机 "
            , "default_operator": "OR"
        }
    }
}

// 注意：query中的or and 是查询时 匹配条件是否同时出现----or 出现一个即可，and 两个条件同时出现default_operator的or and 是对结果进行 并集（or）、交集（and）


query_string：有default_operator连接符的脚本

GET goods/_search
{
    "query": {
        "query_string": {
            "fields": ["title","brandName","categoryName"],
            "query": "华为手机 ", 
            "default_operator": "AND"
        }
    }
}

Java API


    /**
     * queryString
     */
    @Test
    public void testQueryStringQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        //queryString
        QueryStringQueryBuilder query = QueryBuilders.queryStringQuery("华为手机").field("title").field("categoryName").field("brandName").defaultOperator(Operator.AND);
        sourceBulider.query(query);
        searchRequest.source(sourceBulider);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);

        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();

            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);

            goodsList.add(goods);
        }

        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }

布尔查询

boolQuery：对多个查询条件连接。连接方式：
•must（and）：条件必须成立
•must_not（not）：条件必须不成立
•should（or）：条件可以成立
•filter：条件必须成立，性能比must高。不会计算得分
得分:即条件匹配度,匹配度越高，得分越高

eg: 布尔查询：boolQuery
1. 查询品牌名称为:华为
2. 查询标题包含：手机
3. 查询价格在：2000-3000

#boolquery
#must和filter配合使用时，max_score（得分）是显示的
#must 默认数组形式
GET goods/_search
{
    "query": {
        "bool": {
            "must": [
                {
                    "term": {
                        "brandName": {
                            "value": "华为"
                        }
                    }
                }
            ],
            "filter":[
                { 
                    "term": {
                        "title": "手机"
                    }
                },
                {
                    "range":{

                        "price": {
                            "gte": 2000,
                            "lte": 3000
                        } 
                    }
                }
            ]
        }
    }

    } 

#filter 单独使用 filter可以是单个条件，也可多个条件（数组形式）
GET goods/_search
{
    "query": {
        "bool": {
            "filter": [
            {
                "term": {
                    "brandName": {
                        "value": "华为"
                    }
                }
            }
            ]
        }
    }
}

java api

  /**
     * 布尔查询：boolQuery
     * 1. 查询品牌名称为:华为
     * 2. 查询标题包含：手机
     * 3. 查询价格在：2000-3000
     */
    @Test
    public void testBoolQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        //1.构建boolQuery
        BoolQueryBuilder query = QueryBuilders.boolQuery();
        //2.构建各个查询条件
        //2.1 查询品牌名称为:华为
        QueryBuilder termQuery = QueryBuilders.termQuery("brandName","华为");
        query.must(termQuery);
        //2.2. 查询标题包含：手机
        QueryBuilder matchQuery = QueryBuilders.matchQuery("title","手机");
        query.filter(matchQuery);
        //2.3 查询价格在：2000-3000
        QueryBuilder rangeQuery = QueryBuilders.rangeQuery("price");
        ((RangeQueryBuilder) rangeQuery).gte(2000);
        ((RangeQueryBuilder) rangeQuery).lte(3000);
        query.filter(rangeQuery);
        //3.使用boolQuery连接
        sourceBulider.query(query);
        searchRequest.source(sourceBulider);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);
        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            goodsList.add(goods);
        }
        for (Goods goods : goodsList) {
            System.out.println(goods);
        }
    }

聚合查询

指标聚合：相当于MySQL的聚合函数。max、min、avg、sum等
•桶聚合：相当于MySQL的 group by 操作。不要对text类型的数据进行分组，会失败。

# 聚合查询
# 指标聚合 聚合函数
GET goods/_search
{
  "query": {
  "match": {
    "title": "手机"
  }
},
  "aggs": {
  "max_price": {
    "max": {
      "field": "price"
    }
  }
}
} 

#桶聚合 分组
GET goods/_search
{
  "query": {
  "match": {
    "title": "手机"
  }
},
  "aggs": {
  "goods_brands": {
    "terms": {
      "field": "brandName",
          "size": 100
    }
  }
}
}

java api

/**
     * 聚合查询：桶聚合，分组查询
     * 1. 查询title包含手机的数据
     * 2. 查询品牌列表
     */
    @Test
    public void testAggQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        // 1. 查询title包含手机的数据
        MatchQueryBuilder query = QueryBuilders.matchQuery("title", "手机");
        sourceBulider.query(query);
        // 2. 查询品牌列表
        /*
        参数：
            1. 自定义的名称，将来用于获取数据
            2. 分组的字段
         */
        AggregationBuilder agg = AggregationBuilders.terms("goods_brands").field("brandName").size(100);
        sourceBulider.aggregation(agg);
        searchRequest.source(sourceBulider);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);

        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            goodsList.add(goods);
        }

        for (Goods goods : goodsList) {
            System.out.println(goods);
        }

        // 获取聚合结果
        Aggregations aggregations = searchResponse.getAggregations();
        Map<String, Aggregation> aggregationMap = aggregations.asMap();
        //System.out.println(aggregationMap);
        Terms goods_brands = (Terms) aggregationMap.get("goods_brands");
        List<? extends Terms.Bucket> buckets = goods_brands.getBuckets();
        List brands = new ArrayList();
        for (Terms.Bucket bucket : buckets) {
            Object key = bucket.getKey();
            brands.add(key);
        }

        for (Object brand : brands) {
            System.out.println(brand);
        }

    }

高亮查询

高亮三要素：
•高亮字段
•前缀
•后缀
默认前后缀：em

GET goods/_search
{
  "query": {
  "match": {
    "title": "电视"
  }
},
  "highlight": {
  "fields": {
    "title": {
      "pre_tags": "<font color='red'>",
          "post_tags": "</font>"
    }
  }
}
}

java api

/**
     *
     * 高亮查询：
     *  1. 设置高亮
     *      * 高亮字段
     *      * 前缀
     *      * 后缀
     *  2. 将高亮了的字段数据，替换原有数据
     */
    @Test
    public void testHighLightQuery() throws IOException {
        SearchRequest searchRequest = new SearchRequest("goods");
        SearchSourceBuilder sourceBulider = new SearchSourceBuilder();
        // 1. 查询title包含手机的数据
        MatchQueryBuilder query = QueryBuilders.matchQuery("title", "手机");
        sourceBulider.query(query);
        //设置高亮
        HighlightBuilder highlighter = new HighlightBuilder();
        //设置三要素
        highlighter.field("title");
        highlighter.preTags("<font color='red'>");
        highlighter.postTags("</font>");
        sourceBulider.highlighter(highlighter);
        // 2. 查询品牌列表
        /*
        参数：
            1. 自定义的名称，将来用于获取数据
            2. 分组的字段
         */
        AggregationBuilder agg = AggregationBuilders.terms("goods_brands").field("brandName").size(100);
        sourceBulider.aggregation(agg);
        searchRequest.source(sourceBulider);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits searchHits = searchResponse.getHits();
        //获取记录数
        long value = searchHits.getTotalHits().value;
        System.out.println("总记录数："+value);
        List<Goods> goodsList = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        for (SearchHit hit : hits) {
            String sourceAsString = hit.getSourceAsString();
            //转为java
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            // 获取高亮结果，替换goods中的title
            Map<String, HighlightField> highlightFields = hit.getHighlightFields();
            HighlightField HighlightField = highlightFields.get("title");
            Text[] fragments = HighlightField.fragments();
            //替换
            goods.setTitle(fragments[0].toString());
            goodsList.add(goods);
        }
        for (Goods goods : goodsList) {
            System.out.println(goods);
        }

        // 获取聚合结果
        Aggregations aggregations = searchResponse.getAggregations();
        Map<String, Aggregation> aggregationMap = aggregations.asMap();
        //System.out.println(aggregationMap);
        Terms goods_brands = (Terms) aggregationMap.get("goods_brands");

        List<? extends Terms.Bucket> buckets = goods_brands.getBuckets();

        List brands = new ArrayList();
        for (Terms.Bucket bucket : buckets) {
            Object key = bucket.getKey();
            brands.add(key);
        }

        for (Object brand : brands) {
            System.out.println(brand);
        }

    }