# Springboot + ElasticSearch 构建博客检索系统 **Repository Path**: Yenn-2017_admin/esblog ## Basic Information - **Project Name**: Springboot + ElasticSearch 构建博客检索系统 - **Description**: 练习https://www.imooc.com/learn/1161 - **Primary Language**: Java - **License**: MIT - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 0 - **Created**: 2019-11-21 - **Last Updated**: 2023-05-25 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # Springboot + ElasticSearch 构建博客检索系统 #### 介绍 视频练习 Springboot + ElasticSearch 构建博客检索系统 https://www.imooc.com/learn/1161 ### 第1章 课程介绍 本章会介绍学习内容，演示项目效果，梳理课程中使用到的工具，软件，以及本课涵盖的技术栈、重难点。 #### 1-1 课程导学 (08:33) + 课程介绍 - 学习内容和目标 - 项目效果预览![image-20221206150802957](./images/image-20221206150802957.png) + 思维导图![image-20221206150940987](./images/image-20221206150940987.png) + springboot+ es + springboot - 项目搭建 - jpa + mysql - rest + html - js跨域 - es集成 - es curd - 基于es template实现复杂查询 + ES - postman - kibana devtools - dsl + vue - 双向绑定 - 事件 - 数据交互+数据渲染 + 如何学习本课程 + 演示1： ### 第2章 初识ElasticSearch 本章讲解ES的基本概念，适用场景，与关系型数据库的对比。演示ES、kibana的安装与使用。 #### 2-1 ElasticSearch概念和适用场景 (04:46) 1. Elasticsearch概念和适用场景 分布式 全文检索 实时快速 Restful 2. Elasticsearch VS Mysql 3. Elasticsearch安装和部署 #### 2-2 ElasticSearch 数据类型，和关系型数据库的对比 (02:19) 1. Elasticsearch VS Mysql | Mysql | ES | | -------- | -------- | | Database | Index | | Table | Type | | Row | Document | | Column | Field | | Schema | Mapping | 2. Mysql: Select * from user.user_info where name =‘张三’; 3. ES: GET /user/user_info/_search?q=name:张三 #### 2-3 安装 ES、postman、kibana (06:30) 1. Elasticsearch安装和部署 + https://www.elastic.co/cn/ + https://www.elastic.co/cn/elasticsearch/ + 下载：https://www.elastic.co/cn/downloads/elasticsearch + 找到版本：V6.3.2，下载 + | SpringBoot版本 | Elasticsearch版本 | kibana版本 | 中文分词器ik | logstash全量、增量同步 | | -------------- | ------------------------------------------------------------ | -------------------------------------------------------- | ----------------------------------------------------------- | ---------------------------------------------------------- | | 2.0.0.RELEASE | V6.3.2 | V6.3.2 | V6.3.2 | V6.3.2 | | 2.3.7.RELEASE | V7.6.2 | V7.6.2 | V7.6.2 | V7.6.2 | | | https://www.elastic.co/cn/downloads/past-releases#elasticsearch | https://www.elastic.co/cn/downloads/past-releases#kibana | https://github.com/medcl/elasticsearch-analysis-ik/releases | https://www.elastic.co/cn/downloads/past-releases#logstash | + 启动：bin\elasticsearch.bat，单点ES启动 + http://localhost:9200页面显示： + ```html { "name" : "SkNjB2t", "cluster_name" : "elasticsearch", "cluster_uuid" : "Symsw8ZeSze9q_n-CxRWYg", "version" : { "number" : "6.3.2", "build_flavor" : "default", "build_type" : "zip", "build_hash" : "053779d", "build_date" : "2018-07-20T05:20:23.451332Z", "build_snapshot" : false, "lucene_version" : "7.3.1", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } ``` 2. kibana 1. 下载页面：https://www.elastic.co/cn/downloads/kibana + 启动：bin\kibana.bat + **kibana汉化:** 编辑器打开 /config/kibana.yml，添加：`i18n.locale: "zh-CN"` 重启kibana：已经汉化 + server running at http://localhost:5601 #### 2-4 演示postman、kibana对ES的交互 (11:46) 1. PostMan演示 + ![image-20221206162031447](./images/image-20221206162031447.png) + ![image-20221206162101813](./images/image-20221206162101813.png) 1. 查看所有索引 ```shell curl --location --request GET 'localhost:9200/_all' ``` 2. 创建索引-test ```shell curl --location --request PUT 'localhost:9200/test' ``` ```json { "acknowledged": true, "shards_acknowledged": true, "index": "test" } ``` 3. 删除索引-test ```shell curl --location --request DELETE 'localhost:9200/test' ``` ```json { "acknowledged": true } ``` 4. 创建索引-person ```shell curl --location --request PUT 'localhost:9200/person' ``` 5. 新增数据-person-1 ```shell curl --location --request PUT 'localhost:9200/person/_doc/1' \ --header 'Content-Type: application/json' \ --data-raw '{ "first_name": "John", "last_name": "Smith", "age": 25, "about": "I love to go rock climbling", "interests": [ "sports", "music" ] }' ``` 6. 搜索数据-person-id ```shell curl --location --request GET 'localhost:9200/person/_doc/1' ``` 7. 搜索数据-person-name ```shell curl --location --request GET 'localhost:9200/person/_doc/_search?q=first_name:john' ``` - Kibana演示 1. 查询所有：GET _all ![image-20221207160341700](./images/image-20221207160341700.png) 2. 查询person: GET /person/_doc/1 ![image-20221207161405176](./images/image-20221207161405176.png) 3. 查询表person的查询条件为全文检索关键字： ```apl POST /person/_search { "query": { "bool":{ "should":[ { "match": {"first_name": "Eric"} } ] } } } ``` ![image-20221207162324696](./images/image-20221207162324696.png) 4. 查询：OR条件 ```apl POST /person/_search { "query": { "bool": { "should": [ { "match": { "first_name": "Eric" } }, { "match": {"about":"rock"} } ] } } } ``` ![image-20221207162649313](./images/image-20221207162649313.png) 5. 查询：AND效果 ```apl POST /person/_search { "query": { "bool": { "must": [ { "match": { "last_name": "Smith" } }, { "match": {"about":"rock"} } ] } } } ``` ![image-20221207162829185](./images/image-20221207162829185.png) 6. ### 第3章 博客网站全文检索 本章中通过个人博客全文检索的实际场景，让学员了解现有mysql技术的局限性，并初步认识ES的检索性能。 #### 3-1 基于Mysql实现 (07:29) + 基于Mysql实现 + 索引 -> fulltext + 分库分表 -> 结果过滤 #### 3-2 基于ES实现 (04:57) + 基于ES实现 Term index Documents + ![image-20221207165453143](./images/image-20221207165453143.png) + ![image-20221207165654083](C:\Users\yerenyun\AppData\Roaming\Typora\typora-user-images\image-20221207165654083.png) ### 第4章 Mysql、ES 数据同步 本章中会介绍并对比logstash、mysql binlog 订阅、Go语言中间件。演示基于logstash 如何实现 全量和增量数据同步。 #### 4-1 数据同步中间件 (09:02) 1. 全量、增量 2. 几种实现方式 3. 开源中间件 + binlog订阅![image-20221207170209704](./images/image-20221207170209704.png) + alibaba/canal 阿里巴巴MySQL binlog增量订阅&消费组件 + siddontang/go-mysql-elasticsearch - Sync MySQL data into elasticsearch + logstash![image-20221207170422455](./images/image-20221207170422455.png) - id time + #### 4-2 logstash全量、增量同步解决方案 (10:30) 1. 下载：https://www.elastic.co/cn/downloads/logstash 2. 配置 config/mysql.conf ```apl input { jdbc { # jdbc驱动包位置 jdbc_driver_library => "E:\logstash-6.3.2\mysql-connector-java-5.1.31.jar" #要使用的驱动包类，有过java开发经验的应该很熟悉这个了，不同的数据库调用的类不一样。 jdbc_driver_class => "com.mysql.jdbc.Driver" # myqsl数据库的连接信息 jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/blog" # mysql用户 jdbc_user => "root" # mysql密码 jdbc_password => "root" # 定时任务， 多久执行一次查询，默认一分钟，如果想要没有延迟，可以使用schedule =>"* * * * *" schedule => "* * * * *" # *清空上次的sql_last_value记录* clean_run => true # 你要执行的语句 statement => "SELECT * FROM t_blog where update_time >:sql_last_value and update_time< NOW() order by update_time DESC " } } output{ elasticsearch { # es host : port hosts => ["127.0.0.1:9200"] # 索引 index => "blog" # _id document_id => "%{id}" } } ``` ![image-20221207172211244](./images/image-20221207172211244.png) 3. 运行： logstash -f ../config/mysql.conf 4. 示例： ![image-20221208142517079](./images/image-20221208142517079.png) ![image-20221208142643232](./images/image-20221208142643232.png) 运行： logstash -f ../config/mysql.conf ![image-20221208142847564](./images/image-20221208142847564.png) ![image-20221208142926547](./images/image-20221208142926547.png) ![image-20221208143059735](./images/image-20221208143059735.png) 5. SQL迭代 - Select * from t_blog where update_time >:sql_last_value order by update_time DESC; ![image-20221208143213207](./images/image-20221208143213207.png) ![image-20221208143335158](./images/image-20221208143335158.png) - Select * from t_blog where update_time >=:sql_last_value order by update_time DESC; ![image-20221208143414652](./images/image-20221208143414652.png) ![image-20221208143439668](./images/image-20221208143439668.png) - Select * from t_blog where update_time >:sql_last_value and update_time org.springframework.boot spring-boot-starter-data-elasticsearch ``` 2. 测试示例报错： ```apl 022-12-08 17:29:47.447 INFO 6236 --- [ main] o.s.d.elasticsearch.support.VersionInfo : Version Spring Data Elasticsearch: 4.0.6.RELEASE 2022-12-08 17:29:47.448 INFO 6236 --- [ main] o.s.d.elasticsearch.support.VersionInfo : Version Elasticsearch Client in build: 7.6.2 2022-12-08 17:29:47.448 INFO 6236 --- [ main] o.s.d.elasticsearch.support.VersionInfo : Version Elasticsearch Client used: 7.6.2 2022-12-08 17:29:47.449 INFO 6236 --- [ main] o.s.d.elasticsearch.support.VersionInfo : Version Elasticsearch cluster: 6.3.2 2022-12-08 17:29:47.449 WARN 6236 --- [ main] o.s.d.elasticsearch.support.VersionInfo : Version mismatch in between Elasticsearch Client and Cluster: 7.6.2 - 6.3.2 2022-12-08 17:29:48.803 WARN 6236 --- [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning ``` 3. #### 5-6 项目后端 REST API 实现 (17:07) 1. 新建 DataController.java 2. 返回数据表中的博客信息 ，URL: http://localhost:8081/blogs 3. DSL语句 ```apl POST /blog/_search { "query": { "bool": { "should": [ {"match_phrase": { "title": "springboot" }}, { "match_phrase": { "content": "springboot" } } ] } } } ``` ![image-20221212135138502](./images/image-20221212135138502.png) #### 5-7 项目前端 VUE 视图渲染 (21:49) ### 第6章 课程回顾与总结 本章中结合思维导图，回顾课程中用到的技术，并提出几点建议。 #### 6-1 课程回顾与总结 (05:04)