万亿级流量转发：BFE核心技术与实现-图书

推荐0
收藏0
浏览585

万亿级流量转发：BFE核心技术与实现

章淼 (作者)　

书　　号：978-7-121-41565-4
出版日期：2021-08-02
页　　数：280
开　　本：16(170*240)
出版状态：上市销售
维护人：滕亚帆

纸质版￥89.00

本书围绕BFE 开源项目，介绍网络前端接入和网络负载均衡的相关技术原理，说明BFE开源软件的设计思想和实现机制，讲解如何基于BFE开源软件搭建网络接入平台。
本书共17 章，分为四个部分。第一部分为原理篇，介绍BFE 开源项目的概貌，并对网络前端接入技术和网络负载均衡技术做简要介绍；第二部分为设计篇，介绍BFE 开源项目的设计细节，包括BFE 的设计思想、BFE 的转发模型、BFE 的关键机制和HTTPS 的优化等；第三部分为操作篇，说明如何安装和部署BFE、如何在各种使用场景下配置BFE；第四部分为实现篇，从处理流程、插件机制、协议实现等方面说明BFE的实现机制。

万亿级流量转发平台BFE团队首著。

序

从2014年4月写下Go语言版本BFE的第一行代码起，7年多的时间过去了。从2015年1月Go语言版本BFE全量上线开始，BFE至今已经在百度稳定运行了6年多的时间，每天转发请求超过万亿次。

BFE（Baidu Front End，百度统一前端）是百度统一七层流量转发平台，当你访问百度的时候，很可能已经在使用BFE的服务了。
百度的BFE团队始建于2012年。当2012年年底我加入百度的时候，整个团队只有6~7个人。这个团队的创始人是夏华夏同学（现在在美团），他为BFE团队的工作方向做了奠基性规划。BFE初期的转发引擎是基于C语言的，听说是杨震原同学（现在在头条）的大作。
2014年年初，基于各种考虑，我们决定对转发引擎进行重构。这次重构前后花费了3个季度，投入了超过30个人月的资源。在面对多次失败的风险后，Go语言版本的BFE终于出炉了。
这里必须感谢部门领导李硕和团队经理贺锋的大力支持，感谢管理层的高度信任；感谢直接参与的几位同学（李炳毅、魏为、杨思杰、陶春华等），大家都是冒着失败离职的风险，硬着头皮把这个项目做下来的。
我必须要感谢百度。到目前为止，我仍然坚定地认为，百度是中国最适合做技术的公司。百度给了工程师最大的尊重和自由，也愿意为了技术研发承担最大的风险。BFE团队的另一个项目GTC(全局流量调度)，前后研发了5年时间。曾经有一个朋友告诉我，也就只有百度可以给团队这么多的时间，如果在其他公司，一年内做不出来，项目很可能就被取消了。能够在百度、在中国做全球最领先的技术，我感到无比骄傲。

2019年7月，BFE的转发引擎对外开源。项目名称仍保留英文缩写BFE，英文全称更名为Beyond Front End（中文意为“超越前端”）。我们希望通过BFE的开源推动负载均衡技术的发展。
从开源的那天起，BFE就已经开始了新的征程。BFE得到了各方的广泛关注，有不少新增的功能是由百度之外的开发者贡献的。BFE也被一些客户选择用于关键的业务场景，在度小满金融、央视网、招商银行等处都有BFE的身影。作为一个做技术的人，能够让自己所做的工作为社会创造价值，这是莫大的幸福和荣幸。

在BFE开源后，我们不断地收到大家提出的一些问题。网络负载均衡本身是一个比较专业和复杂的技术方向；BFE是为面向工业级使用场景而设计的，在模型和机制上和其他同类软件相比，会更加复杂。以上这些因素让一些使用者和开发者在理解BFE的机制方面遇到了困难。希望通过《深入理解BFE》这本书，能够帮助读者更好地了解网络负载均衡的相关技术，让读者更容易地理解BFE的设计机制和使用方法。

BFE项目，是一群技术人的汗水、梦想和追求。
BFE开源，是为了交流、共享，为全中国、全世界的同行赋能。

感谢各位读者的关注，欢迎大家使用BFE开源项目，并提出反馈或参与开发！

章淼博士
百度BFE团队技术负责人、百度代码规范委员会主席
2021年6月25日写于百度

原理篇
第1 章 BFE 简介·····························································································3
1.1 什么是BFE·························································································3
1.2 BFE平台介绍·····················································································4
1.2.1 为什么需要构建BFE 平台·····················································4
1.2.2 BFE平台的主要功能·····························································6
1.3 BFE开源项目介绍·············································································7
1.3.1 BFE平台的模块组成·····························································7
1.3.2 BFE开源项目中的内容··························································9
第2 章网络前端接入技术简介·····································································10
2.1 什么是网络前端接入········································································10
2.2 网络前端接入面临的挑战································································11
2.3 百度的网络前端接入方案································································13
2.4 网络前端接入技术的发展趋势························································15
第3 章网络负载均衡技术简介·····································································24
3.1 负载均衡的概念···············································································24
3.2 网络负载均衡功能的实现································································25
深入理解BFE
·VI·
3.2.1 机制说明···············································································25
3.2.2 两种方式对比········································································27
3.3 四层负载均衡和七层负载均衡························································28
设计篇
第4 章 BFE 的设计思想················································································33
4.1 BFE转发引擎重构的缘起································································33
4.2 BFE为什么要基于Go 语言·····························································34
4.3 BFE转发引擎的主要设计思想························································37
4.4 BFE和相关开源项目的对比····························································38
第5 章 BFE 的转发模型················································································42
5.1 转发模型概述···················································································42
5.1.1 基本概念···············································································42
5.1.2 转发过程···············································································43
5.1.3 对多租户实现机制的讨论····················································46
5.2 BFE的路由转发机制·······································································47
5.3 条件表达式·······················································································49
5.3.1 设计思想···············································································49
5.3.2 基本概念···············································································50
5.3.3 语法介绍···············································································51
5.3.4 条件原语匹配的内容····························································52
5.3.5 条件原语名称的规范····························································54
5.4 内网流量调度机制············································································55
5.4.1 内网流量调度背景介绍························································55
5.4.2 内网流量调度工作机制························································60
目录
·VII·
5.4.3 内网转发的其他机制····························································65
第6 章与转发相关的关键机制·····································································68
6.1 健康检查机制···················································································68
6.1.1 健康检查的原理····································································69
6.1.2 主动健康检查和被动健康检查············································69
6.1.3 分布式健康检查和集中式健康检查····································73
6.1.4 BFE的健康检查···································································77
6.2 超时设置···························································································77
6.2.1 BFE和客户端间通信的超时················································78
6.2.2 BFE和后端实例间通信的超时············································81
6.3 BFE信息透传···················································································84
6.3.1 客户端IP地址的透传··························································84
6.3.2 其他信息的透传····································································85
6.4 限流机制···························································································89
6.4.1 限流的概念···········································································89
6.4.2 限流的配置···········································································91
6.4.3 分布式限流···········································································91
6.4.4 入口限流和目的限流····························································94
6.4.5 限流和内网流量调度····························································95
第7 章运维相关机制····················································································97
7.1 监控机制···························································································97
7.1.1 日志监控及其问题································································98
7.1.2 BFE的内部状态输出···························································99
7.1.3 统计状态和日志的配合使用··············································101
深入理解BFE
·VIII·
7.2 Web Monitor基础库·······································································101
7.2.1 Web Monitor概述·······························································102
7.2.2 状态变量维护······································································102
7.2.3 延迟统计变量维护······························································104
7.2.4 建立专用的Web服务器·····················································106
7.2.5 注册回调函数······································································107
7.3 日志机制·························································································108
7.3.1 日志类型·············································································108
7.3.2 日志打印的注意事项··························································109
7.3.3 BFE的访问日志·································································110
7.4 配置管理························································································· 111
7.4.1 BFE配置文件的分布·························································· 111
7.4.2 常规配置和动态配置··························································112
7.4.3 动态配置的实现机制··························································113
第8 章 HTTPS 的优化················································································116
8.1 HTTPS优化背景及必要性·····························································116
8.2 HTTPS优化的挑战········································································117
8.3 HTTPS中的优化机制·····································································118
8.4 BFE中HTTPS 相关增强机制························································120
操作篇
第9 章 BFE 服务的安装部署······································································129
9.1 软件安装包下载安装······································································129
9.2 源代码编译方式安装······································································132
9.3 Docker方式安装·············································································133
目录
·IX·
9.4 BFE命令行参数·············································································134
9.5 查看BFE 服务的运行状态·····························································135
第10 章 BFE 服务的基础配置····································································136
10.1 场景说明·······················································································136
10.2 修改基础配置文件········································································137
10.3 转发的配置···················································································138
10.3.1 转发配置流程····································································138
10.3.2 具体案例···········································································139
10.3.3 服务访问验证····································································146
10.3.4 配置的重新加载································································146
第11 章配置负载均衡算法及会话保持·····················································147
11.1 子集群间的负载均衡····································································147
11.2 子集群级别的会话保持································································149
11.2.1 配置实例···········································································149
11.2.2 参数的具体含义································································150
11.3 实例间的负载均衡········································································151
11.3.1 加权轮询配置示例····························································151
11.3.2 最小连接数的配置示例····················································153
11.4 实例级别的会话保持····································································154
第12 章配置HTTPS 和更多协议·····························································155
12.1 设置HTTPS 基础配置··································································155
12.1.1 配置HTTPS 端口·····························································156
12.1.2 配置加密套件····································································156
12.1.3 配置服务端证书································································156
深入理解BFE
·X·
12.1.4 配置TLS 规则···································································157
12.2 配置TLS 会话重用·······································································158
12.2.1 配置会话缓存····································································159
12.2.2 配置会话票证····································································160
12.3 配置TLS 双向认证·······································································161
12.4 对不同安全等级的区分································································163
12.5 支持更多协议···············································································164
12.5.1 HTTP/2配置·····································································165
12.5.2 SPDY 配置········································································166
12.5.3 WebSocket配置·································································167
12.5.4 连接后端服务的协议························································169
第13 章其他常用配置················································································171
13.1 配置重写·······················································································171
13.1.1 开启重写···········································································171
13.1.2 模块配置···········································································171
13.1.3 重写动作详细描述····························································173
13.2 配置重定向···················································································177
13.2.1 开启重定向········································································177
13.2.2 模块配置···········································································177
13.2.3 重定向动作详细描述························································178
13.3 配置限流功能···············································································180
13.3.1 开启限流模块····································································180
13.3.2 模块配置···········································································181
13.3.3 限制特定维度的流量························································182
13.3.4 设置限流门限····································································183
13.3.5 设置限流动作····································································184
目录
·XI·
实现篇
第14 章其他常用配置················································································187
14.1 BFE的代码组织···········································································187
14.2 BFE的进程模型···········································································190
14.2.1 协程的分类·······································································190
14.2.2 并发模型···········································································191
14.2.3 并发能力···········································································192
14.2.4 异常恢复机制····································································192
14.3 请求处理流程···············································································193
14.3.1 连接的建立·······································································193
14.3.2 连接的处理·······································································194
14.3.3 请求的处理·······································································196
14.3.4 请求的结束·······································································198
14.3.5 连接的结束·······································································199
14.4 请求路由实现···············································································199
14.4.1 关键数据结构····································································200
14.4.2 目的租户路由····································································203
14.4.3 目的集群路由····································································204
14.5 负载均衡实现···············································································206
14.5.1 全局负载均衡····································································207
14.5.2 分布式负载均衡································································210
第15 章模块插件机制················································································214
15.1 BFE的回调点设置·······································································214
15.2 BFE内置的扩展模块···································································216
15.3 模块框架的实现机制····································································217
深入理解BFE
·XII·
15.3.1 模块基础类型····································································217
15.3.2 连接/请求处理及回调函数的调用···································223
15.4 如何开发BFE扩展模块·······························································224
15.4.1 配置加载···········································································225
15.4.2 回调函数的编写和注册····················································226
15.4.3 模块状态的展示································································228
第16 章核心协议实现················································································231
16.1 HTTP的实现················································································231
16.1.1 HTTP 代码的组织·····························································231
16.1.2 从用户读取HTTP 请求····················································233
16.1.3 向后端转发请求并获取响应············································236
16.1.4 向用户回复HTTP 响应····················································239
16.2 HTTP2的实现··············································································240
16.2.1 HTTP2 代码的组织···························································240
16.2.2 HTTP2 连接处理模块·······················································241
16.2.3 HTTP2 连接相关协程及关系···········································243
第17 章 BFE 的多进程GC 机制································································249
17.1 模型设计·······················································································249
17.1.1 多进程轮转机制································································250
17.1.2 子进程状态定义································································252
17.2 相关参数的确定············································································253
17.2.1 切换时间参数的选择························································253
17.2.2 子进程数量的计算····························································255
17.2.3 内存消耗的计算································································256