万亿级流量转发:BFE核心技术与实现
  • 推荐0
  • 收藏0
  • 浏览207

万亿级流量转发:BFE核心技术与实现

章淼 (作者) 

  • 书  号:978-7-121-41565-4
  • 出版日期:2021-08-02
  • 页  数:280
  • 开  本:16(170*240)
  • 出版状态:上市销售
  • 维护人:滕亚帆
纸质版 ¥89.00
本书围绕BFE 开源项目,介绍网络前端接入和网络负载均衡的相关技术原理,说明BFE开源软件的设计思想和实现机制,讲解如何基于BFE开源软件搭建网络接入平台。
本书共17 章,分为四个部分。第一部分为原理篇,介绍BFE 开源项目的概貌,并对网络前端接入技术和网络负载均衡技术做简要介绍;第二部分为设计篇,介绍BFE 开源项目的设计细节,包括BFE 的设计思想、BFE 的转发模型、BFE 的关键机制和HTTPS 的优化等;第三部分为操作篇,说明如何安装和部署BFE、如何在各种使用场景下配置BFE;第四部分为实现篇,从处理流程、插件机制、协议实现等方面说明BFE的实现机制。
万亿级流量转发平台BFE团队首著。


从2014年4月写下Go语言版本BFE的第一行代码起,7年多的时间过去了。从2015年1月Go语言版本BFE全量上线开始,BFE至今已经在百度稳定运行了6年多的时间,每天转发请求超过万亿次。

BFE(Baidu Front End,百度统一前端)是百度统一七层流量转发平台,当你访问百度的时候,很可能已经在使用BFE的服务了。
百度的BFE团队始建于2012年。当2012年年底我加入百度的时候,整个团队只有6~7个人。这个团队的创始人是夏华夏同学(现在在美团),他为BFE团队的工作方向做了奠基性规划。BFE初期的转发引擎是基于C语言的,听说是杨震原同学(现在在头条)的大作。
2014年年初,基于各种考虑,我们决定对转发引擎进行重构。这次重构前后花费了3个季度,投入了超过30个人月的资源。在面对多次失败的风险后,Go语言版本的BFE终于出炉了。
这里必须感谢部门领导李硕和团队经理贺锋的大力支持,感谢管理层的高度信任;感谢直接参与的几位同学(李炳毅、魏为、杨思杰、陶春华等),大家都是冒着失败离职的风险,硬着头皮把这个项目做下来的。
我必须要感谢百度。到目前为止,我仍然坚定地认为,百度是中国最适合做技术的公司。百度给了工程师最大的尊重和自由,也愿意为了技术研发承担最大的风险。BFE团队的另一个项目GTC(全局流量调度),前后研发了5年时间。曾经有一个朋友告诉我,也就只有百度可以给团队这么多的时间,如果在其他公司,一年内做不出来,项目很可能就被取消了。能够在百度、在中国做全球最领先的技术,我感到无比骄傲。

2019年7月,BFE的转发引擎对外开源。项目名称仍保留英文缩写BFE,英文全称更名为Beyond Front End(中文意为“超越前端”)。我们希望通过BFE的开源推动负载均衡技术的发展。
从开源的那天起,BFE就已经开始了新的征程。BFE得到了各方的广泛关注,有不少新增的功能是由百度之外的开发者贡献的。BFE也被一些客户选择用于关键的业务场景,在度小满金融、央视网、招商银行等处都有BFE的身影。作为一个做技术的人,能够让自己所做的工作为社会创造价值,这是莫大的幸福和荣幸。

在BFE开源后,我们不断地收到大家提出的一些问题。网络负载均衡本身是一个比较专业和复杂的技术方向;BFE是为面向工业级使用场景而设计的,在模型和机制上和其他同类软件相比,会更加复杂。以上这些因素让一些使用者和开发者在理解BFE的机制方面遇到了困难。希望通过《深入理解BFE》这本书,能够帮助读者更好地了解网络负载均衡的相关技术,让读者更容易地理解BFE的设计机制和使用方法。

BFE项目,是一群技术人的汗水、梦想和追求。
BFE开源,是为了交流、共享,为全中国、全世界的同行赋能。

感谢各位读者的关注,欢迎大家使用BFE开源项目,并提出反馈或参与开发!

章淼 博士
百度BFE团队技术负责人、百度代码规范委员会主席
2021年6月25日写于百度

目录

原 理 篇
第1 章 BFE 简介·····························································································3
1.1 什么是BFE·························································································3
1.2 BFE平台介绍·····················································································4
1.2.1 为什么需要构建BFE 平台·····················································4
1.2.2 BFE平台的主要功能·····························································6
1.3 BFE开源项目介绍·············································································7
1.3.1 BFE平台的模块组成·····························································7
1.3.2 BFE开源项目中的内容··························································9
第2 章 网络前端接入技术简介·····································································10
2.1 什么是网络前端接入········································································10
2.2 网络前端接入面临的挑战································································11
2.3 百度的网络前端接入方案································································13
2.4 网络前端接入技术的发展趋势························································15
第3 章 网络负载均衡技术简介·····································································24
3.1 负载均衡的概念···············································································24
3.2 网络负载均衡功能的实现································································25
深入理解BFE
·VI·
3.2.1 机制说明···············································································25
3.2.2 两种方式对比········································································27
3.3 四层负载均衡和七层负载均衡························································28
设 计 篇
第4 章 BFE 的设计思想················································································33
4.1 BFE转发引擎重构的缘起································································33
4.2 BFE为什么要基于Go 语言·····························································34
4.3 BFE转发引擎的主要设计思想························································37
4.4 BFE和相关开源项目的对比····························································38
第5 章 BFE 的转发模型················································································42
5.1 转发模型概述···················································································42
5.1.1 基本概念···············································································42
5.1.2 转发过程···············································································43
5.1.3 对多租户实现机制的讨论····················································46
5.2 BFE的路由转发机制·······································································47
5.3 条件表达式·······················································································49
5.3.1 设计思想···············································································49
5.3.2 基本概念···············································································50
5.3.3 语法介绍···············································································51
5.3.4 条件原语匹配的内容····························································52
5.3.5 条件原语名称的规范····························································54
5.4 内网流量调度机制············································································55
5.4.1 内网流量调度背景介绍························································55
5.4.2 内网流量调度工作机制························································60
目 录
·VII·
5.4.3 内网转发的其他机制····························································65
第6 章 与转发相关的关键机制·····································································68
6.1 健康检查机制···················································································68
6.1.1 健康检查的原理····································································69
6.1.2 主动健康检查和被动健康检查············································69
6.1.3 分布式健康检查和集中式健康检查····································73
6.1.4 BFE的健康检查···································································77
6.2 超时设置···························································································77
6.2.1 BFE和客户端间通信的超时················································78
6.2.2 BFE和后端实例间通信的超时············································81
6.3 BFE信息透传···················································································84
6.3.1 客户端IP地址的透传··························································84
6.3.2 其他信息的透传····································································85
6.4 限流机制···························································································89
6.4.1 限流的概念···········································································89
6.4.2 限流的配置···········································································91
6.4.3 分布式限流···········································································91
6.4.4 入口限流和目的限流····························································94
6.4.5 限流和内网流量调度····························································95
第7 章 运维相关机制····················································································97
7.1 监控机制···························································································97
7.1.1 日志监控及其问题································································98
7.1.2 BFE的内部状态输出···························································99
7.1.3 统计状态和日志的配合使用··············································101
深入理解BFE
·VIII·
7.2 Web Monitor基础库·······································································101
7.2.1 Web Monitor概述·······························································102
7.2.2 状态变量维护······································································102
7.2.3 延迟统计变量维护······························································104
7.2.4 建立专用的Web服务器·····················································106
7.2.5 注册回调函数······································································107
7.3 日志机制·························································································108
7.3.1 日志类型·············································································108
7.3.2 日志打印的注意事项··························································109
7.3.3 BFE的访问日志·································································110
7.4 配置管理························································································· 111
7.4.1 BFE配置文件的分布·························································· 111
7.4.2 常规配置和动态配置··························································112
7.4.3 动态配置的实现机制··························································113
第8 章 HTTPS 的优化················································································116
8.1 HTTPS优化背景及必要性·····························································116
8.2 HTTPS优化的挑战········································································117
8.3 HTTPS中的优化机制·····································································118
8.4 BFE中HTTPS 相关增强机制························································120
操 作 篇
第9 章 BFE 服务的安装部署······································································129
9.1 软件安装包下载安装······································································129
9.2 源代码编译方式安装······································································132
9.3 Docker方式安装·············································································133
目 录
·IX·
9.4 BFE命令行参数·············································································134
9.5 查看BFE 服务的运行状态·····························································135
第10 章 BFE 服务的基础配置····································································136
10.1 场景说明·······················································································136
10.2 修改基础配置文件········································································137
10.3 转发的配置···················································································138
10.3.1 转发配置流程····································································138
10.3.2 具体案例···········································································139
10.3.3 服务访问验证····································································146
10.3.4 配置的重新加载································································146
第11 章 配置负载均衡算法及会话保持·····················································147
11.1 子集群间的负载均衡····································································147
11.2 子集群级别的会话保持································································149
11.2.1 配置实例···········································································149
11.2.2 参数的具体含义································································150
11.3 实例间的负载均衡········································································151
11.3.1 加权轮询配置示例····························································151
11.3.2 最小连接数的配置示例····················································153
11.4 实例级别的会话保持····································································154
第12 章 配置HTTPS 和更多协议·····························································155
12.1 设置HTTPS 基础配置··································································155
12.1.1 配置HTTPS 端口·····························································156
12.1.2 配置加密套件····································································156
12.1.3 配置服务端证书································································156
深入理解BFE
·X·
12.1.4 配置TLS 规则···································································157
12.2 配置TLS 会话重用·······································································158
12.2.1 配置会话缓存····································································159
12.2.2 配置会话票证····································································160
12.3 配置TLS 双向认证·······································································161
12.4 对不同安全等级的区分································································163
12.5 支持更多协议···············································································164
12.5.1 HTTP/2配置·····································································165
12.5.2 SPDY 配置········································································166
12.5.3 WebSocket配置·································································167
12.5.4 连接后端服务的协议························································169
第13 章 其他常用配置················································································171
13.1 配置重写·······················································································171
13.1.1 开启重写···········································································171
13.1.2 模块配置···········································································171
13.1.3 重写动作详细描述····························································173
13.2 配置重定向···················································································177
13.2.1 开启重定向········································································177
13.2.2 模块配置···········································································177
13.2.3 重定向动作详细描述························································178
13.3 配置限流功能···············································································180
13.3.1 开启限流模块····································································180
13.3.2 模块配置···········································································181
13.3.3 限制特定维度的流量························································182
13.3.4 设置限流门限····································································183
13.3.5 设置限流动作····································································184
目 录
·XI·
实 现 篇
第14 章 其他常用配置················································································187
14.1 BFE的代码组织···········································································187
14.2 BFE的进程模型···········································································190
14.2.1 协程的分类·······································································190
14.2.2 并发模型···········································································191
14.2.3 并发能力···········································································192
14.2.4 异常恢复机制····································································192
14.3 请求处理流程···············································································193
14.3.1 连接的建立·······································································193
14.3.2 连接的处理·······································································194
14.3.3 请求的处理·······································································196
14.3.4 请求的结束·······································································198
14.3.5 连接的结束·······································································199
14.4 请求路由实现···············································································199
14.4.1 关键数据结构····································································200
14.4.2 目的租户路由····································································203
14.4.3 目的集群路由····································································204
14.5 负载均衡实现···············································································206
14.5.1 全局负载均衡····································································207
14.5.2 分布式负载均衡································································210
第15 章 模块插件机制················································································214
15.1 BFE的回调点设置·······································································214
15.2 BFE内置的扩展模块···································································216
15.3 模块框架的实现机制····································································217
深入理解BFE
·XII·
15.3.1 模块基础类型····································································217
15.3.2 连接/请求处理及回调函数的调用···································223
15.4 如何开发BFE扩展模块·······························································224
15.4.1 配置加载···········································································225
15.4.2 回调函数的编写和注册····················································226
15.4.3 模块状态的展示································································228
第16 章 核心协议实现················································································231
16.1 HTTP的实现················································································231
16.1.1 HTTP 代码的组织·····························································231
16.1.2 从用户读取HTTP 请求····················································233
16.1.3 向后端转发请求并获取响应············································236
16.1.4 向用户回复HTTP 响应····················································239
16.2 HTTP2的实现··············································································240
16.2.1 HTTP2 代码的组织···························································240
16.2.2 HTTP2 连接处理模块·······················································241
16.2.3 HTTP2 连接相关协程及关系···········································243
第17 章 BFE 的多进程GC 机制································································249
17.1 模型设计·······················································································249
17.1.1 多进程轮转机制································································250
17.1.2 子进程状态定义································································252
17.2 相关参数的确定············································································253
17.2.1 切换时间参数的选择························································253
17.2.2 子进程数量的计算····························································255
17.2.3 内存消耗的计算································································256

读者评论

图书类别

相关博文

  • 立于山巅!他,凭什么抗住万亿级流量冲击!

    立于山巅!他,凭什么抗住万亿级流量冲击!

    博文小编 2021-08-24

    在云计算时代浪潮下,大规模、高并发的技术架构已成为主流。云计算的高速发展,离不开底层基础设施的创新与改进,传统七层负载均衡架构已无法满足复杂的网络集群。 在云时代巨量请求背景下,在网民数量和互联网流量井喷的时点,2012年,百度技...

    博文小编 2021-08-24
    40 0 0 0

相关图书

趣玩Python:自动化办公真简单(双色+视频版)

本书以数据收集→数据清洗→数据分析→数据可视化→根据数据可视化结果(即图表)做决策为脉络,介绍Python在实际工作场景中的应用,侧重于用Python解决工作中...

 

Excel VBA程序开发自学宝典(第4版)

罗刚君 (作者)

《Excel VBA 程序开发自学宝典(第 4 版)》是 VBA 入门与提高的经典教材。全书包含基础知识部分和高级应用部分。其中,基础知识部分包含 VBA 的基...

¥119.00

Power BI建模权威指南

阿尔贝托·费拉里 (Alberto Ferrari) 马尔·科鲁索 (Marco Russo) (作者) 刘钰 潘丽萍 付大伟 (译者)

如何使用Excel和Power BI高效发现数字背后的信息?在数据分析时如何准确写出所需的公式?如何快速响应各方需求,提升自己的价值……答案是使用“数据模型”。...

 

跟着视频学Excel数据处理:函数篇

曹明武 (作者)

本书主要介绍Excel函数知识。主要内容包括函数基础知识、函数使用小技巧、文本函数、查找引用函数、逻辑函数、日期和时间函数、其他函数,以及函数初级综合案例、函数...

¥59.90

财务分析那些事儿:Power BI财务数据实战

杨晨 (作者)

本书从财务分析师的视角阐述了Power BI在日常财务工作中的应用,主要包括搭建财务模型、制作分析报告、风险控制、制定预算和财务分析案例等方面,对提升数据处理效...

¥79.00

裸演说:你也能成为演说达人

Garr Reynolds(加尔·雷纳德) (作者) 王佑 (译者)

本书作者是国际知名的畅销书作者,先后撰写了《演说之禅》和《设计之禅》等畅销作品,这本《裸演说》是专门写给演讲者的。作者通过生动的示例和形象的比喻回答了演讲者的困...