目录
第Ⅰ部分　Hadoop架构与Hadoop集群介绍
第1章　Hadoop与Hadoop环境介绍............................................................................... 3
Hadoop简介.........................................................................................................................4
Hadoop 的特性............................................................................................................5
Hadoop 与大数据........................................................................................................5
Hadoop 的典型应用场景............................................................................................6
传统数据库系统..........................................................................................................7
数据湖..........................................................................................................................9
大数据、数据科学和Hadoop ..................................................................................10
Hadoop集群与集群计算................................................................................................... 11
集群计算.................................................................................................................... 11
Hadoop 集群..............................................................................................................12
Hadoop组件和Hadoop生态..............................................................................................14
Hadoop管理员需要做些什么...........................................................................................16
Hadoop 管理—新的范式......................................................................................17
关于Hadoop 管理你需要知道的.............................................................................18
Hadoop 管理员的工具集..........................................................................................19
Hadoop 1和Hadoop 2的关键区别....................................................................................19
架构区别....................................................................................................................20
高可用性....................................................................................................................20
多计算引擎................................................................................................................21
xiv 目录
分离处理和调度........................................................................................................21
Hadoop 1 和Hadoop 2 中的资源分配.....................................................................22
分布式数据处理：MapReduce和Spark、Hive、Pig ......................................................22
MapReduce ................................................................................................................22
Apache Spark .............................................................................................................23
Apache Hive ...............................................................................................................24
Apache Pig .................................................................................................................24
数据整合：Apache Sqoop、Apache Flume和Apache Kafka ..........................................25
Hadoop管理中的关键领域...............................................................................................26
集群存储管理............................................................................................................26
集群资源分配............................................................................................................26
作业调度....................................................................................................................27
Hadoop 数据安全......................................................................................................27
总结....................................................................................................................................28
第2章　Hadoop架构介绍............................................................................................. 31
Hadoop与分布式计算.......................................................................................................31
Hadoop 架构......................................................................................................................32
Hadoop 集群..............................................................................................................33
主节点和工作节点....................................................................................................33
Hadoop 服务..............................................................................................................34
数据存储—Hadoop分布式文件系统...........................................................................35
HDFS 特性................................................................................................................35
HDFS 架构................................................................................................................36
HDFS 文件系统........................................................................................................38
NameNode 操作........................................................................................................41
利用YARN（Hadoop操作系统）进行数据处理...........................................................45
YARN 的架构............................................................................................................46
ApplicationMaster 如何与ResourceManager 协作进行资源分配..........................51
总结....................................................................................................................................54
第3章　创建和配置一个简单的Hadoop集群................................................................ 55
Hadoop发行版本和安装类型...........................................................................................56
目录xv
Hadoop 发行版本......................................................................................................56
Hadoop 安装类型......................................................................................................57
设置一个伪分布式Hadoop集群.......................................................................................58
满足操作系统的要求................................................................................................58
修改内核参数............................................................................................................59
设置SSH ...................................................................................................................64
Java 需求....................................................................................................................65
安装Hadoop ..............................................................................................................66
创建必要的Hadoop 用户.........................................................................................66
创建必要的目录........................................................................................................67
Hadoop初始配置...............................................................................................................67
环境变量配置文件....................................................................................................69
只读默认配置文件....................................................................................................70
site 专用配置文件.....................................................................................................70
其他Hadoop 相关的配置文件.................................................................................71
配置文件的优先级....................................................................................................72
可变扩展和配置参数................................................................................................74
配置Hadoop 守护进程环境变量.............................................................................74
配置Hadoop 的核心属性（使用core-site.xml 文件）............................................76
配置MapReduce（使用mapred-site.xml 文件）.....................................................78
配置YARN（使用yarn-site.xml 文件）...................................................................79
配置HDFS（使用hdfs-site.xml 文件）...................................................................80
操作新的Hadoop集群.......................................................................................................82
格式化分布式文件系统............................................................................................82
设置环境变量............................................................................................................82
启动HDFS 和YARN 服务.......................................................................................83
验证服务启动............................................................................................................85
关闭服务....................................................................................................................85
总结....................................................................................................................................86
第4章　规划和创建一个完全分布式集群..................................................................... 87
规划Hadoop集群...............................................................................................................88
集群规划注意事项....................................................................................................88
xvi 目录
安排服务器................................................................................................................90
节点选择的标准........................................................................................................90
从单机架到多机架............................................................................................................91
调整Hadoop 集群.....................................................................................................91
CPU、内存和存储选择的一般性原则....................................................................92
主节点的特殊要求....................................................................................................95
关于服务器大小的几点建议....................................................................................96
集群增长....................................................................................................................97
大型集群指南............................................................................................................97
创建一个多节点集群........................................................................................................98
如何设置测试集群....................................................................................................98
修改Hadoop的配置.........................................................................................................102
更改HDFS 的配置（hdfs-site.xml 文件）.............................................................102
更改YARN 的配置.................................................................................................105
修改MapReduce 的配置........................................................................................109
启动集群.......................................................................................................................... 110
使用脚本启动和关闭集群...................................................................................... 112
快速检查新集群的文件系统.................................................................................. 113
配置Hadoop服务、Web界面和端口.............................................................................. 114
服务配置和Web 界面............................................................................................ 115
设置Hadoop 服务的端口....................................................................................... 117
Hadoop 客户端........................................................................................................120
总结..................................................................................................................................122
第Ⅱ部分　Hadoop应用架构
第5章　在集群上运行一个应用—MapReduce框架和Hive、Pig ............................ 125
MapReduce框架..............................................................................................................125
MapReduce 模型.....................................................................................................126
MapReduce 怎样工作.............................................................................................127
MapReduce 作业处理.............................................................................................129
一个简单的MapReduce 程序................................................................................130
通过运行WordCount 程序理解Hadoop 作业的处理过程..................................132
目录xvii
MapReduce 输入/ 输出目录..................................................................................133
Hadoop 如何展示作业细节....................................................................................133
Hadoop Streaming ....................................................................................................135
Apache Hive .....................................................................................................................137
Hive 数据组织.........................................................................................................138
使用Hive 表............................................................................................................138
将数据导入Hive ....................................................................................................138
使用Hive 查询........................................................................................................139
Apache Pig .......................................................................................................................139
Pig 执行模型...........................................................................................................140
一个简单的Pig 示例..............................................................................................140
总结..................................................................................................................................141
第6章　集群上的应用—Spark框架介绍................................................................. 143
Spark是什么....................................................................................................................144
为什么使用Spark ...........................................................................................................145
速度..........................................................................................................................145
易用性......................................................................................................................147
通用框架..................................................................................................................148
Spark 和Hadoop ......................................................................................................148
Spark技术栈....................................................................................................................149
安装Spark ........................................................................................................................151
Spark 示例...............................................................................................................152
Spark 的主要文件和目录.......................................................................................153
编译Spark 二进制文件..........................................................................................153
减少Spark 日志......................................................................................................153
Spark运行模式................................................................................................................154
本地模式..................................................................................................................154
集群模式..................................................................................................................154
集群管理器......................................................................................................................154
独立集群管理器......................................................................................................155
基于Apache Mesos 的Spark ..................................................................................157
基于YARN 的Spark ..............................................................................................158
xviii 目录
YARN 和Spark 如何协同合作..............................................................................159
设置基于Hadoop 集群的Spark .............................................................................159
Spark和数据获取............................................................................................................159
从Linux 文件系统加载数据..................................................................................160
从HDFS 加载数据.................................................................................................160
从关系型数据库获取数据......................................................................................161
总结..................................................................................................................................162
第7章　运行Spark应用程序....................................................................................... 163
Spark编程模型................................................................................................................163
Spark 编程和RDD ..................................................................................................164
Spark 编程...............................................................................................................166
Spark应用程序................................................................................................................167
RDD 基础................................................................................................................168
创建RDD ................................................................................................................168
RDD 操作................................................................................................................171
RDD 持久化............................................................................................................173
Spark应用的结构............................................................................................................174
Spark 术语...............................................................................................................174
Spark 应用程序的组件...........................................................................................174
交互式运行Spark应用程序............................................................................................175
Spark shell 和Spark 应用程序...............................................................................176
Spark shell ................................................................................................................176
使用Spark shell .......................................................................................................176
Spark 集群执行概述...............................................................................................179
创建和提交Spark应用....................................................................................................180
构建Spark 应用......................................................................................................180
在独立的Spark 集群上运行应用..........................................................................180
使用spark-submit 执行应用...................................................................................181
在Mesos 上运行Spark 应用..................................................................................183
在Hadoop YARN 集群上运行Spark 应用............................................................183
使用JDBC/ODBC 服务..........................................................................................186
配置Spark应用................................................................................................................187
目录xix
Spark 的配置属性...................................................................................................187
运行spark-submit 时的配置...........................................................................187
监控Spark应用................................................................................................................188
使用Spark Streaming处理流式计算...............................................................................189
Spark Streaming 如何工作......................................................................................189
Spark Streaming 示例，又是WordCount ..............................................................191
使用Spark SQL 处理结构化数据...................................................................................192
数据框架..................................................................................................................192
HiveContext 和SQLContext ...................................................................................193
使用Spark SQL .......................................................................................................193
创建DataFrames .....................................................................................................195
总结..................................................................................................................................195
第Ⅲ部分　管理和保护Hadoop数据和高可用性
第8章　NameNode的作用和HDFS的工作原理......................................................... 199
HDFS—NameNode与DataNode之间的交互.............................................................200
客户端和HDFS 之间的交互.................................................................................200
NameNode 与DataNode 之间的通信....................................................................201
机架感知与拓扑逻辑......................................................................................................203
如何在集群中配置机架感知策略..........................................................................204
找出集群的机架信息..............................................................................................204
HDFS 数据副本..............................................................................................................206
HDFS 数据组织和数据块......................................................................................207
数据复制..................................................................................................................207
文件块和副本状态..................................................................................................209
客户端如何读写HDFS数据...........................................................................................213
客户端如何读取HDFS 数据.................................................................................213
客户端如何向HDFS 写数据.................................................................................214
了解HDFS恢复过程.......................................................................................................217
生成戳......................................................................................................................218
租约恢复..................................................................................................................218
块恢复......................................................................................................................219
xx 目录
管道恢复..................................................................................................................219
HDFS中的集中式缓存管理...........................................................................................220
Hadoop 和OS 的页面缓存.....................................................................................221
集中式缓存管理的关键原则..................................................................................221
集中式缓存管理如何工作......................................................................................221
配置缓存..................................................................................................................222
缓存指令..................................................................................................................223
缓存池......................................................................................................................223
使用缓存..................................................................................................................223
Hadoop归档存储、SSD和内存（异构存储）..............................................................225
不同存储类型的性能特点......................................................................................225
对异构HDFS 存储的需求.....................................................................................226
存储体系结构的变化..............................................................................................227
文件的存储首选项..................................................................................................228
设置归档存储..........................................................................................................228
管理存储策略..........................................................................................................232
移动数据..................................................................................................................232
实现归档..................................................................................................................233
总结..................................................................................................................................234
第9章　HDFS命令、HDFS权限和HDFS存储............................................................ 235
使用HDFS Shell命令管理HDFS ....................................................................................235
使用hdfs dfs 实用程序来管理HDFS .............................................................237
列出HDFS 文件和目录.........................................................................................239
创建HDFS 目录.....................................................................................................241
删除HDFS 文件和目录.........................................................................................242
更改文件和目录所有权和组..................................................................................242
使用dfsadmin实用程序执行HDFS操作........................................................................243
dfsadmin -report 命令..................................................................................245
管理HDFS权限和用户...................................................................................................247
HDFS 文件权限......................................................................................................247
HDFS 用户和超级用户..........................................................................................249
管理HDFS存储...............................................................................................................252
目录xxi
检查HDFS 磁盘使用情况.....................................................................................252
分配HDFS 空间配额.............................................................................................255
重新均衡HDFS数据.......................................................................................................259
HDFS 数据不均衡的原因......................................................................................260
运行均衡器以均衡HDFS 数据.............................................................................260
使用hdfs dfsadmin 使事情更简单.........................................................................263
何时运行均衡器......................................................................................................265
回收HDFS空间...............................................................................................................266
删除文件和目录......................................................................................................266
降低复制因子..........................................................................................................266
总结..................................................................................................................................268
第10章　数据保护、文件格式和访问HDFS ............................................................... 269
保护数据..........................................................................................................................270
使用HDFS 回收站防止意外数据删除.................................................................270
使用HDFS 快照保护重要数据.............................................................................272
通过文件系统检查确保数据完整性......................................................................276
数据压缩..........................................................................................................................281
常用压缩格式..........................................................................................................282
评估各种压缩方案..................................................................................................282
MapReduce 的各个阶段的压缩.............................................................................283
Spark 的压缩...........................................................................................................286
数据序列化..............................................................................................................286
Hadoop文件格式.............................................................................................................287
确定正确文件格式的标准......................................................................................288
Hadoop 支持的文件格式........................................................................................289
理想文件格式..........................................................................................................294
Hadoop 小文件问题和合并文件............................................................................294
使用NameNode 联合架构克服小文件问题.........................................................295
使用Hadoop Archives 管理小文件........................................................................295
减小小文件的性能影响..........................................................................................298
使用Hadoop WebHDFS和HttpFS ...................................................................................299
WebHDFS—Hadoop REST API ..........................................................................299
xxii 目录
使用WebHDFS API ................................................................................................300
了解WebHDFS 命令..............................................................................................301
使用HttpFS 网关从防火墙后面访问HDFS .........................................................304
总结..................................................................................................................................306
第11章　NameNode操作、高可用性和联合............................................................. 307
了解NameNode操作.......................................................................................................308
HDFS 元数据..........................................................................................................309
NameNode 启动过程.............................................................................................. 311
NameNode 和DataNode 如何协同工作................................................................ 311
检查点操作......................................................................................................................313
Secondary NameNode、检查点节点、备份节点和Standby NameNode ............314
配置检查点操作频率..............................................................................................315
管理检查点性能......................................................................................................316
检查点的机制..........................................................................................................317
NameNode安全模式操作...............................................................................................319
自动安全模式操作..................................................................................................319
将NameNode 置于安全模式.................................................................................320
NameNode 如何进行模式转换..............................................................................321
备份和恢复NameNode 元数据.............................................................................322
配置HDFS高可用性.......................................................................................................324
NameNode HA 架构（QJM）..................................................................................325
设置HDFS HA Quorum 集群.................................................................................327
部署高可用性NameNode ......................................................................................331
管理HA NameNode 设置.......................................................................................335
HA 手动和自动故障转移.......................................................................................336
HDFS联合.......................................................................................................................338
联合NameNode 的体系结构.................................................................................339
总结..................................................................................................................................340
目录xxiii
第Ⅳ部分　数据迁移、资源分配、作业调度及安全
第12章　将数据导入和导出Hadoop .......................................................................... 343
Hadoop数据传输工具简介.............................................................................................343
通过命令行将数据加载到HDFS ...................................................................................344
使用-cat 命令转储文件的内容..........................................................................344
检测HDFS 文件.....................................................................................................345
从HDFS 或向HDFS 复制或移动文件.................................................................346
使用-get 命令移动文件.........................................................................................347
向HDFS 或从HDFS 移动文件.............................................................................348
使用-tail 和head 命令............................................................................................348
使用DistCp在HDFS集群之间复制数据........................................................................349
如何使用DistCp 命令移动数据............................................................................349
DistCp 选项.............................................................................................................351
使用Sqoop从关系型数据库获取数据...........................................................................353
Sqoop 架构..............................................................................................................354
部署Sqoop ..............................................................................................................355
使用Sqoop 移动数据.............................................................................................356
使用Sqoop 导入数据.............................................................................................356
将数据导入Hive .....................................................................................................367
使用Sqoop 导出数据.............................................................................................369
通过Flume从外部来源采集数据...................................................................................376
Flume 架构简介......................................................................................................376
配置Flume agent .....................................................................................................378
简单的Flume 示例.................................................................................................379
使用Flume 将数据移动到HDFS ..........................................................................381
更复杂的Flume 示例.............................................................................................383
与Kafka交互数据............................................................................................................385
Kafka 的优点...........................................................................................................386
Kafka 是如何工作的...............................................................................................386
设置Apache Kafka 集群.........................................................................................388
将Kafka 与Hadoop 和Storm 集成.......................................................................392
总结..................................................................................................................................393
xxiv 目录
第13章　Hadoop集群中的资源分配.......................................................................... 395
Hadoop中的资源分配.....................................................................................................395
管理集群的工作负载..............................................................................................396
Hadoop 的资源调度器............................................................................................397
FIFO调度器.....................................................................................................................398
容量调度器......................................................................................................................399
队列和子队列..........................................................................................................400
集群如何分配资源..................................................................................................405
抢占申请..................................................................................................................408
启用容量调度器......................................................................................................409
一个典型的容量调度器..........................................................................................409
公平调度器......................................................................................................................413
队列..........................................................................................................................414
配置公平调度器......................................................................................................415
作业是如何被放置到队列中的..............................................................................417
公平调度器中的应用抢占......................................................................................418
安全和资源池..........................................................................................................419
一个fair-scheduler.xml 示例文件...........................................................................419
将作业提交到调度器..............................................................................................421
在队列之间移动应用程序......................................................................................421
监控公平调度器......................................................................................................422
容量调度器和公平调度器的对比..................................................................................422
两个调度器之间的相似之处..................................................................................422
两个调度器之间的差异..........................................................................................422
总结..................................................................................................................................423
第14章　使用Oozie管理作业工作流.......................................................................... 425
使用Apache Oozie调度作业...........................................................................................425
Oozie架构........................................................................................................................427
Oozie 服务器...........................................................................................................427
Oozie 客户端...........................................................................................................428
Oozie 数据库...........................................................................................................428
在集群中部署Oozie ........................................................................................................429
目录xxv
安装和配置Oozie ...................................................................................................430
为Oozie 配置Hadoop ............................................................................................432
了解Oozie工作流............................................................................................................434
工作流、控制流和节点..........................................................................................434
使用workflow.xml 文件定义工作流.....................................................................435
Oozie如何运行一个动作................................................................................................436
配置动作节点..........................................................................................................437
创建Oozie工作流............................................................................................................442
配置控制节点..........................................................................................................443
配置作业..................................................................................................................448
运行Oozie工作流作业....................................................................................................449
指定作业属性..........................................................................................................449
部署Oozie 作业......................................................................................................451
创建动态工作流......................................................................................................451
Oozie 协调器...................................................................................................................452
基于时间的协调器..................................................................................................453
基于数据的协调器..................................................................................................455
基于时间和数据的协调器......................................................................................456
从命令行提交Oozie 协调器..................................................................................457
管理和治理Oozie ............................................................................................................458
常见的Oozie 命令..................................................................................................458
Oozie 故障排除.......................................................................................................460
Oozie cron 调度和Oozie SLA ................................................................................461
总结..................................................................................................................................462
第15章　Hadoop安全................................................................................................ 463
Hadoop安全概览.............................................................................................................464
认证、授权和审计..................................................................................................466
使用Kerberos进行Hadoop认证......................................................................................467
Kerberos 及其工作原理..........................................................................................467
Kerberos 认证过程..................................................................................................469
Kerberos 互信..........................................................................................................470
一个特殊主体..........................................................................................................471
xxvi 目录
将Kerberos 添加到集群中.....................................................................................471
Hadoop 相关的Kerberos 设置...............................................................................476
使用Kerberos 保护Hadoop 集群..........................................................................480
Kerberos 如何验证用户和服务..............................................................................486
管理Kerberized Hadoop 集群................................................................................487
Hadoop授权.....................................................................................................................490
HDFS 权限..............................................................................................................491
服务级授权.............................................................................................................495
基于角色的Apache Sentry 权限设置....................................................................497
Hadoop审计.....................................................................................................................503
审计HDFS 操作.....................................................................................................504
审计YARN 操作....................................................................................................504
Hadoop数据安全............................................................................................................505
HDFS 透明加密......................................................................................................505
加密转换中的数据.................................................................................................508
其他Hadoop安全举措....................................................................................................509
使用Apache Knox 网关保护Hadoop 基础设施...................................................509
Apache Ranger 安全管理........................................................................................509
总结.................................................................................................................................510
第Ⅴ部分　监控、优化和故障排除
第16章　管理作业、使用Hue和执行常规任务........................................................... 513
使用YARN命令管理Hadoop作业..................................................................................514
查看YARN 应用程序.............................................................................................515
检查应用程序的状态..............................................................................................516
Kill 正在执行的作业..............................................................................................516
检查节点状态..........................................................................................................517
检查YARN 的队列状态.........................................................................................517
获取作业的日志......................................................................................................517
YARN 管理命令......................................................................................................518
下线和上线节点..............................................................................................................519
包含和剔除主机......................................................................................................520
目录xxvii
下线DataNodes 和NodeManagers ........................................................................521
重新上线节点..........................................................................................................522
关于下线和重新上线的注意事项..........................................................................523
添加新的DataNode 和NodeManager ....................................................................524
高可用性ResourceManager.............................................................................................524
高可用性ResourceManager 架构...........................................................................525
设置高可用性ResourceManager ............................................................................525
ResourceManager 故障转移....................................................................................526
使用ResourceManager 高可用性命令...................................................................528
执行常规管理任务..........................................................................................................529
将NameNode 移动到不同的主机.........................................................................529
管理高可用性NameNode ......................................................................................529
使用关闭/ 启动脚本来管理集群...........................................................................530
均衡HDFS ..............................................................................................................530
均衡DataNodes 上存储..........................................................................................531
管理MySQL数据库........................................................................................................532
配置MySQL 数据库...............................................................................................532
配置高可用性MySQL ............................................................................................533
备份重要集群数据..........................................................................................................535
备份HDFS 元数据.................................................................................................535
备份Metastore 数据库............................................................................................537
使用Hue管理集群...........................................................................................................537
允许用户使用Hue ..................................................................................................538
安装Hue ..................................................................................................................538
配置集群以使用Hue ..............................................................................................540
管理Hue ..................................................................................................................544
使用Hue ..................................................................................................................544
使用HDFS的附加功能...................................................................................................545
在多宿主网络中部署HDFS 和YARN ..................................................................545
短路本地读取..........................................................................................................546
可挂载的HDFS ......................................................................................................548
使用NFS 网关将HDFS 挂载到本地文件系统....................................................549
总结..................................................................................................................................551
xxviii 目录
第17章　监控、指标和Hadoop日志.......................................................................... 553
监控Linux服务器............................................................................................................554
Linux 系统监控基础...............................................................................................554
Linux 系统监控工具...............................................................................................556
Hadoop指标.....................................................................................................................559
Hadoop 指标类型....................................................................................................560
使用Hadoop 指标...................................................................................................561
收集文件系统的指标..............................................................................................561
使用Ganglia进行监测.....................................................................................................563
Ganglia 架构............................................................................................................563
Ganglia 和Hadoop 整合.........................................................................................564
设置Hadoop 指标...................................................................................................565
Hadoop日志记录.............................................................................................................566
Hadoop 日志消息....................................................................................................566
守护进程和应用程序日志以及如何查看这些日志..............................................568
应用程序日志记录的工作原理..............................................................................568
Hadoop 如何使用HDFS 目录和本地目录............................................................570
NodeManager 如何使用本地目录..........................................................................571
通过日志聚合将作业日志存储在HDFS 中.........................................................576
使用Hadoop 守护程序日志...................................................................................580
使用Hadoop的Web UI进行监控....................................................................................582
使用ResourceManager Web UI 监控作业.............................................................583
JobHistoryServer Web UI ........................................................................................589
使用NameNode Web UI 进行监控........................................................................591
监控其他Hadoop组件.....................................................................................................592
监控Hive .................................................................................................................592
监控Spark ...............................................................................................................593
总结..................................................................................................................................593
第18章　调优集群资源，优化MapReduce作业和基准测试.............................................. 595
如何分配YARN内存和CPU ...........................................................................................596
分配内存..................................................................................................................596
配置CPU 内核数量................................................................................................604
目录xxix
内存与CPU 之间的关系........................................................................................605
配置高性能......................................................................................................................605
推测执行..................................................................................................................605
减少系统上的I/O 负载..........................................................................................608
调整map和reduce任务，管理员可以做什么................................................................608
map 任务调优..........................................................................................................609
输入和输出..............................................................................................................610
reduce 任务调优......................................................................................................613
MapReduce shuffle 进程调优.................................................................................615
优化Pig和Hive作业........................................................................................................617
优化Hive 作业........................................................................................................618
优化pig 作业...........................................................................................................619
对集群进行基准测试......................................................................................................621
使用TestDFSIO 测试I / O 性能............................................................................621
使用TeraSort 进行基准测试..................................................................................623
使用Hadoop 的Rumen 和GridMix 进行基准测试.............................................625
Hadoop计数器.................................................................................................................629
文件系统计数器......................................................................................................629
作业计数器..............................................................................................................631
MapReduce 框架计数器.........................................................................................632
自定义Java 计数器.................................................................................................633
限制计数器数量......................................................................................................633
优化MapReduce ..............................................................................................................633
map-only 与map 及reduce 作业............................................................................634
使用combiners 提升MapReduce 性能..................................................................634
使用partitioner 提高性能.......................................................................................635
在MapReduce 过程中压缩数据............................................................................636
太多的map 和reduce 任务....................................................................................637
总结..................................................................................................................................639
第19章　在YARN上配置和调优Apache Spark .......................................................... 641
在YARN上配置Spark的资源分配.................................................................................642
分配CPU .................................................................................................................642
xxx 目录
分配内存..................................................................................................................642
如何把资源分配给Spark .......................................................................................642
Spark 应用程序的资源分配限制...........................................................................643
将资源分配给驱动程序..........................................................................................645
为执行器配置资源..................................................................................................648
Spark 如何使用内存...............................................................................................652
要注意的事情..........................................................................................................654
集群或客户端模式..................................................................................................656
配置Spark 相关网络参数......................................................................................657
YARN Spark动态资源分配............................................................................................658
动态和静态资源分配..............................................................................................658
如何管理动态资源分配..........................................................................................658
启用动态资源分配..................................................................................................659
存储格式和压缩数据......................................................................................................660
存储格式..................................................................................................................660
文件大小..................................................................................................................662
压缩..........................................................................................................................662
监控Spark应用程序........................................................................................................663
使用Spark Web UI 了解性能.................................................................................663
Spark 系统和Metrics REST API ............................................................................666
YARN 上的Spark 历史记录服务器......................................................................666
从命令行跟踪作业..................................................................................................668
调优垃圾回收..................................................................................................................668
垃圾回收机制..........................................................................................................668
如何收集GC 统计数据..........................................................................................669
调优Spark Streaming应用程序.......................................................................................670
减少批处理时间......................................................................................................670
设置正确的批次间隔..............................................................................................670
调优内存和垃圾回收..............................................................................................671
总结..................................................................................................................................671
第20章　优化Spark应用程序..................................................................................... 673
重新审视Spark执行模型................................................................................................674
目录xxxi
Spark 执行模型.......................................................................................................674
shuffle操作以及如何减少shuffle操作............................................................................676
WordCount 示例（再一次展示）............................................................................676
shuffle 操作的影响..................................................................................................678
配置shuffle 参数.....................................................................................................679
分区和并行性（任务数）..............................................................................................684
并行度......................................................................................................................685
极少数任务的问题..................................................................................................687
设置默认分区数......................................................................................................687
如何增加分区数量..................................................................................................688
使用Repartition（重新分区）和Coalesce（合并）
操作来更改RDD 中的分区数...............................................................................689
两种类型的分区器..................................................................................................690
数据分区和如何避免shuffle ..................................................................................690
数据的序列化和压缩优化..............................................................................................691
数据序列化..............................................................................................................691
配置压缩..................................................................................................................692
Spark的SQL查询优化器.................................................................................................693
优化步骤..................................................................................................................693
Spark 的推测执行功能...........................................................................................695
数据本地化的重要性..............................................................................................696
缓存数据..........................................................................................................................698
缓存容错..................................................................................................................699
如何指定缓存..........................................................................................................699
总结..................................................................................................................................704
第21章　Hadoop故障排除—样例.......................................................................... 705
空间相关问题..................................................................................................................705
处理Linux 文件系统100% 使用的情况...............................................................706
HDFS 空间问题......................................................................................................707
本地目录以及日志目录空间超出..........................................................................707
磁盘容错..................................................................................................................709
处理卡住的YARN作业...................................................................................................710
xxxii 目录
JVM内存分配与垃圾回收策略......................................................................................712
理解JVM 垃圾回收................................................................................................712
优化垃圾回收..........................................................................................................713
Analyzing Memory Usage .......................................................................................713
内存不足问题..........................................................................................................714
ApplicationMaster 内存问题...................................................................................715
处理不同类型的错误......................................................................................................716
处理守护进程失败的情况......................................................................................716
启动Hadoop 守护进程失败...................................................................................717
任务和作业失败......................................................................................................718
Spark作业故障排除........................................................................................................719
Spark 的容错机制...................................................................................................720
杀死Spark 作业......................................................................................................720
一个作业的最大尝试次数......................................................................................720
一个作业最大的失败次数......................................................................................720
调试Spark应用................................................................................................................720
通过日志聚合访问日志..........................................................................................720
当日志聚合未开启时访问日志..............................................................................721
重新审视启动环境..................................................................................................721
总结..................................................................................................................................722
附录A　安装VirtualBox和Linux以及虚拟机的克隆..................................................... 723