YARNarchitecture and resourcemanagement

YARNarchitecture and resourcemanagement

1. YARNIntroduction

YARN (Yet Another Resource Negotiator) is Hadoop 2.xversion引入 resourcemanagementsystem, 用于managementclusterin resource并schedulingjob. 它 出现解决了Hadoop 1.xversioninMapReducearchitecture 局限性, implementation了resourcemanagement and jobscheduling 分离.

1.1 YARN design目标

• resourcemanagement: 统一managementclusterin CPU, memoryetc.resource
• jobscheduling: 根据job resourcerequirements and priority, 将resource分配给job
• many frameworksupport: support many 种计算framework, such asMapReduce, Spark, Flinketc.
• high reliability: throughhigh availabilitydesign, 确保system 连续性
• high scale性: 可以easilyscalecluster规模

1.2 YARN and Hadoop 1.x 区别

in Hadoop 1.xversionin, MapReducearchitecturepackage含两个主要component: JobTracker and TaskTracker. JobTracker负责resourcemanagement and jobscheduling, TaskTracker负责执行task. 这种design存 in 以 under 局限性:

• JobTracker is 单点failure
• resourcemanagement and jobscheduling耦合 in 一起, 缺乏flexible性
• 仅supportMapReduceframework
• scale性 has 限

YARNthrough将resourcemanagement and jobscheduling分离, 解决了这些issues, implementation了更flexible, 更强 big resourcemanagementsystem.

2. YARNcorecomponent

YARNadoptsmaster-slave (Master-Slave) architecture, 主要package含以 under component:

2.1 Resourcemanagementr (RM)

Resourcemanagementr is YARN 主node, 负责全局resourcemanagement and jobscheduling. 它 主要functionsincluding:

• resourcemanagement: managementclusterin resource, includingCPU, memoryetc.
• jobscheduling: 接收jobsubmitting, for job分配resource
• application程序management: managementapplication程序 生命周期, including启动, monitor and 终止
• nodemanagement: monitorNodemanagementr status

Resourcemanagementrpackage含两个主要component:

• Scheduler: 负责resource scheduling, 根据job resourcerequirements and priority, 将resource分配给job
• Applicationsmanagementr: 负责application程序 management, including接收jobsubmitting, 启动ApplicationMaster, monitorApplicationMasterstatusetc.

2.2 Nodemanagementr (NM)

Nodemanagementr is YARN from node, 负责单个node resourcemanagement and task执行. 它 主要functionsincluding:

• resourcemanagement: managementnode on resource, includingCPU, memoryetc.
• containersmanagement: creation, monitor and 销毁containers
• task执行: in containersin执行task
• status报告: 向Resourcemanagementr报告nodestatus

2.3 ApplicationMaster (AM)

ApplicationMaster is 每个application程序 专用management器, 负责application程序 生命周期management. 它 主要functionsincluding:

• job分解: 将application程序分解 for many 个task
• resourcerequest: 向Resourcemanagementrrequestresource
• taskscheduling: 将task分配给合适 node执行
• taskmonitor: monitortask 执行status
• fault toleranceprocessing: processingtask失败 circumstances

2.4 Container

Container is YARNin resource分配单位, package含CPU, memoryetc.resource. 每个task in 一个containersin执行, containersproviding了task执行 隔离environment.

3. YARNarchitecturedesign

YARN architecturedesignsuch asgraph所示:

4. YARNworkflow程

YARNjob 执行过程可以分 for 以 under 几个步骤:

1. jobsubmitting: 客户端向Resourcemanagementrsubmittingapplication程序
2. ApplicationMaster启动: Resourcemanagementr for application程序分配第一个containers, 并 in 该containersin启动ApplicationMaster
3. resourcerequest: ApplicationMaster向Resourcemanagementrrequestresource
4. resource分配: Resourcemanagementr根据scheduling策略, 将resource分配给ApplicationMaster
5. containers启动: ApplicationMaster向Nodemanagementr发送启动containers request, Nodemanagementrcreation并启动containers
6. task执行: containersin执行具体 task
7. statusupdate: task向ApplicationMaster报告执行status
8. jobcompletion: 当所 has task执行completion after , ApplicationMaster向Resourcemanagementr报告jobcompletion, Resourcemanagementr释放resource

5. YARNresource scheduling器

YARNproviding了 many 种resource scheduling器, 用于根据不同 策略分配resource.

5.1 FIFO Scheduler

FIFO (First In First Out) scheduling器 is YARN 默认scheduling器, 它按照jobsubmitting 顺序分配resource. FIFOscheduling器 优点 is simple 易懂, 但缺点 is 不support many queue and priority, 可能导致 long job阻塞 short job.

5.2 Capacity Scheduler

Capacityscheduling器support many queue, 每个queue可以configuration一定 resource容量, 不同queue之间可以共享resource. Capacityscheduling器 优点 is support many 租户, resource隔离 and resource共享, 适合 many userenvironment.

5.3 Fair Scheduler

Fairscheduling器按照公平principles分配resource, 每个queue or job最终会获得公平 resource份额. Fairscheduling器 优点 is support many 租户, resource隔离 and 公平分配, 适合 many userenvironment.

6. YARNhigh availability性

YARNthroughrun两个Resourcemanagementr (Active and Standby) 来implementationhigh availability性. 当Active Resourcemanagementrfailure时, Standby Resourcemanagementr可以自动切换 for Activestatus, 确保system 连续性.

6.1 YARNhigh availabilityconfiguration

要configurationYARNhigh availability, 需要completion以 under 步骤:

1. configurationZooKeepercluster
2. configurationResourcemanagementrhigh availability
3. configurationJournalNodecluster
4. 启动YARNcluster

7. YARNcommands行operation

YARNproviding了丰富 commands行tool, 用于management and monitorYARNcluster.

7.1 clustermanagementcommands

# 启动YARNcluster
$ start-yarn.sh

# 停止YARNcluster
$ stop-yarn.sh

# 查看Resourcemanagementrstatus
$ yarn rmadmin -getServiceState rm1

# 查看Nodemanagementrlist
$ yarn node -list

# 查看Nodemanagementrstatus
$ yarn node -status node1:8042

7.2 jobmanagementcommands

# submittingjob
$ yarn jar application.jar com.example.Application /input /output

# 查看joblist
$ yarn application -list

# 查看jobstatus
$ yarn application -status application_1234567890_0001

# 杀死job
$ yarn application -kill application_1234567890_0001

# 查看joblog
$ yarn logs -applicationId application_1234567890_0001

7.3 queuemanagementcommands

# 查看queuelist
$ yarn queue -list

# 查看queueinformation
$ yarn queue -status default

# updatequeueconfiguration
$ yarn queue -update queue.xml

8. YARNmonitor and management

YARNproviding了 many 种monitor and management方式, includingWeb UI, commands行tool and REST API.

8.1 Web UI

YARNproviding了Web UI, 用于monitorclusterstatus and job执行circumstances. Resourcemanagementr Web UI默认端口 for 8088, Nodemanagementr Web UI默认端口 for 8042.

8.2 REST API

YARNproviding了REST API, 用于programming方式management and monitorYARNcluster. 例such as:

# 获取Resourcemanagementrstatus
$ curl -X GET http://rm1:8088/ws/v1/cluster/info

# 获取joblist
$ curl -X GET http://rm1:8088/ws/v1/cluster/apps

# 获取nodelist
$ curl -X GET http://rm1:8088/ws/v1/cluster/nodes

9. YARNbest practices

9.1 resourceconfiguration

合理configurationYARN resourceparameter, including:

• yarn.nodemanager.resource.cpu-vcores: Nodemanagementr可分配 CPU核数
• yarn.nodemanager.resource.memory-mb: Nodemanagementr可分配 memory big small
• yarn.scheduler.minimum-allocation-mb: containers最 small memory分配
• yarn.scheduler.maximum-allocation-mb: containers最 big memory分配
• yarn.scheduler.minimum-allocation-vcores: containers最 small CPU分配
• yarn.scheduler.maximum-allocation-vcores: containers最 big CPU分配

9.2 scheduling器选择

根据practicalrequirements选择合适 scheduling器:

• such as果 is 单userenvironment, 且jobpriority不 high , 可以usingFIFOscheduling器
• such as果 is many userenvironment, 且需要resource隔离, 可以usingCapacityscheduling器
• such as果 is many userenvironment, 且需要公平分配resource, 可以usingFairscheduling器

9.3 high availability性configuration

for 于produceenvironment, 建议configurationYARNhigh availability, 确保system 连续性.

9.4 monitor and 告警

configurationYARNmonitor and 告警, 及时发现 and 解决issues. 可以using以 under tool:

• YARN in 置 Web UI and REST API
• 第三方monitortool, such asGanglia, Nagios, Prometheusetc.

10. summarized

本tutorial深入介绍了YARN corecomponent, architecturedesign and working principles. YARNserving asHadoop 2.xversion引入 resourcemanagementsystem, through将resourcemanagement and jobscheduling分离, implementation了更flexible, 更强 big resourcemanagement.

YARN corecomponentincludingResourcemanagementr, Nodemanagementr, ApplicationMaster and Container. Resourcemanagementr负责全局resourcemanagement and jobscheduling, Nodemanagementr负责单个node resourcemanagement and task执行, ApplicationMaster负责application程序 生命周期management, Container is resource分配 单位.

YARNsupport many 种scheduling器, includingFIFOscheduling器, Capacityscheduling器 and Fairscheduling器, 可以根据practicalrequirements选择合适 scheduling器. for 于produceenvironment, 建议configurationYARNhigh availability, 确保system 连续性.

YARN 出现使得Hadoopecosystem更加flexible and 强 big , support many 种计算framework, such asMapReduce, Spark, Flinketc., 成 for big dataprocessing important Basicscomponent.