强化Learning

1. what is 强化Learning？

强化Learning (Reinforcement Learning, 简称RL) is 机器Learning 一个branch, 它through智能体 (Agent) and environment (Environment) 交互来Learning最优策略. 智能体through执行动作 (Action) 来影响environment, environment则through反馈奖励 (Reward) 信号来指导智能体 Learning.

1.1 强化Learning basic要素

• 智能体 (Agent) : Learning and 执行动作 主体.
• environment (Environment) : 智能体交互 out 部世界.
• status (State) : environment 当 before circumstances.
• 动作 (Action) : 智能体可以执行 operation.
• 奖励 (Reward) : environment for 智能体动作 反馈信号.
• 策略 (Policy) : 智能体 from status to 动作 map.
• valuefunction (Value Function) : 衡量status or status-动作 for long 期value.
• model (Model) : 智能体 for environment understanding and 预测.

1.2 强化Learning 特点

• 试错Learning: through尝试不同 动作并接收反馈来Learning.
• latency奖励: 奖励可能 in 一系列动作 after 才会获得.
• 序列决策: 需要考虑动作 long 期影响, 而不仅仅 is 即时奖励.
• 不确定性: environment可能 is 随机 , 动作 结果可能不确定.

2. 马尔可夫决策过程

马尔可夫决策过程 (Markov Decision Process, 简称MDP) is 强化Learning 数学Basics, 它 is a describes序贯决策issues framework.

2.1 马尔可夫性质

马尔可夫性质 is 指: 未来 status只依赖于当 before status, 而 and 过去 status无关. 用数学公式表示 for :

P(Sₜ₊₁ | Sₜ, Aₜ, Sₜ₋₁, Aₜ₋₁, ..., S₀, A₀) = P(Sₜ₊₁ | Sₜ, Aₜ)

2.2 MDP 组成

• status空间 (S) : 所 has 可能status collection.
• 动作空间 (A) : 所 has 可能动作 collection.
• 转移概率 (P) : in statuss执行动作a after 转移 to statuss' 概率.
• 奖励function (R) : in statuss执行动作a after 获得 即时奖励.
• 折扣因子 (γ) : 未来奖励 折扣系数, 取值范围 for [0, 1].

2.3 策略 and valuefunction

2.3.1 策略

策略π is from status to 动作 map, 可以 is 确定性 or 随机性 :

• 确定性策略: π(s) = a, 表示 in statuss时选择动作a.
• 随机性策略: π(a|s) = P(A=a | S=s), 表示 in statuss时选择动作a 概率.

2.3.2 valuefunction

valuefunction衡量status or status-动作 for long 期value:

• statusvaluefunction (Vπ(s)) : in 策略π under , from statuss开始 expectation回报.
• 动作valuefunction (Qπ(s,a)) : in 策略π under , from statuss执行动作a after expectation回报.

valuefunction 递推relationships (贝尔曼方程) :

Vπ(s) = E[Rₜ₊₁ + γVπ(Sₜ₊₁) | Sₜ = s]

Qπ(s,a) = E[Rₜ₊₁ + γQπ(Sₜ₊₁, Aₜ₊₁) | Sₜ = s, Aₜ = a]

3. 强化Learning corealgorithms

3.1 基于value method

3.1.1 Q-learning

Q-learning is a无model 强化Learningalgorithms, 它直接Learning动作valuefunctionQ(s,a):

Q(s,a) ← Q(s,a) + α[R + γmaxₐ'Q(s',a') - Q(s,a)]

• α is Learning率, 控制update 步 long .
• γ is 折扣因子.
• maxₐ'Q(s',a') is for under 一statuss' 最 big 动作value 估计.

3.1.2 SARSA

SARSA is a基于策略 强化Learningalgorithms, 它Learningstatus-动作-奖励-status-动作 转换:

Q(s,a) ← Q(s,a) + α[R + γQ(s',a') - Q(s,a)]

and Q-learning不同, SARSAusingpractical执行 under 一动作a' Q值, 而不 is 最 big Q值, 因此它 is aon-policyalgorithms.

3.1.3 DQN (深度Qnetwork)

DQN结合了Q-learning and 深度Learning, using神经network来近似Qfunction:

• usingexperience回放 (Experience Replay) 来 stable 训练.
• using目标network (Target Network) 来reducing训练 不 stable 性.

3.2 基于策略 method

3.2.1 策略梯度

策略梯度 (Policy Gradient) 直接optimization策略πθ, through计算策略 梯度并沿梯度方向updateparameter:

∇θJ(θ) = E[∇θlogπθ(a|s)Qπθ(s,a)]

3.2.2 REINFORCEalgorithms

REINFORCE is a蒙特卡洛策略梯度algorithms, 它using完整 轨迹来估计梯度:

• 收集一条完整 轨迹.
• 计算每个status-动作 for 回报.
• using回报serving as权重, update策略parameter.

3.2.3 Actor-Criticalgorithms

Actor-Critic结合了valuefunction and 策略梯度 优点:

• Actor: 负责选择动作, 基于当 before 策略.
• Critic: 负责assessmentActor 动作, 估计valuefunction.

3.3 基于model method

基于model 强化LearningmethodthroughLearningenvironment model来planning动作:

• modelLearning: Learningenvironment 转移概率 and 奖励function.
• planning: usingLearning to model来mock and planning未来 动作.

3.4 深度强化Learning

深度强化Learning (Deep Reinforcement Learning) 结合了深度Learning and 强化Learning, able toprocessing high 维status空间 issues:

• DQN及其变体: such asDouble DQN, Dueling DQN, Prioritized Experience Replayetc..
• 策略梯度 深度version: such asDDPG, PPO, SACetc..
• model-based深度强化Learning: such as世界model, model预测控制etc..

4. 强化Learning 探索 and 利用

探索 and 利用 (Exploration vs Exploitation) is 强化Learningin coreissues:

4.1 探索

探索 is 指尝试 new 动作, 以发现可能 更 good 策略:

• ε-贪婪策略: 以概率ε随机选择动作, 以概率1-ε选择当 before 认 for 最 good 动作.
• Boltzmann探索: 根据动作value softmax概率选择动作.
• on 限置信区间 (UCB) : 平衡探索 and 利用 策略.

4.2 利用

利用 is 指根据当 before knowledge选择认 for 最 good 动作:

• 选择value最 high 动作.
• 基于当 before 策略选择动作.

4.3 探索 and 利用 平衡

in 强化Learningin, 需要 in 探索 and 利用之间找 to 平衡:

• 过 many 探索可能导致efficiency low under .
• 过 many 利用可能导致陷入局部最优.
• 通常adopts逐渐reducing探索率 策略.

5. codeexample: Q-learningimplementation

under 面 is a usingPythonimplementationQ-learningalgorithms解决 simple 网格世界issues example:

import numpy as np
import matplotlib.pyplot as plt

# 定义网格世界environment
class GridWorld:
    def __init__(self):
        # 4x4网格
        self.grid_size = 4
        # 终止status
        self.terminal_states = [(0, 0), (3, 3)]
        # 动作:  on ,  right ,  under ,  left 
        self.actions = [(-1, 0), (0, 1), (1, 0), (0, -1)]
        self.action_names = [' on ', ' right ', ' under ', ' left ']
    
    def step(self, state, action):
        # 计算 new status
        new_state = (state[0] + action[0], state[1] + action[1])
        
        # check is 否越界
        if new_state[0] < 0 or new_state[0] >= self.grid_size or 
           new_state[1] < 0 or new_state[1] >= self.grid_size:
            new_state = state
        
        # check is 否 to 达终止status
        if new_state in self.terminal_states:
            reward = 0
            done = True
        else:
            reward = -1
            done = False
        
        return new_state, reward, done
    
    def reset(self):
        # 随机初始化status (非终止status) 
        while True:
            state = (np.random.randint(self.grid_size), np.random.randint(self.grid_size))
            if state not in self.terminal_states:
                return state

# Q-learningalgorithms
class QLearningAgent:
    def __init__(self, env, learning_rate=0.1, discount_factor=0.99, epsilon=1.0, epsilon_decay=0.999, epsilon_min=0.01):
        self.env = env
        self.lr = learning_rate
        self.gamma = discount_factor
        self.epsilon = epsilon
        self.epsilon_decay = epsilon_decay
        self.epsilon_min = epsilon_min
        
        # 初始化Q表
        self.q_table = np.zeros((env.grid_size, env.grid_size, len(env.actions)))
    
    def choose_action(self, state):
        # ε-贪婪策略选择动作
        if np.random.uniform(0, 1) < self.epsilon:
            # 探索: 随机选择动作
            return np.random.randint(len(self.env.actions))
        else:
            # 利用: 选择Q值最 big  动作
            return np.argmax(self.q_table[state[0], state[1], :])
    
    def learn(self, state, action, reward, next_state, done):
        # Q-learningupdate规则
        old_value = self.q_table[state[0], state[1], action]
        
        if done:
            next_max = 0
        else:
            next_max = np.max(self.q_table[next_state[0], next_state[1], :])
        
        # updateQ值
        new_value = old_value + self.lr * (reward + self.gamma * next_max - old_value)
        self.q_table[state[0], state[1], action] = new_value
    
    def decay_epsilon(self):
        # 衰减探索率
        self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)

# 训练Q-learning agent
def train_agent():
    env = GridWorld()
    agent = QLearningAgent(env)
    
    episodes = 10000
    steps_per_episode = []
    
    for episode in range(episodes):
        state = env.reset()
        done = False
        steps = 0
        
        while not done:
            # 选择动作
            action_idx = agent.choose_action(state)
            action = env.actions[action_idx]
            
            # 执行动作
            next_state, reward, done = env.step(state, action)
            
            # Learning
            agent.learn(state, action_idx, reward, next_state, done)
            
            # updatestatus
            state = next_state
            steps += 1
        
        # 衰减探索率
        agent.decay_epsilon()
        steps_per_episode.append(steps)
        
        if episode % 1000 == 0:
            print(f"Episode {episode}, Steps: {steps}, Epsilon: {agent.epsilon:.3f}")
    
    return agent, steps_per_episode

# test训练 good  agent
def test_agent(agent):
    env = GridWorld()
    state = env.reset()
    done = False
    path = [state]
    
    print("test训练 good  agent:")
    print(f"起始status: {state}")
    
    while not done:
        # 选择最优动作
        action_idx = np.argmax(agent.q_table[state[0], state[1], :])
        action = env.actions[action_idx]
        action_name = env.action_names[action_idx]
        
        # 执行动作
        next_state, reward, done = env.step(state, action)
        
        print(f"动作: {action_name},  under 一status: {next_state}, 奖励: {reward}")
        path.append(next_state)
        state = next_state
    
    print(f"终止status: {state}")
    print(f"path: {path}")
    return path

# visualization训练过程 and 结果
def visualize_results(agent, steps_per_episode, path):
    # visualization训练曲线
    plt.figure(figsize=(12, 6))
    
    plt.subplot(1, 2, 1)
    plt.plot(steps_per_episode)
    plt.title('训练过程: 每 episode  步数')
    plt.xlabel('Episode')
    plt.ylabel('Steps')
    
    # visualizationQ表
    plt.subplot(1, 2, 2)
    # 计算每个status 最 big Q值
    max_q = np.max(agent.q_table, axis=2)
    plt.imshow(max_q, cmap='hot', interpolation='nearest')
    plt.title('每个status 最 big Q值')
    plt.colorbar()
    
    # 标记path
    for i, (x, y) in enumerate(path):
        plt.text(y, x, str(i), color='white', ha='center', va='center', fontweight='bold')
    
    # 标记终止status
    for (x, y) in agent.env.terminal_states:
        plt.text(y, x, 'T', color='blue', ha='center', va='center', fontweight='bold', fontsize=12)
    
    plt.tight_layout()
    plt.show()

# 主function
if __name__ == "__main__":
    # 训练agent
    agent, steps_per_episode = train_agent()
    
    # testagent
    path = test_agent(agent)
    
    # visualization结果
    visualize_results(agent, steps_per_episode, path)
    
    # 打印Q表
    print("\nQ表:")
    for i in range(agent.env.grid_size):
        for j in range(agent.env.grid_size):
            print(f"status ({i},{j}): {agent.q_table[i, j, :]}")

5. 强化Learning application场景

5.1 游戏

• 棋盘游戏 (such as国际象棋, 围棋) .
• 电子游戏 (such asAtari游戏, 星际争霸, DOTA 2) .
• 游戏AIdesign.

5.2 机器人控制

• 机械臂控制.
• 机器人导航 and 避障.
• 四足机器人 运动控制.
• 人形机器人 平衡 and 行走.

5.3 自动驾驶

• 车辆控制 and pathplanning.
• 交通信号控制.
• 自动驾驶策略optimization.

5.4 financial transaction

• algorithms交易策略.
• 投资组合optimization.
• riskmanagement.

5.5 resource scheduling

• datain心 load balancing.
• 智能电网 能sourcesscheduling.
• 物流 and 供应链management.

5.6 推荐system

• personalized推荐策略.
• long 期uservalue最 big 化.
• many 臂老虎机issues in 推荐in application.

5.7 医疗healthy

• personalized治疗solutions.
• 药物剂量optimization.
• 医疗resource分配.

5.8 工业控制

• 化工过程控制.
• 制造业 producescheduling.
• quality控制 and 预测性maintenance.

6. 强化Learning challenges

6.1 样本efficiency

• 需要 big 量 交互样本才能Learning to good 策略.
• in 真实世界in, 获取样本可能很昂贵 or dangerous .

6.2 stable 性 and 收敛性

• 训练过程可能不 stable , easy 发散.
• valuefunction or 策略可能振荡.

6.3 探索 and 利用 平衡

• such as何 has 效地平衡探索 and 利用.
• in high 维空间in探索变得更加 difficult .

6.4 泛化capacity

• in 训练environmentinLearning 策略可能难以泛化 to new environment.
• environment 微 small 变化可能导致performance big 幅 under 降.

6.5 security性 and reliability

• 强化Learningsystem in 探索过程in可能采取 dangerous 动作.
• 难以保证system security性 and reliability.

7. 互动练习