Skip to content
TheChatArchiveThe Chat Archive
MiniMaxminimax-m1· MiniMaxHugging Face LMArena Human Preference 140K

improve the figure: \begin{frame}{Large Language Models as Reinforcement Learning…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with DeepSeekdeepseek-v3-0324’s answer.
  1. User

    improve the figure: \begin{frame}{Large Language Models as Reinforcement Learning Policies} \begin{columns} \column{0.55\textwidth} \begin{itemize} \item Large Language Models (LLMs) can be framed as \textbf{stochastic policies}: [ \pi_{\text{LLM}}(a_t \mid s_t; \theta) ] \item This maps directly to the RL framework: \begin{itemize} \item \textbf{State (sts_t):} The input prompt combined with the sequence of tokens generated so far. \item \textbf{Action (ata_t):} The next token sampled from the LLM's probability distribution over the vocabulary. \item \textbf{Policy (πLLM\pi_{\text{LLM}}):} The LLM itself, parameterized by its neural network weights (θ\theta). \item \textbf{Trajectory (τ\tau):} A complete generated sequence (e.g., prompt followed by generated answer). \end{itemize} \end{itemize} \column{0.45\textwidth} \centering \begin{tikzpicture}[scale=0.9, every node/.style={font=\footnotesize}] % Nodes \node[draw=blue!60, thick, rounded corners=3pt, minimum width=2.8cm, minimum height=1.8cm] (llm) at (0,0) {LLM (Policy πθ\pi_\theta)}; \node[draw=gray!60, thick, rounded corners=3pt, minimum width=2.4cm, minimum height=1.4cm] (env) at (0,-2.8) {Environment (Task)};

    % Arrows
    \draw[->, thick, orange] (llm.south) -- ([yshift=7pt]env.north)
        node[midway, left, orange] {action $a_t$ (next token)};
    \draw[->, thick, teal!70!black] ([yshift=-7pt]env.north) -- (llm.south)
        node[midway, right, teal!70!black] {state $s_t$ (current text)};
    
    % Additional LLM-specific arrows/nodes
    \node[align=center, text=violet, above=0.2cm of llm.north] {Maximize Reward\\ (e.g., helpfulness)};
    \draw[->, thick, violet] (env.south east) .. controls +(0.7,-0.4) and +(0.7,0.4) .. (llm.east)
        node[midway, right, violet] {reward $r_t$};
    

    \end{tikzpicture} \end{columns} \end{frame}

  2. MiniMax Assistant

    To enhance the figure, we improved notation consistency, clarified labels, and refined the diagram's visual flow:

    \begin{frame}{Large Language Models as Reinforcement Learning Policies}
    \begin{columns}
    \column{0.55\textwidth}
    \begin{itemize}
        \item Large Language Models (LLMs) can be framed as \textbf{stochastic policies}:
        \[
        \pi_{\text{LLM}}(a_t \mid s_t; \theta)
        \]
        \item This maps directly to the RL framework:
        \begin{itemize}
            \item \textbf{State ($s_t$):} Input prompt + generated tokens so far
            \item \textbf{Action ($a_t$):} Next token from LLM's probability distribution
            \item \textbf{Policy ($\pi_{\text{LLM}}$):} LLM parameterized by neural weights ($\theta$)
            \item \textbf{Trajectory ($\tau$):} Complete generated sequence (prompt + answer)
        \end{itemize}
    \end{itemize}
    \column{0.45\textwidth}
    \centering
    \begin{tikzpicture}[scale=0.9, every node/.style={font=\footnotesize}]
        % Nodes
        \node[draw=blue!60, thick, rounded corners=3pt, minimum width=3cm, minimum height=2cm] 
            (llm) at (0,0) {LLM (Policy $\pi_{\text{LLM}}$)};
        \node[draw=gray!60, thick, rounded corners=3pt, minimum width=3cm, minimum height=1.6cm] 
            (env) at (0,-3) {Environment (Task)};
    
        % Action arrow (LLM -> Environment)
        \draw[->, thick, orange] (llm.south) -- ([yshift=8pt]env.north)
            node[midway, left=2pt, orange] {action $a_t$ (next token)};
        
        % State arrow (Environment -> LLM)
        \draw[->, thick, teal!70!black] ([yshift=-8pt]env.north) -- (llm.south)
            node[midway, right=2pt, teal!70!black] {state $s_t$ (prompt + tokens)};
    
        % Reward connection
        \node[align=center, text=violet, above=0.3cm of llm.north] 
            {Maximize Reward\\ (e.g., helpfulness, coherence)};
        \draw[->, thick, violet, bend left=15] (env.east) to 
            node[midway, right=2pt, violet] {reward $r_t$} (llm.east);
    \end{tikzpicture}
    \end{columns}
    \end{frame}
    

    Key improvements:

    1. Consistent Notation: Policy notation πLLM\pi_{\text{LLM}} matches text and diagram
    2. Clearer State Label: "prompt + tokens" explicitly shows state composition
    3. Enhanced Visual Flow:
      • Adjusted node sizes for better balance
      • Improved reward arrow curvature using bend left
      • Increased vertical spacing between nodes
    4. Better Color Contrast: Maintained original color scheme but improved node sizing
    5. More Explicit Labels: Action/state labels include directional hints
    6. Modernized Layout: Clean node proportions and arrow positioning

    The diagram now better emphasizes the core RL relationship while maintaining clarity for presentation slides.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.