Skip to content
The Chat Archive
deepseek-r1-0528· DeepSeek LMArena Human Preference 140K

Design a computational graph for "Recurrent networks with recurrent connections between…

  1. User

    Design a computational graph for "Recurrent networks with recurrent connections between hidden units, that read an entire sequence and then produce a single output" and develop the forward propagation equations for the same.

  2. Assistant

    Computational Graph for Recurrent Network (Many-to-One)

    The computational graph below illustrates a recurrent neural network (RNN) with recurrent hidden connections that processes an input sequence and produces a single output. Here, ( T ) is the sequence length, ( \boldsymbol{x}_t ) is the input at time ( t ), ( \boldsymbol{h}_t ) is the hidden state, and ( \boldsymbol{y} ) is the final output.

        (h_0)────────┐
          │          │
          ▼          ▼
    x₁ → RNN → h₁ → RNN → h₂ → ... → h_{T-1} → RNN → h_T → Output → y
          ▲      ▲      ▲              ▲          ▲          ▲
          │      │      │              │          │          │
          x₂     x₃     x₄            x_T        W_hy      b_y
          │      │      │              │          │
          └──────┴──────┴─────── ... ─┘          │
              (Shared weights W_xh, W_hh, b_h)   │
                                                 │
                                       (Parameters not shared)
    

    Key Components:

    1. Initial State: ( \boldsymbol{h}_0 = \boldsymbol{0} ) (zero vector)
    2. RNN Cell (repeated for ( t = 1 ) to ( T )):
      • Input: ( \boldsymbol{x}t ), Previous hidden state ( \boldsymbol{h}{t-1} )
      • Parameters: ( W_{\text{xh}}, W_{\text{hh}}, \boldsymbol{b}_h ) (shared across time steps)
    3. Output Layer:
      • Input: Final hidden state ( \boldsymbol{h}_T )
      • Parameters: ( W_{\text{hy}}, \boldsymbol{b}_y ) (not shared with RNN cells)

    Forward Propagation Equations

    1. Initialize Hidden State

    [ \boldsymbol{h}0 = \boldsymbol{0} \quad \text{(Zero vector of dimension } h{\text{dim}} \text{)} ]

    2. Recurrent Hidden State Update (for ( t = 1, 2, \dots, T ))

    [ \boldsymbol{a}t = W{\text{xh}} \boldsymbol{x}t + W{\text{hh}} \boldsymbol{h}_{t-1} + \boldsymbol{b}_h ] [ \boldsymbol{h}_t = \sigma(\boldsymbol{a}_t) ] where:

    • ( \boldsymbol{x}_t \in \mathbb{R}^{d} ): Input vector at time ( t )
    • ( W_{\text{xh}} \in \mathbb{R}^{h_{\text{dim}} \times d} ): Input-to-hidden weights
    • ( W_{\text{hh}} \in \mathbb{R}^{h_{\text{dim}} \times h_{\text{dim}}} ): Hidden-to-hidden weights
    • ( \boldsymbol{b}h \in \mathbb{R}^{h{\text{dim}}} ): Hidden layer bias
    • ( \sigma ): Activation function (e.g., (\tanh) or ReLU)

    3. Final Output

    [ \boldsymbol{y} = g \left( W_{\text{hy}} \boldsymbol{h}_T + \boldsymbol{b}_y \right) ] where:

    • ( W_{\text{hy}} \in \mathbb{R}^{o_{\text{dim}} \times h_{\text{dim}}} ): Hidden-to-output weights
    • ( \boldsymbol{b}y \in \mathbb{R}^{o{\text{dim}}} ): Output layer bias
    • ( g ): Output activation (e.g., softmax for classification, linear for regression)

    Dimensionality Summary

    VariableDimensionDescription
    ( \boldsymbol{x}_t )( \mathbb{R}^{d} )Input vector
    ( \boldsymbol{h}_t )( \mathbb{R}^{h_{\text{dim}}} )Hidden state
    ( \boldsymbol{y} )( \mathbb{R}^{o_{\text{dim}}} )Output
    ( W_{\text{xh}} )( \mathbb{R}^{h_{\text{dim}} \times d} )Input weights
    ( W_{\text{hh}} )( \mathbb{R}^{h_{\text{dim}} \times h_{\text{dim}}} )Recurrent weights
    ( W_{\text{hy}} )( \mathbb{R}^{o_{\text{dim}} \times h_{\text{dim}}} )Output weights

    Example with Concrete Values

    • Input: Sequence ( {\boldsymbol{x}_1, \boldsymbol{x}_2} ) where ( \boldsymbol{x}_1 = \begin{bmatrix} 0.5 \ -1.2 \end{bmatrix}, \boldsymbol{x}_2 = \begin{bmatrix} 0.8 \ 0.3 \end{bmatrix} )
    • Parameters:
      [ W_{\text{xh}} = \begin{bmatrix} 0.1 & 0.4 \ -0.2 & 0.3 \end{bmatrix}, ; W_{\text{hh}} = \begin{bmatrix} 0.5 & 0.1 \ -0.3 & 0.2 \end{bmatrix}, ; \boldsymbol{b}h = \begin{bmatrix} 0.1 \ -0.1 \end{bmatrix}, ; W{\text{hy}} = \begin{bmatrix} 1.0 & -0.5 \end{bmatrix}, ; \boldsymbol{b}_y = \begin{bmatrix} 0.2 \end{bmatrix} ]
    • Activation: ( \sigma = \tanh ), ( g = \text{linear} )

    Step-by-Step Calculation:

    1. At ( t = 1 ): [ \boldsymbol{a}_1 = \begin{bmatrix} 0.1 & 0.4 \ -0.2 & 0.3 \end{bmatrix} \begin{bmatrix} 0.5 \ -1.2 \end{bmatrix} + \begin{bmatrix} 0.5 & 0.1 \ -0.3 & 0.2 \end{bmatrix} \boldsymbol{0} + \begin{bmatrix} 0.1 \ -0.1 \end{bmatrix} = \begin{bmatrix} -0.43 \ -0.46 \end{bmatrix} ] [ \boldsymbol{h}_1 = \tanh\left( \begin{bmatrix} -0.43 \ -0.46 \end{bmatrix} \right) = \begin{bmatrix} -0.406 \ -0.430 \end{bmatrix} ]

    2. At ( t = 2 ): [ \boldsymbol{a}_2 = \begin{bmatrix} 0.1 & 0.4 \ -0.2 & 0.3 \end{bmatrix} \begin{bmatrix} 0.8 \ 0.3 \end{bmatrix} + \begin{bmatrix} 0.5 & 0.1 \ -0.3 & 0.2 \end{bmatrix} \begin{bmatrix} -0.406 \ -0.430 \end{bmatrix} + \begin{bmatrix} 0.1 \ -0.1 \end{bmatrix} = \begin{bmatrix} -0.156 \ -0.253 \end{bmatrix} ] [ \boldsymbol{h}_2 = \tanh\left( \begin{bmatrix} -0.156 \ -0.253 \end{bmatrix} \right) = \begin{bmatrix} -0.155 \ -0.248 \end{bmatrix} ]

    3. Output: [ \boldsymbol{y} = \begin{bmatrix} 1.0 & -0.5 \end{bmatrix} \begin{bmatrix} -0.155 \ -0.248 \end{bmatrix} + \begin{bmatrix} 0.2 \end{bmatrix} = \boxed{-0.049} ]


    Key Characteristics:

    1. Recurrence: Hidden states ( \boldsymbol{h}t ) depend on previous states ( \boldsymbol{h}{t-1} ).
    2. Parameter Sharing: ( (W_{\text{xh}}, W_{\text{hh}}, \boldsymbol{b}_h) ) are reused across all time steps.
    3. Sequence Processing: Entire input sequence ( (\boldsymbol{x}_1, \dots, \boldsymbol{x}_T) ) is read before producing output ( \boldsymbol{y} ).
    4. Applications: Sentiment analysis (sequence → sentiment score), sequence classification, etc.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.