Design a computational graph for "Recurrent networks with recurrent connections between…
- User
Design a computational graph for "Recurrent networks with recurrent connections between hidden units, that read an entire sequence and then produce a single output" and develop the forward propagation equations for the same.
- Assistant
Computational Graph for Recurrent Network (Many-to-One)
The computational graph below illustrates a recurrent neural network (RNN) with recurrent hidden connections that processes an input sequence and produces a single output. Here, ( T ) is the sequence length, ( \boldsymbol{x}_t ) is the input at time ( t ), ( \boldsymbol{h}_t ) is the hidden state, and ( \boldsymbol{y} ) is the final output.
(h_0)────────┐ │ │ ▼ ▼ x₁ → RNN → h₁ → RNN → h₂ → ... → h_{T-1} → RNN → h_T → Output → y ▲ ▲ ▲ ▲ ▲ ▲ │ │ │ │ │ │ x₂ x₃ x₄ x_T W_hy b_y │ │ │ │ │ └──────┴──────┴─────── ... ─┘ │ (Shared weights W_xh, W_hh, b_h) │ │ (Parameters not shared)Key Components:
- Initial State: ( \boldsymbol{h}_0 = \boldsymbol{0} ) (zero vector)
- RNN Cell (repeated for ( t = 1 ) to ( T )):
- Input: ( \boldsymbol{x}t ), Previous hidden state ( \boldsymbol{h}{t-1} )
- Parameters: ( W_{\text{xh}}, W_{\text{hh}}, \boldsymbol{b}_h ) (shared across time steps)
- Output Layer:
- Input: Final hidden state ( \boldsymbol{h}_T )
- Parameters: ( W_{\text{hy}}, \boldsymbol{b}_y ) (not shared with RNN cells)
Forward Propagation Equations
1. Initialize Hidden State
[ \boldsymbol{h}0 = \boldsymbol{0} \quad \text{(Zero vector of dimension } h{\text{dim}} \text{)} ]
2. Recurrent Hidden State Update (for ( t = 1, 2, \dots, T ))
[ \boldsymbol{a}t = W{\text{xh}} \boldsymbol{x}t + W{\text{hh}} \boldsymbol{h}_{t-1} + \boldsymbol{b}_h ] [ \boldsymbol{h}_t = \sigma(\boldsymbol{a}_t) ] where:
- ( \boldsymbol{x}_t \in \mathbb{R}^{d} ): Input vector at time ( t )
- ( W_{\text{xh}} \in \mathbb{R}^{h_{\text{dim}} \times d} ): Input-to-hidden weights
- ( W_{\text{hh}} \in \mathbb{R}^{h_{\text{dim}} \times h_{\text{dim}}} ): Hidden-to-hidden weights
- ( \boldsymbol{b}h \in \mathbb{R}^{h{\text{dim}}} ): Hidden layer bias
- ( \sigma ): Activation function (e.g., (\tanh) or ReLU)
3. Final Output
[ \boldsymbol{y} = g \left( W_{\text{hy}} \boldsymbol{h}_T + \boldsymbol{b}_y \right) ] where:
- ( W_{\text{hy}} \in \mathbb{R}^{o_{\text{dim}} \times h_{\text{dim}}} ): Hidden-to-output weights
- ( \boldsymbol{b}y \in \mathbb{R}^{o{\text{dim}}} ): Output layer bias
- ( g ): Output activation (e.g., softmax for classification, linear for regression)
Dimensionality Summary
Variable Dimension Description ( \boldsymbol{x}_t ) ( \mathbb{R}^{d} ) Input vector ( \boldsymbol{h}_t ) ( \mathbb{R}^{h_{\text{dim}}} ) Hidden state ( \boldsymbol{y} ) ( \mathbb{R}^{o_{\text{dim}}} ) Output ( W_{\text{xh}} ) ( \mathbb{R}^{h_{\text{dim}} \times d} ) Input weights ( W_{\text{hh}} ) ( \mathbb{R}^{h_{\text{dim}} \times h_{\text{dim}}} ) Recurrent weights ( W_{\text{hy}} ) ( \mathbb{R}^{o_{\text{dim}} \times h_{\text{dim}}} ) Output weights Example with Concrete Values
- Input: Sequence ( {\boldsymbol{x}_1, \boldsymbol{x}_2} ) where ( \boldsymbol{x}_1 = \begin{bmatrix} 0.5 \ -1.2 \end{bmatrix}, \boldsymbol{x}_2 = \begin{bmatrix} 0.8 \ 0.3 \end{bmatrix} )
- Parameters:
[ W_{\text{xh}} = \begin{bmatrix} 0.1 & 0.4 \ -0.2 & 0.3 \end{bmatrix}, ; W_{\text{hh}} = \begin{bmatrix} 0.5 & 0.1 \ -0.3 & 0.2 \end{bmatrix}, ; \boldsymbol{b}h = \begin{bmatrix} 0.1 \ -0.1 \end{bmatrix}, ; W{\text{hy}} = \begin{bmatrix} 1.0 & -0.5 \end{bmatrix}, ; \boldsymbol{b}_y = \begin{bmatrix} 0.2 \end{bmatrix} ] - Activation: ( \sigma = \tanh ), ( g = \text{linear} )
Step-by-Step Calculation:
-
At ( t = 1 ): [ \boldsymbol{a}_1 = \begin{bmatrix} 0.1 & 0.4 \ -0.2 & 0.3 \end{bmatrix} \begin{bmatrix} 0.5 \ -1.2 \end{bmatrix} + \begin{bmatrix} 0.5 & 0.1 \ -0.3 & 0.2 \end{bmatrix} \boldsymbol{0} + \begin{bmatrix} 0.1 \ -0.1 \end{bmatrix} = \begin{bmatrix} -0.43 \ -0.46 \end{bmatrix} ] [ \boldsymbol{h}_1 = \tanh\left( \begin{bmatrix} -0.43 \ -0.46 \end{bmatrix} \right) = \begin{bmatrix} -0.406 \ -0.430 \end{bmatrix} ]
-
At ( t = 2 ): [ \boldsymbol{a}_2 = \begin{bmatrix} 0.1 & 0.4 \ -0.2 & 0.3 \end{bmatrix} \begin{bmatrix} 0.8 \ 0.3 \end{bmatrix} + \begin{bmatrix} 0.5 & 0.1 \ -0.3 & 0.2 \end{bmatrix} \begin{bmatrix} -0.406 \ -0.430 \end{bmatrix} + \begin{bmatrix} 0.1 \ -0.1 \end{bmatrix} = \begin{bmatrix} -0.156 \ -0.253 \end{bmatrix} ] [ \boldsymbol{h}_2 = \tanh\left( \begin{bmatrix} -0.156 \ -0.253 \end{bmatrix} \right) = \begin{bmatrix} -0.155 \ -0.248 \end{bmatrix} ]
-
Output: [ \boldsymbol{y} = \begin{bmatrix} 1.0 & -0.5 \end{bmatrix} \begin{bmatrix} -0.155 \ -0.248 \end{bmatrix} + \begin{bmatrix} 0.2 \end{bmatrix} = \boxed{-0.049} ]
Key Characteristics:
- Recurrence: Hidden states ( \boldsymbol{h}t ) depend on previous states ( \boldsymbol{h}{t-1} ).
- Parameter Sharing: ( (W_{\text{xh}}, W_{\text{hh}}, \boldsymbol{b}_h) ) are reused across all time steps.
- Sequence Processing: Entire input sequence ( (\boldsymbol{x}_1, \dots, \boldsymbol{x}_T) ) is read before producing output ( \boldsymbol{y} ).
- Applications: Sentiment analysis (sequence → sentiment score), sequence classification, etc.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.