Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

please, didactically solve this Machine Learning exam. Solve all steps and proceed with…

  1. User

    please, didactically solve this Machine Learning exam. Solve all steps and proceed with reasoning over all the exercises. Answer with completeness and soundness. If you need to illustrate, try to draw it using ASCII art. Format your answer in Markdown, do not use Latex formatting.

    % compile 2 times

    \documentclass[a4paper]{article} \usepackage[solutions%show/hide the solutions ]{verifiche} \usepackage{listings} \lstset{basicstyle=\ttfamily, literate=% Allow for German characters in lstlistings. {Ö}{{"O}}1 {Ä}{{"A}}1 {Ü}{{"U}}1 {ß}{{\ss}}2 {ü}{{"u}}1 {ä}{{"a}}1 {ö}{{"o}}1 {á}{{'a}}1 {é}{{'e}}1 {à}{{'a}}1 {è}{{'e}}1 } \usepackage[margin=3cm]{geometry} \usepackage{tabularx} \usepackage{lipsum} %\usepackage{subfigure} \usepackage{subcaption} \usepackage{graphicx} \usepackage{caption} \usepackage{amsmath} % For advanced math typesetting \usepackage{graphicx} % For including images \usepackage{geometry} \captionsetup{font=small, labelfont=bf} \captionsetup[sub]{labelsep=period, subrefformat=brace} \begin{document}

    %\renewcommand{\institutefont}{\huge} \renewcommand{\closedquestionitem}{◯\bigcirc} \renewcommand{\headercandidatenamelabel}{First and last name} \renewcommand{\headerclasslabel}{Student ID} \renewcommand{\headerdatelabel}{} \setlength{\daterulerlength}{0pt} \setlength{\candidatenamerulerlength}{210pt} \setlength{\classrulerlength}{65pt}

    %\institute{}

    \institute{} \asyear[]{\centerline{\includegraphics[]{images/poliba.png}}} \testtype{\normalsize{ \centerline{\textbf{FOUNDATIONS OF MACHINE LEARNING}}\ \centerline{\textbf{ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING} (ML TEST)}\ %\centerline{\textbf{MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE} (ML TEST)}\ \centerline{\normalfont{Prof. Tommaso Di Noia}}\ \centerline{\normalfont{Academic Year 2023/2024 – 8th Exam Session}} %1st Middle Exam}}\ \centerline{\normalfont{06/11/2024}}} } \instruction{\vspace{-0.8em}} \duration[Test duration:]{1,5 hours} \printheading %\centerline{\includegraphics[]{images/poliba.png}}

    \pagestyle{verifiche}

    \begin{esercizio}[label=Question]% Derive the probabilistic interpretation of the linear regression model using the Maximum Likelihood Criterion. Prove that this is \textbf{equivalent} to the MSE loss function. \end{esercizio}

    \begin{esercizio}[label=Question] \renewcommand{\diffsymb}{∙\bullet}

    Given the following neural network for a classification task, perform a feed forward step by filling the table. Suppose you are using the sigmoid as activation function. \underline{Please, detail all the steps}. \centerline{\includegraphics[width=1\textwidth]{images/image.psd.pdf}}\[0.5em]

    \end{esercizio}

    \begin{esercizio}[label=Question] Explain in \textbf{detail} the concepts of bias and variance. Draw the relative learning curves. \end{esercizio}

    \begin{esercizio}[label=Question] Illustrate the context of application and the steps of the Expectation-Maximization Algorithm (EM). \end{esercizio}

    \begin{esercizio}[label=Question] Describe the motivations and explain the aims of the PCA algorithm. Moreover, show the main differences with respect to the Kernel PCA. \end{esercizio}

    % \begin{esercizio}[label=Question]% % Consider the following \textbf{linear regression} model: y=θ0+θ1x1+θ2x2y = \theta_0 + \theta_1 x_1 + \theta_2 x_2. Assume that the current weights are θ0=0\theta_0 = 0, θ1=1\theta_1 = 1, θ2=0.5\theta_2 = 0.5. Given the following set of samples (X,y)(\mathbf{X}, \mathbf{y}), where each row in X\mathbf{X} represents a data point and each entry in y\mathbf{y} represents the % corresponding target value:\

    % \begin{figure}[ht] % \centering % \includegraphics[width=0.4\textwidth]{images/model.png} % \label{fig:decision-tree} % \end{figure} % \begin{equation*} % \mathbf{X} = \begin{bmatrix} % 3 & -2 \ % 1 & 3 \ % -4 & 1 \ % \end{bmatrix}, \quad % \mathbf{y} = \begin{bmatrix} % 1 \ % 3\ % 2 \ % \end{bmatrix}, % \end{equation*}\ % \ % Compute the initial value of loss fuction and a \textbf{single step} of Gradient Descent to update the θi\theta_i parameters with \textit{Full Batch} and in \textit{Stochastic} solution, using a learning rate α=0.01\alpha=0.01. % \end{esercizio}

    \begin{esercizio}[label=Question]% Consider the following \textbf{linear regression} model: ( y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 ). Assume that the current weights are ( \theta_0 = 1 ), ( \theta_1 = -0.5 ), ( \theta_2 = 0.8 ). Given the following set of samples ( (\mathbf{X}, \mathbf{y}) ), where each row in ( \mathbf{X} ) represents a data point and each entry in ( \mathbf{y} ) represents the corresponding target value: \

    \begin{equation*} \mathbf{X} = \begin{bmatrix} 3 & -2 \ 1 & 5 \ -4 & 3 \ \end{bmatrix}, \quad \mathbf{y} = \begin{bmatrix} 1 \ 2 \ 0 \ \end{bmatrix}, \end{equation*}

    Compute the initial value of the loss function and a \textbf{single step} of Gradient Descent to update the ( \theta_i ) parameters with \textit{Full Batch} and in \textit{Stochastic} solution, using a learning rate ( \alpha = 0.05 ). \end{esercizio}

    % \begin{esercizio}[label=Question]% % Use the k-means algorithm and Euclidean distance to cluster the following 8 examples into 3 clusters: % B1=(2,10), B2=(2,5), B3=(8,4), B4=(5,8), B5=(7,5), B6=(6,4), B7=(1,2), B8=(4,9). % The distance matrix based on the Euclidean distance is given below: % \begin{table}[ht] % \centering % \caption{Distance Matrix for Points B1-B8} % \label{distance-matrix} % \begin{tabular}{|c|c|c|c|c|c|c|c|c|} % \hline % & \textbf{B1} & \textbf{B2} & \textbf{B3} & \textbf{B4} & \textbf{B5} & \textbf{B6} & \textbf{B7} & \textbf{B8} \ \hline % \textbf{B1} & 0 & 25\sqrt{25} & 85\sqrt{85} & 13\sqrt{13} & 65\sqrt{65} & 74\sqrt{74} & 90\sqrt{90} & 5\sqrt{5} \ \hline % \textbf{B2} & & 0 & 36\sqrt{36} & 18\sqrt{18} & 25\sqrt{25} & 16\sqrt{16} & 40\sqrt{40} & 20\sqrt{20} \ \hline % \textbf{B3} & & & 0 & 25\sqrt{25} & 2\sqrt{2} & 2\sqrt{2} & 53\sqrt{53} & 41\sqrt{41} \ \hline % \textbf{B4} & & & & 0 & 13\sqrt{13} & 17\sqrt{17} & 65\sqrt{65} & 2\sqrt{2} \ \hline % \textbf{B5} & & & & & 0 & 2\sqrt{2} & 45\sqrt{45} & 25\sqrt{25} \ \hline % \textbf{B6} & & & & & & 0 & 29\sqrt{29} & 29\sqrt{29} \ \hline % \textbf{B7} & & & & & & & 0 & 58\sqrt{58} \ \hline % \textbf{B8} & & & & & & & & 0 \ \hline % \end{tabular} % \end{table} % \[0.5em] % Suppose that the initial seeds (centers of each cluster) are B1, B4 and B7. Run the k-means algorithm for % 1 epoch only. At the end of this epoch show: % \begin{itemize} % \item The new clusters (i.e. the examples belonging to each cluster) % \item The centers of the new clusters % \end{itemize} % \end{esercizio}

    \begin{esercizio}[label=Question] Matteo and Danilo are managing resources on a computer system and need to allocate processes to available servers. The decision on where to allocate each process depends on several factors, such as \textbf{CPU demand}, \textbf{memory requirements}, \textbf{server load}, \textbf{priority level} of the process, and \textbf{availability of backup resources}. Using the features provided in the dataset, build a decision tree model to assist Matteo and Danilo in making the most efficient allocation for the system.

    \begin{table}[ht] \centering \caption{Decision Table for Process Allocation} \label{allocation-decision-table} \begin{tabularx}{\textwidth}{|X|X|X|X|X|X|} \hline \textbf{CPU Demand} & \textbf{Memory Requirements} & \textbf{Server Load} & \textbf{Priority Level} & \textbf{Backup Availability} & \textbf{Allocation Decision} \ \hline Low & Low & High & Normal & Not Available & No \ \hline Moderate & High & Moderate & Normal & Available & Yes \ \hline High & Low & High & Urgent & Not Available & No \ \hline Low & Moderate & Low & Low & Available & Yes \ \hline Moderate & Low & High & Normal & Available & No \ \hline High & High & Moderate & Low & Available & No \ \hline \end{tabularx} \end{table} \end{esercizio}

    \end{document}

  2. Assistant

    Exercise 1

    Question:

    Derive the probabilistic interpretation of the linear regression model using the Maximum Likelihood Criterion. Prove that this is equivalent to the MSE loss function.


    Solution

    1. Linear Regression Model

    Linear Regression Equation:

    The linear regression model assumes that the target variable ( y ) is a linear combination of input features ( \mathbf{x} ) plus some noise ( \epsilon ):

    [ y = \mathbf{x}^\top \boldsymbol{\theta} + \epsilon ]

    Where:

    • ( \mathbf{x} ) is the input feature vector (including a bias term if necessary).
    • ( \boldsymbol{\theta} ) is the parameter (weight) vector.
    • ( \epsilon ) is the noise term.

    Assumptions:

    • The noise ( \epsilon ) follows a normal distribution with mean 0 and variance ( \sigma^2 ): ( \epsilon \sim \mathcal{N}(0, \sigma^2) ).
    • Noise terms are independent across observations.

    2. Maximum Likelihood Estimation (MLE)

    The goal of MLE is to find the parameter values ( \boldsymbol{\theta} ) that maximize the likelihood of observing the given data.

    Likelihood Function:

    Given ( n ) independent observations ( { (\mathbf{x}i, y_i) }{i=1}^n ), the likelihood function is:

    [ L(\boldsymbol{\theta}, \sigma^2) = \prod_{i=1}^n p(y_i | \mathbf{x}_i, \boldsymbol{\theta}, \sigma^2) ]

    Since ( y_i ) is normally distributed:

    [ p(y_i | \mathbf{x}_i, \boldsymbol{\theta}, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left( -\frac{(y_i - \mathbf{x}_i^\top \boldsymbol{\theta})^2}{2\sigma^2} \right) ]

    Thus, the likelihood becomes:

    [ L(\boldsymbol{\theta}, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left( -\frac{(y_i - \mathbf{x}_i^\top \boldsymbol{\theta})^2}{2\sigma^2} \right) ]

    Log-Likelihood Function:

    Taking the natural logarithm to simplify:

    [ \log L(\boldsymbol{\theta}, \sigma^2) = \sum_{i=1}^n \left( -\frac{1}{2} \log(2\pi \sigma^2) - \frac{(y_i - \mathbf{x}_i^\top \boldsymbol{\theta})^2}{2\sigma^2} \right) ]

    Simplifying further:

    [ \log L(\boldsymbol{\theta}, \sigma^2) = -\frac{n}{2} \log(2\pi \sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^n (y_i - \mathbf{x}_i^\top \boldsymbol{\theta})^2 ]

    3. Maximizing the Log-Likelihood

    To find the MLE of ( \boldsymbol{\theta} ) and ( \sigma^2 ), we maximize the log-likelihood with respect to these parameters.

    Focusing on ( \boldsymbol{\theta} ):

    Assuming ( \sigma^2 ) is known or treating it as a constant, maximizing ( \log L ) with respect to ( \boldsymbol{\theta} ) is equivalent to minimizing the sum of squared errors (since the first term is constant with respect to ( \boldsymbol{\theta} )):

    [ \text{Minimize} \quad \sum_{i=1}^n (y_i - \mathbf{x}_i^\top \boldsymbol{\theta})^2 ]

    4. Equivalence to MSE Loss Function

    Mean Squared Error (MSE):

    The MSE loss function is defined as:

    [ \text{MSE} = \frac{1}{n} \sum_{i=1}^n (y_i - \mathbf{x}_i^\top \boldsymbol{\theta})^2 ]

    Equivalence:

    Minimizing the sum of squared errors (SSE) is equivalent to minimizing the MSE, as MSE is just SSE scaled by ( \frac{1}{n} ). The scaling factor does not affect the location of the minimum.

    [ \text{MSE} = \frac{1}{n} \text{SSE} \quad \Rightarrow \quad \text{Minimize SSE} \iff \text{Minimize MSE} ]

    Conclusion

    Maximizing the log-likelihood of the linear regression model under the Gaussian noise assumption leads to minimizing the MSE loss function. Therefore, the probabilistic interpretation using the Maximum Likelihood Criterion is equivalent to minimizing the MSE loss in linear regression.