Both the training MSE and RSS decrease monotonically as more features are considered

[JiaqiLiu]

modeling-examples/linear_regression/l0_regression.html

Line 13183 in 1abb870

\end{equation}<p>This model, by means of constraint 2, implicitly considers all ${{d-1} \choose s}$ feature subsets at once. However, we also need to find the value for $s$ that maximizes the performance of the regression on unseen observations. Notice that the training RSS decreases monotonically as more features are considered (which translates to relaxing the MIQP), so it is not advisable to use it as the performance metric. Instead, we will estimate the Mean Squared Error (MSE) via cross-validation. This metric is defined as $\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}{(y_i-\hat{y}_i)^2}$, where $y_i$ and $\hat{y}_i$ are the observed and predicted values for the ith observation, respectively. Then, we will fine-tune $s$ using grid search, provided that the set of possible values is quite small.</p>

RSS vs MSE

This paragraph mentioned that it is not advisable to use RSS as the performance metric, but MSE via cross-validation.

I think the highlight on MSE over RSS is misleading. Note that, given estimate $\hat\beta$,

$$ \mathrm{RSS} = (y-X\hat\beta)^T(y-X\hat\beta) = \sum (y_i - \hat{y}_i)^2 = n \cdot \mathrm{MSE} $$

So, both the training MSE and RSS decrease monotonically as more features are considered, not only RSS.

Cross-validation

The cross-validation part should be the correct. That is, we use grid search to find best $s$.