I am trying to understand the idea of Loss functions For Regression Task perfectly.
I have read many textbooks and articles, and I came up with questions related to this subject.
Several different uses of loss functions can be distinguished.
a) In prediction problems: a loss function depending on predicted and observed value defines the quality of a prediction.
b) In estimation problems: a loss function depending on the true parameter and the estimated value defines the quality of estimation.
c) Many estimators (such as least squares or M-estimators) are defined as optimizers of certain loss functions which then depend on the data and the estimated value.
Now, since my focus is on Loss Functions For Regression Task
My questions are as follows.
1- Should I write the loss function formula as a function of the parameter or of the variables (L($\theta-\hat\theta))$ OR (L($y-\hat\y))$?
2- Should I consider the Loss function formula for one point or Not (with sums or not)?
My thought is to introduce Loss function first and then to use the standard notation for all Loss functions (least square, absolute value and Huber Loss, Quntile Loss and so on).