Many classification and regression problems are solved in practice by regularized empirical risk minimizers (RERM). The risk is measured via a loss function. The quadratic loss function is the most popular function for regression. It has been extensively studied (cf. [23, 31] among others). Still many other loss functions are popular among practitioners and are indeed extremely useful in specific situations
ALQUIER, P., COTTET, V. et LECUE, G. (2019). Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions. Annals of Statistics, 47(4), pp. 2117-2144.