in category: Machine Learning_tricks4better performance
Source: from the paper by Prof. Domingos :
Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78-87. (PDF)
*****source for the content below: from machine-learning on Coursera by Dr. Andrew Ng
See this post for how regularization can help prevent over-fitting. But how does it affect the bias and variances of a learning algorithm? This post will go deeper into the issue of bias and variances and talk about how it interacts with and is affected by the regularization of your learning algorithm.
If lambda is small then we’re not using much regularization and we run a larger risk of over fitting whereas if lambda is large that is if we were on the right part of this horizontal axis then, with a large value of lambda, we run the higher risk of having a biased problem, so if you plot J train and J cv, what you find is that, for small values of lambda, you can fit the trading set relatively way cuz you’re not regularizing. So, for small values of lambda, the regularization term basically goes away, and you’re just minimizing pretty much just gray arrows. So when lambda is small, you end up with a small value for Jtrain, whereas if lambda is large, then you have a high bias problem, and you might not feel your training that well, so you end up the value up there. So Jtrain of theta will tend to increase when lambda increases, because a large value of lambda corresponds to high bias where you might not even fit your trainings that well, whereas a small value of lambda corresponds to, if you can really fit a very high degree polynomial to your data, let’s say. After the cost validation error we end up with a figure like this.
When I’m trying to pick the regularization parameter lambda for learning algorithm, often I find that plotting a figure like this one shown below helps me understand better what’s going on and helps me verify that I am indeed picking a good value for the regularization parameter monitor.
