They are essentially the same! In fact, linear regression analysis works well, even with non-normal errors. But, the problem is with p-values for hypothesis testing.
After running a linear regression, what researchers would usually like to know is—is the coefficient different from zero? The t-statistics and its corresponding p-value answers the question if the estimated coefficient is statistically significantly different from zero. Now we can see differences. The distribution of estimated coefficients follows a normal distribution in Case 1, but not in Case 2. That means that in Case 2 we cannot apply hypothesis testing, which is based on a normal distribution or related distributions, such as a t-distribution.
When errors are not normally distributed, estimations are not normally distributed and we can no longer use p-values to decide if the coefficient is different from zero. In short, if the normality assumption of the errors is not met, we cannot draw a valid conclusion based on statistical inference in linear regression analysis.
Learn more. Why Normality assumption in linear regression Ask Question. Asked 2 years, 8 months ago. Active 2 years, 8 months ago. Viewed 4k times. Improve this question. Master Shi Master Shi 5 5 silver badges 10 10 bronze badges. It just happens to be the case that when the error is normal, the model coefficients exactly follow a normal distribution and an exact F-test can be used to test hypotheses about them.
The whole world would be probability. Show 6 more comments. Active Oldest Votes. In fact, here's an example of a "uniform error" model fitted to data by hand: It's easy to identify by sliding a straightedge toward the data that the four marked points are the only candidates for being in the active set; three of them will actually form the active set and a little checking soon identifies which three lead to the narrowest band that encompassess all the data.
Many other choices of model are possible and quite a few have been used in practice. Improve this answer. Would you mind adding some links which give more details as to how these variations are used in practice?
While it can be done readily enough using code, I literally opened the plot in MS Paint and identified the three points in the active set joining two of which gave the slope -- and then moved the line half-way toward the third point by halving the vertical distance in pixels and moving the line up that many pixels -- the point being to demonstrate quite how simple this could be.
A child could be taught to do it. Add a comment. See Peter Huber's seminal book Robust Statistics for more information. Martin L Martin L 1 1 silver badge 5 5 bronze badges. It is also not a very restrictive assumption, as many other types of data will behaive "kind-of-normally" Anyway, as mentioned in a previous answer, there are possibilities to define regression models for other distributions. David David 2, 1 1 gold badge 3 3 silver badges 15 15 bronze badges.
However, why is the normal distribution chosen so often? See also here Importance of normal distribution where Galton's bean machines show the principle intuitively. Sextus Empiricus Sextus Empiricus Neil G Neil G The link function does not have to do with generalizing to different distributional assumptions, but with generalizing the linear part that describes the mean of the distribution.
But, that is not the link function. The link function in GLM relates to the linearizing transformation. But this is not a necessity. Violations of linearity or additivity are extremely serious: if you fit a linear model to data which are nonlinearly or nonadditively related, your predictions are likely to be seriously in error, especially when you extrapolate beyond the range of the sample data.
How to diagnose : nonlinearity is usually most evident in a plot of observed versus predicted values or a plot of residuals versus predicted values , which are a part of standard regression output. The points should be symmetrically distributed around a diagonal line in the former plot or around horizontal line in the latter plot, with a roughly constant variance.
The residual-versus-predicted-plot is better than the observed-versus-predicted plot for this purpose, because it eliminates the visual distraction of a sloping pattern. Look carefully for evidence of a "bowed" pattern, indicating that the model makes systematic errors whenever it is making unusually large or small predictions. In multiple regression models, nonlinearity or nonadditivity may also be revealed by systematic patterns in plots of the residuals versus individual independent variables.
For example, if the data are strictly positive, the log transformation is an option. The logarithm base does not matter--all log functions are same up to linear scaling--although the natural log is usually preferred because small changes in the natural log are equivalent to percentage changes. See these notes for more details. If a log transformation is applied to the dependent variable only, this is equivalent to assuming that it grows or decays exponentially as a function of the independent variables.
If a log transformation is applied to both the dependent variable and the independent variables, this is equivalent to assuming that the effects of the independent variables are multiplicative rather than additive in their original units. This means that, on the margin, a small percentage change in one of the independent variables induces a proportional percentage change in the expected value of the dependent variable, other things being equal.
Models of this kind are commonly used in modeling price-demand relationships, as illustrated on the beer sales example on this web site. Another possibility to consider is adding another regressor that is a nonlinear function of one of the other variables.
Higher-order terms of this kind cubic, etc. This sort of "polynomial curve fitting" can be a nice way to draw a smooth curve through a wavy pattern of points in fact, it is a trend-line option on scatterplots on Excel , but it is usually a terrible way to extrapolate outside the range of the sample data.
Finally, it may be that you have overlooked some entirely different independent variable that explains or corrects for the nonlinear pattern or interactions among variables that you are seeing in your residual plots. In that case the shape of the pattern, together with economic or physical reasoning, may suggest some likely suspects. For example, if the strength of the linear relationship between Y and X 1 depends on the level of some other variable X 2 , this could perhaps be addressed by creating a new independent variable that is the product of X 1 and X 2.
In the case of time series data, if the trend in Y is believed to have changed at a particular point in time, then the addition of a piecewise linear trend variable one whose string of values looks like 0, 0, …, 0, 1, 2, 3, … could be used to fit the kink in the data.
Such a variable can be considered as the product of a trend variable and a dummy variable. Again, though, you need to beware of overfitting the sample data by throwing in artificially constructed variables that are poorly motivated. At the end of the day you need to be able to interpret the model and explain or sell it to others.
Violations of independence are potentially very serious in time series regression models: serial correlation in the errors i. Independence can also be violated in non-time-series models if errors tend to always have the same sign under particular conditions, i. How to diagnose: The best test for serial correlation is to look at a residual time series plot residuals vs. If your software does not provide these by default for time series data, you should figure out where in the menu or code to find them.
Pay especially close attention to significant correlations at the first couple of lags and in the vicinity of the seasonal period, because these are probably not due to mere chance and are also fixable. The Durbin-Watson statistic provides a test for significant residual autocorrelation at lag 1: the DW stat is approximately equal to 2 1-a where a is the lag-1 residual autocorrelation, so ideally it should be close to 2. How to fix: Minor cases of positive serial correlation say, lag-1 residual autocorrelation in the range 0.
An AR 1 term adds a lag of the dependent variable to the forecasting equation, whereas an MA 1 term adds a lag of the forecast error. If there is significant correlation at lag 2, then a 2nd-order lag may be appropriate. If there is significant negative correlation in the residuals lag-1 autocorrelation more negative than Differencing tends to drive autocorrelations in the negative direction, and too much differencing may lead to artificial patterns of negative correlation that lagged variables cannot correct for.
If there is significant correlation at the seasonal period e. The dummy-variable approach enables additive seasonal adjustment to be performed as part of the regression model: a different additive constant can be estimated for each season of the year. If the dependent variable has been logged, the seasonal adjustment is multiplicative.
Something else to watch out for: it is possible that although your dependent variable is already seasonally adjusted, some of your independent variables may not be, causing their seasonal patterns to leak into the forecasts. Major cases of serial correlation a Durbin-Watson statistic well below 1.
You may wish to reconsider the transformations if any that have been applied to the dependent and independent variables. To test for non-time-series violations of independence , you can look at plots of the residuals versus independent variables or plots of residuals versus row number in situations where the rows have been sorted or grouped in some way that depends only on the values of the independent variables.
The residuals should be randomly and symmetrically distributed around zero under all conditions, and in particular there should be no correlation between consecutive errors no matter how the rows are sorted , as long as it is on some criterion that does not involve the dependent variable.
0コメント