05. Multiple regression: statistical inference

01. Is the staffing relationship large enough to matter?

A decision needs magnitude and uncertainty.

Suppose a ministry would consider a staffing change worthwhile only if it improved learning by a meaningful amount relative to its cost. A fitted coefficient alone cannot say how accurately the sample measures that amount. Another comparable sample could give a different estimate. Statistical inference uses the sample and stated assumptions to quantify that uncertainty and assess claims about a population quantity.

We continue with the same staffing contrast and outcome units, then build a standard error, an interval and a test in that order of reasoning. An interval helps describe which magnitudes remain compatible with the estimated uncertainty. A test evaluates a specific proposed coefficient under a reference model. The decision also needs a credible causal comparison and a practical benefit threshold; neither is supplied by the software's uncertainty columns.

02. Recall which coefficient the model estimates

Keep its units and adjustment fixed.

The adjusted staffing slope is -1.101 points per additional student per teacher. It describes a linear comparison at fixed English-learner share. A reduction of two in the ratio changes the fitted score by minus two times this coefficient.

Before calculating a test, state the exact target. Testing whether the population slope is zero differs from testing whether it equals minus one point per student per teacher. A policy decision might also concern a two-student change rather than a one-unit slope. Inference becomes easier to interpret when the comparison is specified before the output is examined.

03. State a null before calculating a p-value

The evidence is evaluated under a specific claim.

A null hypothesis is a precise claim about the population. Here it says the staffing slope is zero. The alternative says it differs from zero, either above or below. To measure the distance from the null, subtract zero from the estimate and divide by its standard error. The result is the t statistic. With a slope near minus 1.101 and a standard error near 0.433, the statistic is about minus 2.54.

Imagine repeating the study under the null and the assumptions that justify the test. For this two-sided test, the p-value is the reference probability of a statistic at least 2.54 units away from zero in either direction. We use it to judge how unusual the data would be under that claim. The calculation assumes the null in order to measure that probability.

04. Normal-error exact inference and robust approximations differ

Know which argument supports the reference distribution.

Under a correctly specified linear conditional mean, independent normal errors with constant conditional variance, and nonredundant regressors, conventional regression t and F reference distributions, where F is used for a joint test of several coefficient restrictions, are exact conditional on the regressor values. The t distribution has heavier tails than the standard normal and uses the residual degrees of freedom. Normality here describes the errors, not a requirement that staffing itself be normally distributed.

Heteroskedasticity-robust inference permits error spread to vary with the regressors. Its usual justification is a large-sample approximation, with suitable sampling and finite variances for the quantities being averaged and no small set of observations dominating the fit. It does not automatically allow errors to be related across districts, states or dates. The reference distribution and the covariance formula are distinct choices, and neither fixes a failure of the model's zero conditional-mean requirement.

05. State the reference distribution before choosing a critical value

An exact classical t test uses its residual degrees of freedom.

For the classical exact result, condition on the observed regressors. The errors are independent and normally distributed with mean zero and a common variance. The fit must determine the coefficients uniquely. If the null is true, the difference between the estimate and the proposed value, divided by its conventional standard error, follows a Student t distribution with n minus p degrees of freedom. Here p counts the intercept and all slopes.

A critical value is a cutoff for that reference distribution. For a two-sided five-percent test, place two and a half percent of the reference probability in each tail. Use the positive cutoff as the multiplier for the matching confidence interval. With many degrees of freedom this cutoff approaches 1.96. Robust standard errors use a large-sample justification developed in the next two meetings.

06. Build a confidence interval step by step

Combine the estimate with a stated reference rule.

Start at the estimate, 0.30 in this constructed example. The standard error is 0.12. If the large-sample normal approximation is appropriate, multiply 0.12 by 1.96. The resulting margin is 0.2352. Subtract it from 0.30 to get 0.0648, then add it to get 0.5352.

Think of drawing another comparable sample. Its estimate and its standard error can change, so its interval can change too. A 95 percent confidence procedure aims to produce intervals that cover the same fixed population coefficient about 95 percent of the time. Read the endpoints as magnitudes to discuss under the stated model and uncertainty calculation. If the comparison is biased, changing the interval width does not correct that comparison.

07. One-sided alternatives must express a prior scientific question

The direction of the alternative changes the tail probability.

A two-sided alternative asks whether a coefficient differs from a stated value in either direction. A one-sided alternative asks whether it is greater, or whether it is smaller. For a greater-than alternative, a large positive test statistic is evidence against the null; a large negative statistic is not evidence in the proposed direction.

Specify that direction before observing the estimated sign. The p-value is the appropriate reference-distribution tail probability under the null, not the probability that the null is true. Under a symmetric continuous reference, a correctly directed one-sided p-value is half the two-sided value; a statistic in the opposite direction has a large one-sided p-value. The economic meaning of the tested value and the model's assumptions remain part of the result.

08. Test a substantively meaningful value

A zero null is not the only relevant question.

Suppose a one-point change in score per additional student per teacher is a useful policy benchmark. Now test a slope of minus one. The estimate is much closer to minus one than it is to zero.

Read the restriction at the top of the Stata output. The test uses the distance from minus one and the uncertainty in the fitted slope. An estimate can give strong evidence against zero while remaining compatible with this benchmark. Report the proposed value alongside the result so the reader knows what question the test answers.

. test str=-1

 ( 1)  str = -1

       F(  1,   417) =    0.05
            Prob > F =    0.8151

09. Compute uncertainty for the policy contrast

Scale the SE along with the estimate.

For a two-student reduction, the fitted change is minus two times the staffing slope. Its SE is twice the slope SE, because multiplying a random estimate by a constant multiplies its standard deviation by the constant's absolute value.

The lincom output supplies the contrast, SE and interval together. State that the result is in score points, with the included language composition held fixed. In interaction models, a contrast can combine several coefficients and its variance will also depend on their covariance. Using the model's covariance information avoids shortcuts such as adding standard errors.

. lincom -2*str

------------------------------------------------------------------------------
       score | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
         (1) |   2.202592   .8656943     2.54   0.011     .5009232     3.90426
------------------------------------------------------------------------------

10. A linear contrast uses cross-coefficient covariance

Combine coefficients in the units of the intended fitted comparison.

A linear contrast is a weighted sum of coefficients. Consider a fitted score comparison that reduces students per teacher by two and increases the English-learner share by five percentage points. Multiply the staffing slope by minus two and the language-share slope by five, then add them. This gives the fitted difference in score points.

The estimates come from the same districts, so their sampling variation can move together. Write a and c for the two fixed weights and b one and b two for the two slope estimates. The variance of the weighted sum is a squared times the first variance, plus c squared times the second variance, plus twice a times c times the covariance. The square root of the estimated variance is the contrast's standard error. Stata's lincom uses these pieces from the fitted model. We still need a separate argument to interpret the contrast as the effect of an intervention.

11. Write an uncertainty statement for a policy reader

Include the comparison, units and conditions.

A concise result sentence names the adjusted comparison, gives a feasible policy contrast and reports its uncertainty method. A second sentence explains the design limitation most relevant to causal interpretation. Neither should require the reader to decode a software table.

For example, describe a fitted two-student ratio difference at fixed language composition and give its robust interval in score points. Then explain that staffing may still respond to unmeasured learning needs. This keeps inference and identification connected to the same application while showing clearly which argument supports each claim.

12. DAG practice: uncertainty is not adjustment

Trace the routes before choosing a control.

Select prior education as a control in the training diagram. Which alternative route remains? If the analyst changes conventional SEs to heteroskedasticity-robust SEs, does that close the route or change the training coefficient on an identical specification and sample?

Name the target comparison first. Trace each route through the relevant variables and say whether it is directed, blocked or open under the stated conditioning. Use the diagram controls to test your proposed explanation before opening the worked reasoning.

13. DAG practice: uncertainty is not adjustment: explanation

Explain what conditioning changes in this stated graph.

The route through unmeasured motivation remains open after conditioning on education. The common-cause arrows are properties of the proposed causal story, not of the covariance formula. Switching the SE option does not block them.

On an unchanged model and sample, the OLS coefficient remains the same while the estimated covariance, intervals and tests can change. Robust inference addresses uncertainty under appropriate conditions. A credible causal comparison needs an additional argument about motivation or a design that addresses its role.

14. A joint hypothesis asks whether several restrictions hold together

One coefficient test cannot replace the combined question.

A joint hypothesis places several restrictions on the population coefficients at once. In the district model, we could ask whether language share and district income both have zero conditional-mean coefficients, holding staffing in the equation. Checking two separate p-values does not create a single test with the intended overall rejection probability.

The restricted model imposes both zero coefficients; the unrestricted model includes both variables. Under the classical normal-error assumptions, an F test compares their loss of fit relative to the unrestricted error-variance estimate. Both fits must use the same observations and outcome. Unequal-variance robust joint testing requires an appropriate covariance construction, developed in the variance block.

. test 1.high 1.high#c.str

 ( 1)  1.high = 0
 ( 2)  1.high#c.str = 0

       F(  2,   415) =   98.88
            Prob > F =    0.0000

15. Classical fit-loss F tests require matched models

Restricted and unrestricted fits must answer one nested comparison.

The restricted model imposes the null equalities; the unrestricted model allows the corresponding coefficients to vary. Both must use the same outcome and observations. SSR is the sum of squared residuals. For a constructed example, let restricted SSR be 110, unrestricted SSR be 100, and let two restrictions remove two parameters. With 100 observations and five unrestricted coefficients, divide the SSR increase per restriction, five, by unrestricted SSR per residual degree of freedom, 100 divided by 95. The F statistic is 4.75.

The general formula uses q for the number of restrictions and p subscript U for all unrestricted coefficients, including the intercept. The classical F reference needs the conventional model assumptions. With unequal variance or dependence, use an appropriately estimated covariance and a justified joint test rather than treating a mechanical SSR calculation as universally valid.

16. School staffing: reproduce and explain

Reproduce the evidence, then explain its economic meaning.

Open the school data and the full task below. Start with the two models in the lab. For each one, use a staffing change from twenty-two to twenty students per teacher and report the predicted score difference. State which variables are held fixed.

Add district income to the model with language share. Use the fitted slope to calculate the same two-student contrast with its robust standard error. In your ministry statement, report the magnitude in score points. Then name a difference between districts that could still affect a causal comparison.

17. School staffing: reproduce and explain: explanation

A reproducible result needs both a calculation and an interpretation.

The file has 420 districts. Score is measured in points and str in students per teacher. A reduction of two students per teacher has a fitted contrast of about +4.56 points in the short regression and +2.20 after adjustment for language composition. Both fits use the same districts, so the movement is a model change.

A suitable ministry statement reports these as adjusted or unadjusted associations. It explains that districts may differ in resources or learning needs that also affect teacher recruitment. A credible intervention comparison would need to address that source of staffing selection. The smaller adjusted estimate does not establish that all remaining selection has been removed.

The income-adjusted gain is 0.137551 points with robust SE 0.580319 on 420 districts. It remains an adjusted association.

18. From finite-sample inference to large-sample reasoning

A test begins with a stated population restriction and an uncertainty estimate that matches the sampling model. A confidence interval expresses the precision of the coefficient, while its units and a decision-sized contrast express economic magnitude.

Exact classical reference distributions required strong error assumptions. Next we study what larger samples can justify when those assumptions are relaxed, and why a larger dataset does not repair a systematically misleading comparison.

19. Confidence interval

Try the comparison before opening the explanation.

Further study

An estimate is 0.30 with robust SE 0.12. Use the large-sample 1.96 rule to form a 95% interval and test a zero effect.

First calculate the estimate plus and minus 1.96 standard errors. For a zero null and a two-sided five-percent normal-reference test, check whether zero lies inside that same interval.

20. Confidence interval: explanation

Follow the calculation and interpret its units.

Further study

\(0.30\pm1.96\times0.12\) gives [0.0648,0.5352]. The t ratio is 2.5, so reject zero at approximately 5% under the relevant large-sample assumptions. This says nothing by itself about the economic importance or causal validity of the estimate.

21. A rejection is not an effect-size statement

Statistical significance and practical value answer different questions.

Further study

A statistically significant result passes a chosen testing threshold. A practically important result is large enough, in its application, to matter for a decision. A tiny effect can be estimated very precisely in a large sample; a meaningful effect can remain uncertain in a noisy sample.

For a staffing decision, translate the slope into a feasible ratio change and compare the implied learning difference with costs and relevant alternatives. A point estimate is the single numerical coefficient reported from the sample. Report its interval as well. The policy reader should be able to see both the range of values consistent with the model and the scale of the decision being considered.

22. A confidence interval can contain an important effect

Failing to reject is not proof of no effect.

Further study

If an interval includes zero and also includes effects large enough to matter, the evidence may be too imprecise for the decision. It is inaccurate to report that the study established no relationship. It is also inaccurate to select a large value from the interval and present it as the estimated effect.

Report the point estimate and interval, explain which practical values remain compatible with the model and discuss what data would improve precision. This distinguishes an uninformative result from evidence that any meaningful effect is small. Both conclusions can be useful, but they require different evidence.

23. Economic versus statistical size

Try the comparison before opening the explanation.

Further study

A precisely estimated policy effect is 0.001 outcome units. Is a small p-value enough to recommend the policy?

Use the outcome units and the decision threshold. A small p-value and a worthwhile policy benefit are different claims.

24. Economic versus statistical size: explanation

Follow the calculation and interpret its units.

Further study

No. Report the units, realistic policy contrast, interval, policy cost and relevant outcome scale. Statistical detectability alone neither establishes causal validity nor a worthwhile benefit.

25. More observations help only under the relevant conditions

The information gain depends on what the observations add.

Further study

With independent, suitably informative observations and stable variance, sampling uncertainty often decreases at a square-root rate as sample size grows. Roughly quadrupling the sample can halve an SE in that setting. This is an approximation, not a promise for any enlarged database.

Repeating highly dependent information may add less precision. Increasing observations while leaving a regressor endogenous can make an estimate concentrate around the wrong value. The next large-sample lesson will separate these ideas explicitly. For today, tie any proposed sample expansion to the sampling structure and the comparison it improves.

26. Avoid repeated specification searches for a preferred p-value

The search changes what an isolated test can mean.

Further study

Suppose you try several control sets and report only the one that gives a p-value below five percent. Even if every test is calculated correctly on its own, the reported result was selected from several chances to find an unusually large statistic.

Start with the question and a reason for the main specification. If you explore other reasonable models, report what you changed and how the estimates and intervals moved. When you need a formal claim after many tests, use a testing plan that accounts for that search. This lets the reader distinguish a planned comparison from a result discovered while exploring.

27. Check the reference distribution and scale

An interval is meaningful only for a defined quantity.

Further study

Before interpreting an interval, identify whether it concerns a coefficient, a fitted contrast, an average outcome or an individual prediction. These quantities have different uncertainty calculations. A confidence interval for an average fitted score is not an interval for every district's score.

Also check whether the displayed interval uses a normal approximation or a t reference and whether the covariance option matches the observations. Small rounding differences in endpoints can arise from the reference rule. The important interpretation still depends on the coefficient target, units, sampling conditions and design.

28. Use the evidence to identify the next useful step

Precision, fit and design can require different improvements.

Further study

An imprecise but credible comparison may benefit from more informative observations. A precise but confounded comparison may require better measurement or a different design. A poor functional form may require a model that represents nonlinear relationships more adequately.

These are different diagnoses. Explain why a proposed next step addresses the concern you have actually observed. This habit helps an empirical report remain useful even when the first regression does not settle the policy question. A statistically significant coefficient is not the only possible contribution of an applied analysis.

29. Report magnitude and uncertainty together

Reproduce the evidence, then explain its economic meaning.

Further study

Run lab_04.do. Use the robust staffing coefficient and SE to construct the approximate 95% interval with the 1.96 rule. Then calculate the estimate and SE for a two-student reduction. Explain how a decision about practical benefits differs from a test of a zero coefficient.

Use the fitted two-student gain to assess a null gain of one score point, rather than zero. Calculate the corresponding test ratio and explain its economic meaning. Write and run your own commands, then save the code, execution log and an explanation in your own words. The independent practice file is a worked reference.

30. Report magnitude and uncertainty together: explanation

A reproducible result needs both a calculation and an interpretation.

Further study

The robust staffing estimate is -1.101296, with SE 0.432847. The approximate interval is [-1.9497, -0.2529] score points per additional student per teacher. Multiplying the slope by minus two gives 2.2026 points; its SE is 0.8657, because scaling by a constant scales the SE by its absolute value.

These calculations quantify uncertainty under the model and inference assumptions. Stata's finite-degree-of-freedom interval can differ slightly from the 1.96 approximation. A recommendation also needs a credible causal comparison, the outcome's educational importance, staffing costs and the relevant population. Rejecting a zero slope does not supply those additional pieces of the decision.

The ratio is (2.202592-1)/0.865694=1.389164. The result does not reject a one-point fitted gain using the approximate two-sided 1.96 rule. This is distinct from the zero-slope question and still needs credible identification for a causal claim.