NGBoost: When a Number Isn't Enough

The Prediction You’re Not Getting

Every gradient-boosted model you’ve trained has been lying to you — not maliciously, but by omission.

When XGBoost predicts “this batter will get 1.8 hits tonight,” that number is the mean of an implicit distribution the model learned and never showed you. You’re getting the headline. NGBoost gives you the full story.

What NGBoost Actually Outputs

Standard gradient boosting fits a function f(x) → ŷ: one number per input. NGBoost fits f(x) → θ, where θ parameterizes a full probability distribution over y.

For a Normal distribution assumption (the default), that means fitting two outputs: a mean μ and a standard deviation σ. The result for any input: “I predict 1.8 hits, with uncertainty σ = 0.65.” That σ is what changes everything downstream.

The visualization below shows both what the training process looks like — using the Ames housing dataset as a familiar regression example — and what that richer output looks like at prediction time versus a standard point estimate.

Natural Gradients: Why the Name

Standard gradient boosting computes ordinary gradients of a loss with respect to predictions. This works fine for scalar targets, but breaks down when you’re simultaneously fitting μ and σ — because the parameter space of a probability distribution isn’t flat.

Moving μ by 1 unit has a very different effect depending on how large σ is. Ordinary gradients are blind to this. They treat all directions in parameter space as equivalent, which produces unstable, poorly-calibrated updates.

Natural gradients fix this by pre-multiplying the gradient by the inverse Fisher information matrix — a local measure of the distribution’s curvature in parameter space. The result is updates that are parameterization-invariant: the model trains the same way regardless of how you’ve parameterized your distribution.

In practice: more stable training, faster convergence, and better-calibrated output.

Proper Scoring Rules

The other key ingredient is the loss function. NGBoost optimizes a proper scoring rule — usually negative log-likelihood (NLL). A scoring rule is “proper” when the only way to minimize it is to report your true beliefs about the distribution.

This has a concrete implication: NGBoost has no incentive to underestimate uncertainty to look confident. Standard gradient boosting optimizes MSE or MAE — neither rewards honesty about uncertainty, so neither produces calibrated intervals.

With a proper scoring rule, when NGBoost says “80% confidence interval,” that interval actually contains the true value roughly 80% of the time.

Applied: Batter Hit Predictions

In the MLB system, a point prediction of “1.8 hits” doesn’t help you reason about threshold questions. Does this batter reach the 2-hit prop line? Should he start tonight given the opposing pitcher?

NGBoost answers directly: with μ = 1.8 and σ = 0.65, P(hits ≥ 2) ≈ 38%. That probability is a first-class output, not a post-hoc approximation.

More importantly, the σ varies per batter per game. A consistent contact hitter facing a weak starter might have σ = 0.4. A streaky power hitter in an unknown matchup might have σ = 0.9. Standard boosting flattens that distinction. NGBoost surfaces it.

The Practical Tradeoff

NGBoost is slower to train than XGBoost. Fitting two parameters per sample instead of one, with natural gradient computations layered on top, roughly doubles training time in practice.

For the MLB system, training runs overnight on a schedule anyway — so this is a non-issue. For a latency-sensitive or compute-constrained use case, you’d weigh the richer output against the cost.

The output also assumes a parametric family. Normal works well for continuous targets clustered around a mean. For heavily skewed distributions (e.g., predicting home run count where most values are 0), LogNormal or a zero-inflated model fits better. NGBoost supports swapping distributions; you just need to choose.

The choice to use NGBoost is really a question of whether downstream systems can consume a distribution. If everything just takes the mean anyway, you’ve paid the training cost for nothing. If your system can propagate uncertainty — into lineup optimization, wager sizing, or flagging low-confidence predictions for human review — then NGBoost earns its keep.