OAI Labs — classification-model evaluation | oai-labs.com

From cells to metrics

OAI Labs — classification-model evaluation | oai-labs.com

OAI Labs is a new independent evaluation guide for developers and data analysts. It gives worked formulas for precision, recall and F1, plus a repeatable model-comparison procedure. It is not an AI laboratory or software vendor.

oai-labs.comA single accuracy figure can look strong while recall on the class you care about is weak.Macro, micro and weighted averages answer different questions about multiclass performance.

One example, every metric

Worked example: cells, derived counts and metrics over 20 evaluated cases
QuantityFormulaValue
True positivescell count8
False positivescell count2
False negativescell count4
True negativescell count8
Actual positivesTP + FN12
Predicted positivesTP + FP10
PrecisionTP / (TP + FP)80%
RecallTP / (TP + FN)66.67%
Worked example: cells, derived counts and metrics over 20 evaluated cases

Figures from the worked example

Each figure is derived from the same 20 cases, so the numbers reconcile across panels.

True positive (TP)False positive (FP)False negative (FN)True negative (TN)
The key terms of this guide, drawn to one scale

How do I calculate precision, recall and F1 from a confusion matrix?

Count the four cells, then apply three formulas in order: precision, recall, and their harmonic mean.

A confusion matrix for a binary classifier has four cells. True positives (TP) are cases that were positive and were predicted positive. False positives (FP) are cases that were negative but were predicted positive. False negatives (FN) are cases that were positive but were predicted negative. True negatives (TN) are cases that were negative and were predicted negative. Every evaluated case lands in exactly one cell, so TP + FP + FN + TN equals the total number of cases.

Precision is TP divided by TP + FP. It measures how often a positive prediction is correct. Recall is TP divided by TP + FN. It measures how much of the actual positive class the model captured. F1 is the harmonic mean of the two: 2PR divided by P + R. Accuracy is TP + TN divided by the total number of cases, and it answers a different question from precision and recall because it counts correct negatives as well.

The order matters in practice. Fill the cells first, check that they sum to the evaluation set size, then compute precision and recall before combining them. If you compute F1 first, you lose the ability to see whether a weak score came from too many false positives or too many false negatives, and those two failures usually call for different fixes.

  • Fill TP, FP, FN and TN from the per-case predictions.
  • Check that the four cells sum to the number of evaluated cases.
  • Compute precision = TP / (TP + FP).
  • Compute recall = TP / (TP + FN).
  • Compute F1 = 2PR / (P + R).
  • Report accuracy separately and state the class balance.

Procedure, step by step

The worked example: TP=8, FP=2, FN=4, TN=8

With these four counts, precision is 80%, recall is 66.67%, F1 is about 72.73% and accuracy is 80% over 20 cases.

Start with the cells: TP=8, FP=2, FN=4, TN=8. The total is 8 + 2 + 4 + 8 = 20 evaluated cases. The number of actual positives is TP + FN = 8 + 4 = 12. The number of predicted positives is TP + FP = 8 + 2 = 10. Those two derived counts are the denominators for recall and precision respectively, and it helps to write them down before dividing.

Precision is 8 / (8 + 2) = 8 / 10 = 0.80, or 80%. Recall is 8 / (8 + 4) = 8 / 12 ≈ 0.6667, or 66.67%. F1 is 2 × 0.80 × 0.6667 / (0.80 + 0.6667), which is approximately 0.7273, or 72.73%. Accuracy is (8 + 8) / 20 = 16 / 20 = 0.80, or 80%.

Read the four numbers together. Precision of 80% means two in ten positive predictions were wrong. Recall of 66.67% means four of the twelve actual positives were missed. Accuracy of 80% matches precision here only by coincidence of these counts; in general the two diverge as class balance changes, which is why accuracy alone is a weak summary when positives are rare.

  • Precision: 8 / 10 = 80%.
  • Recall: 8 / 12 = 66.67%.
  • F1: about 72.73%.
  • Accuracy: 16 / 20 = 80%.

Why does F1 sit below the arithmetic mean of precision and recall?

The harmonic mean lies between precision and recall and is pulled toward the smaller value, so an uneven pair produces an F1 below their arithmetic midpoint.

The harmonic mean of 0.80 and 0.6667 is about 0.7273, which is below the arithmetic average of 0.7333. That is the intended behaviour: F1 penalises imbalance between precision and recall. A model that reaches high precision by predicting positive rarely will show weak recall, and F1 will land closer to the weak number than to the strong one.

This property makes F1 useful as a single ranking number when both error types matter roughly equally. It becomes misleading when they do not. If missing a positive case is far costlier than a false alarm, recall deserves more weight than F1 gives it, and you should report recall directly or use a beta-weighted variant that shifts the balance.

F1 also says nothing about true negatives. In a heavily imbalanced problem, a model can post a respectable F1 while misclassifying a large share of the negative class. Always pair F1 with the confusion-matrix cells or with a class-level breakdown so the reader can see where the errors actually are.

Macro, micro and weighted averaging in multiclass evaluation

The three averaging methods answer different questions: per-class equality, per-case accuracy, and support-weighted performance.

Macro averaging computes an unweighted mean of the metric for each label. Every class counts the same regardless of how many examples it has, so a rare class that performs badly drags the macro score down as much as a common class would. Macro scores are the right summary when you care about performance on every label, including the small ones.

Weighted averaging weights each label metric by its support, meaning the number of true instances of that label in the evaluation set. The result tracks overall per-case performance more closely and is dominated by the large classes. It is useful when the evaluation set's class distribution matches the deployment distribution and you want a summary that reflects that mix.

Micro averaging pools the underlying counts across labels before computing the metric, rather than averaging per-label scores. In a single-label multiclass problem, micro F1 equals accuracy, which makes it a per-case measure rather than a per-class one. Choosing among the three is a statement about what you want the number to represent, so state the choice alongside the score.

  • Macro: unweighted mean of per-label metrics — every class counts equally.
  • Weighted: each label metric scaled by its support — large classes dominate.
  • Micro: pool counts across labels first — in single-label multiclass, micro F1 equals accuracy.

Questions readers actually ask

Can F1 be high while accuracy is low?
Yes. F1 ignores true negatives, so on an imbalanced set a model can score well on F1 for the positive class while misclassifying many negatives, which pulls accuracy down. Report both and show the cells.
What happens when precision is undefined?
Precision is undefined when there are no positive predictions, because TP + FP is zero. Scikit-learn documents a zero_division parameter for controlling behaviour when a metric denominator is zero; fix the policy before comparing runs.
Which averaging should I use for a multiclass report?
Use macro when every label matters equally, weighted when the evaluation class mix reflects deployment, and micro when you want a per-case measure. In single-label multiclass, micro F1 equals accuracy.
Is 80% accuracy in the worked example good?
It cannot be judged alone. The same 20 cases give recall of 66.67%, meaning four of twelve actual positives were missed. Whether that is acceptable depends on the cost of a false negative in the task.

Sources and further reading

References are limited to standard metric definitions and the documented zero-division parameter; no vendor claims are made.