Model Performance Metrics

Model Performance Metrics 


1. MAE — Mean Absolute Error

Formula:

MAE=1







n∑i=1n∣yi−y^i∣\text{MAE} = \frac{1}{n}\sum_{i=1}^{n} |y_i - \hat{y}_i|

Meaning:

  • It measures the average magnitude of errors in a set of predictions, without considering their direction (positive or negative).

  • Lower is better.

  • It’s in the same units as the target variable.

✅ Intuitive: “On average, my predictions are off by X units.”
⚠️ Downside: It treats all errors equally — large and small errors have the same weight.

2. MSE — Mean Squared Error

Formula:

MSE=1n∑i=1n(yi−y^i)2\text{MSE} = \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2

Meaning:

  • It measures the average squared difference between predicted and actual values.

  • Larger errors have exponentially more weight because of the squaring.

✅ Highlights large errors — useful when large deviations are especially undesirable.
⚠️ Less interpretable since it’s in squared units of the target.

3. RMSE — Root Mean Squared Error

Formula:

RMSE=MSE=1n∑i=1n(yi−y^i)2\text{RMSE} = \sqrt{\text{MSE}} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}

Meaning:

  • It’s the square root of MSE, converting it back to the same units as the target variable.

  • Still penalizes large errors more heavily than MAE.

✅ Common and intuitive metric — “average size of the error.”
⚠️ Sensitive to outliers.

4. R² Score — Coefficient of Determination

Formula:

R2=1−∑i=1n(yi−y^i)2∑i=1n(yi−yˉ)2R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}

where yˉ\bar{y} is the mean of the actual values.

Meaning:

  • Measures how well the model explains the variability of the target variable.

  • Range: −∞-\infty to 11

    • R2=1R^2 = 1: perfect prediction

    • R2=0R^2 = 0: model predicts no better than the mean

    • R2<0R^2 < 0: model is worse than just predicting the mean

✅ Explains “goodness of fit”
⚠️ Can be misleading for non-linear models or small sample sizes.

Quick Summary Table

MetricFormulaPenalizes Large ErrorsSame Units as TargetInterpretation
MAE( \frac{1}{n}\sumy_i - \hat{y}_i)❌ No
MSE1n∑(yi−y^i)2\frac{1}{n}\sum (y_i - \hat{y}_i)^2✅ Yes (squared)❌ NoAvg. squared error
RMSE1n∑(yi−y^i)2\sqrt{\frac{1}{n}\sum (y_i - \hat{y}_i)^2}✅ Yes (squared root)✅ YesAvg. error magnitude
R²1−SSresSStot1 - \frac{\text{SS}_\text{res}}{\text{SS}_\text{tot}}—UnitlessVariance explained


What should be standard values of these to call out model is good

That’s a great and practical question — and the answer depends on your data’s scale and context, but here’s how you can interpret these metrics in general.

⚙️ 1. MAE (Mean Absolute Error)

  • No fixed “good” value, because it depends on your target variable’s scale.

  • As a rule of thumb:

    • MAE should be as low as possible.

    • A good model usually has MAE that’s <10% of the average value of your target variable.

      MAE ratio=MAEyˉ<0.1\text{MAE ratio} = \frac{\text{MAE}}{\bar{y}} < 0.1

  • Example: If your target values average around 100, an MAE of ≤10 is generally quite good.

⚙️ 2. MSE (Mean Squared Error)

  • MSE’s magnitude depends on the square of your target variable, so it’s hard to interpret directly.

  • Use it mainly for comparison between models — the smaller, the better.

  • Often replaced with RMSE, which is easier to interpret.

⚙️ 3. RMSE (Root Mean Squared Error)

  • Interpreted in the same units as your target.

  • A good RMSE is typically 10–20% of the range or mean of the target variable.

  • Lower RMSE than MAE suggests your model is handling outliers well.

✅ Good model:

RMSEyˉ<0.2\frac{\text{RMSE}}{\bar{y}} < 0.2

⚠️ Watch out: If RMSE is much larger than MAE, it means you have outliers or large errors that need attention.

⚙️ 4. R² (Coefficient of Determination)

This one has more universal interpretation:

R² ValueInterpretation
1.0Perfect predictions
≥ 0.9Excellent — model explains most variability
0.75–0.9Good — strong predictive power
0.5–0.75Moderate — acceptable depending on context
< 0.5Weak — model doesn’t explain much variance
< 0Model worse than predicting the mean

💡 Rule of thumb:

  • For engineering / physical systems, expect R² ≥ 0.9.

  • For social sciences or human behavior data, R² ≥ 0.6 can already be quite good due to inherent noise.

🧭 Quick Summary

MetricIdeal DirectionGood Rule of Thumb
MAE↓ Lower better<10% of mean(target)
MSE↓ Lower betterCompare between models
RMSE↓ Lower better<20% of mean(target)
R²↑ Higher better>0.75 good, >0.9 excellent

Comments

Post a Comment

Popular posts from this blog

YouTube Tutorials & Blogs for 6-Month Speaker Development

  Complete Resource Guide with Links: YouTube Tutorials & Blogs for 6-Month Speaker Development This comprehensive guide provides direct YouTube channel URLs, specific video tutorials, blog article links, and online resources for each section of your 6-month speaker development plan. Month 1: Foundation & Self-Assessment Voice & Breath Control YouTube Channels (WITH DIRECT LINKS) San Diego Voice and Accent - Julie Cunningham Channel URL: https://www.youtube.com/channel/UCz9cIPlVfRkHl94X9e4l8jA Website: https://sandiegovoiceandaccent.com Course Platform: https://courses.sandiegovoiceandaccent.com Key Videos to Watch: "The Basics of Voice Production Part 3: Respiration Exercises" "Six Steps to a Stronger Voice" course American accent fundamentals series Professional voice training and American English pronunciation Backstage - Voice training for actors Search on YouTube: "Backstage breath control exercises for actors...
Read more

AI Engineering Stack

Image
 AI Engineering Stack Our AI practice must be structured around the three layers of the AI application stack to enable scalable and reliable application development Three layers of the AI application stack to enable scalable and reliable application development Application Development AI inference Prompt engineering Context Construction Evaluation Model Development Inference optimization Dataset engineering Modelling & Training Evaluation Infrastructure Compute management Data management Serving Monitoring Below is a detailed elaboration of each layer in the diagram you provided, expanding on the corresponding bullet points on the right side. This can be used directly in a technical presentation , architecture document, or strategy deck for an AI/ML or Agentic Enterprise practice. AI Practice Architecture: Layer-by-Layer Explanation 1. Application Development Layer This layer focuses on building end-user AI applications and agentic systems by orchestrating models, context, and...
Read more

GenAI Interview Question

 GenAI Interview Question What are hallucinations in LLMs, why do they occur, and how do you detect and mitigate them in production systems? In a high-throughput production system, when would you choose a list over a dictionary (or vice versa), and why? In a performance-critical Python service, how do best- and worst-case list operations impact latency? Which similarity measures are used to compare embedding vectors, and how do you choose the right one for production RAG systems? How are embedding vectors stored and indexed inside a vector database, and what trade-offs affect speed, memory, and recall? What is re-ranking in a RAG pipeline, when should it be introduced, and what trade-offs does it create? How would you design a secure enterprise chatbot with role / pay-grade based access control? How is vector retrieval fundamentally different from SQL queries? What is LangGraph, how does it differ from LangChain, and how does agentic RAG change system design? In a production RAG sy...
Read more