How Composite Indices Can Be Manipulated — and How to Build Them Right
Introduction
Composite indices like the Human Development Index (HDI) or India’s SDG Index by NITI Aayog are powerful tools. They summarize complex realities — like health, education, financial inclusion — into a single, digestible score. Policymakers, media, and the public often take them at face value to judge progress and make comparisons across states or countries.
But beneath the glossy rankings lies a fundamental risk: composite indices can be easily manipulated. By selectively choosing indicators, tweaking targets, or structuring weights, institutions can paint a rosier (or gloomier) picture to serve political or strategic narratives.
This article examines the problems and risks of such manipulation and lays out a set of best-practice safeguards — with real-world examples.
The Problem: Indices Can Be Cooked
1. Discretion in Indicator Selection
Every index starts with a question: What should we measure? But when agencies have wide leeway to choose or drop indicators, they can skew the results.
- Example: NITI Aayog’s SDG Index includes “% of households with bank accounts” under Goal 8 (Decent Work and Economic Growth). While this may reflect financial inclusion, it’s also highly correlated with per capita income. Adding such indicators inflates scores for wealthier states without adding much new information.
2. Unequal or Implicit Weighting
Even if all indicators are “equally weighted,” stacking some categories with more indicators gives them disproportionate influence.
- Example: If Goal A has 10 indicators and Goal B only 3, then Goal A effectively dominates the final score — even if both are supposed to be equally important.
3. Gaming Targets and Scales
Scores are often normalized between a minimum and maximum value (or a 2030 target). Agencies can set easier targets, raising state scores artificially.
- Example: If you set a modest 2030 target for electrification that most states have already achieved, it becomes a free boost in the index.
4. Opaque Methodologies
When the indicator-selection process and scoring formulae aren’t publicly disclosed or frequently change without explanation, it opens the door to undetected manipulation.
Why This Matters
Manipulated indices can:
- Mislead the public and media;
- Reward poor performance or penalize real progress;
- Undermine trust in public data;
- Allow central authorities to favor certain states or policies.
As the saying goes: “What gets measured, gets managed” — so if the measurements are flawed, the management will be too.
Homegrown Metrics or Strategic Tailoring? India’s Divergence from Global Indices
In countries like India, the push for customized indices is often framed as a rejection of “Western-biased” methodologies. Policymakers frequently argue that global frameworks fail to capture India’s unique developmental context, prompting a preference for locally tailored alternatives. However, this shift also raises concerns about methodological cherry-picking. For example, in the SDG India Index, NITI Aayog diverged from several UN-recommended indicators. Instead of using standard global metrics like “proportion of seats held by women in national parliaments,” it included domestic metrics such as female police personnel or women’s participation in local bodies — often with more favorable numbers. Similarly, the National Multidimensional Poverty Index (MPI) uses twelve indicators (instead of the UN’s ten), introducing criteria like landholding and bank account access that tend to downplay rural deprivation. The Atal Innovation Mission’s index also deviates from the Global Innovation Index by heavily emphasizing incubators and startup counts — metrics that favor urban states — while downplaying patents or R&D spending. While such adaptations may reflect local priorities, they also give policymakers room to select indicators that paint a more optimistic picture, often at the expense of global comparability and empirical rigor.
The Solution: Building Indices with Integrity
To prevent gaming, index builders should adopt scientific, transparent, and reproducible methods. Here’s how.
1. Pre-Registration of Methodology
Just like in clinical trials, the rules for building the index — indicator list, data sources, weightings, normalization methods — should be fixed and published before data collection.
- Example: The UNDP’s HDI has maintained a consistent formula since 2010. Any changes are subjected to multi-year expert reviews.
2. MECE Indicator Design
Choose indicators that are Mutually Exclusive and Collectively Exhaustive (MECE). This avoids double-counting and ensures full coverage of the concept being measured.
- For example, avoid including both “GDP per capita” and “bank account ownership” unless it’s proven they reflect distinct development aspects.
3. Causal Relevance, Not Just Correlation
Every indicator should have proven causal relevance to the outcome the index claims to measure. Including indicators just because they correlate with a positive trend opens the door to manipulation.
- Use basic causal techniques like:
- Granger causality tests
- Instrumental variables
- Panel regressions with controls
4. Statistical Checks: PCA or Factor Analysis
If you have dozens of indicators, use Principal Component Analysis (PCA) or Factor Analysis to:
- Reduce dimensionality
- Identify redundancy
- Derive optimal weights based on variance explained
- Example: The World Bank’s Worldwide Governance Indicators use factor models to aggregate related metrics into broader governance pillars.
5. Robustness and Sensitivity Analysis
Publish tests showing how rankings change when:
- Indicators are added or removed
- Weights are varied
- Alternative normalizations are used
If a state’s rank collapses just by dropping one metric, the index is not robust.
6. Open Data and Reproducibility
Publish the raw data, the code used to calculate scores, and detailed documentation. Allow independent auditors and researchers to reproduce results.
- Example: The OECD’s Better Life Index lets users adjust indicator weights live on their website, showing how rankings change transparently.
Conclusion
Composite indices are not inherently flawed — but they are easily weaponized unless built with rigor and transparency. In a data-driven world, trust in metrics is paramount.
If institutions like NITI Aayog want their indices to carry real weight — and not just be bureaucratic PR tools — they must commit to methodological transparency, causal integrity, and statistical soundness.
Otherwise, we risk mistaking data-driven illusions for meaningful progress.
