Clustering the market, and admitting it is only a map
RMT-denoised correlation clustering with the Mantegna ultrametric. The clusters re-discover GICS sectors, but cluster-aware allocation does not beat 1/N out of sample after costs.
00Overview
Correlation clustering promises a diversification skeleton: group the stocks that move together. This project builds that skeleton properly, denoising the correlation matrix first, then asks the honest follow-up question: does knowing the clusters actually help you allocate?
01Method
- Marchenko-Pastur edge filtering to denoise the correlation matrix before clustering.
- Mantegna ultrametric distance, d = sqrt(2 (1 - rho)), which satisfies the metric axioms (property-tested).
- Hierarchical average-linkage clustering, k-means and embedding methods.
- k chosen by the gap statistic against a phase-randomised null (Tibshirani 1-SE rule).
02The honest test
As a diagnostic the clusters work: they recover GICS sectors with an adjusted Rand index of 0.4 to 0.7. As a trading edge they do not: the out-of-sample Sharpe difference between cluster-aware allocation and 1/N is not statistically significant, and the deflated Sharpe is ~0 once the number of trials is honestly counted. The clusters are a map, not an edge.