How Good Are Crystal Generative Models?

Continuous Metrics and Substitution-Based Novelty Analysis

Masahiro Negishi · Future of Materials Design — Singapore, 2026-07-01

QR code linking to these slides
Slides

Inverse design using generative models

Role of generative models in inverse design of materials Pipeline from human-selected candidates to computational screening to experimental synthesis. An animation-ready overlay replaces the first stage with generative models, optionally conditioned on target property, with feedback from screening and synthesis. Human-selected candidates Computational screening Experimental synthesis Generative models Target property
  • Sample candidates from larger chemical space
  • Optionally condition generation on target properties
  • Update the model from screening / synthesis feedback

Generative models 101

Three generative modeling regimes: unconditional, conditional, and reinforcement learning fine-tuning.
  • Once trained, the model generates numerous candidates efficiently.

Rigorous evaluation is necessary

  • Which model is most suitable for a given application?
  • Are we heading in the right direction in model development?
Taxonomy of crystal generative models.
[1]
[1] Li et al., "Materials Generation Survey," arXiv 2025

Two types of evaluation metrics

  • Application- / lab-specific requirements
    • Do candidates satisfy the target property?
    • Are candidates synthesizable with your lab's equipment?
  • Universal prerequisites ← focus of this talk
    • Do models generate diverse, novel, and stable candidates?

Outline

  1. Introduction
  2. Improving the most widely-used metric, "SUN"
  3. Measuring the search space of generative models
  4. Conclusion

Three prerequisites

SUN is binary

Novelty Uniqueness Stability x in A1-xBx d d AB3 AB A Ehull : generated samples : training samples : reference samples ΔEf [energy/atom] A2B1 B U(x) = 0 or 1 × N(x) = 0 or 1 × S(x) = 0 or 1 = SUN(x) = 0 or 1 cU(x) ∈ [0, 1] cN(x) ∈ [0, 1] cS(x) ∈ [0, 1] cSUN(x) ∈ [0, 1]
  • Highly dependent on heuristic thresholds
  • Fails to distinguish marginal passes from strong passes

Making U and N continuous

Uniqueness

d

$\mathrm{U}(x) \in \{0,1\}$:1 if $x$ does not match any preceding generated sample via $\texttt{StructureMatcher}$

$\mathrm{cU}(x) \in [0,1]$:average $d_c$ to the other generated samples

Novelty

d

$\mathrm{N}(x) \in \{0,1\}$:1 if $x$ does not match any training sample via $\texttt{StructureMatcher}$

$\mathrm{cN}(x) \in [0,1]$:$d_c$ to the nearest training sample

$d_c$ = linear sum of the Average Minimum Distance (structural) and the Element Mover's Distance (compositional). [2,3]

[2] Widdowson et al., "Average Minimum Distances," MATCH Commun. Math. Comput. Chem. 2022
[3] Hargreaves et al., "Element Mover's Distance," Chem. Mater. 2020

cU and cN better capture sample distributions

Binary uniqueness and novelty compared with continuous cU and cN distributions.
  • CDVAE has few exact $\texttt{StructureMatcher}$ duplicates, but is highly concentrated in continuous metric space.
  • Best models match MP20-test uniqueness while exceeding its novelty.
  • cU and cN also have theoretical advantages (robust to atomic-coordinate perturbations, invariant to sample permutation, etc.)

Making S continuous

x in A1-xBx AB3 AB A Ehull ΔEf [energy/atom] A2B1 B
Continuous stability shaping functions as a smooth function of energy above hull.
  • Binary stability fails to capture the gradation of stability.
  • An overly strict binary threshold may filter out all but the most conservative samples.

c.f. ~20% of experimentally synthesized MP crystals satisfy $E_\mathrm{hull} \ge 0.1$ [eV/atom].

cS offers a smoother score distribution

Binary stability and continuous cS score distributions.
  • With cS, fewer samples are merely removed.

A single unified metric: cSUN

Binary SUN and continuous cSUN metric comparison.
Sun and cSUN scores of MatterGen samples
$\mathrm{SUN}(x) = \mathrm{S}(x)\,\mathrm{U}(x)\,\mathrm{N}(x) \in \{0,1\}$
$\mathrm{cSUN}(x) = \mathrm{cS}(x)^{w_S}\,\mathrm{cU}(x)^{w_U}\,\mathrm{cN}(x)^{w_N} \in [0,1]$
  • cSUN offers a detailed ranking of candidates.
  • Customizable weights enable the prioritization of specific components over others.

Customizable weights allow flexible evaluation

  • Investigating the S-N tradeoff by setting the weights as below:$(w_S,\, w_U,\, w_N) = (w_S,\, 1,\, 2 - w_S)$
  • More weight on stability $\to$ CDVAE $\downarrow$, MP20 test $\uparrow$.
  • MatterGen and Chemeleon2 consistently hold the top positions.
cSUN ranking sensitivity as stability and novelty weights change.

Top-5 MatterGen samples identified with cSUN

Top five MatterGen samples ranked by cSUN.
  • Novel compositions absent from the training set, with low $E_\mathrm{hull}$.
  • High diversity - not biased toward specific compositions or structures.
  • (b)~(d) are not minor variations of training structures.

Outline

  1. Introduction
  2. Improving the most widely-used metric, "SUN"
  3. Measuring the search space of generative models
  4. Conclusion

Do generative models expand the search space?

Contradicting arguments

  • Generative models can explore a wider material space than conventional methods.
  • AI-generated crystals often have already-known structural motifs despite their novel compositions.
Search-space schematic
Do generative models genuinely expand the search space beyond conventional strategies?

Reconstruct GenAI samples via substitution

GenAI crystal Duplicate Substituted Unmatched
DuplicateIdentical to a training sample.
SubstitutedReproducible by applying elemental substitution to a training sample.
UnmatchedMatched by neither criterion (novel structure).

Step 2: Choosing structurally similar training samples

Step 1: Detecting training duplicates

Step 2: Choosing structurally similar training samples

Step 3: Choosing chemically similar training samples

Step 4: Substituting atoms and relaxing

Step 5: Identifying substituted structures

More than 80% are Duplicate or Substituted

Model Duplicate (%) Substituted (%) Unmatched (%)
MatterGen 20.5 60.6 18.9
DiffCSP++ 28.7 63.0 8.3
WyckoffTransformer 26.6 59.7 13.7
Crystalite 29.6 60.5 9.9
Chemeleon2 8.3 80.3 11.4
MP20 Test 5.4 88.4 6.2
  • Unmatched rate $\leq$ 20%, even for SoTA Crystalite and RL-finetuned Chemeleon2.
  • MP20 Test:
    • 88.4% Substituted reflects the historical practice of materials discovery.
    • 5.4% Duplicate means dataset curation is necessary.

Unmatched in low-symmetry, Duplicate in high-symmetry

Duplicate, Substituted, and Unmatched shares by crystal system.
MatterGen samples (metastable, SMACT-valid)
Generative models expand little beyond substitutions in high-symmetry regions, but show higher exploratory potential in low-symmetry regions.

Interpolation and memorization

UMAP and KDE view of generated and training crystal distributions.
  • Low-symmetry → train-dense region → interpolation (Unmatched)
  • High-symmetry → train-sparse region → memorization (Duplicate)

High-symmetry space is not inherently limited

Wyckoff prototype coverage by space group.
  • Wyckoff prototype: a multiset of Wyckoff labels — e.g. $\{\!\!\{1a,1b,3d\}\!\!\}$ for cubic perovskite $\mathrm{CsPbI_3}$.
  • MP20 covers only a small fraction of the full prototype space, even in high-symmetry regions.
  • MatterGen explores non-MP20 prototypes more effectively in low-symmetry systems.

What Unmatched samples actually look like

Three examples of Unmatched generated crystal structures.
  • (a) O and F link locally plausible polyhedra into a complex global structure — interpolation.
  • (b), (c) Unmatched, yet easily reproducible by classical evolutionary algorithms.

Outline

  1. Introduction
  2. Improving the most widely-used metric, "SUN"
  3. Measuring the search space of generative models
  4. Conclusion

Summary

  • cSUN better captures sample distributions, gives a detailed candidate ranking, is theoretically robust, and supports customizable weights.
  • MatterGen and Chemeleon2 expand the stability–novelty Pareto front of cSUN.
  • Yet current models do not expand the search space beyond substitution-based search in high-symmetry regions.
Continuous SUN (Stable, Unique, and Novel) Metric for Generative Modeling of Inorganic Crystals
Paper: https://iopscience.iop.org/article/10.1088/2632-2153/ae7d85
Code: https://github.com/WMD-group/xtalmet
Substitution-Based Analysis of Structural Novelty for Generative Models of Materials
Paper: https://arxiv.org/abs/2606.23166
Code: https://github.com/WMD-group/xtaledit
QR code linking to these slides
Slides

Acknowledgements

Thank you for your attention.

Prof. Aron Walsh
Prof. Aron Walsh
Dr. Hyunsoo Park
Dr. Hyunsoo Park
Kinga Oliwia Mastej
Kinga Oliwia Mastej
Imperial College London Funai Foundation Isembard Leonardo

Masahiro Negishi · m.negishi25@imperial.ac.uk

References

  1. [1] Li et al. "Materials Generation in the Era of Artificial Intelligence: A Comprehensive Survey." arXiv, 2025.
  2. [2] Widdowson et al. "Average Minimum Distances of periodic point sets - foundational invariants for mapping periodic crystals." MATCH Commun. Math. Comput. Chem., 2022.
  3. [3] Hargreaves et al. "The Earth Mover's Distance as a Metric for the Space of Inorganic Compositions." Chem. Mater., 2020.