When does synthetic data harm a machine-learning project, and how should teams evaluate it?

What synthetic data is actually doing

Synthetic data is data produced by a rule, simulator, statistical model, or generative model rather than directly observed from the target population. That broad label hides several different activities:

• Augmentation adds samples to real training data, often to vary pose, background, noise, wording, or class balance.
• Replacement trains an analytic or predictive model mostly or entirely on generated records because real records are scarce, restricted, or expensive.
• Simulation samples from an explicit process, such as a physics renderer, a digital twin, or a queueing model, where teams can control inputs and labels.
• Test-data generation creates non-production fixtures, stress cases, or demos that exercise software without claiming to represent the real population.
• Privacy-preserving release tries to let people analyse or share useful patterns without exposing people in the source data.

These uses have different standards of evidence. A physics simulator can intentionally cover a dangerous condition that has not yet occurred often, provided the simulator is validated against real sensors and operating conditions. A model fitted to hospital records cannot legitimately make a missing patient group appear simply by sampling it many times. It only resamples and transforms information already learned, including gaps and errors.

The closest useful framing is data curation, not data multiplication. Synthetic examples change the training objective by changing how often the learner sees each pattern. That can be useful when a team knows which pattern should be emphasised and can verify the result in real data. It becomes harmful when duplication is mistaken for new evidence, or when a convenient generative score replaces the actual deployment outcome.

How the harms arise

Distribution mismatch and missing tails

A generator estimates an approximation to the source distribution, not the world that the eventual system will encounter. The approximation may be excellent around dense, ordinary cases and poor at the boundaries: rare classes, unusual feature combinations, temporal shifts, sensor failures, and high-severity events. Those boundaries are often exactly where a classifier needs reliable support. A global similarity score can be high even if it smooths away a clinically important interaction or misses an entire minority mode.

This is not only a theoretical concern. Alaa, Van Breugel, Saveliev, and van der Schaar separate synthetic-data quality into fidelity, diversity, and generalization. A sample can look realistic yet fail to cover the real distribution, or cover it by reproducing records too closely. Their framework is a useful reminder that one aggregate number cannot establish all three properties. How Faithful is your Synthetic Data?

For tabular data, matching every column's histogram and a few pairwise correlations can still miss higher-order dependencies, constraints, and rare conditional relationships. A data set may match the rate of loan defaults and the distribution of income while getting the interaction among income, location, tenure, and default wrong. For scientific inference, the damage can be more serious than a lower prediction score: a 2024 study found that treating tabular synthetic records as observed data can yield unacceptably high false-positive rates, even when an estimator is unbiased. The Real Deal Behind the Artificial Appeal

For images, realism can conceal missing textures, defective anatomy, impossible reflections, or a narrow range of contexts. In an image-classification study, targeted Stable Diffusion images sometimes helped but were matched or beaten by retrieving relevant real images from the generator's upstream corpus. The researchers trace part of the gap to generator artifacts and inaccurate task-relevant details. The Unmet Promise of Synthetic Training Images

For text, fluent output can omit dialects, document types, error patterns, and adversarially ambiguous phrasing that matter in production. For sequences such as claims, clicks, or clinical events, a marginally realistic event distribution can still break timing, persistence, and event-order dependencies. In each modality, assess the conditional distribution relevant to the task, not only visual plausibility or an average distance between embeddings.

Bias amplification, shortcuts, and labels that do not travel

If the original data underrepresents a group or encodes a historical decision, an ordinary generator can reproduce the imbalance and then make it more influential by oversampling its learned pattern. Directed oversampling can help a class balance problem, but it cannot establish that the generated minority records have the correct within-group variation, label relationship, or causal context. A model may receive more examples while learning a less faithful version of the group.

Synthetic pipelines introduce a second source of bias: the generator's assumptions. Prompt choices, selection filters, a rendering engine, an annotator model, a safety filter, and a labeler can each systematically exclude cases. For example, a vision generator may associate a target disease label with a scanner mark or a background style. A downstream classifier then predicts the label from that artifact. It can look strong on a synthetic validation set because the shortcut is present in both training and test samples, then fail as soon as real images lack it.

This is why an average accuracy increase is insufficient. Report performance, calibration, false-positive and false-negative rates by protected or operationally important subgroup, and report them in the rare, severe, or shifted slices that matter to the decision. A subgroup average can also hide a harmful intersection, such as night images from one device for older patients. Where labels are generated by an LLM or another model, audit a stratified human-reviewed sample and identify whether the label is based on the intended evidence rather than generator wording or metadata.

Generator artifacts and misleading aggregate similarity

Artifact learning is a specific form of shortcut learning. In images it can be a watermark-like pattern, implausible texture, cropping convention, or rendering style. In tabular data it can be impossible value combinations, rounded values, missingness patterns, or an identifier that leaked through a preprocessing join. In text it can be repetitive phrasing, model-specific formatting, citation hallucinations, or a system prompt fragment. The predictive model has no reason to ignore an artifact that conveniently predicts the target.

Aggregate checks are still useful, but they are screening tests, not release criteria. A small Wasserstein distance, KS statistic, FID-like score, or correlation error can hide poor precision in a safety-critical slice. Conversely, a synthetic data set can differ globally for benign reasons yet preserve the features a narrowly defined downstream task needs. Measure sample quality, coverage, and record-level duplication separately, then decide using real downstream performance. The separate fidelity, diversity, and generalization dimensions proposed by Alaa and colleagues make this distinction explicit. How Faithful is your Synthetic Data?

Privacy leakage and memorization

Calling data synthetic does not by itself make it anonymous. An overfit generator may reproduce a training record or allow an attacker to infer whether a person was in its training set. That risk can be greatest for unusual records, which are valuable to the model and often belong to underrepresented groups. In a realistic black-box setting, the DOMIAS membership-inference attack was substantially more successful than prior approaches, especially for uncommon samples. Membership Inference Attacks against Synthetic Data through Overfitting Detection

Memorization is not restricted to tables. Carlini and colleagues demonstrated extraction of individual training images from diffusion models, including photographs of people, and earlier work extracted hundreds of verbatim sequences from a language model. Extracting Training Data from Diffusion Models Extracting Training Data from Large Language Models More specifically, research on LLM-generated synthetic text found membership inference can target the fine-tuning data even when the attacker sees synthetic text rather than the generator itself. The Canary's Echo

Differential privacy is different from a marketing claim that data is synthetic. When the entire training and release mechanism has a valid differential-privacy guarantee, it bounds an individual's influence under stated assumptions and parameters. That guarantee has a utility cost and must be implemented and accounted for correctly. NIST cautions that non-differentially-private synthetic data generally offers only informal privacy protection and may be vulnerable to attacks. NIST SP 800-226 Even with differential privacy, teams should document the privacy parameters, composition across releases, threat model, and task-utility loss.

Feedback loops, contamination, and model collapse

Training a model on output from an older model is not the same as ordinary augmentation. If each generation replaces the real data with finite samples from the preceding generator, sampling error and generator bias can compound. Shumailov and colleagues show that recursive training can induce distribution shift and ultimately remove tails of the original distribution, a phenomenon they call model collapse. AI models collapse when trained on recursively generated data

This result does not establish a universal synthetic-data percentage threshold. The impact depends on the generator, filtering, task, real-data anchor, mixture policy, and repeated generations. It does establish a governance requirement: preserve a provenance-labelled reservoir of human- or sensor-observed data, track generated lineage, and prevent accidental re-ingestion of previous outputs. Web-scale text and image collection needs deduplication and synthetic-content detection or labelling where possible. Otherwise a benchmark, training set, or evaluation corpus can be contaminated by the same generated material, producing an optimistic and circular result.

A validation design that can catch the problem

Keep a real evaluation set independent of data generation and model development. Lock it before selecting a generator, prompt, mixture ratio, sampling temperature, labeler, or downstream model. The generator must not train on it, and analysts should not repeatedly tune against it. If a test set is used for many iterations, it becomes a development set. Reserve a final time-based, site-based, device-based, or prospectively collected holdout when deployment conditions make that feasible.

Build the experiment around the intended decision

Write a short use-case specification first: who or what will receive the prediction, the target label and label horizon, the deployment population, the plausible distribution shifts, and the harm of each error. Pre-specify success thresholds, including subgroup and tail thresholds. The goal is to prevent the generator from becoming the implicit definition of what counts as realistic.

Then make separate partitions of real data. The generator may fit only the authorised real training partition. A development partition can select methods, but the locked holdout remains inaccessible until the design is fixed. Split by person, household, document family, time, site, or device as appropriate. Random row-level splits are often invalid for sequential records or near duplicates, because they let a generator and a classifier benefit from the same individual or template across partitions.

Compare training strategies, not just data sets

For the same downstream model and tuning budget, compare at least these conditions:

1. Real-only training.
2. Synthetic-only training.
3. Real plus synthetic training at several predeclared mixture ratios.
4. A simple non-generative baseline, such as conventional augmentation, class weighting, real-data retrieval, or a domain simulator.

Evaluate all conditions on the same locked real holdout. This is commonly described as train-on-synthetic, test-on-real, or TSTR. It is more informative than training and testing on synthetic data because it asks whether the synthetic distribution transfers to reality. A reverse experiment, train-on-real and test-on-synthetic, can diagnose whether generated samples are recognisable, but it cannot demonstrate deployment utility. The need to evaluate task-specific utility rather than generic data quality is also the premise of work on post-processing private synthetic data for user-selected measures. Post-processing Private Synthetic Data for Improving Utility on Selected Measures

Report uncertainty, not only the best point estimate. Use paired confidence intervals or a pre-specified resampling procedure for the real holdout. Break out precision, recall, AUROC or AUPRC where appropriate, calibration error, decision-curve utility, and error rates at the actual operating threshold. For rare-event tasks, prevalence-aware precision and recall can be more decision-useful than accuracy. Compare calibration before and after any post-processing or confidence thresholding.

Inspect the slices that average metrics hide

Define slices before looking at outcomes. At minimum, include class frequency bands, protected and operational subgroups, intersections of important subgroups, sites and devices, time periods, corruption or stress conditions, and high-cost errors. For a medical classifier, a useful tail slice might be rare conditions at a new clinic. For a fraud detector, it might be new merchants with a low base rate. For an LLM, it might be short, misspelled, multilingual, or policy-sensitive inputs.

Test the synthetic data itself at several levels:

• Univariate and conditional fidelity: category rates, ranges, missingness, label prevalence, conditional distributions, and domain constraints.
• Dependence and temporal fidelity: important joint distributions, higher-order interactions, event order, persistence, and known causal or physical invariants.
• Coverage: distance or density diagnostics by subgroup and tail region, plus manual review of the most novel and nearest generated records.
• Artifact and shortcut probes: train a discriminator to distinguish real from synthetic data, inspect its most predictive features, remove suspected artifacts, and test whether downstream performance survives counterfactual edits.
• Downstream utility: TSTR performance by slice and operating threshold, compared with real-only and simple baselines.

No single diagnostic is enough to approve the dataset. A real-vs-synthetic discriminator can detect a difference that does not matter for the target task. Its value is as a probe for differences that deserve explanation. Conversely, a weak discriminator or good-looking embedding plot is not evidence that tail labels and causal relationships are correct.

Make privacy testing adversarial

Privacy evaluation should match what is released and what an attacker could know. Test exact and near-duplicate matches after canonicalisation, nearest-neighbour exposure, membership inference, attribute inference where relevant, and reconstruction or extraction attacks for the model family. Report attack success overall and for outliers, rare combinations, and sensitive subgroups. Do not use a single distance-to-closest-record threshold as proof of privacy. It can miss inference from many weak signals and can be distorted by high-dimensional distance.

If the project claims differential privacy, obtain a review of the full mechanism. Record the adjacency definition, sampling assumptions, privacy accounting method, epsilon and delta, all repeated releases, and the actual code path. If it does not claim differential privacy, describe the protection modestly as empirically tested risk reduction, not anonymisation. That distinction is especially important when synthetic data will be shared outside the original trust boundary.

Require provenance and an ablation trail

For every generated batch, retain the generator and version, training-data source and access authority, prompts or simulation parameters, filters, seeds where practical, labeler, transformations, quantity, mixture ratio, and intended use. Label synthetic records so that later data collection cannot silently feed them back into a future generator. Record the real-data freeze date and the identifier of every holdout that was excluded.

Run ablations that vary one design choice at a time: no synthetic data, each generator, each quality filter, each label source, and each mixture ratio. Test whether the apparent gain remains when suspected artifacts are removed and whether it exists across random seeds and holdout sites. An improvement that appears only at one mixture, one seed, or one generator prompt should be treated as a hypothesis to investigate, not a deployment result.

Decision table for common uses

The table's distinction matters. Simulation can be the strongest option for a well-understood mechanism because it exposes assumptions and lets teams explore counterfactual conditions. Replacement is the most demanding claim because it asks generated samples to stand in for unknown real variation. Test fixtures generally need functional coverage, not population fidelity, so they should not be sold as representative data.

Example

Hypothetical example: A radiology team has few verified examples of an uncommon finding. It uses a diffusion generator to create extra images and a model-assisted labeler to tag them. The team keeps an untouched real test set split by hospital and date, including rare cases and images from the newest scanner.

The team trains the same classifier on real-only data and on real data plus synthetic data at 10%, 30%, and 60% synthetic proportions. It evaluates each version on the untouched real images, reports sensitivity and calibration for the uncommon finding by hospital and scanner, and asks specialists to review synthetic examples nearest to real patients and examples the classifier finds most influential. It also tests whether a simple image crop or generator detector predicts the label unusually well.

If a 30% mixture improves sensitivity on both hospitals without worsening calibration or subgroup performance, and the privacy and artifact checks pass, augmentation is defensible for that scope. If the gain exists only on the synthetic validation data, or disappears on the new scanner, the correct conclusion is not that more generated images are needed. The generator has exposed a coverage gap that requires targeted real collection, a better simulator, or a narrower deployment claim.

Practical release checklist

• State the intended decision, population, label definition, and the real-world failure modes that matter.
• Keep an independently sourced, locked real holdout out of generator fitting, data selection, and iterative tuning.
• Compare real-only, synthetic-only, and several real-plus-synthetic mixtures against simple alternatives.
• Report downstream utility and uncertainty by subgroup, tail case, site, time, and operating threshold.
• Test conditional relationships and constraints, not merely marginals or a single similarity score.
• Inspect synthetic-origin artifacts and label shortcuts with discriminators, counterfactual edits, and human review.
• Perform privacy attacks appropriate to the released data or model, including tests for rare records and memorization.
• Track lineage, generator versions, prompts or parameters, filters, mixture ratios, and all subsequent releases.
• Prevent synthetic records from contaminating future training and benchmark data.
• Stop or narrow use when the locked real evaluation shows a material utility, fairness, calibration, or privacy regression.