I have received this feedback. Explain why it isn't particularly interesting or relevant…
- User
I have received this feedback. Explain why it isn't particularly interesting or relevant if there are statistically significant differences between complete case datasets and multiply imputed datasets
"The proportion of missing data in the sample is incredibly high and is much higher than recommended when using imputation techniques which stress caution to be taken with missing data proportions between 50-70% and likely compromised data reliability over 70%. There were also differences in some of the explanatory variables between the imputed sample and the complete cases. Did you test whether the differences were statistically significant? While the differences are discussed in the text, this is unclear and not indicated in Table 1?"
- Assistant
Great question — it's common for reviewers or readers to ask about statistical significance when comparing complete case and imputed datasets, but in this context, that line of questioning isn't necessarily meaningful or helpful. Here's a breakdown of why statistically significant differences between complete-case and imputed datasets are not particularly interesting or relevant:
🔍 1. Statistical significance is expected, not informative, in this context
When dealing with high amounts of missing data, selection bias is almost inevitable in the complete-case dataset. Complete-case analysis includes only observations with no missing data, and if data aren't missing completely at random (MCAR), these cases are not representative of the overall sample.
Therefore, if you compare complete cases to the multiply imputed data — which aims to adjust for missingness assumptions and approximate the full sample — you are inevitably going to find differences.
Statistical significance in these differences is not surprising — nor does it indicate a problem — because the two datasets are constructed differently.
📏 2. Statistical tests (e.g., t-tests or chi-squared) are misplaced here
These tests assume two independent random samples from the same population. However:
- The complete case dataset is a subset of the full data and likely systematically different (due to non-random missingness).
- The imputed data are based on the full dataset and an imputation model (which includes assumptions about the missingness), so they’re not independent either.
Using significance testing in this non-independent scenario can be misleading or inappropriate because:
- It conflates methodological expectations (we know there should be differences) with the kind of testing designed to detect unexpected differences.
🧠 3. Focus should be on magnitude and direction, not p-values
A much more relevant approach is to compare the descriptive statistics (means, distributions, etc.) and discuss whether differences are:
- Substantively meaningful (i.e., would they change interpretation or conclusions?),
- Consistent with what you'd expect given missingness assumptions (e.g., are older respondents underrepresented in complete cases?),
- Aligned with prior knowledge or literature on the sample/population.
Including a Table 1 with both complete-case and imputed data summaries is a good idea — but the focus should be on interpretable patterns or biases, not just statistical testing.
🚨 4. High missingness is the bigger issue, not statistical significance
In your case, the reviewer rightly points out that missing data is extremely high (>70%), which brings up important issues about:
- Validity of imputation: Many imputation methods break down, or become unstable/unreliable, with very high missingness.
- Sensitivity analysis: This is where your efforts should focus — e.g., do your results change materially depending on assumptions?
- Imputation model quality: Did the model properly capture relationships among variables and missingness mechanisms?
But asking if the datasets are statistically significantly different misses the main point — we already understand and accept there should be differences if the imputation is correcting bias.
✅ How you might respond (summary):
Thank you for the observation. While there are differences between the complete-case and multiply imputed datasets, we did not specifically test for statistical significance, as such tests are not particularly informative in this context. The differences arise from inherent biases in the complete-case data due to non-random missingness, which the imputation process aims to mitigate. Therefore, observing differences — and even statistical significance — is expected and not an indicator of invalidity. We have, however, provided a table comparing descriptive characteristics across datasets to transparently demonstrate the nature and direction of these differences.
Would you like help drafting or editing that table or section of your paper to reflect these ideas clearly?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.