VAYAS¹: An Age-Stratified Zero-Shot Audit of Hindi Speech Recognition

1. Introduction

Automatic speech recognition (ASR) systems are increasingly deployed as the primary interface for voice assistants, transcription tools, and accessibility technology. Prior audits of commercial ASR in English have found substantial disparities in error rate across demographic groups defined by race (Koenecke et al. 2020), and structural audits of other speech and vision systems have shown that undisclosed performance gaps across a benchmark’s underlying population are common when systems are evaluated only in aggregate (Buolamwini and Gebru 2018). Age is a comparatively under-studied axis for this kind of audit, and Hindi, despite being among the most widely spoken languages in the world, has very little public infrastructure for answering this question at all: the most commonly used open Hindi ASR benchmark, Common Voice, turns out to have almost no confirmed elderly speakers once its self-reported age field is actually counted (Section 3).

This paper reports a zero-shot audit (no system is fine-tuned or otherwise modified) of four ASR systems on Hindi speech, following the general blueprint of building a benchmark and auditing existing systems against it rather than training anything new (Buolamwini and Gebru 2018), and adopting the matched-cohort design of Koenecke et al. (2020): an elderly group and a same-source, same-conditions control group, so that age is the only systematic difference between the two samples being compared. We find one dominant result (a large, consistent quality gap between systems, whose interpretation is complicated by a training-data contamination risk we flag but cannot resolve) and one narrower, suggestive but not statistically confirmed signal (a small elderly disadvantage specific to one system, driven partly, not wholly, by an identifiable failure mode in its decoder). We treat both as worth reporting in full, including where they fall short of confirmation: a mostly-null result across three of four systems is not a failure of the audit, it is information about where an age-based accessibility problem does and does not currently show up at the sample size this study could run.

2. Related Work

Koenecke et al. (2020) audited five commercial ASR systems on African American Vernacular English versus white American English using matched speaker cohorts, finding substantially higher word error rates for Black speakers across every system tested; the matched-cohort design used here (same corpus, same recording conditions, one demographic axis varied) follows that methodology directly. Buolamwini and Gebru (2018) established the broader structural pattern this work follows: constructing a benchmark with genuine population coverage and auditing already-deployed systems against it, rather than proposing a new model. Pellegrini et al. (2012) is the closest prior study of age effects specifically, examining the impact of age on ASR performance in a non-English setting; we adopt its motivating question but, as discussed in Section 3, could not replicate its finer decade-level age banding given our source data’s constraints. Thennal D K et al. (2025) motivate reporting both WER and CER together rather than WER alone, since word- and character-level error can diverge in morphologically rich languages; Hindi’s own morphology (case-marked postpositions, extensive compounding) makes this decomposition relevant here even though the original motivation was a different language family.

On the systems side, this audit covers Whisper (Radford et al. 2023), a general-purpose multilingual encoder-decoder model; MMS (Pratap et al. 2023), Meta’s massively multilingual CTC-based system; and two Hindi-specialized systems from AI4Bharat, IndicConformer (AI4Bharat 2025) and IndicWhisper (Bhogale et al. 2023), both fine-tuned or trained specifically for Indian languages. Both AI4Bharat systems draw, directly or indirectly, on IndicVoices (Javed et al. 2024) as training data, a fact with direct consequences for this audit’s own eval corpus (Section 3.4).

3. Data

3.1 Source selection

We initially considered Mozilla Common Voice Hindi (cv-corpus-26.0-2026-06-12, validated split) as the elderly-cohort source, since it is the most widely cited open Hindi ASR corpus and provides decade-level age brackets. Counting its validated split directly, however, shows this is not viable: of 369 total speakers, 145 fall in the under-60 control range, 222 left the (optional, self-reported) age field blank, and only 2 speakers are confirmed aged in their 60s, with zero confirmed in their 70s or 80s. Applying an n$\,\geq\,$8-speakers-per-band stopping rule, every elderly band fails outright.

We therefore used IndicVoices-R Hindi (Sankar et al. 2024), the Hindi subset of AI4Bharat’s IndicVoices-R corpus, as the source of record for both the elderly and control cohorts. A full, column-selective read of all 10 parquet shards (368 unique speakers, 26,318 utterances total, no audio decoded during this scan) gives real, exact per-age-band counts:

The elderly (60+) band clears the n$\,\geq\,$8 threshold with wide margin, and the combined under-60 pool (318 speakers) is more than sufficient for a same-corpus, duration-matched control cohort (Section 3.3). Figure 1 and Figure 2 summarize this comparison.

3.2 Age-band resolution and speech type

IndicVoices-R’s age_group field is a 4-class label (18-30, 30-45, 45-60, 60+) with no finer resolution available from public metadata. Unlike the decade-level scheme used by Pellegrini et al. (2012) and originally targeted by this project, there is no way to distinguish speakers in their 60s, 70s, or 80s from this source. We therefore report a single collapsed elderly (60+) band against a single control (< 60) band, rather than the finer scheme we had originally intended; this is a real resolution loss we state plainly rather than silently substituting a coarser claim for a finer one.

We manually inspected a sample of verbatim/normalized/task_name fields to confirm speech type before treating this as a spontaneous-speech corpus. Two signals both indicate genuine spontaneous, topic-elicited speech rather than read-aloud script: (1) task_name values are topic prompts (“KYP - Cooking”, “Daily Life”, “Alexa Commands”), not transcript references; and (2) verbatim preserves colloquial spelling and at least one visible disfluency/false-start that the cleaned normalized field scrubs. We use verbatim as the gold reference throughout, since it is the field that actually preserves what a spontaneous-speech evaluation should measure against.

One caveat carries through every result in this paper: IndicVoices-R applies a speech-enhancement (denoising/dereverberation) preprocessing step before release, which may suppress some of the acoustic degradation an age-based audit is trying to measure. We flag this as a standing limitation (Section 7), not a resolved issue.

3.3 Cohort construction

The elderly cohort is all 50 confirmed 60+ speakers in IndicVoices-R Hindi. The control cohort is drawn from the same corpus’s 318-speaker under-60 pool via duration-matched random sampling: control clips are shuffled and greedily accumulated until their total duration meets or exceeds the elderly cohort’s total duration, so that the two cohorts are comparable in scale as well as source. Sourcing both cohorts from the same corpus, rather than pairing IndicVoices-R elderly speakers against a Common Voice control group, is what makes this a matched-cohort design in the sense of Koenecke et al. (2020): the only systematic difference between the two samples is age, not platform, recording setup, or elicitation protocol.

Two of the four systems (MMS, IndicConformer) were run against the full elderly pool (3,899 clips) plus an equal-duration control sample (3,787 clips), 7,686 clips total. The other two (Whisper large-v3, IndicWhisper) measured 2–4x slower than real-time on the GPUs available for this study (an RTX 3050 laptop GPU, 4GB VRAM, for development; Kaggle-hosted NVIDIA P100/T4 instances for the full batch runs), so were run against a smaller, still duration-matched, non-overlapping union of three sampling passes (elderly and control positions from each earlier pass were explicitly excluded before sampling the next, using the same deterministic scan order and a fresh random seed per pass; see scripts/ in the released code, which recovers each pass’s exact sample by replaying its seed rather than only checking for accidental overlap after the fact): 4,807 unique clips total, comprising 2,434 elderly and 2,373 control clips.

Duration-matching alone does not equalize per-speaker clip counts. Elderly speakers contribute far more clips each than control speakers: 77.98 clips/speaker (elderly) vs. 11.95 (control) in the MMS/IndicConformer pool, and 48.68 vs. 7.58 in the Whisper/IndicWhisper pool. Since our statistics aggregate to one mean per speaker before computing confidence intervals and effect sizes (Section 4.3), control speaker-level means are computed from far fewer clips each and are correspondingly noisier; this is the direct cause of at least one visibly wide control-cohort confidence interval in Table 2 (IndicConformer control CER). Age remains the only demographic axis deliberately varied between cohorts, but it is not the only difference in how each cohort’s statistics are computed, and we do not claim otherwise. Balancing clips per speaker, not just total duration, is the clear next step for this sampling design (Section 7).

3.4 Training-data contamination risk

A serious caveat applies specifically to IndicConformer and IndicWhisper, one we were not able to fully resolve before running the audit. IndicVoices-R, this paper’s evaluation corpus, is explicitly described by its own authors as derived from IndicVoices (Javed et al. 2024) via an audio-restoration pipeline (demixing, dereverberation, denoising) applied to the same underlying recordings, not an independently collected corpus (Sankar et al. 2024). Publicly available AI4Bharat documentation states that their ASR models, including IndicWav2Vec and IndicWhisper, are trained in part on IndicVoices; we could not obtain an authoritative, model-specific statement confirming or ruling out IndicConformer’s exact training set, despite checking its Hugging Face model card, its GitHub organization, and AI4Bharat’s own ASR project page, none of which publish a dataset-level training manifest.

This means IndicConformer’s result in this paper cannot be presented as a clean zero-shot comparison without qualification: if its training data included IndicVoices, it may have seen the same underlying recordings and transcripts as some or all of this paper’s evaluation clips (post-enhancement audio, pre-enhancement audio, or both), which would inflate its apparent accuracy relative to the two systems in this audit with no plausible IndicVoices exposure (Whisper large-v3, MMS). One partial, non-conclusive counter-data-point: IndicConformer also produced an exact-match transcript on a single Common Voice Hindi clip during pilot validation (a corpus wholly unrelated to IndicVoices), suggesting at least some of its accuracy is not corpus-specific, but a single clip cannot settle a corpus-level contamination question, and we do not treat it as doing so. Given this, we report IndicConformer’s result descriptively rather than causally: it is the best-performing system on this benchmark, as measured, and we explicitly do not claim this reflects a fair, contamination-free zero-shot comparison against the other three systems. Framing its advantage as an architecture or training-recipe insight (rather than a possible data-leakage artifact) is not supportable without the speaker- or utterance-level training manifest AI4Bharat has not published, and we do not make that framing (see Section 6). We raise this as a general recommendation rather than a complaint specific to AI4Bharat, whose IndicVoices, IndicVoices-R, and Vistaar releases are themselves valuable, well-documented contributions this paper relies on directly: publishing a speaker- or utterance-level training manifest alongside released ASR checkpoints, or at minimum a clear per-model statement of exactly which corpora contributed training data, would let audits like this one resolve exactly this kind of question rather than having to flag it as an open risk.

4. Methodology

4.1 Systems audited

All four systems are run zero-shot: no fine-tuning, no prompt engineering beyond specifying the target language where the API supports it.

• Whisper large-v3 (Radford et al. 2023) (openai/whisper-large-v3, $\sim$1.55B parameters): general-purpose multilingual encoder-decoder, called directly via WhisperForConditionalGeneration rather than the higher-level pipeline() wrapper, after the latter surfaced three separate real integration bugs during development (a missing system ffmpeg dependency, a torchcodec library mismatch, and a genuine transformers bug in its dict-input preprocessing path). No explicit beam size, temperature-fallback, or repetition-penalty configuration was set beyond library defaults; this gap in disclosure applies equally to IndicWhisper below and is discussed as a limitation (Section 7) directly relevant to Section 5.3.

• MMS-1B (Pratap et al. 2023) (facebook/mms-1b-all, $\sim$1B parameters): CTC-based, part of Meta’s Massively Multilingual Speech project. MMS ships one model with per-language adapter weights rather than one checkpoint per language; the Hindi adapter is loaded explicitly via processor.tokenizer.set_target_lang("hin") and model.load_adapter("hin") before every transcription call.

• IndicConformer (AI4Bharat 2025) (ai4bharat/indic-conformer-600m-multilingual, 600M parameters): AI4Bharat’s Conformer-based multilingual Indic ASR model, run with its CTC decoding head (the model also supports RNNT decoding; CTC is what this audit used throughout).

• IndicWhisper (Bhogale et al. 2023): AI4Bharat’s Hindi Whisper fine-tune, distributed as a direct checkpoint archive (not a standard Hugging Face Hub repository) per the Vistaar project. That archive ships two Hindi checkpoints, fine-tuned from different Whisper base sizes; we confirmed via each checkpoint’s own config.json that this audit used whisper-large-hi-noldcil ($d_{model}=1280$, 32 encoder and 32 decoder layers, matching Whisper large’s architecture, $\sim$1.55B parameters), not the archive’s other whisper-medium-hi_alldata_multigpu checkpoint ($d_{model}=1024$, 24+24 layers, $\sim$769M parameters). We flag this deliberately: our original loading code selected whichever checkpoint a directory scan happened to return first, which is filesystem-order-dependent and was not a considered choice at the time; we verified after the fact which one actually ran, and have since fixed the loader to select explicitly rather than by scan order.

Two further systems in the originally planned seven-system set were excluded after real, documented attempts rather than assumption: IndicWav2Vec’s LM-decoding dependency (pyctcdecode$\rightarrow$kenlm) does not build against the Python version used in this environment, and Meta’s Omnilingual ASR ships no Windows-compatible native backend wheel at all. Both are platform gaps in the development environment used for this study, not judgments that the systems are unusable in principle. A fifth system, Sarvam (a billed API), remains in-scope but deferred pending an explicit cost decision, and is not included in the results below.

4.2 Metrics

For every clip with a non-empty gold reference, we compute Word Error Rate (WER) and Character Error Rate (CER) via jiwer (Vaessen 2024), using its default whitespace word tokenization and raw character-sequence comparison (no Hindi-specific text normalization is applied; this is a documented choice, not an oversight). We additionally report the WER$-$CER divergence (Thennal D K et al. 2025), per-clip substitution/insertion/deletion counts, and a speaking-rate proxy (reference word count divided by clip duration).

We use IndicVoices-R’s verbatim field as the gold reference throughout, since it preserves disfluencies and false starts rather than the cleaned normalized field. This is a deliberate choice for measuring spontaneous-speech ASR, but it has a real methodological consequence: no system in this audit is trained to emit disfluencies, so any age-related difference in disfluency rate could in principle produce a WER/CER gap attributable to reference style rather than acoustic recognition failure, in the same direction this paper’s central hypothesis (elderly speech is harder to recognize) would also predict. We tested this directly by re-scoring every clip against normalized references and recomputing every effect size in Table 3: all 8 effects keep the same sign under both reference styles, and IndicWhisper’s WER/CER pattern in particular is essentially unchanged ($d=0.136$/$0.181$ normalized vs. $0.139$/$0.187$ verbatim; still nominally significant, still short of surviving Benjamini–Hochberg correction, under both scorings). The disfluency confound does not explain away this paper’s age-effect signal, which is a real strengthening of the IndicWhisper result even though it remains statistically suggestive rather than confirmed (Section 4.3).

4.3 Statistics

Every aggregate statistic below is reported as an (estimate, CI_low, CI_high, n) tuple, never a bare point estimate. Per-clip metrics are first aggregated to one mean value per speaker (not pooled across clips directly), since multiple clips from the same speaker are not independent observations; 95% confidence intervals are then obtained via bootstrap resampling of speakers (10,000 resamples). Elderly-vs-control effect sizes are reported as both Cohen’s $d$ (parametric, mean-based) and a rank-biserial correlation derived from a two-sided Mann-Whitney U test (nonparametric, robust to skew) on the same per-speaker distributions, with $p$-values from the same test. With 4 systems $\times$ 2 metrics, this paper runs 8 such tests; we apply a Benjamini–Hochberg correction across all 8 and report both raw and corrected $p$-values (Table 3), rather than letting the smallest raw $p$-values stand uncorrected. We also report each test’s minimum detectable effect (MDE) at 80% power given its actual sample sizes, since a null result from an underpowered test is not evidence of no effect.

5. Results

5.1 Error rate by system

Figure 3 and Table 2 show WER and CER for all four systems, split by age cohort. IndicConformer has the lowest error rate by a wide margin (WER 17.1% elderly / 18.3% control) despite being architecturally the smallest of the four models tested (600M parameters vs. $\sim$1B–1.55B for the others), but this comparison carries the training-data contamination risk flagged in Section 3.4 and is not presented as a clean zero-shot result. MMS-1B and Whisper large-v3 are comparable to each other (WER $\approx$ 35–36% in both cohorts). IndicWhisper, an India-specialized fine-tune of a Whisper large-architecture checkpoint (Section 4.1), is instead the weakest of the four systems by a substantial margin (WER 46.4% control / 50.5% elderly), despite sharing IndicConformer’s exposure to AI4Bharat’s training data ecosystem; that a same-provenance system performs worst, not best, is itself a data point against treating training-data overlap as a full explanation for IndicConformer’s result. Section 5.3 investigates IndicWhisper’s result rather than taking it at face value.

5.2 Age-cohort effect sizes

Figure 4 and Table 3 report Cohen’s $d$ and rank-biserial $r$ for every system/metric combination. Effect sizes are small throughout ($|d| < 0.2$ in all eight system$\times$metric pairs), below this sample’s own minimum detectable effect of $d = 0.427$ at 80% power (Section 4.3), so every null result here is genuinely uninformative about whether a small true effect exists, not evidence that none does. Three raw $p$-values fall under .05 (IndicWhisper WER and CER, Whisper large-v3 CER), but none survive Benjamini–Hochberg correction across the 8 tests (all corrected $p \geq 0.10$; Table 3). We report IndicWhisper’s WER/CER pattern as suggestive rather than confirmed: both raw $p$-values are the smallest in the table, both point the same direction (elderly disadvantage), and one of the two survives even after excluding IndicWhisper’s degenerate-output clips (CER, $p=0.040$; WER does not, $p=0.085$; Section 5.3), a plausible real signal at the sample size this audit could run, but not one we can call statistically confirmed. Whisper large-v3’s CER effect flips sign between Cohen’s $d$ ($-0.045$, elderly slightly better on raw data) and the rank-biserial estimate ($+0.183$, elderly slightly worse by rank), a real, not erroneous, consequence of a mean-based and a rank-based statistic disagreeing under a skewed distribution, and does not survive correction either; we report both statistics rather than picking the one that tells a cleaner story.

5.3 IndicWhisper’s repetition-loop failure mode

Manual inspection of IndicWhisper’s highest-WER clips (several exceeding WER $>$ 1.0, mathematically possible only via runaway insertion) revealed genuine decoder degeneration: a short reference phrase transcribed correctly, followed by a repeated token or short phrase looping tens to hundreds of times (e.g. a three-word phrase repeated 9 times, degenerating into a single-syllable loop of 30, then 100+, repetitions). This is a known failure mode of encoder-decoder ASR models under greedy or lightly-constrained decoding, particularly on short inputs, and is not a computation artifact in our pipeline.

We quantify this directly by counting clips with WER $>$ 1.0 as a “degenerate-output rate” (Figure 5, Table 4). This threshold is a deliberately simple, mathematically motivated proxy (WER $>$ 1.0 is only reachable via runaway insertion), not a repetition-specific detector (e.g. max-$n$-gram-repeat count); we manually confirmed a sample of flagged clips shows genuine repetition, but the threshold is also biased toward shorter references, which cross it with fewer excess words than longer ones. IndicWhisper degenerates on 7.16% of its clips, 16–80x the rate of the other three systems individually (0.09%, 0.34%, 0.46%; the range is wide because the other three systems’ own rates differ by a factor of five from each other). Degenerate clips do skew shorter on average (6.67s vs. 9.91s), but given the length-bias just noted, we treat this as a secondary, illustrative observation, not as independent confirmatory evidence.

The by-cohort breakdown matters for two reasons. First, IndicWhisper’s degenerate rate is only modestly higher for elderly speakers (7.60% vs. 6.70%): the pathology is not an elderly-specific failure mode, it is a general short-clip failure mode that elderly speakers hit slightly more often, consistent with (though not proof of) the speaking-rate relationship in Section 5.4. Second, IndicConformer’s own degenerate clips are entirely concentrated in the control cohort (7 of 7, 0.18% vs. 0.00% elderly); this, not a general control-cohort noise property, is what produces IndicConformer’s unusually wide control CER confidence interval in Table 2 (a handful of outlier clips dominating a per-speaker mean), and is a second concrete illustration of why the per-speaker clip-count imbalance noted in Section 3.3 matters in practice, not just in principle.

Recomputing WER/CER with degenerate clips excluded (Table 5) shows this pathology explains part, not all, of IndicWhisper’s disadvantage: trimmed WER drops from 50.5% to 42.1% (elderly) and 46.4% to 38.9% (control), and the elderly-control WER gap itself shrinks from 4.18 to 3.20 percentage points, a 23% reduction, not the majority of the gap. IndicWhisper remains clearly the worst-performing system of the four even after trimming, and re-running the significance tests on trimmed data (Section 4.3’s method, unchanged) shows the elderly-disadvantage signal itself is sensitive to this choice: CER stays nominally significant ($d=0.290$, $p=0.040$) but WER does not ($d=0.233$, $p=0.085$), where both were nominally significant untrimmed. We report this instability directly rather than choosing whichever version of the analysis looks cleaner: the pathology is a genuine, partial explanation, not a full one, and the age-effect claim built on top of it is correspondingly fragile.

5.4 Speaking rate and error rate

Figure 6 shows binned mean WER against speaking rate (reference words per second) for each system. Faster speech is associated with lower WER for all four systems. Since WER’s denominator is reference word count, and our speaking-rate proxy is also reference words divided by duration, part of this association is a length-normalization artifact rather than a purely acoustic/phonetic effect: controlling for reference word count via partial correlation attenuates Whisper large-v3’s raw Spearman $\rho=-0.287$ to $\rho=-0.214$, still a real association but roughly a quarter smaller once the length confound is accounted for; the already small IndicConformer and IndicWhisper correlations are essentially unchanged by the same control ($\rho=-0.005 \to -0.005$ and $\rho=-0.070 \to -0.046$ respectively). We had originally speculated that slower, more hesitant speech is “plausibly more common among elderly speakers” as a mediating mechanism; testing this directly (per-speaker mean speaking rate, elderly vs. control) shows it is not supported in this corpus: elderly speakers’ mean rate (2.58 words/sec) is only marginally lower than control’s (2.63 words/sec), and the difference is not significant (Mann-Whitney $p=0.41$). The speaking-rate/WER association is real but not a demonstrated channel for this paper’s age comparisons specifically.

6. Discussion

The central finding of this audit is not the one it originally set out to test cleanly. Rather than a uniform elderly-speaker accessibility gap, we find a large system-quality gap (IndicConformer vs. the other three) whose interpretation is compromised by an unresolved contamination risk (Section 3.4), and a narrower, suggestive, not statistically confirmed age-linked signal specific to one system (IndicWhisper). Neither is the clean result the matched-cohort design was built to produce, and we think reporting both honestly, including where they fall short of confirmation, is more useful than reporting either as settled.

The near-null age effect for MMS-1B, IndicConformer, and Whisper large-v3 is not evidence that elderly-speaker ASR accessibility is a solved problem in general: every one of these null results sits below this sample’s minimum detectable effect ($d=0.427$–$0.430$ at 80% power; Section 4.3), so the honest reading is “no large effect detected at this sample size,” not “no effect exists.” IndicWhisper’s WER/CER pattern is the kind of result a matched-cohort audit is designed to surface, but it does not survive multiple-comparison correction, is itself below the sample’s MDE (meaning if the effect is real, this sample likely overstates its size), and is only partially robust to excluding the system’s own repetition-loop pathology (CER survives retesting on trimmed data, WER does not; Section 5.3). We report it as a real candidate for a follow-up, adequately powered study, not as a confirmed finding of this one.

We explicitly do not draw the conclusion IndicConformer’s raw numbers might otherwise invite – that architecture or training-recipe choices explain its advantage over three larger models. Given the documented, unresolved possibility that its training data overlaps this evaluation corpus (Section 3.4), a data-leakage explanation is at least as plausible as an architectural one, and we do not have the evidence to distinguish them. What this audit can say without qualification is narrower: on this benchmark, as measured, IndicConformer produced the lowest error rates of the four systems tested, and whether that reflects genuine generalization or prior exposure to related data is an open question this paper raises but does not resolve.

7. Limitations

• Training-data contamination risk (Section 3.4) is the most serious open item in this paper: IndicConformer’s and, on the same public documentation, possibly IndicWhisper’s training data may overlap IndicVoices-R’s source recordings, and we could not obtain an authoritative training manifest to confirm or rule this out. IndicConformer’s headline result should be read with this caveat foremost, not as a footnote.

• Per-speaker clip-count imbalance (Section 3.3): duration-matching does not equalize clips per speaker (elderly speakers contribute 6.5–6.9x more clips each than control speakers across the two system pools). This inflates the effective precision of elderly per-speaker means relative to control ones and is the direct, confirmed cause of at least one unusually wide confidence interval in this paper’s results (IndicConformer control CER, Section 5.3).

• Gold-reference confound, tested: we score against verbatim transcripts, which preserve disfluencies no audited system is trained to produce, raising the possibility that an elderly-control disfluency-rate difference alone could inflate WER/CER gaps. We re-scored every clip against normalized references to test this directly (Section 4.2); all 8 age-effect signs are unchanged and IndicWhisper’s pattern is essentially identical under both scorings. This does not resolve the multiple-comparison caveat, but it does rule out disfluency scoring as the source of the signal that exists.

• Decoding configuration is underspecified: beam size, temperature fallback, and repetition-penalty settings were left at library defaults for Whisper large-v3 and IndicWhisper and were not recorded per-run. Since Whisper’s reference implementation ships a temperature-fallback mechanism specifically to reduce repetition loops, some share of Section 5.3’s degenerate-output rate may reflect this audit’s configuration rather than the checkpoints’ intrinsic behavior under their own developers’ recommended settings.

• Unequal sample sizes across systems (7,686 clips for MMS/IndicConformer vs. 4,807 for Whisper/IndicWhisper, driven by compute constraints, Section 3.3) mean the degenerate-output-rate comparison in Table 4 is not drawn from identical clip pools across all four systems, though the elderly/control ratio is held constant within each pool.

• Statistical power: every null age-effect result in this paper is below its own minimum detectable effect at 80% power ($d \approx 0.43$); we report this explicitly (Section 6) rather than letting a null result read as stronger evidence than it is.

• Age-band resolution collapses to a single 60+ vs. under-60 split; the finer 60s/70s/80s scheme originally targeted, and used by Pellegrini et al. (2012), is not recoverable from either public source checked.

• Speech enhancement: IndicVoices-R applies denoising/dereverberation before release. This may suppress acoustic degradation an age-based audit is trying to measure, but denoising can also distort breathy or jittery elderly voices in ways that manufacture rather than hide a gap; we do not know which direction dominates here, and flag both rather than only the suppression case. Any elderly-vs-control gap reported here is a gap after this preprocessing, not a raw-audio result.

• A fifth system, Sarvam, a billed commercial API, remains explicitly deferred rather than included or silently dropped, pending a cost decision outside the scope of this paper.

• Two further systems (IndicWav2Vec, Meta’s Omnilingual ASR) were excluded due to real, documented platform incompatibilities in the development environment used (a Python version/native-library mismatch and a lack of Windows wheels, respectively), not a judgment that they are unusable in principle.

• Single-language scope: this audit covers Hindi only. The original two-family (Indo-Aryan/Dravidian) design was dropped when a viable Tamil data source could not be confirmed; the claim this paper can support is a single-language proof of concept and an explicit call for replication in other languages, not a cross-family comparison.

8. Conclusion

We audited four zero-shot Hindi ASR systems on a matched elderly/control cohort drawn from IndicVoices-R Hindi, a corpus we selected specifically because the field’s more commonly used Hindi benchmark, Common Voice, turns out to have almost no confirmed elderly speaker representation once actually counted (a majority of Common Voice Hindi speakers leave the age field blank, so this is a claim about confirmed representation, not a claim that no elderly speakers exist in that corpus at all). We find IndicConformer to have the lowest error rates of the four systems, though this result carries an unresolved training-data contamination risk we could not rule out and do not present as a clean zero-shot comparison; IndicWhisper to be the weakest, partly (about a quarter of its elderly-control gap) due to an identified decoder repetition-loop pathology on short clips; and small age-cohort effects across all four systems that do not survive correction for multiple comparisons, with IndicWhisper’s pattern the one candidate worth a dedicated, adequately powered follow-up study rather than a confirmed finding here. We release the full audit pipeline, matched-cohort sampling code, and per-clip results at https://github.com/kshubham090/vayas-kosh so this protocol, including the parts of it that did not resolve cleanly, can be checked, extended, and improved on.

References