Recent research suggests that episodic memory is associated with systematic differences in the localization of neural activity observed during memory encoding and retrieval. The retrieval-related anterior shift is a phenomenon whereby the retrieval of a stimulus event (e.g., a scene image) is associated with a peak neural response which is localized more anteriorly than the response elicited when the stimulus is experienced directly. Here, we examine whether the magnitude of the anterior shift, i.e., the distance between encoding- and retrieval-related response peaks, is moderated by age, and also whether the shift is associated with memory performance. Younger and older human subjects of both sexes underwent fMRI as they completed encoding and retrieval tasks on word-face and word-scene pairs. We localized peak scene- and face-selectivity for each individual participant within the face-selective precuneus (PCU) and in three scene-selective (parahippocampal place area [PPA], medial place area [MPA], occipital place area [OPA]) regions of interest (ROIs). In line with recent findings, we identified an anterior shift in PPA and OPA in both age groups and, in older adults only, in MPA and PCU also. Of importance, the magnitude of the anterior shift was larger in older than in younger adults. The shift within the PPA exhibited an age-invariant across-participant negative correlation with source memory performance, such that a smaller displacement between encoding- and retrieval-related neural activity was associated with better performance. These findings provide novel insights into the functional significance of the anterior shift, especially in relation to memory decline in older age.