Cognitive decline is a severe concern of patients with mild cognitive impairment. Also, in patients with temporal lobe epilepsy, memory problems are a frequently encountered problem with potential progression. On the background of a unifying hypothesis for cognitive decline, we merged knowledge from dementia and epilepsy research in order to identify biomarkers with a high predictive value for cognitive decline across and beyond these groups that can be fed into intelligent systems. We prospectively assessed patients with temporal lobe epilepsy ( N = 9), mild cognitive impairment ( N = 19), and subjective cognitive complaints ( N = 4) and healthy controls ( N = 18). All had structural cerebral MRI, EEG at rest and during declarative verbal memory performance, and a neuropsychological assessment which was repeated after 18 months. Cognitive decline was defined as significant change on neuropsychological subscales. We extracted volumetric and shape features from MRI and brain network measures from EEG and fed these features alongside a baseline testing in neuropsychology into a machine learning framework with feature subset selection and 5-fold cross validation. Out of 50 patients, 27 had a decline over time in executive functions, 23 in visual-verbal memory, 23 in divided attention, and 7 patients had an increase in depression scores. The best sensitivity/specificity for decline was 72%/82% for executive functions based on a feature combination from MRI volumetry and EEG partial coherence during recall of memories; 95%/74% for visual-verbal memory by combination of MRI-wavelet features and neuropsychology; 84%/76% for divided attention by combination of MRI-wavelet features and neuropsychology; and 81%/90% for increase of depression by combination of EEG partial directed coherence factor at rest and neuropsychology. Combining information from EEG, MRI, and neuropsychology in order to predict neuropsychological changes in a heterogeneous population could create a more general model of cognitive performance decline.