Contextual information is of paramount importance in medical image understanding to detect and differentiate pathologies, especially when interpreting difficult cases. Current computer-aided detection (CAD) systems typically employ only local information to classify candidates, without taking into account global image information or the relation of a candidate with neighboring structures. In this work, we improve the detection of hard exudates in retinal images incorporating contextual information in the CAD system. The context is described by means of high-level contextual-based features based on the spatial relation with surrounding anatomical landmarks and similar lesions. Results show that a contextual CAD system for hard exudate detection is superior to an approach that uses only local information, with a significant increase of the figure of merit of the Free Receiver Operating Characteristic (FROC) curve from 0.840 to 0.945.