Toledillo Castle is a Muslim defensive construction. R. Fernández González chronologically places this fortress in the Umayyad emirate, and it would continue to be expanded during the Caliphate period. It was rebuilt in the eleventh and twelfth centuries in the face of the Christian thrust from the north.
The Castle is located on the eastern flank of the municipality of Peñaflor, five and a half kilometres northeast of the town. It sits on a small plateau on a spur of the Sierra Morena mountain range that descends gently in a north-south direction to the Guadalquivir Valley. This is in stark contrast with the steep slopes of the easternmost flank through which the River Retortillo flows, which acts as a moat and provides natural protection to the Castle as its southern flank.
Under the Generic Declaration of the Decree of 22 April 1949, and the Law 16/1985 on Spanish Historical Heritage, the General-Directorate for Cultural Assets of the Andalusian Government declared Toledillo Castle an Asset of Cultural Interested. It was listed in the General Catalogue of Andalusian Historical Heritage on 25 June 1985. In 1993, the Andalusian Government granted special recognition to the castles in Andalusia.