In the 19th and 20th centuries there was speculation concerning a pre-medieval origin for the castle, but small excavations in the early 20th century found no evidence. The castle does occupy a position similar to that of possible hillfort sites in the Chilterns, and the presence of earlier earthworks within the medieval fortification is not discounted. Geophysical survey within the outer bailey revealed a series of buried ditches forming an irregular grid on markedly different alignment to that of the bailey; these may be the remains of an earlier field system, possibly Iron Age.
The Castle Hill dominates the village of Totternhoe and the lower land all around it. It is hardly surprising that it was the site of a Norman castle and may have been used for defensive structures in the centuries preceding the Norman Conquest or even in prehistory.