The Old Hall was lived in until 1810, when there was a serious fire destroying the South East wing. The Noels then moved into a nearby farmhouse, which is part of the present Hall. Wings were added in 1841 and 1850. In 1867 stone from the Old hall was used to build a Roman Catholic Church adjacent to the Hall to mark the present Earl's conversion to the Catholic faith. Tradition says that squatters working on the Hall in 1841 build the cottages at Stamford End for themselves.

The legend of the Mistletoe Bough is said to have originated at the Old Hall. The story refers to the Christmas of 1700, when there was merrymaking at the hall. A young lady taking part in a theatrical performance was asked to lie in an oak chest, representing a coffin. Later when the chest was opened she was found to be dead.