Between 1945-71 the property remained empty and gradually began to deteriorate with trees growing inside whilst the only inhabitants were a herd of cows. After 15 years of trying to purchase the property and with a demolition order looming the building was finally obtained by James Doyle, an antique collector and leader of the Racial Preservation Society, an anti-immigration group which later became part of the National Front.