The Ogilvies held Beldorney from the late 15th century until it was granted in 1545 to George Gordon and Janet Rose. He was the eldest of three illegitimate sons of Adam Gordon, Dean of Caithness, the third son of Alexander, 1st Earl of Huntly. It is likely that the castle was built between 1552, when the Crown confirmed the transfer, and 1562 when the estate was confiscated for three years after the battle of Corrichie. In the mid 17th century the Gordons of Beldorney got into financial difficulties. A relative named John Lyon obtained the superiority but was hated by the tenants and was murdered by them in the Braes of Abernethie in 1667. John Gordon, 5th laird, spent a year confined in Edinburgh Tollbooth before he was cleared of complicity in the crime. He and his wife Anne have left their initials and the date 1679 on the courtyard entrance, so somehow they must have found the funds to build this court with low ranges on either side. In 1689 Beldorney was pillaged by Highland Jacobites during their pursuit of the Williamite forces under General Mackay. In the 1790s the estate was sold to Thomas Buchan of Auchmacoy who in turn sold it to Sir William Grant. After the Grant male line failed in 1919 the estate was sold to Sir Thomas Birkett for whom alterations were made. Further works were carried out for Commander Vivian Robinson, who purchased the castle after Sir Thomas died in 1959.