The lands originally belonged to Ballindallochs of that ilk but were in the hands of John and Barbara Gordon by the time the 16th century Z plan tower house was built - a stone with their names and date of 1546 were found behind paneling in 1818. Conversely, the castle was besieged and captured in 1590 by James Gordon, a kinsman of the Earl of Huntly. This was a result of a quarrel between John Grant, tutor of the young laird Patrick Grant, and the latter’s mother.
A pretty, young woman dressed in crinoline haunts the Pink Tower, one of the rooms of the castle. She is a benign presence and never did any harm to the several visitors who saw her. A green lady haunts the dinning room. Nobody knows exactly who she is, but she has been seen many times.
A man haunts the castle too. He is supposed to be General James Grant, a member of the family who died in 1806. He comes back every night, riding a magnificent white horse, as if even in death he wishes to keep a proprietary eye on his home and land. Sometimes, he wanders in the castle, and goes to the storeroom that used to be the dungeon.
The fourth ghost is a sad figure of a young woman often seen crossing the Old Avon Bridge, to the letterbox. Unable to accept rejection by the man she loved, she died of sadness wasting away writing to him everyday. In death, she continued to post new letters to her beloved. The workers employed in the construction of a new bridge, just beside the old one, had seen her very often.