Aylmer House is now known by the name of its original owner, the Reverend William Aylmer. Aylmer was the third son of Sir Fenton Aylmer of County Kildare and the third Anglican vicar to reside in Akaroa. He arrived in New Zealand in 1851 and was apparently so horrified by Lyttelton and Christchurch that he was on the point of returning to Ireland. However, he found Akaroa charming, and remained there until his death in 1893. He was vicar of St Peter's Church until his retirement in 1873. Aylmer was also elected to the first Provincial Council as one of Akaroa's two representatives in 1853.

The original part of the house is a low square single-storey building, with an encircling verandah. Built of timber, with a shingle roof, it was possibly designed and built by Samuel Farr (1827-1918), a colonial architect and builder, who worked in Akaroa for 12 years before moving to Christchurch and establishing himself as an architect there. Aylmer's house was typical in form of the simplified Georgian cottages built by a number of early New Zealand colonists, with its hipped roof and symmetrical elevations.

Around 1900 a two-storey wing was added by the then owner, Greacen Black. Black was a draper in Akaroa, opened a crayfish cannery, and then took up farming. The timber wing he added runs at right angles to the original house, has a gable roof and is illustrative of later colonial houses.

Aylmer House is still used as a family home and has spacious grounds. Probably the grandest house in Akaroa at the time it was built, both wings illustrates the early Pakeha architecture of New Zealand, and the evolving nature of family homes. Historically the house is associated with Aylmer, a prominent Akaroa colonist.