Karori Cemetery was established in 1891 to replace the severely overcrowded Bolton Street Cemetery. The first burial took place in August that year but it was another six months before regular burials began. In the interim the Wellington City Council, which was responsible for the cemetery, set about developing the necessary infrastructure.

Shelter was originally constructed as a shelter for mourners and was probably the second purpose-built structure at the cemetery; the first being the sexton's cottage, long since replaced. The proposal to build a shelter was first debated by the Council's Cemetery Committee in August 1891 but a decision was deferred. The City Surveyor, George Wiltshire (1846-1905), submitted a design for the building, and estimated the cost at £90 to £100. The lower of the two tender prices was £119.19.6, from S. Hemingway, and he was awarded the contract.

Built for the use of all, it was described by the sexton as having a tower with louvres and it came in for heavy criticism for not being able to keep out the weather. In early 1892 the shelter was enclosed with windows and a door. By at least 1904 the building was also known as the mortuary chapel.

Initially the shelter was used by most denominations, but by the 1950s most burials took place in the far-flung corners of the cemetery, so the building became almost exclusively used by the Jewish community, which retained its burial section nearby. The building soon became named, informally, the Jewish Chapel. This association with the Jewish community was reinforced during this period when the Jewish community took responsibility for the building's maintenance, in return for a lease on the building. It became known as the Jewish Prayer House. The building narrowly escaped demolition in 1965, and in 1967 the Jewish community paid for it to be upgraded.

In 1977 the Jewish community moved to a new building at Makara cemetery and the building's condition deteriorated. Its restoration was supported by Ian McCutcheon, director of Wellington City Council's Parks and Recreation department, at a cost of $3000. The work undertaken included upgrading the foundations, installing a concrete floor, and the construction of a new roof.

In more recent times the shelter has been subject to further restoration but is largely unused at present. It is planned (2003) to use it as an interpretation centre, as part of an initiative to transform the cemetery into an historical park.