According to St. Peter's records, the worshippers have been meeting since the summer of 1852, when Joseph H. Dulles fathered a group of various denominations for worship on Sunday afternoons at Indian Rock at the shore. The first church was built in July 1869 at a cost of $4,000, only for a "great northeast gale" to demolish the structure in September of the same year.
However, the very next day the church trustees met and resolved to build a new church. This same church, built in English Gothic style, stands today, with the same stained glass Tiffany windows. The church has been in the National Register of Historic Places since 1982.
A preschool is also run by the church on the property. A rector's house on the property was converted into the schoolhouse in the 1970s. An annex completed in the late 2000s connects the church with the schoolhouse, and also provides office space. Sermon podcasts are available on the church's website.