The Old Corner Bookstore.
Some have billed this as the first brick building in Boston, but that fact is in dispute. What is not in dispute is that this is a wonderful relic from years past.
And looking as it does today:
Google Maps search done by /u/MyApplePie
A stereogram of the Old Corner Bookstore taken in the 19th century
“Threatened with demolition in 1960, the building was “rescued” through a purchase by Historic Boston, Inc. for the sum of $100,000.Historic Boston is a not-for-profit preservation and real estate organization that rehabilitates historic and culturally significant properties in Boston’s neighborhoods so they are a usable part of the city’s present and future. The building is listed on the National Register of Historic Places and is Boston Landmark under the auspices of the Boston Landmarks Commission.” – Wikipedia
The Old Wolf has spoken.
Playing in the World Game 
I thought the oldest brick building was on Marshall St .occupied by William davenport the first town Cryer of boston
The James Blake House is the oldest building in Boston. Whether the brick building you mentioned was constructed before 1661 is uncertain. It is a very old building nonetheless.