Great points, Pete.
Yes, the interesting point is the legal nonconforming use has been in place and in use openly for at least 40 years when the zoning ordinance changed. It was 40 years ago the use was grandfathered. I know for a fact that all of the adjoining property owners have purchased and moved in within the last 40 years. So I believe what your leading to is:
Those surrounding owners had the onus of due diligence when they investigated the property at their individual time(s) of purchase. Such onus allows the LL to place some (if not a majority) of the blame at the neighbor's feet for failing to know that the property possessed the unique characteristics (a multifamily dwelling) that it had. The use was open and notorious. It is unfair, now, at this time, to gather up a disgruntled group and claim that the subj prop is imposing external obsolescence on the whole street. Is that your train of thought?
Further, since the use has been in place for decades, they can't seek relieve from the appeal board in regards to the use. What they might do is seek relief from the ad valorem tax assessor?
Am I in sync with you?