Not necessarily. They could have been built prior to zoning, Variances can be granted. And different reigns of royalty at the planning and building department can interpret things different ways. And sometimes there are details within the code that can turn what appears to be illegal into something that just requires an administrative permit and a nominal fee. etc., etc., etc., etc., etc., etc.
OK Mike, I will meet you 1/2 way. If the home was the Flintstones house and built before there were zoning laws, or in a county where banjo's play Deliverance in the background and they do not have or require permits, let alone can spell it, then you have point.
If a variance was granted, there would be a record and that too would then be a legal use, although it may not or not be rebuilt if destroyed.
In areas which have zoning laws, require permits and have records of permits, which is most areas, apparently not all but most, there are not may ways for the royalty to interpret a permit, either it is there or it is not.