Is Automatic Data Population Really Saving Time?

[RCA]

You want the answer to why auto adjustments are allowed (by the GSEs):

"Francis Galton’s 1906 “ox-weighing” observation, now regarded as the classic early demonstration of the wisdom of crowds.

At a livestock fair in Plymouth, England, 787 people entered a contest to estimate the dressed weight of an ox. The actual weight was 1,198 pounds.

Galton initially emphasized the median estimate—what he called the “middlemost” estimate—which was 1,207 pounds, only nine pounds too high. A later calculation of the arithmetic mean of all the guesses produced approximately 1,197 pounds, just one pound below the actual weight.

The marbles-, jelly-beans-, or candies-in-a-jar experiment is a common modern classroom replication of Galton’s result, rather than usually being the original experiment.

The underlying explanation is that, when guesses are made independently, some people overestimate and others underestimate. Their errors partly cancel when averaged:





If the errors are not systematically biased in one direction, their average may be close to zero.

This phenomenon works best when the group has:

• reasonably independent judgments,
• diversity of information or perspectives,
• no strong common bias,
• and a sensible method of aggregation.

It can fail badly when people influence one another, share the same mistaken assumption, or all receive misleading information. The usual name to search for is “Galton’s ox experiment,” “Vox Populi,” or “the wisdom of crowds.”


it seems this is what automatic submission of appraisal adjustment for different features amounts to.

Yes, Bert—that is essentially the same statistical idea, provided the adjustment submissions are sufficiently independent and come from reasonably competent appraisers.

Suppose many appraisers independently estimate the market adjustment for a feature—for example:

• a second bathroom,
• an additional garage space,
• superior condition,
• a view,
• a larger lot,
• or a swimming pool.

Each submitted adjustment can be represented as:


where:

• θ is the underlying market-supported adjustment,
• bi is the appraiser’s systematic bias,
• ei is random estimation error.

Averaging the submissions gives:



With a large number of independent estimates, the random-error component eˉ may become small. That is the same error-cancellation mechanism behind Galton’s ox-weight experiment.

Why it could work well for appraisal adjustments​

Individual appraisers may possess fragments of useful information that are difficult to capture in one formal model:

• experience with buyer reactions,
• paired-sale observations,
• knowledge of local neighborhoods,
• prior appraisal assignments,
• agent interviews,
• sensitivity to feature interactions,
• and judgment about whether a feature is typical or unusual.

No single appraiser may estimate the adjustment precisely, but the collective estimate could be more stable than most individual estimates.

In effect, the system would treat appraisers as a distributed market-sensing network.

But a simple average would often be inadequate​

The principal problem is that appraisal adjustments are not generally universal constants. A bathroom adjustment might differ substantially according to:

• price range,
• neighborhood,
• property size,
• existing bathroom count,
• house age,
• buyer population,
• date of sale,
• and whether the market is supply-constrained.

Thus, the useful object would not be:

average bathroom adjustment
but something more like:



where A is the submitted adjustment and X contains the relevant property and market characteristics.

For example:



This is closer to a conditional crowd estimate than a raw crowd average.

Independence is the critical issue​

The wisdom-of-crowds effect weakens if appraisers are not independently estimating the adjustment.

For example, appraisers may all be influenced by:

• the same lender guidance,
• the same appraisal software defaults,
• the same published adjustment tables,
• the same prior reports,
• the same MLS conventions,
• or common industry rules of thumb.

In that case, the errors are correlated. Averaging correlated errors does not make them disappear nearly as quickly.

If everyone uses the traditional “$10,000 per bathroom” rule, 5,000 submissions do not represent 5,000 independent estimates. They may represent one assumption copied 5,000 times.

Selection bias would also matter​

Submitted adjustments would not be a random sample of market judgments. They would arise only when:

• an appraiser chose a particular comparable,
• the comparable differed on that feature,
• the appraiser believed an adjustment was necessary,
• and the reporting system required or permitted the adjustment.

This could produce serious selection effects.

For example, a pool adjustment might be submitted primarily in assignments where pools are especially important. The average submitted adjustment could therefore overstate the pool’s effect in the broader market.

The adjustment direction must be standardized​

Another complication is sign convention.

If the subject has a feature and the comparable does not, the adjustment may have one sign; if the comparable has the feature and the subject does not, it has the opposite sign.

The system would need to convert all submissions into a common interpretation, such as:

Estimated marginal contribution of the feature to the comparable property’s market value.

Otherwise, positive and negative adjustments might cancel mechanically even when the appraisers agree perfectly.

Percentage adjustments may aggregate better than dollar adjustments​

Raw dollar adjustments are often not comparable across price levels.

A $25,000 view adjustment has a very different meaning in:

• a $400,000 market,
• a $1 million market,
• and a $5 million market.

It may be preferable to collect both:

Dollar adjustment
and



Even percentage adjustments are not necessarily stable, but they can improve comparability.

Medians and trimmed means may be better than ordinary averages​

Appraisal submissions could contain:

• typographical errors,
• unusual luxury properties,
• inexperienced judgments,
• software-generated defaults,
• or extreme but legitimate market conditions.

Therefore, the ordinary mean could be unstable.

A better aggregation system might report:

• median,
• trimmed mean,
• interquartile range,
• number of observations,
• standard deviation,
• geographic coverage,
• time period,
• and confidence or reliability score.

Expertise weighting is possible, but potentially dangerous
One could weight submissions according to demonstrated reliability:



Weights might reflect:

• sample support supplied by the appraiser,
• consistency with later sale evidence,
• geographic experience,
• appraisal review outcomes,
• or historical predictive accuracy.

However, weighting by credentials, years of experience, or appraisal volume alone might not improve accuracy. A highly experienced appraiser may simply have a more firmly established bias.

The best weighting criterion would be empirical forecasting or validation performance, not professional status.

There is an important difference from Galton’s experiment​

In Galton’s experiment, there was a clearly observable final truth: the ox had an actual measured weight.

For many appraisal adjustments, there is no directly observed final answer. The market contribution of a fireplace or view is latent. It must itself be inferred from transactions.

Therefore, averaging appraisal adjustments can produce a stable consensus, but consensus is not automatically the same as market truth.

A crowd can agree closely and still be wrong.

That distinction is central:



The submissions need to be validated against market evidence.

The strongest version would combine crowd estimates with transaction models​

A powerful system could combine:

1. Appraiser-submitted adjustments
2. Hedonic or nonparametric estimates from sales
3. Repeated-sales or matched-pair evidence
4. Broker or buyer survey evidence
5. Out-of-sample validation

The appraiser submissions could serve as prior information, while the transaction data update those beliefs.

In Bayesian form:



That would be more defensible than either pure machine estimation or pure professional consensus.

 

[RCA]

You want the answer to why auto adjustments are allowed (by the GSEs):

"Francis Galton’s 1906 “ox-weighing” observation, now regarded as the classic early demonstration of the wisdom of crowds.

At a livestock fair in Plymouth, England, 787 people entered a contest to estimate the dressed weight of an ox. The actual weight was 1,198 pounds.

Galton initially emphasized the median estimate—what he called the “middlemost” estimate—which was 1,207 pounds, only nine pounds too high. A later calculation of the arithmetic mean of all the guesses produced approximately 1,197 pounds, just one pound below the ...

Automatic collection of appraisal adjustments could indeed become a practical wisdom-of-appraisers system. It might reveal useful central tendencies, local norms, dispersion, and uncertainty that no individual appraiser can observe alone.

But it works only if the system:

• preserves independence, (It doesn't really)
• records detailed context, (It doesn't)
• standardizes adjustment meaning, (It doesn't)
• detects copied or default values, (It doesn't)
• uses robust statistics, (It doesn't)
• separates market segments, (It doesn't necessarily)
• and validates the aggregated estimates against actual transaction behavior. (It doesn't)

So, clearly the GSEs have been allowing this practice. And we ALL KNOW how little they think of appraisal accuracy. This is their way of saying "Crowd Intelligence" trumps individual appraiser valuation.

- The Sorry State of Affairs. Yes!!

So how much are GSE Appraisals Worth? They are GARBAGE.

Aivre follows the same practice - in even more ways.

 

[RCA]

MLS is a sales tool for RE agents, so the MLS does not have a vested interest in appraisal-type accuracy.

The RE agent ineptness and unreliability, which is the MLS listing service, is a form of job security for us lol. The fact that churn-and-burn appraisers might use AI to populate and then those appraisers will not take the time to verify or check themselves - it might or might not catch up with them, but at some point it could..

The most important data comes from the assessor's office. And, in some counties and cities, they are not that reliable.

Data	Primary source / responsible party	Who is responsible for accuracy?
Square footage	County assessor; MLS listing	Assessor for its records; listing broker/agent for MLS
Room counts	County assessor; MLS listing	Assessor for its records; listing broker/agent for MLS
Lot size	County assessor / recorded documents; MLS	Assessor/recording records for official data; broker/agent for MLS
Sale price	Recorded transaction / assessor records; MLS	Recording/title records and assessor for their respective records; MLS reporting broker for MLS
Ownership	County recorder/title records; assessor	Recorder/title system is the authoritative ownership chain; assessor maintains ownership for property-tax purposes
MLS presentation of any of these	MLS	Listing broker/agent is primarily responsible; MLS compliance can investigate and enforce its rules

Now ask if the appraiser has a duty to report incorrect square-footage to the Assessor and MLS. That answer is no.

Very few appraisers can accurately draw floor plans or report square footage. From responses on this forum, it should be clear that a sizeable number of appraisers could really care less about accuracy to the 1/10th of a square foot or thereabouts. They simply state so in their posts. A few do state that they report in accordance with ANSI standards.

I did a lot of appraisals between 2002-2008—and saw many other appraisals and sketches. In fact, the worst ones I ever saw were from MAIs; they often relied on the tax roll and created diagrams that fit the measurements and square footage. If they had a college degree, you could bet that their measurements were most likely never different from assessor calculations - and yet without any other basis. They never even did measurements (although they would lie that they did!). You could tell by looking at the diagrams that they virtually never matched the building footprint. The reason was that if the square footage values differed, there was a good chance they would have to argue with a reviewer, and that could lead to other problems. They were told what to do by their supervisors, ... who might eventually regret their roles. - So, the "system" is often the main culprit. The 'System' is rigged to a certain behavior that prioritizes the system's speed and efficiency with respect to relatively short-term goals of business survival.

Can anyone really trust the true validity of an appraisal? In the last analysis, on second thought, I think the answer is no. People want to trust the appraisal to get things done. That is just expediency. So, they live with the doubt. And the corrupt system we have thrives on that desire for expediency, at least short-range expediency. But over time, the bias towards higher prices (at least in the case of residential transactions) gums the system, making it less efficient.

The question is: What would the world look like if all appraisals were really fairly accurate estimates of Market Value? IMO, transactions would happen much faster, the economy would turn faster, and things would run smoother. But that is hard to prove.

 

[Zoe]

There is a difference between programs that autopopulate data, such as Spark or a data source, and AI programs such as AIVRE or AppraisalPilot.

In 2.6, I did not use auto-population because it did not seem to save that much time, and if it saved 15 minutes, well, I can spend another 15 minutes on an appraisal. Inputting the data is not robotic; I was analyzing and thinking about it. However, 3.6 has so many data fields and lacks the ability to transfer comps in the fields; I will plan to use a program like DataSource to fill in the data, and then I will edit, verify, etc. Assuming I stay in business lol.

The AI programs are different - though they also auto-populate (I assume), what they also do is , if prompted, write narrative, pick comps, analyze, assign condition ratings from the photos, etc. I went on the AppraisalPilot site, and they said the AI will even "analyze" the sales contract! Anyone who sends a confidential sale contract to AI needs to do something else. The problem with AI is that it is open souce -htf else could these startups all get hold of it so fast - which means once the data is out there from us, bye bye any control over it, who it is shared with etc.

The AI provides, imo, significant appraisal assistance, and it should be disclosed by the appraiser if they used it as such, but I bet they won't. And the GSEs, as far as I know, do not have any plans to monitor specifically for that.

IDK how much time it saves - if a churn-and-burn appraiser briefly reads it and signs, it will save time. If the appraiser verifies, picks their own comps, edits, etc it will not save much time, if any.

Yeah, there is security issues but why can't a company like alamode or click forms or ACI or whoever offer the same thing included in the form software? You know their software programmers are every bit as skilled as any of these people like Aivre or AppraisalPilot or datasource or spark.

It seems with the market share they already have, it would be easy to do anything another one of these companies offer.