Measure an AI analyst against your own team, not against a benchmark. On every deal it reviews, count the calls your team kept, the ones your team changed, what your team raised that the memo missed, and whether each earlier correction was applied. Put those four counts in a one-page report every week and decide on the trend.
Updated 22 October 2026. Published by Titleman.
What "accuracy" means here
An AI analyst does not give a buy or sell verdict. Its memo is a set of calls: an assumption it took, a figure it could not verify, a risk it flagged, a comparable it used. Your team reads the memo and does one of three things with each call: keeps it, changes it, or adds one the memo missed.
That is what we count. Accuracy is the match between the analyst's calls and your team's decisions on the same deal. A generic score on someone else's deals tells you nothing about whether it reads a deal the way your team does.
The weekly scorecard
| Count | How it is counted | Where it comes from |
|---|---|---|
| Deals reviewed | Memos delivered this week | The memos |
| Calls kept | Calls your team left unchanged, out of all calls in the week's memos | The memo against your team's final version |
| Calls changed | Calls your team corrected, with the correction | Your replies and edits |
| Missed | Points your team raised that no memo had | Your replies and committee notes |
| Corrections applied | Corrections applied in a later memo, out of corrections sent | The later memos, each marked "as you asked" |
Every count can be traced back to a memo and a reply. If a number in the report cannot be traced, it does not go in the report.
How the review screen makes the count possible
Each finding in a review is its own item, with the quoted line, how it was checked, and a button to resolve it. Your team's choice on each finding is the raw material of the match.
On the test deal in this screen, a feasibility memo for a two-parcel site, the review raised 55 findings: 31 marked as needing attention and 24 verified. Each of the 31 is a call your team would keep, change or dismiss, and that choice is what the scorecard counts.
Week by week: what to look at
- Week 1. The red lines. Did every memo check them? Most changes this week are your thresholds and assumptions, because the analyst knows the market but not yet your firm. The count that matters most is corrections applied.
- Weeks 2 to 4. Calls changed should fall on the kinds of points you already corrected. New changes should be new kinds of points, not repeats.
- Month 2 and after. Missed points matter more than changed ones. Look at what your committee asked that no memo covered, and correct for that.
We do not print a target curve as a fact. The page how your analyst learns describes the typical path; your report shows your own.
The day-6 report in a probation
On a 7-day probation the first report comes on day 6 and has three lines:
- Reviewed: N deals.
- Match: your team kept X of the calls in those memos; here is where it did not.
- Corrections applied: M out of the corrections you sent.
N, X and M come from your week only. The decision on day 7 is made on them.
What not to measure
- Speed alone. A fast memo that your team rewrites is slower than no memo.
- Pages written. A longer memo is not a better one.
- Number of findings. Many findings can still miss the one your team cares about.
When a correction is not applied
A correction your team sent that does not show up in a later memo is not a learning-curve point. It is an error on our side, it is fixed, and the fix is in the next report. If corrections are applied and the match still does not move, that is a fact for your decision, and we would rather you see it in week one than in month three.
How to set it up
- Name one person who reads the memos and corrects them by reply.
- Keep your team's final version of each deal, so calls can be compared.
- Send the committee notes, so missed points are counted.
- Read the report each week, and decide on the trend, not on one memo.
FAQ
How do you measure the accuracy of an AI analyst in real estate?
Against your own team's decisions on your own deals. For each memo, count the calls your team kept, the ones it changed, what your team raised that the memo missed, and whether each earlier correction was applied. Report it every week.
Why not use a general accuracy benchmark?
A benchmark measures someone else's deals and someone else's judgment. The question your firm needs answered is whether the analyst reads a deal the way your team does, and only your decisions can answer it.
What is in the weekly report?
Deals reviewed, the share of calls your team kept, what it missed, and corrections applied out of corrections sent, with each count traceable to the memo and the reply it came from.
What should not be measured?
Speed alone, pages written and the number of findings. A long memo with many findings can still miss the one your team cares about.
What if the match does not improve?
Look at the corrections first. A correction that was not applied next time is an error on our side and is fixed. If the corrections are applied and the match still does not move, that is a fact for the decision at the end of the probation.
Read next: hiring an AI analyst for real estate: the first 7 days, how an AI underwriting analyst learns your firm's style and AI agents vs software for CRE firms. To get a week of counts on your own deals, request a demo.