Baseline · v80.91
117 passed · 11 failed
Candidate · v90.84
108 passed · 20 failed
Change−0.0713 worse· 4 better · 111 same
| Case | Expected | v8 | v9 | Change | Judge note |
|---|---|---|---|---|---|
| What was Northwind's revenue last quarter? | Cites the Q3 filing | Pass0.96 | Fail0.41 | −0.55 | States a figure that is not in any source |
| Who leads procurement at Ardent Freight? | Names the VP from the CRM | Pass0.92 | Fail0.48 | −0.44 | Names a person from an older news item |
| Summarise the last three calls with Kestrel | Three dated call notes | Pass0.90 | Fail0.52 | −0.38 | Merges two calls into one |
| Does Lakeshore use a competitor today? | Says unknown, suggests asking | Pass0.88 | Fail0.55 | −0.33 | Guesses a vendor without evidence |
| List open opportunities over $50K | Four records from the CRM | Pass0.94 | Fail0.61 | −0.33 | Misses one record |
| When does the Arbor contract renew? | March 2027, from the contract | Pass0.97 | Fail0.66 | −0.31 | Gives the signing date instead |