Forty-seven chat conversations with the Entrata Super Agent payments sub-agent, graded on the ELI+ rubric. This review covers what residents actually ask, how the agent performs, and where a Leasing Center overflow tier would plug in.
Residents are not mostly asking "what's my balance." Of ~31 genuine resident conversations, the top intents are "explain this charge" (10), "the portal won't let me pay" (8), and late fees (5). Simple balance and due-date lookups were almost entirely our own testers.
The AI is accurate but can't close. Functional accuracy averages 95, yet only 26% of conversations reach closure and 60% end with the resident disengaging. The agent is read-only by design: it cannot waive, fix, enroll, or reconcile a charge against the lease.
70% of conversations end with "contact the property office." No warm transfer, no ticket, no callback. That handoff point is exactly where Leasing Center overflow would sit.
It's chat-only and slow. p95 response time averages 11 seconds; 28 of 47 failed the 10-second bar. Nothing here is ready for a phone call, which is LC's channel.
Primary intent per conversation, genuine resident conversations only (n = 31). Sixteen internal test scripts ("What is my session ID?", repeated "Can I pay rent three months ahead?") are excluded here and shown in the appendix.
Residents arrive confused or upset about a specific charge, blocked by the portal, or wanting a fee removed. Those three buckets are roughly 75% of real traffic, and all three need either context the AI lacks (lease terms, why a charge posted, portal status) or authority it isn't given (waive, fix, enroll).
The use case we assumed drives volume, a quick balance or due-date lookup, did not show up on its own from a real resident in this sample.
| # | Intent | Convos | How residents phrase it | Avg score | Can the AI resolve it alone today? |
|---|---|---|---|---|---|
| 1 | Explain a charge / bill is higher / duplicate charge | 10 | "What is the $43 LockDIQ for on my ledger?" · "Why are the charges showing twice" · "Why do I have a $68 reimbursement fee" · "That's not the amount I signed my new lease for" · "They told me November was free" | 78 | Partially. Reads the ledger correctly but can't say why a charge posted or check it against the lease. 7 of 10 were score-capped; 8 of 10 ended in an office referral. |
| 2 | Portal / payment failure | 8 | "when i click Pay Now, the popup is blank" · "He won't let me pay my rent" · "payments unavailable at this time" · "Why is my split pay not working" · "It says failure in savings" | 84 | No. Zero visibility into portal status. Every one deferred to the office or "try again later." |
| 3 | Late fees | 5 | "When is a late fee assessed?" · "I would like to dispute my late fee charges" · "Can you eliminate late charges" · "I was charged a $50 fee because my bank made an error" | 92 | Half. Schedule and amount questions score 94–100. Dispute and waiver requests always route to the office. |
| 4 | How / where / with what do I pay | 4 | "Different methods to make payment" · "What was the card with which I made payments" · "Is there any way I can pay my rent thru here" | 78 | Yes for mechanics. One hallucinated payment-method fee (capped at 50). |
| 5 | Flex / Split Pay / payment plan / hardship | 3 | "Does this property still offer Flex rent" · "I lost my job and am unable to pay rent this month. Can we work something out?" | 89 | Read-only. Can report an existing plan; cannot enroll, modify, or counsel. Flex questions get a generic office referral. |
| 6 | Refund / deposit confirmation | 1 | "I'm not seeing the three letter code to initiate the refund of $75" | 98 | Yes (answered from payment history). |
| — | Balance / due date test runs | 4 | "How much do I owe this month?" · "When is my rent due?" | 92 | Yes, 1–2 turns. No real resident asked this on its own. |
| — | Policy questions test runs | 9 | "Can I pay rent three months ahead?" · "When is the second rent reminder sent?" | 69 | No. No policy knowledge; one run looped the same clarifying question 5×. |
| — | Off-topic | — | "When I am trying to finalize renewal" · "Español?" · leaking faucet | — | No renewals routing exists. Renewal question was deflected: "I can't confirm that detail here." |
Scores are ELI+ composite (50% functional, 50% realism) with hard caps when a required criterion fails.
| Composite score | 81 mean · 87 median |
| Functional accuracy | 95 · 2 hallucination flags |
| Realism (tone, listening, efficiency) | 89 |
| Reached closure (Phase 5/5) | 12 / 47 (26%) |
| Converted · Abandoned by resident | 16 (34%) · 28 (60%) |
| Ended with "contact the property office" | 33 / 47 (70%) |
| Real escalation / handoff language | 3 / 47 |
| Latency p50 · p95 · max | 8.6s · 11.1s · 20.6s |
| Failed ≤10s p95 bar | 28 / 47 |
| Turns per conversation | median 2 · max 12 |
| Agent LLM cost per conversation | $0.001 – $0.18 |
Required rubric criteria failed, by number of conversations.
Resident pushes back ("I already paid that," "that's not what I signed") → agent recites the ledger again → resident leaves. Three conversations were flagged as agent-caused stalls.
The AI's own guardrails make the case. Its rubric states it "never waives / refunds / reverses, approves a plan, enrolls." Every conversation that needs one of those ends at the same place: "I can give you the property office phone or address." That office referral is the overflow opportunity.
| Resident need | AI today | LC overflow agent could… | What the agent UI needs |
|---|---|---|---|
| Explain a charge against the lease | Reads ledger, can't explain or reconcile | Verify resident, read ledger and lease, explain, open a ticket if wrong | Resident verification · ledger view · lease summary (rent, concessions, fees) · escalation to site |
| Portal won't accept payment | "Try again later / call the office" | Walk through payment, confirm method on file, take payment by phone if permitted, log the portal issue | Payment-method view · known-issue status · support ticket |
| Late-fee dispute / waiver | Quotes the fee, refers out | Quote schedule, capture dispute, route to site with context, or waive within client-defined policy | Late-fee schedule · dispute form · client waiver rules (if delegated) |
| Flex / Split Pay / hardship | Reports existing plan only | Explain options at that property, route enrollment or hardship to site with notes | Property payment-program config · warm transfer / callback |
| Renewal questions | No routing — deflected | Answer renewal status, offer terms, schedule follow-up | Renewal offer data (net-new for LC) |
Small sample: 47 conversations, one day, ~5 client accounts, 16 of them internal test scripts. All files carry the scenario label payment_001_balance_and_late_fee because they were bulk-imported; the detected-topics field includes noise that isn't real. Latency is per agent turn and includes the nested payments sub-agent call. A handful of judge criteria returned unparseable responses.
Cap kinds: bot_tell = robotic phrasing / duplicate replies (cap 70) · payment_policy = same unresolved issue twice without escalation (cap 70) · critical = hallucinated fee or unbacked action claim (cap 50) · action_failure (cap 50). Phase: P5 = closed with next step; P4 = correct boundaries but no close.
| # | Intent | First resident question | Score | Cap | Phase | Outcome | p95 |
|---|