Apr 2026 – Aug 2026
Support analytics with LLMs
13.5 months of customer chats and 1,130 call transcripts, read with LLMs to show what customers need
- AI
- Data analysis
- Duration
- 4 months
- Company
- Boxette Group
- Role
- Data engineer and analyst
Overview
Boxette’s full customer-chat history, loaded into Postgres, with every recently active contact classified by LLMs. The results, together with AI-rated call transcripts, became reports and a recommendations deck for management.
Tools
- Node.js
- PostgreSQL
- Gemini
- Claude
- Python
What customers actually ask
Management did not know what customers actually write and call about, which questions a bot or an FAQ could take over, where the knowledge base had gaps, or how fast operators answered. The data sat in Respond.io, which has no bulk export of message text, and in the call-scoring database.
From the API to Postgres
Node.js extractors pull every Respond.io message per contact through the API, with cursor pagination and watermark-based incremental updates, into a dedicated Postgres schema. SQL steps link messages to conversations, split them into contact episodes and compute service levels. The contacts were then exported as compact transcript batches for the models.
Calibrated models, ranges where they disagree
Each contact was classified for topics, deflectability, unresolved issues, sentiment and bot failures, mostly with Gemini Flash, with Claude used for the reference sets. Every model was calibrated against a reference set, and metrics where the models disagree are reported as ranges. A separate five-step pipeline (Gemini 3.1 Pro with context caching, then a Claude Opus summary) compared AI-rated call transcripts with the support agent’s knowledge base to find missing topics. The results became a Georgian report, an Excel workbook, a follow-up list of 1,180 unresolved contacts with negative sentiment and a 19-page recommendations deck, partly produced by Claude agents I orchestrated.
Result
Loaded 622,079 messages, 148,129 conversations and 23,064 contacts covering 13.5 months, and analysed all 10,392 contacts active in the last six months, 8,524 of them through Gemini for $96.02 with 0 errors. Tracking came up for 40% of contacts, customs declarations for 27% and tariffs for 24%, and by a conservative estimate at least 56% raised issues a good FAQ or bot could resolve. Across 1,130 calls, only 28.4% of 1,910 topic mentions were fully covered by the knowledge base, and 33.1% not at all.