What it is
ChattyTracker watches what AI chatbots say. You give it a list of questions and, optionally, some names or terms you care about. It asks several chatbots those questions, saves the answers, and repeats on a schedule. Then it shows what changed between runs, whether your terms were mentioned, and whether the mention was positive or negative. It is built as a multi-user web app with teams and roles.
How it works
- Sign in and create a project.
- Add the questions or phrases to track, and any terms to watch for.
- Pick which AI models to ask.
- Start a scan. You see an estimated cost first, and each scan uses credits.
- Answers from each model come back and are saved as a scan.
- Compare any scan with the previous one, see where your terms appear, and read the sentiment of each mention. A Playground page lets you try a one-off prompt and keeps the history.
What exists today
- Sign-in with roles, team invitations and password reset
- Projects, tracked phrases, watch terms and a credit balance
- Scan trigger, scan list and scan detail pages
- Scan-to-scan comparison
- Mention detection and a beta sentiment check
- A scheduler and background job queue
- Playground with saved history, per-team branding settings, and a dashboard
- Browser tests with screenshots, and server setup files for staging and production
What isn't there yet
- Scans against live AI models were marked "ready to test" but not confirmed working end to end
- Not deployed anywhere
- Sentiment is a beta feature
- No charts of change over time; comparisons are between two scans
- The project's status notes are optimistic, so the real state needs a fresh check
- The project folder has many progress notes to clean up
Under the hood
- Back end in Node.js with Express, PostgreSQL for data and Redis for the job queue
- Front end in Svelte with Vite and Tailwind
- All model calls go through OpenRouter, so one key reaches many chatbots
- Runs locally with Docker Compose; server configs are drafted for hosting