Hiring for AI Features: 7 Red Flags and 8 Deliverables You Should Require
AI features can be genuinely transformative-better support experiences, smarter onboarding, faster internal workflows. But they can also become an expensive demo that never turns into a reliable product.
At Jensen Technologies, we’ve been building and maintaining web and mobile applications for many years. Recently, that increasingly includes AI components: search, summarisation, chat, recommendations, document automation, and “agent-like” workflows. The projects that succeed aren’t the ones with the flashiest promises-they’re the ones with clear requirements, measurable outcomes, and operational discipline.
Below are 7 red flags to watch for when hiring for AI features, followed by 8 concrete deliverables you can require in a statement of work so decisions are evidence-based (even if you’re not technical).
7 red flags when hiring for AI
- No definition of success. If you hear “it’ll get smarter over time” but there’s no KPI, baseline, or minimum acceptable performance, you’re buying hope.
- No evaluation plan. Credible teams talk about test datasets, metrics, and acceptance thresholds. Vague “we’ll test it” is not a plan.
- Hand-wavy privacy and data retention. If they can’t say what data is sent where, what is stored, for how long, and who can access it, risk accumulates fast.
- No fallback behaviour. AI will be uncertain sometimes. It can also fail (API outages, rate limits, unexpected inputs). A production feature needs a safe, predictable fallback.
- Latency and cost are afterthoughts. Response times and per-request costs can quietly make an AI feature unusable-especially at scale.
- Everything stays in prototype mode. If the approach never progresses beyond demos, there’s often no ownership of monitoring, deployment, or long-term maintenance.
- They can’t explain tradeoffs in plain language. You don’t need buzzwords-you need clear reasoning. A good partner can explain why they chose (for example) retrieval-augmented generation (RAG) vs fine-tuning vs a rules + AI hybrid.
8 concrete deliverables to require (and why they matter)
These deliverables turn “AI as a vibe” into an engineering project you can manage, compare, and ship.
- 1) Problem statement + definition of done.
Include the user journey, what the AI is responsible for, and what it is not. Add a measurable KPI such as resolution rate, cost per case, time saved, lead qualification rate, or deflection rate. - 2) Approach document (model + method).
A short write-up explaining the chosen technique: prompts only, RAG, fine-tuning, rules + AI, or a combination. It should name constraints such as languages, tone, and domain specificity. - 3) Evaluation plan with thresholds.
Specify which metrics matter (e.g., accuracy, citation correctness, hallucination rate, toxicity, or “must not answer” compliance) and what “good enough to launch” means. - 4) A small evaluation dataset you can own.
Even 50–200 representative cases is valuable. The key is that it’s based on your domain and can be reused to assess future changes. - 5) Sample evaluation report.
Ask for a first report early. It should show results, failure modes, and what will be improved. This is where hype meets reality-in a good way. - 6) Safety and guardrails specification.
Define boundaries: refusal rules, sensitive topics, prompt-injection handling, data-leak prevention, and what citations or sources are required. - 7) Privacy + data flow diagram.
This should clearly state what is logged, what is stored, where it lives (regions), retention periods, and access control. It should also cover whether prompts or documents are used to train third-party models. - 8) Production-readiness plan.
Monitoring, rate limiting, alerts, rollback strategy, and ownership of ongoing improvements. Include a plan for human review if the feature affects customers, compliance, or high-stakes decisions.
A simple rule: a good AI partner won’t resist these deliverables. They’ll welcome them-because this is how you build something dependable, maintainable, and safe.
How Jensen Technologies approaches AI features
When we build AI into web and mobile apps, we treat it like any other critical system: we define success, test it against real use cases, design robust fallbacks, and make cost/latency visible. We also focus heavily on product fit-many “AI problems” are solved better with improved UX, search, content structure, automation, or a smaller scoped model.
If you’re considering AI features for your website or app, get in touch with Jensen Technologies. We’re happy to talk through your idea, help you shape requirements, or support you from prototype to production-without the surprises.