Module 3, Lesson 5: How to Evaluate New AI Tools as They Appear
Listen to this lesson
Module 3, Lesson 5: How to Evaluate New AI Tools as They Appear
Estimated reading time: 7 minutes
New Tools Every Week
If it feels like a new AI tool launches every other day, that's because it does. Your LinkedIn feed is full of people telling you about the latest must-have AI app. Your inbox has newsletters ranking the "top 50 AI tools of the week."
Most of these tools won't matter to you. Some will. The skill isn't keeping up with every launch — it's knowing how to quickly evaluate whether a new tool is worth your time.
This lesson gives you a practical framework for doing exactly that.
The Evaluation Framework
When you hear about a new AI tool, run through these five questions before investing your time.
1. What problem does it solve — and do I have that problem?
This sounds obvious, but it's the step most people skip. A tool can be genuinely impressive and still be completely irrelevant to your work.
Before signing up for anything, ask yourself:
- What specific task does this tool help with?
- Is that a task I actually do?
- How am I currently handling that task?
- Am I unhappy with how I currently handle it?
If you can't clearly answer these questions, the tool probably isn't for you right now — no matter how impressive the demo looks.
Example: A tool that generates AI-powered data visualisations is remarkable if you regularly present data. If you don't, it doesn't matter how good it is.
2. Does it do something the big tools can't already do?
ChatGPT, Claude, Gemini, and Copilot are increasingly capable. Before adopting a new specialist tool, check whether one of the tools you already use can do the same thing well enough.
Many "new AI tools" are essentially wrappers — they take a general-purpose AI model, add a specific interface, and charge you for it. That's not always bad (sometimes a purpose-built interface is genuinely better), but it's worth knowing what you're paying for.
Ask yourself:
- Could I achieve this result with ChatGPT or Claude using the right prompt?
- Is this tool using the same underlying model as something I already have access to?
- What specifically does this tool add beyond what the general-purpose assistants offer?
3. What's the business model?
Understanding how a tool makes money tells you a lot about its sustainability and incentives.
Free with a paid tier: The most common model. Evaluate the free tier on its own merits — is it useful, or just a teaser? If you'd need to pay, is the price justified?
Free, no obvious revenue: Be cautious. The company may be subsidised by investors (which means it could change dramatically or shut down), or your data might be the product. Check the privacy policy.
Open source: Generally trustworthy and sustainable, but may require technical knowledge to use. Good for people who value transparency and control.
Enterprise-only: Not relevant for individual use, but worth knowing about if your organisation is evaluating tools.
Red flags:
- No clear privacy policy
- Requiring excessive permissions (access to all your files, contacts, etc.) for a simple task
- "Free forever" promises from tiny startups with no visible revenue
- Aggressive upselling within the free tier
4. What are real people saying (not the launch hype)?
Every AI tool launch comes with a wave of excitement. Founders post glowing threads. Influencers share breathless reviews. This tells you very little about whether the tool is actually useful in practice.
Better sources of honest evaluation:
- Reddit communities (like r/ChatGPT, r/artificial, r/productivity) — people there tend to be candid about what works and what doesn't
- YouTube reviews from independent creators — look for people who test tools practically, not just demonstrate the highlights
- Comments and replies on launch announcements — often more informative than the announcement itself
- Wait two weeks. The most reliable evaluation strategy is simply waiting. If a tool is genuinely useful, people will still be talking about it in a fortnight. If it was just hype, it'll be forgotten.
5. What are the data and privacy implications?
Every AI tool processes your input somehow. Before you paste your work into a new tool, understand:
- Where does your data go? Is it processed locally or sent to cloud servers?
- Is your data used for training? Many tools use your inputs to improve their models unless you opt out.
- What's the data retention policy? How long do they keep your data?
- Who can access it? Just the tool, or also third parties?
This matters more for some tasks than others. Asking an AI tool to explain a concept? Low risk. Pasting your company's financial data into a new tool you found on Twitter? Worth pausing to check the privacy policy first.
A Practical Checklist
When you encounter a new AI tool, score it quickly:
- I have a real problem this tool could solve
- It does something my current tools can't do (or does it meaningfully better)
- The business model makes sense and isn't a red flag
- Real users (not just the company) say it's useful
- The privacy policy is acceptable for my use case
- It's been around for at least a few weeks and people are still using it
If you can tick most of these boxes, it's worth trying. If not, move on. There'll be another tool next week.
Tools That Have Stood the Test of Time
For reference, here are some tools beyond the big four that have proven their value over time, as of early 2026:
- Perplexity AI — AI-powered search with citations
- Notion AI — AI writing and organisation within the Notion workspace
- Otter.ai — Meeting transcription and summaries
- Midjourney — Image generation (artistic quality)
- GitHub Copilot — Code assistance
- Grammarly — Writing assistance and grammar checking (AI-enhanced)
- Canva Magic Studio — AI features within Canva's design platform
- Descript — Audio/video editing with AI transcription
These have been around long enough, with large enough user bases, that they've demonstrated real staying power. They're not going anywhere soon.
The Meta-Skill
Ultimately, what this lesson teaches isn't about any specific tool. It's about developing judgment — the ability to quickly assess whether something new is worth your attention.
In a world of constant AI product launches, the most valuable skill isn't knowing every tool. It's knowing which ones to ignore.
Key Takeaways
- Most new AI tools won't be relevant to you — and that's fine. Focus on tools that solve problems you actually have.
- Check whether your existing tools can already do the job before adopting something new.
- Understand the business model and privacy implications before trusting a new tool with your data.
- Wait for real user reviews rather than relying on launch hype — give it a fortnight.
- The ability to evaluate tools quickly is more valuable than knowing every tool that exists.
Practical Exercise
Evaluate a new tool.
- Find a recently launched AI tool. You can look on:
- Product Hunt (producthunt.com) — filter by "Artificial Intelligence"
- Twitter/X — search for "new AI tool"
- Any AI newsletter you follow
- Run it through the five-question evaluation framework from this lesson:
- What problem does it solve, and do I have that problem?
- Does it do something my current tools can't?
- What's the business model?
- What are real people saying?
- What are the data/privacy implications?
- Write a brief evaluation (5–10 sentences) with your conclusion: would you try this tool, or pass? Why?
This exercise builds a skill you'll use for years. The specific tool doesn't matter — the evaluation habit does.
Knowledge Check
1. What should you check before assuming a new AI tool is genuinely useful?
- a) How many social media followers the company has
- b) Whether real users (not just the company) report finding it useful after the initial hype
- c) Whether it has a mobile app
- d) How much venture capital the company has raised
Answer: b) Whether real users report finding it useful — launch hype from the company and influencers tells you very little about practical usefulness.
2. Why should you check the business model of a new AI tool?
- a) To make sure the company is profitable
- b) To understand how they make money — because if there's no obvious revenue, your data may be the product, or the service may not last
- c) To decide whether to invest in the company
- d) Because free tools are always lower quality
Answer: b) Understanding the business model helps you assess sustainability and whether your data is being used in ways you're not comfortable with.
3. Before adopting a specialist AI tool, what's a good first step?
- a) Buy the most expensive plan immediately
- b) Check whether a tool you already use (like ChatGPT or Claude) can handle the same task well enough
- c) Read only the company's own marketing materials
- d) Wait until there's only one tool left in that category
Answer: b) Check whether your existing tools can already do the job — many specialist tools are wrappers around the same models you already have access to.
Visual overview