Vendor evaluation

Vendor Evaluation

Overview

This skill covers how B2B marketers should evaluate, select, and manage vendors — including AI tools, martech platforms, creative agencies, PR firms, and intent data providers. It also addresses tech stack auditing and consolidation. All practices are sourced exclusively from Exit Five podcast guests across 9 episodes; no general best practices have been added.

---

AI Tool Evaluation

Start from the Specific Problem, Not the Tool

When selecting AI tools, work backwards from the specific problem you are trying to solve rather than choosing based on general capability or brand familiarity. Identify what a given tool excels at — not what it can do generally — then combine complementary tools for better output. For example, one tool may be excellent at following detailed instructions and using structured formats, while another excels at rendering output visually; combining both can produce better results than using either alone. Avoid the trap of using one tool for everything simply because it is familiar. (Source: Jess Lytle, Episode #319)

Require an AI Component in All New Tool Procurement

When evaluating your marketing tech stack, establish a corporate mandate that any new tool brought on must have some AI component built in. During procurement, require both a business case and a team need statement for each tool, and evaluate how it connects to or is additive to your existing stack. This prevents tool sprawl by forcing intentional decisions about what gets added. (Source: Sara Ajemian, Episode #288)

How to Run the Evaluation Process

(Note: this is contested — see Where Experts Disagree)

Two structurally different approaches have been recommended by guests:

Individual leader trial first: As a leader, personally sign up for free trials and evaluate new tools before sharing them with the team. This filters out the majority of tools that look good on the surface but underdeliver in practice. Only recommend tools you have personally tested, and frame recommendations as informative suggestions rather than mandates, allowing teams to opt in rather than forcing adoption. (Source: Kris Rudeegraap, Episode #159)

Formal cross-functional committee: Establish a formal process for employees to propose and evaluate new AI tools before adoption. Create a committee including IT/security and people team leaders to vet tools. Allow employees to pilot tools on free trials without ingesting proprietary company data, then bring promising tools to the committee for fast-track evaluation decisions. This balances experimentation with security and compliance guardrails. (Source: Bill Glenn, Episode #328)

See Where Experts Disagree for a full discussion of which approach to use and when.

---

Tech Stack Auditing and Consolidation

Audit and Consolidate to Free Budget for Campaign Spend

If you inherit a budget with excessive technology spend, conduct a martech audit to identify redundant, underutilized, or non-strategic tools. Consolidate vendors where possible and cut tools that do not directly support your company goals. This frees up annualized software costs that can be reallocated to campaign spend and GTM activities. Prioritize tools that directly enable your core marketing motions over nice-to-have platforms. (Source: Rowan Tonkin, Episode #197)

---

Evaluating Agencies

Creative Agencies: Evaluate by Portfolio, Not Promises

When shopping for a creative agency or designer, prioritize reviewing their actual work output — portfolio, case studies, and past projects. If you like their work, hire them and trust them to deliver in that style. If you do not like their work, find a different agency. Do not hire an agency whose portfolio does not match your vision and then try to direct them toward a different aesthetic; that mismatch will produce friction and mediocre results. (Source: Dave Gerhardt, Episode #153)

PR Agencies: Demand Specificity in the Pitch Meeting

When evaluating PR agencies, ask them to come with specific ideas that include named journalists and specific publications they plan to pitch. This reveals whether the agency has real relationships and a concrete strategy, versus generic pitches. Request this specificity during the initial pitch meeting to assess the quality of their network and strategic thinking. (Source: Priscilla Barolo, Episode #193)

---

Evaluating Intent Data Vendors

Intent data is a category where vendor quality varies dramatically and where the wrong choice wastes significant budget. Apply rigorous vetting before committing.

Ask Whether the Data Is Shared or Resold

Ask vendors directly: Do you share or resell this data to other vendors? If the answer is yes, the data is commoditized — your competitors using different vendors may be receiving identical signals under different brand names. (Source: Chris Rack, Episode #150)

Probe What Constitutes a "Signal"

Ask vendors: What constitutes a signal in your data? If the answer is "website visits" or "bitstream data," that is intent-to-learn, not intent-to-purchase. Better signals include reading case studies or comparison reports, visiting review sites, or other actions that indicate active evaluation. Most intent data conflates these two categories, making it unreliable for ABM targeting. Be skeptical of vendors claiming to have solved intent; search platforms are among the only channels that reliably capture true purchase intent. (Source: Chris Rack, Episode #150)

Trace the Data Back to Its Source

Ask three specific questions when evaluating intent data providers: (1) Where do you get your data? (2) Do you buy it or create it yourself? (3) If you buy it, who do you buy it from? Many providers purchase the same data from the same one or two sources, meaning competitors using different vendors receive identical signals. Providers unable to clearly answer these questions should be treated with skepticism. Proprietary or self-created signals offer competitive differentiation. (Source: Chris Rack, Episode #140)

---

Where Experts Disagree

Should AI/New Tool Evaluation Be Done by Individual Leaders Personally Trialing Tools, or Through a Formal Cross-Functional Committee Process?

Support summary: 1 vs 1

This is a genuine structural disagreement, not a matter of emphasis. The two approaches center authority in fundamentally different places — one in individual judgment, one in institutional process.

---

Position 1: Individual leader trial first (Kris Rudeegraap, CEO/Co-founder, Sendoso — Episode #159, July 2024)

Leaders should personally sign up for and trial new tools before recommending them to the team. This personal filtering process eliminates the vast majority of tools that look good on the surface but underdeliver. Recommendations should then be framed as suggestions, not mandates, allowing teams to opt in. Rudeegraap argued that leaders recommending only personally tested tools builds credibility with their teams.

---

Position 2: Formal cross-functional committee (Bill Glenn, CMO, CData Software — Episode #328, February 2026)

New tool adoption should go through a structured, formal process involving a cross-functional committee including IT/security and people team leaders. Employees can pilot tools on free trials without ingesting proprietary data, but promising tools must be brought to the committee for fast-track evaluation before full adoption. Glenn advocated for this structure specifically because individual experimentation without guardrails creates security and compliance risks — particularly when proprietary company data could be ingested by unapproved AI tools.

---

Context dependency: The individual-leader trial approach may be more appropriate for smaller companies or non-AI tools with lower security risk. The committee approach is explicitly designed for AI tools at companies with meaningful compliance and data security obligations. However, both guests are giving general advice about tool evaluation processes, and the approaches are structurally in conflict regardless of company size — one centers individual judgment, the other requires institutional approval.

Trend note: The more recent guest (Bill Glenn, 2026) advocates for the committee/governance approach, while the earlier guest (Kris Rudeegraap, 2024) advocates for individual leader trials. This may reflect growing security concerns around AI tool adoption specifically, which have intensified since mid-2024.

Why it matters: Choosing the wrong evaluation model can either slow down tool adoption with bureaucratic overhead or expose the company to data security and compliance risks. Take a deliberate stance on which tradeoff you are willing to accept before recommending an approach to a user.

---

What NOT To Do

• Do not choose AI tools based on general capability or brand familiarity. Work backwards from the specific problem you are trying to solve. (Source: Jess Lytle, Episode #319)
• Do not use one tool for everything just because it is familiar. Combine complementary tools when they produce better output together. (Source: Jess Lytle, Episode #319)
• Do not add new tools to the stack without a business case and a team need statement. Forcing intentional decisions prevents tool sprawl. (Source: Sara Ajemian, Episode #288)
• Do not hire a creative agency whose portfolio does not match your vision and then try to redirect them. That mismatch leads to friction and mediocre results. (Source: Dave Gerhardt, Episode #153)
• Do not accept generic PR agency pitches. If they cannot name specific journalists and publications in the pitch meeting, treat that as a red flag. (Source: Priscilla Barolo, Episode #193)
• Do not accept intent data vendor claims at face value. Ask specifically what constitutes a signal and whether the data is resold. Website visits and bitstream data are not purchase intent. (Source: Chris Rack, Episodes #150 and #140)
• Do not assume different intent data vendors are providing differentiated data. Many purchase from the same one or two underlying sources. (Source: Chris Rack, Episode #140)
• Do not allow employees to ingest proprietary company data into unapproved AI tools during evaluation pilots. (Source: Bill Glenn, Episode #328)
• Do not retain tools that are redundant, underutilized, or non-strategic. Consolidate and reallocate that budget to campaign spend. (Source: Rowan Tonkin, Episode #197)

---

Sources

Episode	Guest	Guest Context	Date
Episode #140	Chris Rack	—	2024-05-13
Episode #150	Chris Rack	—	2024-06-17
Episode #153	Dave Gerhardt	—	2024-06-27
Episode #159	Kris Rudeegraap	CEO/Co-founder, Sendoso	2024-07-18
Episode #193	Priscilla Barolo	—	2024-11-14
Episode #197	Rowan Tonkin	—	2024-11-28
Episode #288	Sara Ajemian	—	2025-10-06
Episode #319	Jess Lytle	—	2026-01-08
Episode #328	Bill Glenn	CMO, CData Software	2026-02-11