Let’s see if this is a familiar story:
You are in the process of looking for a new marketing services provider so naturally, you send out an RFP to or request information from, a few agencies. One of the biggest things you are looking for is an agency that has strong experience with the channel(s) you are trying to enhance. So you ask for 3 relevant case studies as a part of the proposal. You select an agency that had really strong case studies and a proposal that wows you and then 3-6 months into the relationship with the new agency you realize you made a mistake. Timelines are slipping, deliverables don’t have the kind of strategic sharpness that you saw in the pitch, even worse the new solution has launched but isn’t generating results.
If you have ever experienced this then you know there is truth to this. It generally doesn’t matter what an agency has done in the past for someone else. Someone else is not YOU. Of course, it is important to make sure that your agency has a base level of competence in the subject matter areas you are engaging them for and case studies are a window into the capabilities of the agency. But here is the difficult truth: that is not a selection criterion, it is table stakes.
Here are a few reasons why case studies do not make good agency evaluation materials:
1. Case studies can be exaggerated and at their worst can be just a type of clickbait. How do you validate their accuracy, authenticity and real ROI? In many cases, you can’t.
2. They are inherently artifacts of the past. Agencies undergo constant change, staff turnover, and evolving areas of focus and so does the marketing landscape. Case studies speak to what the agency used to do, not what they are capable of right now.
3. They are not about YOU and your unique challenges! They may be somewhat similar, but similar is not the same.
So yeah there are some problems with case studies but what is a better way to evaluate an agency’s effectiveness?
1. Find out who is actually going to be delivering the work on a day-to-day basis. At big agencies that is the senior, mid-level and junior staff – usually director level and down. At small agencies, the executive team is often directly involved with ongoing work product delivery. Find out what the case is and ask yourself do they get your business? Are the subject matter experts (whether it’s a small agency executive or a big agency intern) capable of articulating you’re positioning within their unique medium? These are the people who will have the greatest impact on your business and especially when it comes to big agencies are often NOT the people who are pitching you. Agencies after all, are just a composite of people with varying levels of talent and experience.
2. In the pitch does the agency only show you bright shiny objects or stay in the philosophical realm? Or can they translate big ideas into clear tactical executions?
3. Stop doing massive high-risk RFPs and sticking yourself long term with an agency that isn’t right for you. A/B test your agencies! For example. If you are looking for a new search marketing agency give each of your top 2 agency choices a campaign (paying work) for 90 days. The partner that is the right fit for you will very clearly rise to the top and you will have real data and a track record to inform that decision of who to partner with long term. If you are looking for a creative agency, paid spec work is a fair and much more accurate way to get campaign solutions that will actually work in the real world versus shiny ideas from pitches that very rarely make it to market in their original form.
4. Ask for case studies. Wait what? Yes, you read that right, continue asking for case studies but when you do, ask for the person who actually does the work i.e. the PPC strategist or UX Planner to explain the case study. Don’t focus on their presentation skills or how many times they say “um”. Focus on their command of the subject matter and the medium. Do they know it inside and out and are they passionate about it? Are they smart enough to understand your positioning and requirements? Or do they fumble through and struggle to relate it to your business? Now you are getting somewhere in understanding if the agency is really capable of delivering results for you.
So let’s clarify. Case studies are worthless IF you don’t understand how to evaluate them, place too much importance on them, or use them as actual selection criteria. They remain a critical “table stakes” source of base capability assessment and can serve an important role, just not the role that most people use them for.
It’s one of the reasons why my agency is a “Rescue Agency”. Because I see this happen every single day. Clients run into this situation a lot and need additional outside help to either get the struggling agency on track, end up shifting the work to another agency in their roster that was never well suited to do the work, or again find themselves needing to run another agency search.
What do you think about the role of case studies?