Direct Mail Testing Unlearned: Myths, Methods, and What Actually Works

I’m Brad Kugler. I’m the CEO and co-founder of DM2O and I’ve been doing this for eight

years and I’m just finally beginning to learn this industry. I’m kidding. I actually think I know something about it

at this point. But I’m going to defer to to my uh trusted panelists here are way

more experienced than I am. Morgan, why don’t you start and and introduce yourself and tell us what you do here?

Sure. I’m Morgan DiGiorgio and I’m the CRO for Direct Mail 2.0 and who’s Mailing

What and we are marketing experts that specialize in omni channel marketing and the improvement of direct mail

campaigns. Brad is being very modest here today and uh through our 15

technology platform competitive intelligence database and newly released into public beta DM2O.AI

platform that is free right now for a limited time. So if you haven’t tested that out yet, please do. Um, I’ll keep

it short and sweet. Looking forward to this webinar today and hearing from our stellar panelists.

All right, Eamon, over to you. Tell us who you are and what you do. And I think I got your name right. So, uh,

yeah, nailed it. Eamon Barisone. So, I’m a principal solutions engineer at Lob and um, I’ve been in the direct mail

space for about 5 years and testing is very close to my heart. My background is in online advertising for companies like

Amazon and um a lot of the learnings I had from testing in in digital I’ve been

very excited to bring into the world of direct mail. So yeah, looking forward to chatting today.

Thank you, Eamon. And and John, tell us who you are and then I’ll I’ll I’ll do a

proper intro to what’s going to happen next after you quickly introduce yourself. Okay. Well, I’ve been a direct mail

consultant for well since 1981, I guess. And uh primarily worked with cataloges

in my career, about 80 different cataloges. And uh over the course of that time, um I’ve had the privilege of

deciding which customers of some of the greatest cataloges in America get the

last catalog. That’s how I explained it to Dick Cabella. And so I’ve pioneered

machine learning and which is kind of like testing on steroids. And I also

have have done a lot of what I called audits where I walk into a company and try to spot something that was um

interesting or testable or or uh or something that could change and have

seen uh I have I think 36 documented turnarounds uh doing that and I have

seven cases where uh we changed them the valuation of companies so much that they

could sell for millions or go public. Uh, and I’m sort of proud of that. I’ve

also been pestering Brad about testing and so he forced me to exam re-examine

my position. Thank Thank you, John, for that. Uh, let’s go straight to Stephen. Sorry for

our technical issues. It looks like you are uh able to get through the maze of complexities and were able to join us.

So, we’re just doing a quick intro. Steve, why don’t you tell us who you are and what you do? and I want to thank you

for showing up too. Oh, thank you for having me. I’m Steve Arensvich. I’m the VP of client strategy

for our direct marketing uh practice here at RR Donnelly. Uh I’ve been here about I’ve been in this contiguous

business through mergers and acquisitions about 31 years and um have a uh a fervent uh appreciation for

direct all things direct marketing and again testing is the bedrock of that. So I’m really excited to be uh speaking

with you uh about this today. Um we do a lot of testing. We are you know

aggressive proponents of test marketing for our for our cross-section of clients

and we apply I think every methodology known demand to try and accomplish uh

you know new winners for our clients. So uh so again it’s a very important component of the direct marketing

discipline. Thank you Steve. So, so just before I give the floor back to John to do a

quick presentation on what we’re all here for, I wanted to give two minutes on on why we’re here and why it came to

this. Uh Morgan and I and John have crossed paths many times over the last

few years and and he is absolutely fervent and a a crusader for the cause

of testing. Okay. Now, there’s nothing wrong with that, but I have to say his

his his affection for this subject and his knowledge had gotten me. He what he says

is basically right. Now, I don’t know that it applies on all circumstances or

all situations, but testing is important and a lot of people don’t do it. Now, we’re here with Quad, who isn’t here

yet, and Ronnelly and and Aean has talked about testing, but I have to say, a lot of the smaller marketers, it is

really not something that’s in their wheelhouse for money, for the size of campaigns, and budget-wise, whatever.

But, you know, John and I and others have had some spirited debates over the years. So, we got together and say,

well, what let’s write let’s write a white paper on how to properly do this. You know, he’s got the experience. We

contributed some things. We we we we we started this project six months ago. We

thought it would be a piece to have that anybody can use. This is not a proprietary piece. It’s a I would say

it’s bas basically John’s gift to the to the industry and and I applaud him for

that and I really want to give him kudos for taking the countless hours he did to

put this together and go back and forth with us. So this piece has not existed until John’s, you know, burning passion

to create it and and and I basically just wanted to enable him and set him loose to create this piece and give it

as a gift to anybody in the in the industry. The goal here is to discuss it, to debate it, and see if we can

maybe poke some holes into it or agree with what we want to agree. So, with that being said, this is about learning

and um I’m going to turn it over to John who’s got about a nine and a half minute presentation and uh I’ve told him I’ve

got my my hook to pull him off if it goes any longer than that and then we’re going to have a spirited debate. So,

John, go ahead, share your screen and we will sit here quietly. Okay. And I got my timer going. Uh okay.

Well, thank you for the wonderful introduction. And I do want to add that I have printed millions of cataloges with RR Donley, one of my favorite

printers in the world. I actually did a a motivational speech, my only one ever, and got a standing ovation from your

Salesforce and a lot of also a lot of stuff with Quad. Okay, so you can get the slides

and you can find out more about about what else I’ve done. So when I started this with Brad and and Morgan, I said,

“Okay, well, what is testing?” And then how does that connect up with the

scientific method? Why does rigorous testing matter? And if you’re a vendor, I hope some of the people on here, you

can put in the comments if you want, uh, if you’re an agency or if you’re a printer or something like that,

testing can get you bigger circulation, can get you longer term customer client

relationships. And I’ve seen it happen over and over and I’m very impressed with for example Gund who I think is a

male is a uh direct mail 2.0 client of yours. I think they they mentioned that

but they’re emphasizing that with their new c with their new clients and I’ve been you know fighting with them just as

much as with you about how that can that can open a client’s mind. And so this is

super important I think and if you’re a marketer and working at a company uh in

the marketing and I hope there’s some digital marketers here uh we’re going to talk a little bit about how testing can

get you bigger budgets and can really blow the doors off things. So I went to my library behind me and I pulled off

the some of the best books on direct marketing and what I found was that

there was no uh definition of testing. I I was a little shocked. Um, there was

plenty on how to test and what you could test, uh, but really never touched the

quality of the testing. So, I asked Chad GPT and, uh, it said, you know, it’s

evaluating something and to determine its quality. And that’s a perfectly valid test. And I’ve been guilty of of

calling out, you know, people say there’s no good tests on your whole, you know, all your case studies are are no

good. But the truth is testing is is used in a very generic sense. And so I’m

going to argue that that’s okay, but there’s better you can do and the more you understand what you’re trying to

accomplish, the the the bigger budgets you can generate. Um what the CEO wants

for five I was at a general agency for two years and we had five we won five out of five of the pitches which is was

unheard of. And in every case, the CEO afterwards would say, “I like that guy

who talked about accountable advertising. That’s what we want. We want predictable marketing.” And we

hear, you know, the buzzword, and this was probably 15 or 20 years ago, was predictive modeling. Okay? And, you

know, everybody wants to talk about prediction because you never bet enough on the winning horse, right? How much is

knowing the future for sure worth? If you think about the Back to the Future movie number two where he had the race

programs, made a fortune just knowing who would win. Uh, but in our world, in

the world of humans, science and engineering are able to figure out how to get a rocket to the moon. They’re fig

they’re able to figure out how to build houses that don’t collapse when we get a foot or two of snow, which we just did.

And the way they do it is by identifying, isolating and validating causal forces. Okay? And my my undergrad

was in a lot of philosophy of science and a lot of history of science. And it’s it’s almost comical to to read what

ear the early scientists thought was the cause of things. You know, Aristotle

thought things got happier as they more joyful. That attracted them to the earth. But you know, but we’ve what

we’ve done instead with the scientific method and and the and the scientific revolution is we’ve identified the the

actual forces which produce repeatable effects which is what your CFO wants. I

would say that most testing falls under the under the category of observation.

Okay. So how do we get from testing to observation? This was a a catalog that I

worked on long long ago, 1986, and we had great clients and I would hand this

out to friends who owned companies and I’d say, “Hey, you know, you might use

some of this kind of stuff.” And they’d say, “Why would I want a Pontiac hat?” And I’d say, “No, no, no. You you need

to you need to put you you can put your logo on there.” They said, “Oh, yeah. We buy that stuff.” Well, after it happened

about a dozen times, I I figured, hey, people are not getting what we’re doing. And so, I went to the creative director.

I said, ‘We need a test that isolates the your imprint here message. And so, if you look at these two cataloges, I

put this up at DAS, 100 people, most people don’t even get what’s different about the two two cataloges. But the

hypothesis was that people didn’t understand the logos. And so, we were going to tell them more clearly. And

what happened was we did a 600,000piece test, I think it was at Donley actually, and the the the catalog with the your

imprint here, the ugly catalog, did 40% better. Okay. And that gives us an

explanation of the causal force of being clear. It also disproves the the the

industry maxim that creative only has a 10% impact.

And it’s only by taking the time to to test your data. Your data will will

produce observation. But all data is historical. There is no data from the future. Never. No matter what people

talk about real- time data. It’s still it’s just newer fresher data. And so in

order to get to the future, we have to generate theory. Now what’s the what’s the cost of this? Well, Hammocker CFO,

may they rest in peace, said, “I don’t like testing because if A beats B, why did we even do B, right? Why did we do

300,000 cataloges with the with the company logos on them? It eroded our profits.” And it’s true, but the problem

is we didn’t know that we were eroding the profits. And so that one test took a

company from printing a million cataloges a year to printing six million cataloges while I was there. Okay? Okay.

And ended up selling it for millions. And so, and there’s also a destructive cost of not testing. JC Penney lost

five, four, I’m sorry, billion dollars of company valuation. Nordstrom’s quit mailing coupons and and the foot traffic

dried up. They could have tested. They could have said, “Let’s try a few stores, but no, they just stopped it

all.” Jill wanted to go fully digital, so they stopped their catalog. And the president got on the stock call and

said, “You know, it was a big mistake. we went too fast, too soon, and she got fired. Musician’s friend, while we were

working with him said, “Why are we mailing all these cataloges?” And we said, “Well, let’s find out.” And so, we

developed a holdout test where we didn’t mail 30,000 of the of the roughly half a

million customers. And what we found was for every dollar that they put into into

printing a catalog, they got $3 of bottomline ebida profit. And that kept

them mailing through the through some really tumultuous economic financial

times. So prediction, which is what the CEO wants. It’s it’s

invaluable, is testing. The gold standard testing assures prediction. Now, I’m not saying there isn’t good

valid testing in all kinds. You know, I’ve done it with a one-man show and I explained his market to him and he and

he had the, you know, it revolutionizes business. But if you talk to your clients about prediction, it develops

their long range vision. It can grow your budget with them. It can grow your circulation if you’re a printer. And if

you’re a marketer in a company, it can get you bigger budgets. And with permission of Brad, I said,

“Okay, if you’re interested in me coming into your company and looking around, we’ll have a special offer for you.

There’s a QR code. You can reach out to me at LinkedIn.” Okay, I beat it. 9 minutes. Fantastic, John. Thank you very

much. It was well prepared. And I do want to say, well, we’re going to we’re going to put up everybody’s contact

information at the end with some QR codes so that you can contact these people with specific questions. Uh if

you guys do have a question or anything, rather than put it in the chat because there’s a lot of people just introducing

themselves, put it in the Q&A so that I can I can disperse these uh equally to

our finely esteemed panelist. Uh in the meantime, I want to get started and ask John a simple question since he probably

put in more work on this than anybody else. Why is this a passion to you? What what

what drives you to continue to beat the drum for this for 30, 40 years? What

what is it that drives this? Uh, you know, I went to a little Bible

college down the road called the University of Wisconsin and I was trying to prepare for ministry. So, I took

these courses in history of science and philosophy of science. I ended up getting an MA and my thesis is on

an actual scientific problem. And uh and and and a pastor friend told me I should

go into business instead of ministry because I liked ideas more than people. And yeah, and so I gave it a go and and

got fired. And uh then prayed for a miracle job and lo and behold, I became a marketing consultant. And my first one

of my first big projects, which is on the wall behind me, if I zoomed out a little bit, uh was a catalog. and I got

paired up with a seasoned uh list broker and he handed me six data cards and

said, “Try these and let me know what happens.” And I said, “Well, can you tell me anything about these lists?” And he said, “Don’t worry about it. We don’t

worry about we don’t worry about any of the extra stuff. We just worry about what worked and what didn’t.” Well,

I knew from my philos philosophical background that there’s more to it than that. that if you really want to

understand prediction, you really want to isolate the causal variables. And what was interesting was this was an a

computer supply catalog, which the category had just exploded from the time we started to the time we mailed it. And

there were like three before we started and there were like 60 by the time we got in the mail, believe it or not. And

so the lists what we identified was the computer lists failed because everybody

was renting those and getting delued. And the normal business lists like or normal, you know, like business week,

they didn’t work either because they were just too general. It was the middle ones where they’d have a a couple of,

you know, articles on computers. Those were the ones that worked. And that that

perspective philosophical or that observation drove that that company into

multi-million dollar growth. And so and it was like, hey, even without marketing

courses, you know, I can do this. I can test. I can figure it out. I can let the

market tell me what works. And uh I’ve never recovered from that and had some

ex explosive wins, you know, by finding one or two good ideas that we could test, verify,

and then roll out like crazy. And the the white paper includes several of those stories so I won’t duplicate.

Yeah. So hopefully people will take advantage and read it. But let me let me get some feedback and and I’ll I’ll

start with uh Steve here. Is there an experience that you would

point to or a short story where uh you did direct mail testing and the results

were memorable and it’s something you can share? I mean, if you can mention client names, great. If you can’t, then

you can maybe say an industry. Well, it’s it’s fascinating to me because again, I I’ve walked the same

very similar to path to John and his name is iconic in this business, but I I also graduated from school as a as a

philosophy major and fell into marketing. Um trend. I see a trend there. Yeah. So, we so we definitely a little

further south than University of Michigan, Loyola, but uh but uh uh the the idea being that I think it was David

Oggov that says, you know, uh direct marketing is the science of advertising. So, so

any positive experience that the the one of the great passions that we have in direct marketing is the win and and

pointing to this EBID growth. This is, you know, the ability to to to affect that change for our clients is really

the the jazz that that good direct marketers bring to their clients table. In terms of, you know, it’s it’s the

it’s the un it’s it’s the surprise test result that really uh has moved the

needle for me personally uh uh the counterintuitive result that that drives

and I think John pointed to one, you know, uh pretty doesn’t always win. Okay. So, so oftent times we have uh

creative design agencies and CEOs and top marketers who want to put you know this brand identity and this va massive

color and this overwhelming uh promotional presentation versus what we

refer to as a blind treatment and in very and often most of the time the blind treatment wins. Um, interestingly

enough, now a recent example of a very large uh uh food service uh inhome food

service direct marketer. I’m not going to mention the name, but uh they they are out there and and they were uh uh

going through the the life cycle of creative um uh mandating more color in

the piece and more promotion, more branding, more identity in the piece. And um just for the sake of jazz, we

decided to take that very colorful uh uh self uh uh closedend mailer and put it

in a blind envelope with some very innocuous operational official looking

stuff on it and it generated a 35% lift. Okay, so well in advance covering the

cost of the envelope and well, you know, delivering that ebida that John spoke of to the bottom line. Again,

counterintuitive. Nobody under the sun would have said you take this fancy looking thing versus this, you know,

even if you hold it up to focus groups, they would all say, “Oh, I want the one that’s more informative.” But the one

that intrigues you, that gets you into the package, that causes you to have handle time, that’s the one that wins.

And and those are some of the some of the experiences that I think, you know, have followed us in in almost every

experience with client testing. Yeah, that makes sense. Totally. And and to Aean here, uh, you know, you have a

also a deep direct marketing background. Is there situations and and I’m hoping to develop a little controversy here and

and and stir the pot a little bit, so I admit my fallacy. What do you recommend

testing in all cases? And if not, what would be the cases that you would bypass the

initiative to test? So, I wish I had a hot take here and

could say here’s a time where you ought not test, but if you’re sending marketing mail, so of course like transactional mail, you know, sure, no

need to maybe no need to test there, but I really tried to rack my brain and tried to imagine a scenario in marketing

mail in which I wouldn’t recommend running some kind of test. But the thing is is you can always be testing

something. Now, maybe the volume of the campaign is going to be really small and you’re not going to have stat sig or

something like that. So, maybe you can’t always run a good test, but I always have a thought that as long as you’re

aware that the test you’re running, if you’re aware of the limitations of it, you can still take those learnings back

from it and use those as directional to form a hypothesis for a test at scale or

a test with, you know, good confidence in stat sig. So, um, I wish I could say

there’s a good time to not test, but I just think always test something. I just I can’t imagine not wanting to test at

least one little variable. So, so, and let’s take a typical

I’m a I’m a car dealer and I’m local or regional.

You know what? What would be in a test, you want a controlled environment, right? There’s one thing you want to test. It’s a creative or a headline or

something else. What would be an example of a size of a test? Oh, they’re just they’re selling Land Rovers to 5,000

people within a county. Yes, it’s a high ticket item, but do you have grounds if

you’re really only mailing to 5,000 qualified people to do a test? And if so, what is the test size? I think

yeah I mean I I don’t have a magic number for a test but I usually think like if you’re not sending 30k mailers

or something like that like that’s probably on my low end of where can I ensure that we really can

trust the learnings but okay I would that’s I was fishing for a number you know everybody’s and and

there’s nothing set in stone you know but I would imagine that a mailing that’s in this four digits

may not justify a test unless you’re doing innovate in innov uh what do I

call it you’re you’re you’re in you’re adding something to each mailing that may change the results and it’s

subsequent mailings iteration that’s what I’m talking iteration of that mailing you know

even on a small mailer though I might test something small and then

get the learnings back and know that this you know this might be anecdotal

this might not significant, but even that still might inform the way I think about my next large test. I don’t know

if John or Steve, you would agree with that or if that’s creating too much too much noise. But that’s my that’s still

my thought. I I agree. I think I think in reality, if if we’re honest, even that 5,000

piece mailer that doesn’t have a test cell against it is a test. Anything you do in direct marketing is a test. You’re

going to read those results. you’re going to try and analyze what that is and you’re going to try and figure out what to do next.

So, I think the point that was originally made is that if we apply the stabilizing factors, uh the scientific

elements, the the practice of testing in a disciplined way, you get more out of the test. But, uh just throwing throwing

your mail out there, you’re going to read those results and your test results are going to be what they are. So, I

think that that’s that’s an argument to say you must you are always testing. you must always test uh and and and even

though it’s difficult with sample size, I think stacking uh uh experiences

together, the the goal that we’re looking for in proper testing is is is uh is the gravity of the result to to

compare one pile versus another pile. And we need enough to say the more we

get, the more confident we are. But again to to Aean’s point even if we’re

using a directional or anecdotal type result that’s something we can build on

and uh it’s not the most desirable but we have to make lemonade whenever we can. Right. So yeah I I agree.

We’re definitely getting a lot of interaction from the audience and I I I don’t want to de

divest too much from that. So but I want to ask Morgan this this is where this is where we start kicking up some dust.

when you add a digital campaign, let’s face it, this is a digital ad world and

most marketing is done digitally at this point. So, if you’re doing a direct mail

campaign and many people do it with an omni channel focus where you have a digital component, how does that change

the testing? And I’m going to let Morgan do this because I know we have a lot of experience on this. We don’t have the

So, I’m going to rely on the testing guys and I’m going to rely on Morgan’s experience to try to create a little bit

of a debate here on this because you’re adding variables and I understand that. So, how do you do a legitimate test in

this situation? Go ahead, Morgan. Before I segue into that though, I do want to comment when you were, you know, asking

about the the size of the test because in our environment and a lot of instances, most people when they are

testing something, they are trying to test an incremental lift by adding

digital or adding omni channel on top of a direct mail campaign. And there’s always a little bit of push back there

because clients are pushing back because they don’t want to spend more. So, we know that testing is effective, but they

don’t want to spend an additional amount of budget in order to test whether or not this omni channel works. But if they

do, let’s say traditionally they’re mailing 500,000 pieces a month or a million pieces a month and they say,

“Okay, well, let’s test adding omni channel on top of it then, but we we’ll just test it on a a 5,000 piece mailer.”

How are you going to get uh that’s not a large enough cell in order to identify

the KPIs that you need to see if there really was an incremental lift? It you’re comparing apples to oranges. If

you’re typically mailing 500,000 pieces, how are you going to gauge a lift off of a 5,000 piece mailer, right? It just

doesn’t make any sense. So, I think that quantity is important and something that

needs to be uh correlated to what it is exactly you’re trying to extract from that test. And it has to be in alignment

with what your current marketing budget traditionally is. Right now adding omni channel makes testing more important and

a lot more complicated. Right? Because now channels are contaminating each other. So

you don’t have a control unless you pick the control as one specific thing. So yeah, in a mail only world you can test

version A versus B and you can measure the response. In an omni channel world now you have to think in terms of

assignment. Who was assigned to receive the omni channel? Who was not assigned

to receive the omni channel? And most importantly, did your control group accidentally get hit by retargeting,

email, SMS, platform, uh, optimization? Because most marketers are using

multiple channels and that that that really is complicating things. So, you need to isolate what it is you’re truly

testing. And with direct mail 2.0, know when we’re trying to see if omni channel

drove a lift in response. Uh most of the time I see people doing this wrong. Okay? And I will tell you one of the

most common mistakes. They’ve got two different lists that they’re mailing to and they’re not isolating where these

people are being driven to. We’ve got list A and we’ve got list B. And guess what? They are all being driven to the

same website where every single prospect is being retargeted with digital ads.

And then guess what? They are all getting the digital enhancement and the supposed control group you paid for

digital marketing for to see if it’s lifting that group’s response is getting minimized and diluted by displaying ads

to the mail group that you didn’t want to display ads to. So it’s impossible. You’d have to create a landing page or some other

correct. They need to be completely isolated. There is digital marketing out there where you can do specifically onetoone targeting by list upload or you

have to isolate specific landing pages and not be gauging any kind of response by anybody that went to the homepage. if

you are truly trying to identify if there was an incremental lift. Yeah, it it is difficult. And we we get

asked this all the time and I’ll tell you that we’ve had this happen recently where where a a reseller is like, “Well,

show me your case studies. Show me the proof of the improvement and and it’s so hard to isolate a test.” And even if a

if a client does this, we’re never sure that they even did it right. So it it’s

a it’s a very very complicated math problem with many many variables. So so

when asked for this type of thing, it’s difficult to give it especially if it’s been co-mingled or muddled with a pure

direct mail but still had retargeting going on on the website through some digital agency. You know it’s it it

casts a nullification on the whole process. So all right to me.

Yeah. Well, I I wanted to see if if some of our our more seasoned experts, Steve

and John, had a comment about the specific part of it about the omni channel component to testing.

You know, it’s it’s it’s the it’s the emerging category. I mean, we’re we’re we RD is is now selling digital and and

mail in combination. I think what Morgan said is is really taking that mail file

and then uploading it and dropping digital on top of that mail file, you know. So, so that direct connection

rather than generally mailing a file and then going separately and finding the

similar persona that that’s where variance comes in. So the so the first thing is that the second thing is

there’s some technology enablers like uh you know we’re we’re proponents of flow code and we’re proponents of some of the

other technologies where where that direct connection whether you responded by the banner ad or whether you

responded by the mail piece uh is controllable and that metric is now available to us. So that’s been a

tremendous boon to some of the complexities that you spoke of. But I think the bottom line is yes, the big

mistake that we see is sending people to the common website and and and in today’s age again that’s where

technology kicks in. It it’s not as expensive uh to to establish these these

uh landing environments that are unique and that correspond to the impetus that sent you there. Not only do they raise

response, but they also help you organize your your your management of the respondents and and testing

evaluation. Yeah. And that’s a good point. I I’m I’m going to get to you, John, but I want to I want to acknowledge some of the the

discourse that’s going on in the chat and Toby Daggerheart in particular, but

yes, QR codes, UTMs, specific landing pages, that is the way to do it, you

know, and and that requires a team effort to set up and make it very controlled environment, but it can be

done. I’m just saying it adds complexity. I’m not saying it’s impossible. It’s just next level to

really do a truly scientific control group test. So yeah, John, please

comment on omni channel in testing. There’s there’s a good case in the white paper, the lovesack case where the VP of

marketing called and said um you know we our catalog isn’t doing that well. uh

and I think he was calling to fire me and he said why why should we mail

something to our existing customers who already know what we do when our main

problem is brand recognition which is was well well documented by the

consultants that he’d hired and I said would you really like to know the answer he said what do you mean he was from

Mountain Dew you know he said how could you know the answer and I said well what we’ll do is not mail some of the people

you and we’ll we’ll mail the cataloges. We’ll keep track of who got it and who didn’t. And I’m I’m a huge proponent of

whole of hold out testing the mail which we can control and identifying for sure

the the causal forces. You know, there’s a priority of media that some media gets

you much better information. All all the mass media

throws itself away, but male can’t throw itself away. It sits on the table. You

have to make you have to have engagement. You have to make a decision. There’s uniqueness. And so and so if we

find the big things and settle those in mail and that’s in the Loveack case,

then you can use the omni channel to pour gasoline on it and you can make a

fortune. You know, Love Sack went from 75 million to 100 million in one year and they had and it was all same store

sales. I mean, how many retailers can do that? And it wasn’t a great economic period either, right? I mean, it it you

know, and later the founder uh Sean said, “Our marketing team, they figured

out how to allocate their resources and they also said to me privately, you

taught us so much.” Right? So, think about the hierarchy. Think about I know

it takes longer to find it out in mail. I know it’s putts, but if you could

really find out something for sure and get your theory down, get your

explanation, your theory down, get your CEO on board with it, then you can just

pour money into into social and into mass media and you can just make a fortune. uh which is if you get to the

question seven on the economics I have one point I want to make besides and then I it’s coming it’s coming we’re we’re

going to go on to question um and I’m going to I’m going to see if if Aean has anything for this are there any horror

stories in testing where things went wrong and what could what did you learn from it and and if one of you guys also

have something I don’t know who’s got a story for what so so if let me know

this the idea is that let’s teach somebody from our own mistakes. That’s what I’m hoping to get out here.

I I have one that comes to mind. Um very large homegoods company and they said,

“We want to do some testing within audiences.” So, they were basically

upselling current customers, right? Or or they customers they already had in their database. and they broke out their

this they broke out uh their audience group into four different subsets and said we’re going to send the same

campaign to these four different audience subsets and we’re going to try and get an understanding of which profile responds to this kind of offer

the best. Um but there was some problems with the setup. First is that breaking

their audience into four made each of those test groups too small to really

trust the learning. So, I guess going back to what we were talking about earlier, um, which we flagged and we

said, you know, I’m not sure this is a big enough test to draw a lot of strong conclusions from.

And then the second thing that they didn’t really account for is looking at

the looking at those at the folks that they selected within those audiences and understanding their past purchase

behavior. So, it wasn’t really a balanced um audience test against other factors

that also applied to the folks within those audience groups. They were testing against that one factor being the audience group they were a part of but

not accounting for other things. Um so they ran a test,

they drew some conclusions from it based on a small number of results, but those

results being much stronger in one of the four quadrants. Again, I kind of said to them, “Look, this might be good

to think about for future testing, but I don’t think this is strong enough to

draw this strong of a conclusion on.” And they said, “Oh, that’s okay.” They then took all their budget and pushed it

on that audience for a large campaign. The campaign did not perform well. And so the followup to that was now having

to go back and doing a similar campaign to the first one, but in a way that was going to be um that was gonna be

basically had to go back to the drawing board and do another audience test um and do it more intelligently. Um, but I

think that that budget that they wasted on that second send was truly unnecessary and they could have um

iterated on that first test to do a better second audience test and then poured money into the learnings from

that. Interesting. Can you can you divulge what kind of money was wasted by not doing that? Is it issue?

It was the the mail volume was in the millions for the the second send

and that was based off of um that was based off of testing

I I can’t remember exactly but I want to say the initial test was like 25,000 in

each audience group or something. So they just took a huge learning off of

relatively small initial. Yes. Comparative to the to to the whole

thing. All right. So I want to I want to just do some housekeeping here. We’ve got about 13 audience questions here. I want

to take the next seven or eight minutes to go through some of our questions and let you guys comment. So I’m going to

jump around a little. I want to open this one up and I’ll let Morgan

because she’s only spoke once. Is testing something that AI will replace

or make obsolete in the future? I want to see what Morgan says and then we’ll let Steve and John make quick comment on

that. No, I I don’t think that AI will replace testing, but it absolutely will make

testing more powerful, right? because we have access to this artificial

intelligence that will help us generate hypotheses faster. We can predict

segments, personalize creative or optimize offers which will help us uh

experience cost savings. But you’re still going to need realworld experiments to prove what actually

causes lift. Marketing is full of variables, okay? Seasonality, competition, platform changes, creative

fatigue, and and AI can help guide you. And it is such a lucrative tool. Um, and

you know, I again I I have to put a plug here. I I strongly encourage you to try

dm2.ai. We have ran almost two billion pieces of mail through our platform. We

have something that most people don’t have in our database and that is response rate data. Uh,

real attribution to actual mail pieces. Yeah. And which LLMs do not have.

Correct. and and tagged every element of a DM campaign from font to offer,

creative, you know, color, you name it. So, you know, when testing becomes a

barrier because of cost, but we know how important it is, leveraging AI to

mitigate some of those costs up front by ensuring that we’re leveraging the most appropriate mail piece by utilizing AI

or the most appropriate audiences by uh you know, grabbing data from CRM and

things like that is just so beneficial and helpful. But it will not truly uh

make testing obsolete. testing will still be the referee.

Thank you. Let let me give Steve some air time on that point. Do you have anything to comment on AI? I I just to just to quicken the you

know, yes, I agree everything that Morgan said, AI is a fabulous tool. It’s it’s across our business. The analytic

services are enhanced. The decision-making on what to test can be enabled. But the fundamentals of

testing, what did I do? What did I get for it? What do I do next? When applied to direct marketing, you know, that is

irreplaceable. and and I don’t see a near-term uh AI alternative to that.

No. Yeah. I mean, some of these kids coming up in their 20s that are relying on AI for companionship and everything else

they do in life, I I think they may differ. I think they may see AI as a

solution for almost anything. But, uh I I agree that without specific data on attribution for specific respondents and

creative and type of campaigns, that is not in these generalized LLMs. So yeah, I there’s nothing that beats uh

you know the the actual knowing the predictive nature of knowing your money on the line and forecasting for profit.

You have to have live experience there and I don’t see a uh virtual

alternative. Good. John, I did want to get to you. You wanted to speak on the economics of

sample size and testing. Am I correct? Well, let me get let me give you 90 seconds to

to opine on that. I I’ll put a a link in the comments. I watched a a great video from

Veritassium, who I like. It’s a math kind of channel on YouTube, and they were talking about whether you’re in a a

normal distribution world. Um, Mark Ritzen had a great article on Walmart

versus Target and said, you know, Walmart is concentrated on execution and

done a great job and grown like crazy. Um, if you’re in a world where you just

incrementally improve, then testing, just be consistent. Just keep at it. But

if you’re in a world uh of of uh of you know the big get bigger those kind of

worlds a test can catapult your business

out of its category and can multiply by tenfold in months your profitability and

in and marketing is at the center of that and AI is good at the the the

optimization, but most of what it finds as correlation is spurious and the the the the

anomaly is usually what gives you the breakthrough. And that’s what the human

will spot that AI will never spot. And I’ve gone headto-head against

against AI modelers for decades, believe it or not, and they don’t win. they they

they they only win on paper, but when you put it in the mail, they don’t. And that’s Morgan’s point, and I think that’s the greatest one. Um,

fair enough. Fair enough. All right. I’m going to jump to the last question here, which I’m trying to give some takeaways

to the to the audience and to history here, and I’ll let you each answer this as briefly as you can. What do people

seem to test most often that might not be the first thing to test? And I’m looking at the control, the audience,

the creative, the timing, the geography. What What is it that I guess people seem

to miss? And you you guys can each have a stab at it and I’ll just shut up and and and we’ll start with Aean and we’ll we’ll go around the group here.

Yeah, I’ll go quick. Um I do think testing creative is important. I don’t think like I don’t want to downplay the

importance of testing different creative factors against each other, different messages against each other. Um, but I

tend to see um, our customers bring that to the table as the first thing they’re thinking about testing. And I often

advise them to think about testing audience or testing offer and value prop. Um, maybe even before testing

content. Okay, great. Uh, Mo, you want to take a

stab? We’ll listen. I actually was going to say the same thing. I feel like people start with creative because it’s tangible, but it’s

not the first lever I pull. It’s usually audience and offer. You know, there’s nothing more important than who you’re

marketing to and then what you’re marketing to those people. But, um, I also think that it’s really important in

testing for people to pay attention to incrementality. So, you know, marketers are measuring

response, but response really isn’t the point. you can get response that maybe

would have happened anyway, you know, but like for us, the question is, you know, did the mail or the omni channel

enhancement create additional conversions that you wouldn’t otherw otherwise get, right? Um, and I I just

think that as opposed to looking at, hey, we got orders, uh, looking if you

got or created an incremental profit is extremely important to pay attention to.

All right, Steve, your turn. what what what should Yeah, I again I could totally concur. It’s it’s an absolute uh agreement, but

I think that clients spend a lot of time uh on the side of economizing what can I

do to take cost out and maintain the response and that overweighs their balanced testing. Uh you know,

oftentimes to what John said, the breakthrough comes from something that’s counterintuitive. Sometimes when we put more cost into the delivery of the

piece, we get a exponential response and we need to evaluate that as a future

investment. Uh and and testing in general is an investment category for our clients that that then turns it away

from being an expense to something that I need to nurture and get more profit and more out of. So, so testing too much

on the downside is is is uh is going to get you uh you know only limited uh uh

performance results. Great. Thank you, John. Tell me what

what are we testing? That’s the number one thing people love testing format which is relatively hard. There were comments in

the thing about what about letters versus postcards that you know that it’s hard because they’re not the same and

and you can do it. It’s just tricky. um or medium, you know, digital versus

male, that’s also difficult or design, you know, changing the pictures and the and the type face and that sort of

thing. And there’s a usually it’s low impact. I would recommend that all digital marketers marketers do a DM hold

out test. I would also recommend turning off your digital for one week, which would probably pay for your DM P test,

and see if anything happens. That’s what happened with Uber. They found out nothing happened, and it can often pay

for that test. And so that’s what that’s what I would suggest. Excellent. Thank you. All right. So, I’m

going to share everybody’s little um QR code while and we’re going to go through some audience questions here. We have

quite a few and I’m going to just start at the top. Uh and again, if you’ve got an answer for our audience here, please

just yell it out. Uh an anonymous attendee asked, “What books do you recommend for direct marketing?” Anybody

have a suggestion here? I had a slide, but it went by pretty fast. If you get the slides, there’s a a good stack of

them. E, everybody will everybody will get a copy of the slides. They’ll get a link to download the um the the the white paper

as well as a copy of the of the webinar, which will be on our website probably

tomorrow, I’m guessing. All right. Another anonymous asked, “What are your thoughts on direct response ads in

direct mail? Can it work? Why don’t we clarify direct response ads because I think people may have differing ideas on

those. Anybody want to comment on direct response ads in direct mail?

I guess be careful not to just take an ad and put it in an envelope because I’ve seen that fail colossally. You know, get

somebody that’s got direct mail, you know, like I worked with Oakley. They

didn’t have a response device on it in their mailing. you know, you just can’t

believe what people will do. So, get somebody with some knowledge to help you put that ad into an envelope or

whatever. Yeah. Clearly, there’s technique associated to the mail channel that’s different than the banner channel and

the email. So, each one of them has creative rules of the road that accentuate the results. So, so you can’t

just take a TV spot and convert it into a direct mail piece. You’ve got to, you know, take the offer and the intent of

the message and and they’ve got to work in tandem together. Yeah. All right.

There’s core elements uh each direct mail piece really should encompass. And

if you’re looking to see some successful direct mail campaigns and what’s working for specific verticals and industries,

you can always tap into the who’s mailing what database. Also,

the shameless plug exists here. All right. Uh next one. Lacy Lee, it’s

understandable that B2B and B2C audiences respond differently. Do you have any insights into uh I think it

means direct mail testing strategies you’ve seen succeed in B2B versus B TOC?

Does anyone have a comment on this? Yeah, at the risk of being obvious, um I

think with B2B, a lot of it is about testing the persona and testing the personas of the people that you’re

reaching out to within these businesses. Um, that’s what I’ve seen end up, you

know, drawing back good information for for our customers in B2B marketing is understanding the persona that they’re marketing to and testing different

personas within those businesses. So, so when marketing to a business, I’d like to share

I just want to clarify when marketing to a business, you still want to find an individual in that business, not just

send it blindly into an entire company. Correct? You need an identity within that business. Okay.

All right, John. Maybe not. Maybe not. I’ve had success in just doing title

slugs outperforming individual names, especially if the job is not core like

in that imprinted merchandise. Somebody’s running the the picnic this year that orders from you. The next year

it’s somebody different and so it gets thrown in the trash. And so picnic planner will do will outperform the

person. And for Adobe, for a division of Adobe, we actually tracked because they

had so many names, we actually tracked how many pieces of mail we put into a

building, an office building, regardless of business name. And we modeled on that

because in graphic arts, there’s an area of Chicago where the graphic artist

offices are. And in my agency, I had three different company names for my own

person because they wanted to do direct this and direct that. And so sometimes,

you know, the opposite works really, really well. And we went from mailing, we went from mailing half a million a

month to mailing a million a month. We raised the revenue, the profit per piece by 74%. And it it we went from a $25

million company to a $50 million anal annualization in four months by by

modeling on the business address, not on the company name or the person. Believe it or not, I can

Good, good, good, good info. This is great insight and and there’s a lot of uh wisdom in the amount of years you’ve

been doing this. So, I’m happy to hear that. All right, next one. Uh Barb

Fettino asks, “How do you persuade clients to test? it can be so costly. What do you suggest is the best way to

sell the idea to them? Who’s who’s had some experience with that?

Before we before we uh uh uh as we recommend a test, uh any credible direct

marketer would would would come to the table saying here’s what we expect or here’s what you’ll get if this test

wins. We have to set up the performance parameters before the test. So, so it becomes a a financial decision. is it

worth it for me to invest this amount into the test to gain this uh advantage or this upsell or this upside. So, it’s

really a financial discussion that drives that. And again, with with our large clients, uh we we uh uh we gain

from successful client testing. We get more volume. We get more work. So, we’re

in this together with our clients and we work uh collaboratively to incentivize

uh stable and productive testing. So uh even to the point of of participating in

those things uh when when it makes sense for us as a business. So uh that that uh

uh uh is is is an important component. Interesting. That’s great. Aean, did you have a comment on that?

No, honestly nothing nothing to add to that. That that was that covered everything I would say.

All right, Morgan. No, same thing. I mean, what’s the value that you’re going to receive once you identify the data needed from the

testing and how is that going to boost overall conversions from your direct mail marketing budget and what kind of

revenue does that drive for your organization? So, it’s an opportunity cost conversation. It’s just as Steve

said, it’s financial. Yeah, we work with break even analysis. We want to show that in a financial

numeric way to say absolutely this is what you stand to gain and and then it becomes a business decision rather than

and if it doesn’t come to fruition you you said you partner what would you eat some of the costs? Do you refund them?

Do you credit them? I mean what’s the mechanics there? Uh no I think that what we try and do is defay the some of the uh spiked cost in

testing. So we try and amortize the cost of testing over the extended jobs and and extended relationship. We like to

annualize with our clients the testing budget so that we’re able to apply

testing across the board and we’re we’re if it makes sense and we stand to gain

volume from it, we will invest in the somewhat of the cost of that test

differential. So if you know sometimes we we will offer a client a test uh

module at the same price as the roll out if we feel this is a well- constructed scientifically based odds on chance of

winning. So, so again really the results are we we share in the results you know and again if you’re if you’re world

class you’re getting three maybe four uh 30 maybe 40% of your tests to work. Uh

so so there’s a there there are downsides on testing but we have to manage that and the wins on an

annualized basis will definitely cover and much more uh uh uh delivery and

cover the cost of those lost tests. So so it’s it’s it’s looking at it from a financial perspective. Very insightful.

Okay, I’m going to have to pick and choose questions due to time because I like to keep these things to the required time, but I want people to get

their money’s worth. I’ve got another question from an anonymous attendee. What element in direct mail frequency,

size, etc. would you say would make a successful campaign with direct mail? No

digital. So, what’s most important, frequency or size? I’d love to hear any

opinions on this. Wow. I’m going to go frequency. What let’s say you guys because

repetitiveness in marketing works. Does anyone have a differing opinion?

I would agree. You know, it’s it’s a very broad-based question, but you know, we believe in like the rule of seven,

you know, where you need to penetrate a household or penetrate a prospect a number of times before they make an

optimal, you know, decision or optimized decision. But size is a factor. I I

don’t want to discount the relationship between mailbox busting and and presentation values as well. Uh but but

certainly I think if having to choose between one or the other, it’s it’s sort of a frequency multi-touch uh evaluation

that probably would win more often than not. Okay. I agree. It’s I’ve recommended folks

with a certain male volume budget divide that in three and hit people three times rather than hit everybody once.

Yes, I would agree. I would agree too. All right, wi with that I I don’t want to keep our our guest any longer. I have

more questions, but uh just as a as a closer, I want to thank the panelists.

It’s been very insightful. Uh I I really appreciated uh John sharing his his

decades of wisdom in the form of a white paper and allowing us to put our name on it. Again, all of this will be sent to

you via email in the next 24 hours. All of the information will be on our dm2.com website as well. Please do check

out dm2o.ai. It’s a free public beta till the end of the year. Test it out. We are giving it for free to help teach

this AI to be smarter about direct mail. And of course, you have everybody’s QR code. Feel free to ask them questions

directly. And uh again, thank everyone. I am I’m pleased that you all came. And

uh have a great and happy new year and holidays. All right, guys. Bye-bye.

Thank you. Thank you.