This Issue's TLDR...

• You don't need statistical significance (most of the time)
• You do need image clarity
• You might need AI ads that actually work

👉 Did someone forward you this newsletter? First of all, give them a crisp high five when you see them. Second, head over here to subscribe and read past issues.

​

SPONSOR TrueOps Why are you paying more than a 10% commission to your FBA reimbursements provider? Especially now that Amazon only reimburses based on manufacturing costs. Seriously, I want to know. Are you sticking with your provider because they claim to recover more $? Are you sticking with your provider because some well-known "Amazon expert" (that might not even sell on Amazon anymore) is always recommending that provider? Look, man, you do you. But, for me, I'm going to pocket the extra cash from only paying 10% vs 25%. ​ Pay Just 10% with TrueOps Today

If you don't spend hours on LinkedIn each day, you may have missed my post yesterday, in which I attempted to answer the question:

Does Optimum Nutrition's prominent "#1 Creatine Brand in the USA" badge actually improve purchase intent with shoppers?

My tiny, 5-respondent sample poll showed... it didn't. No impact.

Naturally, the comments section lit up.

My good buddy and newsletter homie ​Max​ (aka The ASIN Audit Guy) said:

"5" US Based respondents? Come on Jon Derkits

And, he was right to point that out!

He even went a step further and re-ran my PickFu poll with a larger sample (n=50) and produced the opposite result!

Outside of the question that some of us have around what Max will do to win an online argument (😉 Joshua Rawe), yesterday's events perfectly illuminated a core debate that dogs us Amazon nerds:

How much data is truly "enough" before you make a decisive move? When does waiting for "statistical significance" become an expensive exercise in missing opportunities?

The truth is, traditional A/B testing dogma, with its academic pursuit of 95% confidence, often gets it wrong for our world. We need speed. We need profit. And, what I'll argue today is that sometimes waiting for that perfect p-value means leaving serious money on the table.

Why Old-School A/B Testing Fails Us

Let's be clear: We're not running academic studies. We're running high-velocity businesses on the most competitive eCommerce platform on the planet. Our goal isn't just to prove a hypothesis; it's to make more money, faster. (At least, that's mine. YMMV.)

The obsession with "statistical significance" often demands massive sample sizes and painfully long test durations. Even Amazon's own "Manage Your Experiments" (MYE) tool, while powerful and increasingly vital, defaults to 8-10 week durations or waiting "to significance," which, for me, can still feel like an eternity when you're trying to outmaneuver competitors or capitalize on a fleeting trend.

I believe that, for tactical, everyday Amazon optimizations -- e.g., listing tweaks, image tests, price adjustments, A+ content variations -- this traditional approach to wait for statistical significance is a profit killer. It's too slow, too rigid, and honestly, a bit naive for the real world of Amazon.

Enter "Test & Roll": Your Secret Weapon for Agile Growth

So, what's the antidote?

It's a pragmatic, two-stage experimental design from a Wharton marketing professor (weird, right?) called "Test & Roll."

It's built for speed and maximizing profit, not just proving a hypothesis to some academic review board.

The core idea is simple:

Instead of waiting for absolute certainty across a large sample, you test on a small, representative portion of your audience, identify the likely winner, and then roll that winning variation out.

Put differently, you end your MYE early, once you have the earliest indication of the likely winner.

It's about making decisions that drive profit, even if the "academic" significance isn't off the charts. It's about minimizing "regret" – the profit you know you're losing by not acting on a strong directional signal.

(For the academic deep dive, you can check out the original paper on "Test & Roll" here: Test & Roll: An experimental design to maximize profit from A/B Tests)

Why "Test & Roll" is Your Unfair Amazon Advantage

As always, you need to test this for your own brand.

But, my experience is that approach is superior to letting experiments run signficantly beyond the 4-week minimum, because:

• Speed to Market: You get winning variations live faster. No more waiting weeks or months for marginal gains to hit some arbitrary statistical threshold. When you know something is likely better, you deploy it.
• Profit Maximization: This methodology is engineered to optimize for expected profit, not just statistical certainty. It inherently balances the cost of testing with the benefit of deploying a better version, quickly.
• Actionable Insights: The focus shifts from "is it significant?" to "which version makes more money now?" This drives a culture of continuous, profitable optimization, rather than endless analysis paralysis.

Implementing "Test & Roll" on Your Amazon Listings with MYE

Here's where this "Test & Roll" concept really adds muscle to your MYEs:

• Strategic Test Design: Instead of just any test, apply "Test & Roll" thinking to what you test. Focus MYE on high-impact, potentially radical changes where the "roll" phase will yield significant returns. (e.g., Main Images)
• Interpreting Results with a Profit Lens: While MYE gives you statistical probability, your "Test & Roll" mindset helps you make a business decision. Sometimes, a "75% probability Version A is better" with clear positive projected impact is "enough" to roll, even if Amazon's tool wants more data. Remember, MYE even warns against "peeking" early, but for us, it's about informed, agile decision-making based on business impact, not just academic purity.

In sum:

Stop Debating, Start Testing (and Rolling)

FRIENDS OF B@A SmartScout Scott Needham @itsScottNeedham Friend: "Are blogs any good at bringing traffic nowadays?" Me: "How do you think ChatGPT and Google learn? They just read what's on the internet." I'm excited to make blog traffic to Amazon products visible. Inside SmartScout you can now see where on the internet a product is 12:3 PM • Jun 10, 2025 1 Retweets 1 Likes Read 1 replies ​ ​ My good friend Scott Needham and the SmartScout team continue to surprise and delight me with new features. The most recent one? Detailed mapping of outside link traffic to any given ASIN. This is super helpful...when spying on competitors, investing in PR, or just pushing your affiliate channel. I'll share some specific ways that I'm using this data in an upcoming B@A issue. In the meantime, go subscribe to SmartScout and play around! ​ Use Code "Jon25" to Get 25% Off Your Subscription

BEST from LinkedIn

CONFESSION: I had never heard of the Clari-Fi Blur Test until I stumbled upon the post below from my aforementioned buddy, Max.

But, it makes total sense.

And also aligns well with ideas that I've preached for years like:

"Mobile First" (Ok, I stole that from Uncle Jeff)

"The 3-Second Test"

If you're unfamiliar with the Clar-Fi Blur Test (like I was), go read Max's post, and play around with your images in the University of Cambridge's free tool.

BEST from X

Super practical, and actionable, guidance here.

There's a lot to unpack and follow in this thread, so I'll shut up and let Fraser cook.

Fraser Cottrell | Ad Creative For Meta & TikTok @sourfraser Your AI ad scripts sound like ChatGPT because you're using the tools wrong. Here's the exact workflow we use at Fraggell to generate scripts that actually convert: (Spoiler: We make ChatGPT write its own prompts) 12:29 PM • Jun 6, 2025 7 Retweets 47 Likes Read 8 replies

​

BEST from YouTube

This video, The Art of Marketing (Full Masterclass), is fantastic.

I watched it twice (2x speed).

For those who don't have a need for speed, and just want the high points in written form, here's a TLDR:

• Positioning is Paramount: Clearly defining how your product differentiates itself from competitors and appeals to a specific audience (through price, unique features, or even counter-culture branding) is the most critical starting point for any marketing strategy.
• Optimize the Marketing Funnel: A brand's online presence, from initial awareness to final purchase, should be viewed as a "funnel" that needs continuous optimization. This includes minimizing customer drop-off on websites and strategically using "soft" calls to action (like email sign-ups) to nurture leads.
• Leverage Email and Retention: Effective email marketing, encompassing both regular campaigns and automated sequences, is crucial for engaging customers. Furthermore, focusing on retaining existing customers and enhancing their experience (often supported by strong customer service) is a key, yet frequently overlooked, driver of long-term value.
• Embrace Creative Scale and Content: Modern marketing demands a high volume of diverse content for various platforms, languages, and ad variants. Marketers must be adept at content creation, understanding platform nuances, and strategically using influencer marketing (both free seeding and paid partnerships).
• Master SEO for Organic Growth: Search Engine Optimization (SEO) is presented as a vital, long-term investment. By optimizing website content for specific keywords and building strong backlinks, businesses can generate significant, free organic traffic that continuously fuels growth.
• Support Retail Presence: For brands in physical stores, successful marketing extends to retail support. This involves providing data to convince retailers, implementing strong merchandising strategies, and offering comprehensive training to sales associates to ensure product visibility and sales.

My favorite line from the masterclass?

"Almost all roads lead to content."

​

​

​

I had 3 separate people reach out and ask if I was the one that posted recently about a marketplace for Amazon shell accounts.

(I was indeed)

Given the frequency at which this question came up, here, once again, is the marketplace that my friends over at AIA Assets have created for "shell" accounts on Amazon (both Vendor and Seller).

I repeat what I said when I first wrote about this:

If you don't know what this is, well, you shouldn't be buying.

But, if this would be helpful to you, you now have a resource!

​​​