Before taking my master’s in business analytics, I was comfortable letting AI do most of the work. If a tool could automatically recommend products and help increase sales, that sounded useful enough to try. I understood the potential benefits, but I didn’t spend as much time asking how it generated those recommendations or what data it used.
Learning more about models and algorithms changed that. I became curious about what was happening behind the recommendations and how we could check whether they actually helped the business. I also started paying more attention to the control we had over the process. Those questions became more practical while I was working with product recommendations for an online retailer.
Increasing AOV was already one of the executive team’s goals. An AI recommendation app seemed like a reasonable way to support it by helping customers find relevant products and encouraging them to add more items to their basket. But after a few months, the ongoing cost became harder to justify. I started researching alternatives that were more cost-efficient, and that search made me think more carefully about what we needed from the tool beyond its AI features.
Understanding the data behind the recommendations
One limitation that stood out to me was the tool’s 90-day data window. Recent purchases are useful, but I wanted to understand what customers had bought together across our longer historical data. A product with limited sales in the past three months might still have a relevant connection to another product, particularly if demand changes throughout the year. Looking only at recent activity could leave some of those relationships out.
We also wanted customers to discover products beyond our core bestsellers. Increasing basket size mattered, but so did giving more of the catalogue a chance to sell. If recommendations kept drawing attention to products that were already performing well, we could miss opportunities to introduce customers to other relevant items.
That made access to our own data more important to me. I wanted the option to develop and adjust recommendations using the purchase history we already had. I wouldn’t assume that including all historical data automatically makes a model better because older patterns may no longer reflect how customers shop. Having access to that history would, however, let us investigate those patterns and decide what was still useful.
Having room to adjust the approach
As I learned more about machine learning, I started asking which algorithms these tools were using. That information wasn’t always clear, which made it harder to understand how closely their approach matched what we wanted to achieve. I became more interested in whether we could bring in different data sources, create our own recommendation logic, and export results for further analysis.
The practical side mattered just as much. We needed to implement recommendation widgets across different placements and make changes without turning each adjustment into a complicated task. A tool could have useful features, but we still had to work with it regularly and assess its ongoing cost.
The platform we eventually moved to gave us more flexibility in these areas. We could use our own data sources and logic while using the platform to display recommendations and run tests. For me, that meant I could investigate what customers actually buy together, develop recommendations from those patterns, and adjust the approach as we learned more. It gave us room to combine the platform’s capabilities with our own understanding of the business.
Checking whether the recommendations help
Finding that two products are frequently purchased together gives me a reason to test a recommendation. I still need to see how it performs when shown to customers, which is why A/B testing became an important requirement. We wanted to compare our recommendations with the existing setup and check whether customers added more items, average order value improved, or products beyond the usual bestsellers received more purchases.
I would consider broader product sales a worthwhile result even if average order value stayed the same. It would support our goal of spreading sales across more SKUs and give us more information about products with limited recent activity. Over time, those purchases could help us understand their place in the catalogue more clearly and inform future recommendations. That improvement would still need to be tested rather than assumed.
I would also look at the wider business results. Selling a broader range of products needs to make sense alongside conversion, profitability, and the cost of running the tool. If the recommendations didn’t improve performance, I would review the results, adjust the model or logic, and run another test. But I would set a threshold for how much time and effort to spend before moving to another conversion optimisation idea. The ability to keep adjusting something is useful, provided we know when continuing is worthwhile.
What I would clarify before choosing a tool today
Looking back, the goal of increasing AOV was already there. What became clearer later were the requirements for pursuing that goal and evaluating the tool’s contribution. Today, I would spend more time defining what we expected recommendations to achieve, which data we needed, how we would test the results, and what ongoing cost would be reasonable.
I would also think about where human judgement belongs. Someone still needs to decide whether an older purchasing pattern remains relevant, which products deserve consideration, and whether the outcome of a test is useful enough to continue. A model can help us identify relationships, but applying those findings requires an understanding of the business and its priorities.
My advice would be to avoid jumping into AI just because it’s cool. I understand the appeal because I was once comfortable letting the tool handle the work without asking many questions. Data can seem boring compared with an AI feature, but understanding the data you have and being able to use it gives you more room to improve the business. Its quality and relevance matter as much as its volume.
I still see a lot of value in AI tools. My expectations have changed as I’ve learned more about them. I now want to understand how a tool fits the goal, what we can control, and how we can judge its contribution. Pairing that capability with human judgement gives us a better basis for deciding what to keep, what to adjust, and what to try next.