Behind the paper: Yudi Zhang finds a more nuanced way to analyze the customer journey

Customers who are shopping on Amazon often navigate a wealth of options before landing on their chosen purchase. When and why does a customer decide to place their order? Yudi Zhang is interested in how the shopping experience translates to purchases.

In a paper that was accepted at the International Conference on Learning Representations 2024 Workshop on AI4Differential Equations in Science, Zhang and her colleagues demonstrated a novel approach to the problem of estimating the most important parts of the customer journey. The paper was written while Zhang was an Applied Scientist at Amazon Web Services (AWS), which she joined in the fall of 2023.

Zhang, who has a PhD in statistics from Iowa State University, joined Amazon Ads as an Applied Scientist in July 2024. Here she talks about the paper and her research interests.

I think what people are studying here is really interesting. It offers rich data for me to work on advanced recommendation systems and personalization, which directly shape customer experiences. These types of systems exist at a lot of big companies, so Ads is a very good place to develop professionally. Amazon invests in scientists by supporting conferences and research, so that keeps us at the cutting edge of the advertising field, and the Ads division has had a lot of growth recently.

My primary focus lies in recommendation systems and personalization, creating models and algorithms that can show people products they might be interested in. Recommendation systems generally have two parts. The first is sourcing, where we retrieve as many relevant products as possible to show the customer. The second part is ranking, where we have a number of products that we want to put in a particular order so that more people will click on ads that are most relevant for them. I'm working on the sourcing part.

The focus is on multi-touch attribution, which is a research area that tries to understand customer journeys and identify which interactions contribute most to the final conversion, which here means a click or a purchase. In an digital retail space, customers often engage with multiple touchpoints before making a purchase: They view some ads, they browse some product pages, they read the reviews. All of those different actions and interactions are considered touchpoints.

Traditionally, models capture last-touch attribution or first-touch attribution, meaning they assign the most importance to the first or last step for the final conversion action. But the customer journey is very complicated. Most of the time, some of the middle touchpoints are more important in driving customers' decisions. So the paper sought a more comprehensive modeling solution to understand how these interactions will work together throughout the entire customer journey.

I think the most exciting part of the paper is that it uses the attention mechanism to model the attribution. A simple multi-attribution model just treats each interaction as isolated or gives each one equal weight. But this attention mechanism looks at the entire sequence of the touchpoints, and it will dynamically assess the importance of each one within the customer journey. Many AI-based tools today are trained based on this attention mechanism, which is a layer that can identify from big data which point is more important. It also works pretty well for our attribution task.

That's one part of the paper, and the other part is the neural ordinary differential equation (ODE). Other types of models might use a time series to model a customer journey, where each step takes place at relatively regular intervals. But in real life, the time intervals in a customer journey can vary widely. You might look at one thing today and another thing in 10 seconds—or in 10 days. The ODE is able to capture these irregular time gaps.

We tested the model on customer interactions with different AWS marketing channels, such as paid search and natural search, and tried to identify which channel was more important for conversions. It performed much better than traditional attribution methods.

Though the research focused on marketing channels for AWS, the method can be applied to advertising as well. The impact spans three areas. First, understanding the sequence and timing of customer interactions helps us determine which ads work best together and in what order. This knowledge could support strategies that initiate engagement through awareness-focused ads and then guide potential customers to ads that drive conversions. Second, knowing the effectiveness of each touchpoint enables advertisers to personalize their campaigns with greater precision. For instance, we could deliver ads to customers who have previously shown high engagement with certain ad formats and content, increasing the relevance and impact of each touchpoint in the customer journey. And third, as we gather data on how different touchpoints influence customer behavior, we can build predictive models that estimate the likely impact of campaigns before they’re launched.

Amazon Ads has a very large impact with the flexibility to explore some innovative ideas, and our division contributes a lot to the company's overall growth, which is exciting. I have access to a lot of very high-quality data sets, which is very valuable for building robust machine learning models. This allows me to do experiments with advanced techniques, so it's a very good experience here.

Scientists at Ads also collaborate closely, and we have a network to share ideas and accelerate innovation. In a recommendation system project, for example, scientists with strengths in machine learning might team up with experts in natural language processing or causal inference to create a well-rounded solution. We regularly peer-review models, validate results, and explore alternatives to ensure accuracy and efficiency. We also work hands-on with software engineers to integrate models into Amazon’s infrastructure, optimizing for real-time performance and scalability, and we partner with product managers to align models with business goals, directly impacting user experience and engagement.

Amazon has a lot of talented people, and you can learn a lot in the process of collaborating with them.

I want to enhance the interpretability of our models—to focus on transparency so that advertisers not only see the performance metrics but also understand the rationale behind the insights. If we can provide more interpretation to advertisers, then we can provide an even more valuable or relevant experience to customers as well.