Behind the paper: Kelly Paulson uses algorithms to spot learning opportunities across business units

Kelly Paulson took a job with Amazon fresh out of grad school. She had earned her PhD in economics from the University of California San Diego and initially saw the job as "something fun" to do before going on to something else. But she liked it too much to leave, and 11 years later, she is Senior Manager of Economists and Applied Scientists at Amazon Ads. The paper she co-authored, Multi-task combinatorial bandits for budget allocation, won best paper at the 2024 AdKDD conference. Here Kelly discusses the paper and her work at Amazon Ads.

I joined the Amazon Kindle team in 2013 when I had just completed my dissertation, and it was the first year Amazon was recruiting junior economists right out of grad school. I was attracted to the data. I had been working on statistical techniques for learning about how people make choices, and, of course, Amazon already had one of the biggest data sets in the world of people making choices.

At the time, I had job offers across tech, government, and academia, and I didn’t expect to stay in Seattle long-term. But since I was interested in having hands-on experience with Amazon’s rich data, I thought I would defer my offers and do something fun for a year. By the end of that year, I really liked the work and decided to stay. I wanted to move into a part of Amazon where there were more people thinking about how to extract information from really big data sets. So I moved into Amazon Ads as the first scientist to work on measurement.

I focused on understanding how customers were influenced by ads and how to transmit that information to advertising optimization systems. Amazon’s unique signals are central to our value proposition. When a shopper sees an ad for shoes, are they more likely to buy those shoes? If we have the opportunity to show a similar shopper the same ad tomorrow, should we do it?

But a critical piece of that was how we could summarize insights in a way that was useful to advertisers. Could we do useful return on investment calculations for them? What could they learn about their back-to-school campaigns that they could incorporate into their holiday campaigns?

It was the early days of thinking about how Amazon could jointly optimize across shoppers and advertisers by helping advertisers bring more useful content to shoppers at the right time.

My background is in econometric theory, with a particular focus on causal models in quantitative marketing. In the past couple of years, I've really been interested in how to develop a causal signal and feed it into a broader set of systems that are making marketing decisions.

That intersection of causal measurement and reinforcement learning is very unique to Amazon. We have such a big scale that we need to use a lot of tools like reinforcement learning to make decisions. We're a very data-driven company, and we want to make decisions based on causal signals, not correlational signals.

Other companies usually focus on one side or the other. The intersection is very interesting and will be everywhere in the future. Amazon is at the forefront of that area.

As is pretty common in the tech industry, we use Amazon's own needs to pressure-test new products that we may eventually sell externally. I'm in the Ads organization, but I work on marketing for a diverse set of Amazon businesses outside of Ads. For example, we may want to raise content streamers’ awareness of a new Prime Video series or raise shoppers’ awareness of holiday deals.

So my team supports Amazon's internal businesses using Amazon Ads products to meet their marketing objectives. The team is diverse, with marketing domain experts that have close relationships with business leaders in addition to the scientists and engineers. Together we identify opportunities to use algorithms in advertising systems and to see patterns that other people can't see.

The idea for this paper was to see if we could learn anything from having an algorithm analyze information across different products. We show that when we use reinforcement learning techniques, we can essentially help one product do better by using what other products have learned and in a very real-time way. We've been running this algorithm daily.

Let's say we already have a bunch of marketing campaigns for different devices and then we're launching a new device for the holidays. Where do we start? This algorithm is a way we can take what's known from the other marketing we've done for existing devices and start the new campaign with an optimized setup.

The main thing is that the improvement in performance is pretty high. At the AdKDD conference, a lot of the other papers reported a 2% or 3% improvement in click-through rate. Our improvement in total clicks averaged 18%. We also showed a cost-per-click reduction of 12.7%.

It goes to show that even when there's intent for products to be coordinated, it's just really hard, and there's a huge business opportunity in using algorithms like this to help products systematically learn from each other. Marketing experts for different products can share what they’ve learned with each other in "lunch and learns" and develop playbooks of best practices—but humans coordinating with each other is expensive, and it's hard to keep nuanced insights up to date in a rapidly evolving industry. This is a principled and systemic way that companies can be better coordinated and more efficient.

Even outside of tech companies, there are lots of large companies with complex marketing needs and global footprints. But a lot of the academic literature is just looking at isolated parts of the business problem. So it’s exciting that if you're a company with diverse products, you can get different parts of your business all pulling in the same direction using reinforcement learning.

Three graduate students took time during their dissertations to work with us as interns on this paper. And all three of them got Amazon offers; two have already started, and one is going to start early this year.

One thing I really like is the ability to develop my ideas into programs. Often I'll be working on one project and realize that something else was a lot more important for the business. We have a process where you write a hypothetical press release and answer some frequently asked questions—called a PR/FAQ—about a future product or program Amazon could have.

Through that, I've had several opportunities to formulate a problem I want to solve, think about the solution, write it up, circulate the idea to colleagues, and originate these new science-powered products. It's fun to see products that now have teams of 25 people and to know I was the one who put the business case forward and got the initial funding.

In my role, I can balance questions about how we measure the impact of ads on customers with how we operate our optimization systems that will lead to better outcomes for our customers. When I talk with other companies, I feel like they're trying to put me in a box; they need me to either measure something or manage an optimization system. At Amazon, I have the flexibility to operate across the whole problem space and really think about how to drive the best outcome for customers, putting the optimization systems and the measurement systems together.

Another thing that's unique about applied scientists here is that they're focused on delivering software—it's not just doing research. They’re building products that are durable and that help solve a long-term business need.

My team thinks about Amazon's own advertising needs every day, and we test existing ad products at large scale and identify opportunities to do better. Because Amazon is one of the biggest advertisers in the world, we learn a lot. We’re not trying to sell existing ad products; we’re finding ways to improve them. That customer obsession, along with having the flexibility to design the best solution, is novel for the industry.