Marketing Cloud maturation means your Databricks, Snowflake, AWS, Azure and GCP instances are going to be more connected to real-time data streams and apps. For brands this data flows in the form of precious first-party data and unlocks a new world of marketing possibilities.
Data modeling, machine learning and real-time algorithmic decisions will become the norm for marketers in the coming years via apps connected to the brand’s data warehouse. With the cloud, not only will brands maintain first-party data sovereignty, they can compose applications with their data to optimize marketing. The future is machines deciding what to say to who in real-time at every touchpoint and interactions powered by unending flow of customer data. If this sounds like what Google and Amazon have built it is. This is how we got the cloud in the first place.
Using first party data – namely behavioral and transactional data – from visits to your website or app in real-time, has been something smart marketers have been doing for a long time. Many years ago when first introducing the idea of personalization to marketers I would talk about “Kairos.” The Greeks defined Kairos as “the right, critical or opportune moment.” It is an important part of Aristotle’s Rhetoric which is about persuasion. What is marketing if not persuasion?
Early Real-Time
At Offermatica (later Omniture Test&Target after acquisition and now Adobe Target) we pioneered the use of tags to define audience segments, real-time triggers and created the “m-box” technology (still in use!) to test and target rules based content serving. Back then keywords were one of four best ways to segment (and sub-segment) in real-time along with sources, geos and visitor events/actions.
Here’s an oldie but goodie where we targeted a message to visits that came from Spanish language search engines. Conversion rate improved over 50% from this source. No need for rocket science here, just relevance.
Another oldie but goodie is the kick-ass Kayak homepage. It was almost impossible to optimize this page better until they reinforced the visitor source and keyword query on the landing and then took that data in real-time along with visitor location derived from the browser to auto-fill the form.
Around 2013 Google started obfuscating keyword data in the referring URL and site analytics because of “privacy.” This eliminated what were typically the highest converting real-time segments and undoubtedly led to more commercial search queries which was great for Google. Less relevance on landing pages meant more people going back and doing searches.
Losing keyword data has left an indelible drag on conversion rates ever since. However we learned as marketers what Google knew as a Search Engine. Using behavioral data for real-time segmentation and matching was a formula unique to the web and powerful for anticipating and meeting the needs of consumers.
Platform Time
Fast forward a decade and immediacy is a more important part of user experience than ever. Increased use of the web and social apps in particular trains our brains to make millisecond choices. Reducing latency of those choices becomes essential for improving conversion metrics. We see it with ApplePay and Amazon One-Click. We see it in same day delivery and Chat-GPT. We see it on TikTok and BeReal. We have not seen it all that much in Marketing. That will now change because of the cloud and connected apps.
Most data that could serve as the foundation for real-time modeling and targeting is presently delivered in batch. There is of course Real-Time Bidding but that is a technical specification for Display Advertising. It is worth noting AdTech has multiple parties making multiple decisions integrated into a single end-point (the header) in less than 200ms for what they should say and who they should say it to and how much they are willing to pay to deliver the message. There is some great cross-learning from AdTech to MarTech on this topic but I will save that for another post.
Of course platforms like Google, Facebook and Amazon have been using real-time data streaming for years to improve advertising and recommendations. Google itself is the original real-time technology of the web making rules-based requests to an index, getting a scored/ranked results set and then continuing to update the scoring model based on what does or doesn’t happen. Way back (can it be almost 20 years!) in 2004 Google started using its first party cookie to personalize the results.
Recommendation systems have probably had the most noticeable use of real-time data on the web. Amazon suggested products was an early leader in this area and I guess if it ain’t broke for the last 15 years don’t fix it, just add advertising.
Low latency relevance explains how Bytedance/TikTok redefined what is possible in a real-time recommendation model with its For You Page. Their data model includes machine vision for in-video feature definition and has created one of the most disruptive media properties of all-time.
Not much has been released about the FYP algo works but the first paper I’ve seen about it was recently shared. Interestingly Bytedance used one of AdTech company Criteo’s open datasets to train the model. As mentioned above, AdTech has had a headstart in real-time decisioning over MarTech. We also learned the fun fact that the internal name of the FYP recommendation algo is “Monolith.”
YouTube recommender also stands out because of the scale of usage and data collected. For many of us the right side of YouTube is our navigation and I’ve argued that it is the most important algorithm in the world (for better and for worse) as it leads so many of us down rabbit holes.
Intent is Real-Time
Interestingly only this week I noticed YT introducing chips as shown above to segment recommendations. It is also a great data collection strategy (the most overlooked yet important strategy for many) as it acquires more explicit first-party data about consumer preferences to update serving and suggestion models for me.
Chips also lend themselves to mobile UX and are used widely by Google. Mobile demands low latency relevance that only algos can provide. This faceting of suggestions is interesting. As a consumer I like it, especially the “watched” as it is a great recovery tool. Recovery being one of the two forms of intent along with discovery.
As marketers we want to understand the real-time intent audience so moving at the speed of customers requires an investment in real-time systems. As you strategize data collection and triggers for the use cases I will get into below, it is important to keep in mind the goal of your visitor at every step. This way data collection and messaging can be the most relevant. Below is an intent framework I created that can be helpful. Through these paths are Awareness, Consideration and Transaction.
All of this is to say that the use cases for building data models using real time streams to make a decision are clear and powerful. While platforms and well funded companies were able to build streaming applications over the past decades, progress with cloud based data infrastructure and open-source, namely the maturation of Kafka (originally developed by LinkedIn), and other Apache projects like Spark (and its offspring Databricks), Storm and Beam and of course AWS (Kinesis), Snowflake, GCP (Dataflow), Azure (Stream) and too many more to mention have created the inflection point in marketing we are at today.
There is only now the opportunity for brands, marketers to develop, test and tune algorithmic models with their own first-party data streams. The impact to customer relevance and thus the results of your marketing can be as powerful to your business as the YouTube and TikTok recommendation algorithms are to theirs.
Use Cases for Building Marketing Cloud Apps with Streaming Data
Over the past few years it’s been interesting to watch the naming conventions emerge. AWS Apps became Connected Apps (nice jiu-jitsu by Snowflake there) and in-house CDPs became composable CDPs. I like to think of all this as the Marketing Cloud (not stack!) so that’s what I’m calling the solutions to these use cases.
First I want to share a use case I helped solve with an app built for an e-commerce retailer. Then I will review some other use cases so you can optimize your marketing to the “Kairos” of your customers. The model below delivered a 24% lift in volume of high value by 1) being able to identify them and 2) being able to increase bids to remarket to them in near real-time. Everyone on the retailer’s marketing team got promotions. 🙂
Use Case: Predicting High Value Customers in Real Time
Predicting high value customers (p-HVC) in real-time is a great use case to optimize a number of factors that influence marketing/advertising strategies and tactics. It’s possibly the most important use case for ROAS (return on ad spend) which is too often correlated or averaged to transactions from every type of customer, including those that may have negative value due to promotions, returns or cancellations.
Performance reports from agencies and internal teams often show all clicks and visits as averaged to a metric. This is of course how the platforms report back to us. It is also one way the platforms maintain an advantage over marketers. The data the platforms don’t share is often the most valuable. That every site visitor has a different value before they get to our site is a fact. The same is true for when they leave. Yet most brands never know these values let alone in real-time.
The value of that consumer is determined by the platform in a number of ways but always based on historical and real-time data on events and behaviors. As marketers we should want to understand the true-value of visitors in real-time too. Ideally we know who looks like they have the potential to be our highest value customers no matter what ad set or group they came from. Then we can activate tactically with that knowledge. If you don’t know what defines your customer value bands already you should run some batch data modeling and discover what behaviors high value customers exhibit early in their customer journey using clustering or regression.
There are few key technical and data considerations when building this type of app with your data warehouse.
First you need a real-time stream of entity and event based data from a data collector like Snowplow, Amplitude or GA4. Be prepared to create some new custom events and likely adjust those schemas over time as you get smarter about what events signal a p-HVC. I really like the Snowplow technology for this because you can easily update your event and entity schemas and not lose historical data.
You will also want the customer file from MDM (Master Data Management) or the CDP (Customer Data Platform) ideally keyed to a single customer ID that works across all channels IDs.
You will want as much product data associated with that golden customer record as possible. If product data is not connected to your customer index and/or the browse buy events of the customer behavioral data you will want to join these.
You’ll ideally have 15 months of site behavioral data with IDs to help with training the model. You’ll also want to ensure that transactions are getting into the database at low latency as this is ultimately how the model will learn in real-time once it is deployed.
You will need to determine the variables that might determine who your highest value customers are. You need to determine if these variables exist, where they exist and if they don’t exist can you add them to a schema. Since we are trying to predict new customer value on anonymous traffic, a very useful dataset to help this model is being able to see “anonymous-to-known” behaviors. Something like anon visit count can be a highly predictive variable. Make sure you are back propagating anonymous behavior once that customer is known and that it has its own attribute.
You will want to create a list of features, rank them as they relate to expected performance in determining High Value Customers. Don’t be surprised if you have over a 100 features. This is the most fun and most critical part of model development. If you have a smart person working with you they may want to use a Gram-Schmidt process for feature ranking.
You’ll then want to create your model over the training data that you’ve acquired to refine it. Once you feel it working fairly well you can deploy it and set up tag integrations and triggers to send these predicted high value customers to your channels where there can be a customer audience that adjusts your bids.
The real-time component of these audiences is especially important for SEM (Search Engine Marketing) performance. If you are like most websites there is a high probability that many of your highest value customers started their relationship with your brand by coming from a Search. There are also lots of potential high value customers that left your site, went back to Google and did another search. You will want to increase your bid for that person so they come back or see your brand again as an impression for their query.
Once you get smart about predicting your best customers, you can experiment with bid strategies for awareness campaigns as well as consideration/retargeting campaigns. You will also want to use this real-time scoring to adjust your content and personalization of the site experience.
This is of course a summary outline of requirements and just one person’s methodology. There are a lot of specifics especially around the data flow that will be unique to your infrastructure and cloud. Data aggregations, logic, gateways, transformations and read/write in the web app/data layer often present their own unique challenges. The above is not easy, but nothing really valuable is. If you are not doing this and your competition is you are at a huge disadvantage for growth.
There are a bunch of interesting tools in the space that can be helpful. For low-latecy lookups/matching Message Gears has a mature product called Engage and last week Hightouch released a personalization API . Also start-ups like Pecan.ai for low code batch training, Continual.ai for deployment and retraining batch models onto streams and Orac Labs for real-time model training and prediction on streams can be helpful building, feeding, training and deploying the data models in a private cloud.
Be prepared to create a POD for this and at least a quarter to stand up and another quarter to tune. That may seem daunting with everything else you are doing to increase conversion and ROAS but acquiring more high value customers is arguably the most important thing you can do for your business.
Additional “Always-On” Marketing Use Cases
Here are some additional real-time marketing use cases you should consider building apps to solve:
Churn Prediction
Build streaming ML that probability models customer churn based on website and/or other channel behavior (or lack thereof). This way marketing can try to “win-back” someone before they have left forever or gone to a competitor and become someone else’s LTV (LifeTime Value). This also helps to increase LTV so you can bid/spend for CAC (Customer Acquisition Cost).
Customer Journey Optimization
Understanding what stage your customer is in is incredibly important to determine how best to help them make a purchase decision. Awareness, Consideration and Purchase Intent are all different mindsets for a customer. Yet our content and pages rarely if ever speak intelligently to each stage across a site/app click-path, let alone across channel and the ad creative > landing page > site experience.
First party data behavioral data like site visit and content consumption and anon-to-known events along with the contextual and behavioral dimensions attached can be highly journey predictive. This data is also useful in media-mix modeling as different channels generally appeal to different consumer stages. For example TV ads (and resulting brand search spikes) are very awareness centered. Social Media often performs well in consideration stages. SEM, especially product and endemic keywords are good signals of purchase intent.
Predicting these stages in-session can have huge conversion rate benefits and help build brand through great UX.
Lead/Visitor/Cohort Scoring Models
This one is generally a little less complex to create than the customer journey models and more for the B2B crowd – though building apps here can be very helpful for B2C with download conversion, digital to physical conversion such as Auto, Home Buying and true lead based businesses like Insurance.
Real time call center data and other support systems like chat can be added into these models to improve accuracy. In fact, the more event touch points the better. When I’ve seen these models not work it is because the inputs are not sufficient. This gets into the importance for data collection strategy e.g. customer event creation not just for this use case but all of these use cases. Too often I see lead scoring models (even recently from large data consultancy) that use a small sample of events and they just end up being wrong.
Next Best Action
Determining the one best choice from a number of options is what predictive models are really good at, especially multi-arm bandit algorithms using reinforcement learning. Any part of the sequence of events where options can change is a candidate for next best action. Think of areas like shopping cart journey, registration/sign-up/onboarding. Other use cases involve message sequencing/storytelling. Gaming is also a great framework to think about places where next-best action can be beneficial to your brand.
Price Optimization
This tugs on people’s fairness strings as we have seen with Amazon and more recently with Ticketmaster and the kerfuffle with Dynamic Pricing algorithm tuned to real-time supply and demand.
Behavioral data as well as inventory and other real-time dimensions should have an affect pricing. This is what a free-market economy is based on. I’ve also seen smart marketers who scrape competitor price data and use that to modify their own pricing in real-time. Smart af!
Taking it a step further, streaming data can be used to predict the effectiveness of promotions in real-time to maximize profitability and prevent full-price customers turning into discount shoppers. It can monitor conversions in real-time for discount usage to understand if loss-leader promotions are being used by one-and-done bargain hunters (thus lowering your overall customer LTV) or are you really creating repeat customers that will increase LTV.
Recommendations – Content, Products, Cross-Sell/Up-Sell
Merchandising and product mix. Product matching for purchase and cross-sell/up-sell. All are great and classic use cases for real-time data.
Closing the Loop
If you made it this far in the post you must be very interested in real-time applications for marketing and advertising! That also means you are thinking about building leading edge practices with technology and data that get smarter over time. Not everyone is there yet but no question there are first mover advantages. Data is a motor sport.
We’ve waited a long time for marketing technologists to become a thing. I do believe that the Marketing Cloud has finally ushered in that era. The cloud offers what many are referring to as “composability” which is the foundation for building applications to solve marketing use cases incrementally. This is after all the reason your CTO has invested in the cloud to begin with. I can assure you that once you start the work it will become foundational as the components you stand up can build numerous applications to solve multiple use cases and the knowledge you acquire through the process will continue to grow and provide advantage.
I also understand that solving these use cases in a transformational ways requires others to get onboard with the need, the strategy and the tactics. I hope this post can be helpful for marketing and data teams trying to position and get sponsorship for cloud marketing composability and first-party data apps using real-time data as POC (proof-of-concept) with your marketing, data and engineering teams.
jonathanmendezblog.com 
Leave a Reply