Written by David Ruckman, Senior PM, RDA
I wanted to post about a recent project that I am pretty excited about.
Have you ever looked at the Google Analytics Real-Time dashboard, and enviously thought to yourself "boy, I'd sure like to have something like that in Sitecore!"? Yeah, me too. I looked around and it seems no one else has tackled this - so I decided to develop this new functionality myself.
Real-Time allows you to monitor activity as it happens on your site. The reports are updated continuously and each visit and hit is reported seconds after it occurs. For example, you can see how many people are on your site right now, which pages or events they're interacting with, and which goal conversions have occurred.
I've found GA's real-time reports to be a very valuable tool over the years for many reasons:
- Lets me get a glimpse of overall system health
- Allows me to monitor whether new and changed content on the site is being viewed
- Helps me understand the usage of my site through event tracking
- Allows me to see whether a one-day promotion is driving traffic to the site, and which pages these users are viewing
- Lets me monitor the immediate effects on traffic from a blog/social network post or tweet
- Helps me monitor goal completions as I test changes on the site
- Brings me joy and comfort to see users interacting with my site in real-time!
So, if you can use Google Analytics' real-time reports, why do you need them in Sitecore too? Because Sitecore is a stand-alone analytics platform and should not depend on third-party software to make it work fully and properly - I should be able to get everything I need, analytics-wise, right in the Sitecore shell. And also because my Sitecore real-time report runs in the back-end, so it's more reflective of actual system health - it will show me the actual data flowing into the Sitecore databases, not just what gets sent to GA in a beacon. And lastly, because I like having the ability to tailor the real-time report to my exact organizational needs and add any new features I want to it.
The question then became how to pull this off in Sitecore 9. One way might be to tap into MongoDB or the SQL Shard databases to pull interactions and contacts as they are added. But I have an even better way, because I have developed a Google Tag Manager integration with Sitecore which was already capturing a rich set of analytics data on the content delivery server and sending it to Sitecore through xConnect. So I decided to build a real-time report in Sitecore that leverages that GTM integration.
Note: The GTM Integration is another exciting story that I will be blogging about separately in the near future.
So, without further ado here is the recipe I cooked up to provide Real-Time analytics reporting in Sitecore:
1. Set up the GTM integration to capture real-time analytics
The GTM integration allows me to develop tags in Google Tag Manager and have those tags (campaigns, events and goals), sent to Sitecore when they are triggered. This way I can collect analytics easily and quickly without needing to do ANY development in Sitecore. The way it works is that you set up tags in GTM like you normally would, and those tagged events are sent to Google Analytics, and simultaneously sent to Sitecore via a custom REST API.
So step 1 was to set up the API in Sitecore and develop all my GTM tags for the project. Here are some of the tags I developed for my demo site:
So now we have analytics data being sent to an API in Sitecore, and we can use that data for our real-time report. But first we somehow need to get that real-time data from the content delivery roles to the content management role (where the real-time report will live)...
2. Set up a message bus to send real-time data to the Sitecore content management role
Sitecore 9 comes with a very robust service bus built on Rebus.NET. The bus is a message transport mechanism which allows application roles to communicate across boundaries. We need to set up our CD instances to push messages onto the bus, and our CM to subscribe to those messages.
Shout out to Pete Nararra and his excellent blog post from SUGCON EU 2019 which helped me ramp up on developing a solution for the Rebus message bus: https://sitecorehacker.com/2019/04/08/wheels-on-the-rebus-go-queue-queue-queue/
I have tested this message bus pattern both locally on a development VM, and in Azure PaaS scaled scenario using the Azure Service Bus, and it works great in both cases. You can configure the bus to use SQL or in-memory storage models. It's very scalable and keeps up well, even under heavy load. To optimize resource usage on the bus I did not set up any kind of retryer - I figured this data is transitory in nature anyway, and if you lose a message here and there it's not the end of the world.
So now we have analytics data being sent to the CM, so it's time to...
3. Create a Real-Time report in Sitecore
I developed the report using JavaScript and implemented the report in Sitecore as a new app in the LaunchPad.
Invoking the app brings up the report right in Sitecore. When new messages are received from the message bus, the report is updated.
And viola! Real-Time reporting in Sitecore which shows everything that's happened in the last 5 minutes on your Sitecore website.
So, to bring the whole real-time report recipe together, here is what I'm doing:
1. The client browser requests a page from Sitecore.
2. Sitecore responds with the page content AND the base GTM script.
3. Once loaded, the script calls out to GTM.
4. GTM responds with all the event listeners for the page.
Now we are all set up and waiting for an event happen. When the visitor triggers an event (like triggering a campaign, clicking a button, or whatever), the following happens:
5. The event is sent to Sitecore via REST API, where it is then sent to xConnect.
6. Sitecore places a message on the service bus containing the event data.
7. Sitecore CM picks the message up from the bus and makes it available to the real-time report.
8. The real-time report displays the event.
Hope you found inspiration or good information in this post. If you have any questions, or are interested discussing real-time Sitecore reports or GTM integration for your organization, please don't hesitate to reach out.
Original Source: http://integryx.net/post/2021/02/25/real-time-analytics-for-sitecore
