Monitoring SaaS Performance from the End User

Posted by on November 1, 2016

When folks talk about end user or ‘real user’ monitoring, they usually mean injecting code (often Javascript) into an application to measure user behavior and timing. This can be great to complement an application monitoring strategy. But this sort of method doesn’t work when the application in question isn’t yours (as in the case of Software-as-a-Service apps). This method also fails to provide a detailed understanding of connectivity.

To bridge the gaps in visibility into applications you no longer own and network connectivity which increasingly influences overall user performance, we built Endpoint Agent. Now that Endpoint Agent is generally available, we’ll be penning a few posts about you can put it to use. First up, we’ll dig into how you can monitor SaaS applications with data collected directly from the users of those services.

Choose Your SaaS Applications to Monitor

You likely have a lot of SaaS applications in use in your organization, some deployed to all employees and others used by just a few functions or teams. In general, we find that organizations get value out of monitoring applications that are mission-critical to their business. Here are some popular apps to inventory in your organization:

  • Sales and Support: Salesforce, Oracle CRM, Dynamics 365, Zendesk
  • HR: Workday, Oracle HCM, SAP SuccessFactors, ADP
  • Supply Chain: Oracle ERP and SCM, Coupa
  • Finance: NetSuite, Intacct, Xero, Zuora, Ariba, Concur, Docusign
  • Collaboration: Office 365, Sharepoint, OneDrive, Google Apps, Box, Dropbox
  • ITSM: ServiceNow, Okta, OneLogin, ADFS

Note that Endpoint Agent today monitors the performance of web-based SaaS applications. That means that web interfaces of email, files sync and chat apps will be monitored, but thick clients installed on the OS will not trigger monitoring. Depending on how your team uses applications like Outlook, Dropbox, Box or Skype for Business, keep this in mind.

The Outcomes You Want to Measure

When it comes to monitoring SaaS applications, there are usually two key outcomes you care about: reachability and availability. From these two basic outcomes you can quickly define a checklist of data that can be helpful to validate that both of these are true, and if not, why.

Here is a good starting point for a checklist of SaaS connectivity and availability:

  • Can I lookup the domain name?
    • Is the DNS query successful?
    • How long does it take?
  • Can I establish a network (TCP) connection?
    • Is there loss? Latency? Jitter? Where in the network?
    • Is there a VPN? A proxy? A CDN?
    • Is there a route to the destination?
  • Can I establish an HTTP connection?
    • Does the server respond?
    • How long is the response time? Wait time?
    • Can I receive data from the server?
  • Is the HTTP response successful?
    • Did I receive a valid response?
  • Beyond the root page, are other pages accessible?
    • Can I login?
    • Can I retrieve and create data?
    • Can I search?

While this may seem like a lot of data to collect, it is critical to understanding what portion of an application or network has a fault, and platforms like ThousandEyes will do the heavy lifting for you.

Two Ways to Monitor SaaS Applications

Monitoring a SaaS application means monitoring a service that you don’t own and can’t directly instrument. Therefore, you have two broad approaches to gather performance data on app and network connectivity:

Script transactions at periodic intervals Record actual behavior from the end user
Data Collection Define fixed vantage points to monitor from, typically in your offices

Define actions for the script to emulate

Define periodic testing intervals
Define monitored users, but record wherever they access the SaaS app

Record whichever pages and actions a user takes

Record data whenever a user visits
Data Output Clear, comparable baselines that are designed to mimic usage

Useful for benchmarking and sampling performance
Noisier performance data that reflects real usage and behavior

Useful for individual troubleshooting and segmenting performance

You can use one or both of these approaches to meet your needs. As you can see, they are very complementary. The trade-off exists in how you instrument your monitoring strategy and whether you value clear baselines that may or may not reflect user conditions, or actual user behavior which may or may not be easily or clearly compared over time and users.

Scripting Web Transactions

Web Transaction tests run at preset frequencies, say every 5 minutes, and use a script to take actions on a browser. It’s like having a bunch of users all taking the same actions in the application to benchmark overall performance and find functions that aren’t working as expected, like authentication or search.

Figure 1
Figure 1: A Selenium web transaction script for buying a television on Amazon.

In ThousandEyes, scripted Web Transaction tests use open-source Selenium scripts. You can write your own script, import an existing script or record a script directly from your browser using our Recorder. Our Enterprise Agents, software-based sensors that you can place in branch offices, then run these scripts within a browser at defined frequencies. You can also compare this data with Cloud Agents in your area for an external comparison.

Scripted web transactions will give you:

  • Completion rates of the full transaction
  • Transaction timings overall and per step
  • Timings per page
  • Page load and DOM load timing
  • Object sizes and timings (blocked, DNS, connect, send, wait, receive) on each page
Figure 2
Figure 2: Web transaction timings per step and per page.

Take a look at previous posts on scripting Basic Web Transactions and Advanced Web Transactions to dive into the details of how to make your transaction scripts effective and resilient against changes in your front-end code.

Recording User Sessions

In comparison to scripted web transactions, Endpoint Agents record user sessions directly from a user’s browser. This means that User Session data reflects the actual performance and sequences of users, rather than ‘clean room’ transactions that wipe the cache and take an identical sequence of events each time. You should therefore expect much more variability in User Session data: for example, when a user wanders outside of wireless range or already has a large file cached in their browser. But it is exactly this variability that makes it useful for troubleshooting individual sessions and interpreting aggregate performance data.

Figure 3
Figure 3: User Session data generated by Endpoint Agents reflects pages loads as they happen.

You define exactly which domains (SaaS or on-premises applications) and networks are monitored. Within these bounds, recorded User Sessions capture the following data:

  • Number of sessions and pages per user
  • Page success rate
  • Page response time
  • Page load and DOM load timing
  • Object sizes and timings (wait, receive) on each page
  • Loss, latency, jitter and connection path for each session
Figure 4
Figure 4: Page load information for a user session while logging into Salesforce.

With Endpoint Agent data, you’ll likely also want to use reports to aggregate data over longer periods of time and over many users to understand overall performance. Each of the metrics listed above can be sliced in various ways to create reports, such as in Figure 5.

Figure 5
Figure 5: Weekly aggregated performance data for Salesforce usage.

For more information on collecting user session data, check out our recent webinar on Monitoring SaaS Application Performance using Endpoint Agents.

Monitoring Your SaaS Applications

With the two approaches above, you’ll have your bases covered when it comes to troubleshooting, benchmarking and monitoring performance. But let’s face it, each SaaS application can be very different in how it functions and performs. With that in mind, stay tuned in the coming weeks for tips on monitoring specific SaaS applications such as Salesforce, Dynamics 365, Office 365 and WebEx.

Processing...