The 5G QoE measurement challenge can be summarized as follows:

• 5G use cases, particularly mMTC and URLLC, demand a new way of measuring QoE for each individual application.
• QoE has a different meaning in machine-type communication. A machine or connected thing’s QoE needs to be interpreted.
• Each application brings a specific set of network performance characteristics that need to be monitored.
• Monitoring becomes more important to operators for critical 5G applications.
• 5G testing adds a new dimension for both data acquisition and post-processing.
• Active testing and understanding of QoS per use case, allied to network operations, will be a key enabler of future 5G business cases.

Without knowing the underlying service and device requirements, network quality engineers do not know what aspects and metrics should be treated as important key performance indicators (KPI). So we need to compile the set of parameters and KPIs, and define thresholds, creating good/bad KPI limits for each application and use case. This is still at a very early stage, but one approach would be to start at the PHY and logical layers and work up to the apps and use cases, defining what needs to be measured.

One positive aspect is that this sort of understanding of use cases QoE can drive the Network Operations Center (NOC) to Service Operations Center (SOC) transformation that operators are making. The SOC is important because it is via this service-driven environment that operators hope to differentiate their customer experiences. For example, a car manufacturer can understand which operators’ network is best suited for its connected or autonomous cars. So we can see that active testing and understanding of QoE per use case, allied to network operations, will be a key enabler of future 5G business cases.

Implications for test solution providers

Considering the operators’ challenges , test equipment providers must produce solutions that can measure multiple virtual networks at the same time and the same location with different methodologies.

An examination of the metrics required to characterize 5G use cases quickly reveals that measuring QoE becomes more complex and more demanding in terms of the data acquisition of additional measurement parameters with greater precision in the RAN. This also drives the need to provide post-processing analytics that encompasses new models for quality of service (QoS) and QoE measurements for network benchmarking, optimization and monitoring.

The consequences of degradation or loss of service for some mission-critical 5G use cases go way beyond an unsatisfactory voice call to a friend or a YouTube video freezing when you stream it on your phone. The critical nature of some applications demands that test solutions must be independent, transparent and traceable to certified international standards and not aligned to proprietary techniques or individual network equipment vendors.

Testing 5G QoE

5G introduces a new dimension and type of use cases; not only the physical test equipment required to sample the network, e.g., a wider bandwidth scanner, but also the methodology of what parameters to test for a specific application and how to post-process the data. There will be new KPIs that contribute to the evaluation of QoS and other factors that feed into QoE. QoE can be built up from the lower layers and use a model to define how QoS maps into QoE.

The key question is what does good QoE look like for a sensor in an industrial IoT deployment, or a connected car, or a VR device, or any specific 5G use case? In these use cases, we need to have a way of understanding whether QoE is good or bad and what the thresholds are. For a simple example, take call setup time. What is an acceptable “setup time” for a sensor alarm, or an autonomous car, or in remote medical use cases?

What may have been well defined in previous use cases, for example, a subscriber streaming a movie on their smartphone, may well not be transferable to 5G use cases. Attention will have to be given to the range of acceptable values of QoE for each specific application, below which it becomes a problem and above which it brings no additional value.

Therefore, it is apparent that testing 5G QoE, particularly for applications other than enhanced Mobile Broadband (eMBB), will require more metrics to be acquired with greater precision that will need to be post-processed more quickly and with greater complexity.

Standardization of testing

International standards organizations, such as the ITU and ETSI, are actively evolving their test models to cover the changes demanded by 5G, and this is something R&S is very actively engaged in. However, building 5G methodologies and standards is going to be complex when we contemplate all the use cases and remember that operators already have 200-400 core key performance indicators (KPI) to monitor.

The sheer amount of KPIs makes the understanding of quality of experience (QoE) in a granular way very difficult. Therefore, this is another change in the quality of service (QoS) environment driven by 5G, where there will be many more parameters to monitor, often in real time.

To evaluate and benchmark networks, KPIs are required that truly reflect the network’s performance so that based on such KPIs it is possible to define a fair and transparent performance scoring method. ETSI has taken the driver’s seat to discuss and define best practices for network benchmarking and scoring that enables the network to be characterized in a single, unified metric.

The method provides the operator with visibility of the status of their network and identifies the factors that influence quality. The factors and weightings that influence the scoring method will be adapted for each 5G use case and application. The robust fundamental methodology will provide the industry with an independent reference against which 5G QoS/QoE can be measured.

The test approach should cover two aspects. First, to release test solutions that enable measuring the technical aspects of 5G networks such as coverage, performance and operation; and second, to enhance our existing QoE methodologies to encompass 5G use cases.

The solutions for testing technical aspects of the 5G RAN are already being used by operators as they move from trials to the commercial deployment of 5G. The QoE solutions from lower layers up to signaling are being developed in partnership with the operators and standardization bodies.

In conclusion, we need to establish an understanding of what the requirements of each 5G use case are. Then we can build out key parameters and KPIs required to meet those requirements and their range and limits. Once we understand why we are testing certain parameters, we will have the test methodology to quantify QoE according to those KPIs.