Development of the Tool for Evaluation of Quality of Mobile Data Services

Mobile operators are interested to deliver excellent quality mobile services in order to differentiate themselves from other mobile radio networks. Therefore monitoring of the perception of the subscribers (accessibility, reliability and quality of a service) is extremely important. Mobile operators usually do not have technical feedback from the subscribers on the network services quality. However the customer perceptions do not always correlate with statistics, obtained by network performance counters or drive tests. Professional tools for the assessment of QoS in radio access networks (Ascom TEMS, Q-voice, etc.) are oriented for the deep analysis of the system parameters. They are used for the drive testing by the operators but are not suitable for the implementation in to mobile terminals of all customers as they are very complex and have high price level. Tools freely available on the Internet have poor functionality and are not configurable (no assurance and low reliability of the measurements). Performance benchmarking tool named VNTT (VGTU Network Testing Tool) is presented and analyzed in this paper. VNTT is a software application that interacts with mobile terminal and measures parameters related to the network and services quality. Measurements of parameters are made in passive and active modes. The application runs in the background and is transparent to the user. It could be used in home and roaming networks. When the application detects a sequence of network or service limitations, the feedback form could be displayed on the screen of the user terminal. Additionally the customer can use this form to provide feedback to the operator about the quality of received service and behavior of mobile applications. The operator correlating network and subscriber information on the quality of service could identify regions where investments and optimization will provide the highest benefits in terms of improved customer satisfaction. Main parameters needed to describe mobile network performance in formal way are widely analyzed in many works [1–4]. Selection of the measured parameters for the VNTT and benefits of analyzing these QoS characteristics presented below.


Introduction
Mobile operators are interested to deliver excellent quality mobile services in order to differentiate themselves from other mobile radio networks.Therefore monitoring of the perception of the subscribers (accessibility, reliability and quality of a service) is extremely important.Mobile operators usually do not have technical feedback from the subscribers on the network services quality.However the customer perceptions do not always correlate with statistics, obtained by network performance counters or drive tests.
Professional tools for the assessment of QoS in radio access networks (Ascom TEMS, Q-voice, etc.) are oriented for the deep analysis of the system parameters.They are used for the drive testing by the operators but are not suitable for the implementation in to mobile terminals of all customers as they are very complex and have high price level.Tools freely available on the Internet have poor functionality and are not configurable (no assurance and low reliability of the measurements).
Performance benchmarking tool named VNTT (VGTU Network Testing Tool) is presented and analyzed in this paper.VNTT is a software application that interacts with mobile terminal and measures parameters related to the network and services quality.Measurements of parameters are made in passive and active modes.The application runs in the background and is transparent to the user.It could be used in home and roaming networks.
When the application detects a sequence of network or service limitations, the feedback form could be displayed on the screen of the user terminal.Additionally the customer can use this form to provide feedback to the operator about the quality of received service and behavior of mobile applications.
The operator correlating network and subscriber information on the quality of service could identify regions where investments and optimization will provide the highest benefits in terms of improved customer satisfaction.
Main parameters needed to describe mobile network performance in formal way are widely analyzed in many works [1][2][3][4].Selection of the measured parameters for the VNTT and benefits of analyzing these QoS characteristics presented below.Most important parameters describing performance of the mobile network are not only actively measured data throughput, delay and service access time [5][6][7][8][9][10][11][12], but also passively measured information about serving cell and signal level, used access technology in use, information of the terminal manufacturer and model, network availability and stability in idle mode.Additionally to the information about network and performance of its services -information of the customer location is also very important [8,9].It allows monitoring and improving services quality in the usual locations of the customers.
VNTT supports measurement of defined parameters in GSM, GPRS/EDGE and UMTS/HSPA networks, however any other mobile networks could be analyzed with minor software changes.
Measurements of the main parameters of mobile data services and verification of the results achieved by the analyzing at the packet level as well as comparison with other tools are presented in this paper.

VNTT description
Complete measurement system consists of software client part so named VNTT and the server part which may vary depending on the circumstances.Overall system describing picture is presented at the Fig. 1.VNTT client could be installed at the customer computer and also could be implemented as an application at the mobile phone, when data services are accessed using smart phones.
Main VNTT difference from other available tools is that it performs permanent QoS monitoring and in parallel customer has a possibility to initiate a performance test of mobile data services in the customer location and to complain if quality of service is unacceptable.
Measurements of the parameters are made in accordance to the ETSI (ETSI TS 102 250-2 V1.If GPS data is not available then obtained mobile cell number could be used to define the customer location. Active mode measurements are made by using libraries of the command line tool "cURL" (http://curl.haxx.se),active mode measurement results are:

Verification of measurement accuracy of VNTT software client
In order to test the measurement accuracy of the VNTT, throughput measurement results were compared with the results obtained by the interface speed monitoring tool set "NetWorx" -monitor application "SpeedMeter".Data throughput was measured with the both tools simultaneously, while downloading predefined data files from the server.Results of the experiment are presented at the Fig. 3 below.

Fig. 3. Comparison of SpeedMeter and VNTT data transfer speed measurement results
As we could observe from the comparison resultsthey are almost identical, average difference between results is 0.02kB/s.
Additional analysis at the packet transmitting level was done with WireShark [8].Comparison of WireShark and VNTT measurement results of the test file download time are presented in the Fig. 4 below.Active measurements involve upload and download of the test data.In order to measure available data throughput -test file must be within proper size.
Example of strong relation of the file size and throughput is presented at the Fig. 5, where throw same mobile channel files of different size are send.From the shape of the approximated curve we can observe that throughput had logarithmic dependency from the file size, general formula for such case is where C(x) is throughput; x -is transmitted file size and coefficients a ≈53 and b≈-147, then This dependency is valid until same technology with same configuration of time slots is used for data transmitting.During tests it was observed that in order to measure maximum channel throughput -minimum size of the test file should be 5MB for the downlink measurements, when testing HSPA3.6 access technology.Data packets round trip time also depends on the packet size.In Fig. 6 each measurement with same packet size was performed 100 times and average values are presented graphically.From Fig. 6 we could see that delay rapidly decreases from the 400 Bytes and then again starting to grow.This could be explained as for the different packet sizesdifferent WCDMA/HSPA traffic channels are used.Therefore in the VNTT measurements of the packet round trip time are performed using two packet sizes: 200 Bytes to evaluate RTT for light applications (chat, web browsing) and 1000 Bytes -for heavy applications (video conferencing and streaming).

Conclusions
Monitoring of the perception of the subscribers (accessibility, reliability and QoS) is extremely important [5,6,7].However the customer perceptions do not always correlate with statistics obtained by network performance counters or seldom performed drive tests.VNTT was designed in order to collect the missing part of QoS information from the real customers.
Main VNTT difference from other available tools is that it performs permanent QoS monitoring and in parallel customer has a possibility to initiate a performance test of mobile data services in the customer location and to complain if quality of service is unacceptable.
VNTT active tests show the best performance during semi-stationary measurements (usual behaviour of mobile data users in their usual in-door locations).Fast movement condition raise deviation of the radio parameters and cell changes which could lead to measurement of different QoS parameters in different cells -different locations.
During the field tests Active test parameters were evaluated and predefined for the GSM/GPRS and UMTS/HSPA technologies.Wrong selection of the test files and packets sizes could result that the maximum performance of the technology will be never archived.

Benefits
For mobile operator • Network coverage and quality in usual customers location (in-door at work and at home) • Network coverage and quality in roaming networks • Handsets quality, sensitivity and failures • Usage of different services and its quality • Customer experience and satisfaction For the customer • Permanent monitoring of quality of subscribed services • Easy to complain if quality of service is unacceptable • Easy to test performance of mobile data services in the customer location by initiating an Active mode • QoS dependant pricing and SLA could be implemented between mobile operator and customer Collected parameters Passive mode • Network availability in idle mode • Information about serving cell and signal level • Used access technology (GSM, UMTS, LTE) • Customer location (serving cell and GPS data) • Information of the terminal model and customer ID Active mode (additionally) • Average data throughput in uplink and downlink at the user location • Data packets delay and packet losses • Service access time of predefined WEB page • Quantity of the interrupted data sessions • Reasons why data session sessions were aborted http://dx.doi.org/10.5755/j01.eee.113.7.621

Fig. 1
Fig. 1 Network performance measurement system based on user terminal equipment with installed VNTT -VGTU Network Testing Tool

Fig. 2
Fig. 2 Architecture of the VNTT software Definition of the customer location is part of the passive measurements using GPS data which is collected by listening communication port where GPS receiver is attached.Parsed and saved GPS parameters are: UTC time, Latitude + Longitude, GPS quality indicator + HDOP, Height.If GPS data is not available then obtained mobile cell number could be used to define the customer location.Active mode measurements are made by using libraries of the command line tool "cURL" (http://curl.haxx.se),active mode measurement results are: 1.Average data throughput in uplink and downlink at the user location is measured.Different files are sent in uplink and down-link.Settings are: FTP servers; test files; quantity of repeating.Results: average speed of the file uploading and downloading and failure rates; 2. Data packets delay and packet losses are measured with ping engine.Measurements parameters are: addresses of the RTT servers; size of the packet in bytes; quantity of the packets.Parameters measured: quantity of the send received and lost packets as well as packets round trip delay time; 3. Service access time of pre-defined WEB page is measured using web client.Settings are: http server address; quantity of repetitions; time to wait for the answer.Results are: speed of the content downloading and time to access complete WEB page; 4. Data session related events (number and duration of the sessions; setup/blocked/dropped success rates; reasons for abnormal terminations) are recorded after predefined sequences of such test requests.

1 .
Average data throughput in uplink and downlink at the user location is measured.Different files are sent in uplink and down-link.Settings are: FTP servers; test files; quantity of repeating.Results: average speed of the file uploading and downloading and failure rates; 2. Data packets delay and packet losses are measured with ping engine.Measurements parameters are: addresses of the RTT servers; size of the packet in bytes; quantity of the packets.Parameters measured: quantity of the send received and lost packets as well as packets round trip delay time; 3. Service access time of pre-defined WEB page is measured using web client.Settings are: http server address; quantity of repetitions; time to wait for the answer.Results are: speed of the content downloading and time to access complete WEB page; 4. Data session related events (number and duration of the sessions; setup/blocked/dropped success rates; reasons for abnormal terminations) are recorded after predefined sequences of such test requests.

Fig. 4 .
Fig. 4. Comparison of WireShark and VNTT time measurement results Average difference between measured results, presented at the Fig. 4 is 0.01 seconds, what can be explained by the accuracy of the OS clock and different averaging accuracy of the WireShark and VNTT engines.

Field tests and parameterization
of the VNTT Passive measurements are made without uploading and downloading additional files, mainly measured are position of the user, cell ID and network status.

Fig. 5 .
Fig. 5. Data throughput dependency on the transmitted file size

Fig. 6 .
Fig. 6.Data packet round trip time dependency on the transmitted packet size

Table 1 .
Selection of the measured parameters