Metrics & Challenges For Testing Streaming Applications In 2019

Image for post
Image for post

Streaming rich media has become an integral part of our daily lives. From watching tutorials on YouTube, Udemy etc. to playing RPGs(Role-Playing Games) on the internet, a major percentage of internet traffic nowadays spends their data on browsing through audio and video contents. With the data speed increasing day by day, media streaming has become the primary way of spreading information to the crowd.

Ensuring your media content runs seamlessly for every user who is visiting your website, irrespective of the device being used to relish the media content. If only it was as easy as picking a pie! The performance of a streaming application may vary depending on the network speed and browser, device, lag, format, frames and many more factors. This article, we will be looking into the challenges related to testing streaming applications in 2019 so your website or web-app doesn’t fail to deliver a delightful user experience.

What Is A Streaming Application?

Even when we talk about music streaming, the shift is clearly evident from audio cassettes and CDs to cloud-based, streaming applications such as Apple Music, Google Play Music and many more.

Technically speaking, instead of watching the media in a browser, streaming applications are advantageous because instead of consuming large data, they come with certain pre-downloaded features and download the required elements in the background while streaming.

When we think about the term Streaming applications then the first thing that pops to our mind is the video streaming websites or web applications. However, the scope of streaming applications also offers Gaming, Podcasts, Music etc.

Why Streaming Applications Became A Huge Hit?

Well, at present we have made the advancement towards Voice assistance. Just a few clicks or even voice command using Amazon’s Alexa can get you whatever content you want on your Smart TV, Desktop or handheld device.

Some renowned video streaming applications:

Some renowned gaming streaming applications are:

Fun Fact: If you are familiar with role-playing action games, you must have played or heard about Assassin’s Creed series and their latest addition to the anthology, Assassins Creed Odyssey. For people who don’t have a highly configured system required to run the game, Google is creating something that is going to be a revolution in the world of gaming. In Oct 2018, Google collaborated with Ubisoft to work on Project Stream. And after 3 months of Beta testing, they were successful to enable players from around the world to access and play Assassins Creed Odyssey streaming via the cloud.

Very soon, a day will come when we don’t need to buy an expensive graphics card for running games. Just like movies and TV series, they are also going to be accessed via a streaming application.

Streaming applications have taken rich media entertainment to a new level by providing users with the access to their favourite tv series, games, sports channel, and news channel at their fingertips. We have relied on streaming media primarily because we can access it from any time, any place as long as we have internet access of course.

Why Should We Test Streaming Applications? Aren’t They All About Internet Speed & Connectivity?

  • Completely breakage free
  • Can handle huge internet traffic
  • Flawlessly streams data under very low network condition
  • Operates across all devices without any responsive issues
  • When used on the desktop, no matter what browser is used does not cause any cross-browser compatibility issues.

However, all these scenarios can be checked only when you run appropriate testing scenarios that test all the required conditions. That is why all kinds of testing of streaming applications is a compulsion. But testing them can be a bit challenging since the tester has to think completely out of the box, sometimes from a user’s perspective and imagine scenarios when the application may cause breakage.

Metrics To Check While Testing A Streaming Application

Buffering Time — In YouTube you may have seen whenever data speed is low, a grey bar starts filling up before the video is played. This is known as the buffering bucket. Measuring the buffering time is very important to check how your app is performing under low network connectivity.

Image for post
Image for post

Bit Rate — Bitrate is a very important metric that measures the quality of your video. It can be measured by checking the number of bits transferred per second. The bit rate of a video is directly proportional to the resolution of the video.

Lagging Time — Once buffering is completed, the video is supposed to play smoothly if the download speed of your network keeps up with the average bit rate. Often you may have seen that whenever the video timeline reaches the buffering level, the video comes to a halt, starting more buffering sequence. Lagging time is an important metric that measures the total waiting time while playing a video. This also includes the initial waiting time for buffering.

Data Consumed — This metric includes the data spent by the user while using the application or playing any specific video. This helps to determine the demand for streamed data.

Lagging Ratio — This is the ratio of video play time and buffering time. It keeps into account the initial buffering time as well. The ratio is usually very low but it never falls below 1.

Platform Compatibility — This is a critical metric for every website or web-app but is highly significant in case of a streaming application. As a streaming application, you target users throughout the globe and making sure that your media rendering stays intact along with fast buffer is crucial. Sometimes the UA(user agent) of the end-user may lead to abrupt streaming.

Subtitles Synchronization — Subtitles are indispensable for a streaming web application or website if you wish to spread virality of your media content. Targeting an audience throughout the globe, you need to make sure to keep them away from accent specific dilemma. You can do that with the help of subtitles, but often subtitles tend to be out of synchronization which ends up being very annoying for the end user.

In order to ensure complete customer satisfaction, run the appropriate tests before launching it or releasing any updates. However, testing streaming applications can be a little tricky as you may have to predict the real-time user interaction with your web application. There are some common roadblocks or challenges of testing streaming applications. Let me address them for you.

Challenges Of Testing Streaming Applications

Response Time in Different Network

This is kind of challenging because most testing teams do not have access to test an application across different networks of varying data speed. Also, there are other conditions like when a customer is travelling, his device may move from one network to another. Conditions like that hugely impact the information flow and impacts on the app’s performance.

Lack of Devices For Cross Device Testing

The ideal scenario, in that case, is to use a cloud-based, cross-device and cross browser testing platform like LambdaTest where you can seamlessly test your application across hundreds of devices in parallel, brought up by VM(Virtual Machines) hosted through LambdaTest servers on the cloud.

Issues Occurring Due to End User Interaction

In YouTube, you may have seen often that under low network whenever a video is played at 240p and you tried to change it to 720p, it just halts and goes to an infinite buffering sequence. While testing, you have to keep in mind the behaviour of the end user. They may not be so tech-savvy and without understanding the network’s performance if they see that when they try to change the resolution, the player halts, they may give a negative review for your application.

Keep scenarios like that in mind during testing and suggest the developers develop fixes accordingly. The fixes may come in form of error message or disabling resolution changing on low network speed.

Cross Browser Compatibility Testing Of A Streaming Application

You can use virtual windows or emulators, but when the application runs in real time, an unforeseen error may arise which the emulator cannot detect. Often I have seen multiple websites causing breakages when running on older versions of Mac and Safari.

LambdaTest provides you with manual and automated cross browser testing through Selenium grid consisting more than 2000+ browsers and their different versions, all running on multiple operating systems such as Mac, Windows or Linux for desktops & iOS or Android for mobile website testing.

Is Your Streaming Application Durable Enough To Entertain Excessive Traffic?

With such a huge population of the world, it gets challenging for a tester to guess the right number of maximum user count that can access a specific video at any given point of time. Load and performance testing of a streaming application seriously demands some out of the box approach.

Is Your Streaming Application Capable Of Handling Streaming Techniques?

Thankfully those days are gone. We have MPEG DASH, Smooth by Microsoft, Adobe’s HDS and Apple’s HLS. All of these are dynamic streaming technologies allow transferring of large chunks of data using very little bitrate.

Wrapping It Up

So, keep yourself updated as a tester with all the latest technologies and if you are the product owner, make sure that no matter what it relates to, whether your app is a gaming platform or a video/music streaming application, or an application for hosting any kind of services, it overcomes all the challenges and can be accessed by the user anywhere without any lag or complications.

Originally published at LambdaTest

Author Arnab Roy Chowdhury

Image for post
Image for post

Written by

Product Growth at @lambdatesting (

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store