对于网络视频质量度量标准的探索毕业论文外文翻译

《对于网络视频质量度量标准的探索毕业论文外文翻译》由会员分享，可在线阅读，更多相关《对于网络视频质量度量标准的探索毕业论文外文翻译（23页珍藏版）》请在装配图网上搜索。

1、A Quest for an Internet Video Quality-of-Experience MetricAthula Balachandran Vyas SekarAditya AkellaSrinivasan Seshan Ion StoicaHui ZhangCarnegie Mellon UniversityUniversity of Wisconsin MadisonStony Brook UniversityUniversity of California BerkeleyABSTRACTAn imminent challenge that content provide

2、rs, CDNs, thirdparty analytics and optimization services, and video player designers in the Internet video ecosystem face is the lack of a single “gold standard” to evaluate different competing solutions. Existing techniques that describe the quality of the encoded signal or controlled studies to me

3、asure opinion scores do not translate directly into user experience at scale. Recent work shows that measurable performance metrics such as buffering, startup time, bitrate, and number of bitrate switches impact user experience. However, converting these observations into a quantitative quality-of-e

4、xperience metric turns out to be challenging since these metrics are interrelated in complex and sometimes counter-intuitive ways,and their relationship to user experience can be unpredictable.To further complicate things, many confounding factors are introduced by the nature of the content itself (

5、e.g., user interest, genre). We believe that the issue of interdependency can be addressed by casting this as a machine learning problem to build a suitable predictive model from empirical observations. We also show that setting up the problem based on domain-specific and measurement-driven insights

6、 can minimize the impact of the various confounding factors to improve the prediction performance.Categories and Subject DescriptorsC.4 Performance of Systems: measurement techniques,performance attributesGeneral TermsHuman Factors, Measurement, Performance1. INTRODUCTIONWith the decreasing cost of

7、content delivery and the growing success of subscription and ad-based business models(e.g., 2), video traffic over the Internet is predicted to increase in the years to come, possibly even surpassing television based viewership in the future 3. An imminent challenge that all players in the Internet

8、video ecosystemcontent providers, content delivery networks, analytics services, video player designers, and usersface is the lack of a standardized approach to measure the Quality-of-Experience (QoE) that different solutions provide. With the “coming of age” of this technology and the establishment

9、 of industry standard groups (e.g., 13), such a measure will become a fundamental requirement to promote further innovation by allowing us to objectively compare different competing designs 11,17.The notion of QoE appears to many forms of media and has a rich history in the multimedia community (e.g

10、., 9, 10,14, 15). However, Internet video introduces new effects interms of measuring both quality and experience:Measuring quality: Internet video is delivered using HTTP-based commodity technology over a largely unreliable network via existing CDN infrastructures. Consequently, the traditional enc

11、oding-related measures of quality like Peak Signal-to-Noise Ratio are replaced by a suite of quality metrics that capture several effects introduced by the delivery mechanismbuffering, bitrate delivered, frame rendering rate, bitrate switching, and startup delay 6, 33.Measuring experience: In the co

12、ntext of advertismentand subscription-supported services, the perceptual opinion of a user in a controlled study does not necessarily translate into objective measures of engagement that impact providers business objectives. Typical measures of engagement used today to approximate these business obj

13、ectives are in-the-wild measurements of user behavior; e.g., fraction of a particular video played and number of visits to the provider 6, 33.To obtain a robust QoE measure, we ideally need a unified and quantitative understanding of how low-level quality metrics impact measures of experience. By un

14、ified, we want to see how the set of quality metrics taken together impact quality, as opposed to each metric in isolation. This is especially relevant since there are natural tradeoffs between the metrics; e.g., lower bitrate can ensure lower buffering but reduces the user experience. Similarly, by

15、 quantitative, we want to go beyond a simple correlational understanding of “metric M impacts engagement”, to a stronger statement of the form “changing metric M from x to x changes engagement from y to y”.Unfortunately, the state of the art in our understanding of video QoE is limited to a simple q

16、ualitative understanding of how individual metrics impact engagement 19. This leads to severe shortcomings for every component of the video ecosystem. For example, adaptive video players today resort to ad hoc tradeoffs between bitrate, startup delay, and buffering 16,20,32. Similarly, frameworks fo

17、r multi-CDN optimization use primitive QoE metrics that only capture buffering effects without accounting for the impact of bitrate or bitrate switching 28, 29. Finally, content providers do not have systematic ways to evaluate the cost-performance tradeoffs that different CDNs or multi-CDN optimiza

18、tions offer 1.We observe that there are three key factors that make it challenging to obtain a unified and quantitative understanding of Internet video QoE:Complex relationships: The relationships between the quality metrics and the effective user experience can be quite complex and even counter-int

19、uitive. For example, while one would naturally expect a higher video bitrate leading to better user experience, we observe a non-monotonic relationship between the two.Metric dependencies: The metrics themselves have subtle interdependencies and have implicit tradeoffs. For example, although switchi

20、ng bitrates to adapt to the bandwidth conditions can reduce buffering, we observe that high rates of switching can annoy users.Impact of content: There are many confounding factors introduced by the nature of the content itself. For example, different genres of content such as live and videoon-deman

21、d (VOD) show very different viewing patterns. Similarly, users interest in content also affects their tolerance non-trivially.Our goal in this paper is to identify a feasible roadmap toward developing a robust, unified and quantitative QoE metric that can address these challenges. We have two intuit

22、ive reasons to be hopeful. The challenges raised by complex relationships and subtle interdependencies can be addressed by casting QoE inference as a machine learning problem of building an appropriate model that can predict the user engagement (e.g., play time) as a function of the various quality

23、metrics. The second issue of content-induced effects can be addressed using domain-specific and measurementdriven insights to carefully set up the learning tasks.Our preliminary results give us reason to be optimistic.For example, a decision tree based classifier can provide close to 50% accuracy in

24、 predicting the engagement. Carefully setting up the inputs and features for the learning process could lead to as high as 25% gain in accuracy of the prediction model.Figure 1: Overview of the Internet video ecosystem; a robust QoE metric is critical for every component in this ecosystem.The rest o

25、f this paper is organized as follows. Section 2 describes how a standardized QoE metric would impact the different players in the video ecosystem. Section 3 describes the main challenges in developing a QoE metric. Section 4 makes the case for a predictive model for developing a QoE metric. In Secti

26、on 5, we present some preliminary results before discussing various challenges in Section 6. We conclude in Section 7.2. USE CASES FOR VIDEO QOEWe begin with a brief overview of the Internet video ecosystem today and argue why there is an immediate need for a standardized QoE metric and how this imp

27、acts the different players in the video ecosystem (Figure 1).Content providers like HBO, ABC, and Netflix would like to maximize their revenues from subscription and adbased business models while trying to minimize their distribution costs. To this end, content providers have business arrangements w

28、ith CDNs (e.g., Akamai, Limelight) and also with third-party analytics (e.g., Ooyala 8) and optimization services (e.g., Conviva 5). A robust QoE metric enables content providers to objectively evaluate the cost-performance tradeoffs offered by the CDNs and the value that such third-party services o

29、ffer.Content Distribution Networks need to allocate their distribution resources (e.g., server and bandwidth capacity) across user population. They need standard metrics to demonstrate superior cost-performance tradeoffs. CDNs also need such metrics to guide the design of their delivery infrastructu

30、res to minimize their delivery costs while maximizing their performance 24.Recent studies have argued the case for cross-CDN optimization 28, 29 and there are already commercial services (e.g., Conviva 5) that provide these capabilities. These services need standard measures to demonstrate quantifia

31、ble value to the content providers. An open challenge that such optimization frameworks face is the choice of a suitable quality metric that needs to be optimized 29.Similarly, third-party video analytics services need concrete ways to translate their insights with respect to user demographics and u

32、ser behaviors into quantitative engagement effects.Video player designers have to make conscious tradeoffs in their bitrate adaptation algorithms. For example, moving to a higher bitrate may offer better engagement but increases the risk of buffering that is known to annoy users. Similarly, user stu

33、dies suggest that users cannot tolerate too frequent bitrate switches as it impacts their perceptual experience 18. The lack of a systematic understanding of video QoE forces player designers to use ad hoc adaptation strategies without a clear optimization goal 16, 20, 32.Ultimately, the success of

34、this ecosystem depends on the users experience. Increasingly, the same content (TV shows, movies) is available from multiple providers (e.g., Apple iTunes, Amazon, Hulu Plus, Google). Beyond issues of content availability, users would prefer services that give them a better cost-experience tradeoff.

35、 Another issue relates to the recent introduction of ISP bandwidth quotas 4; an understanding of video QoE enables users and delivery providers to better customize the experience under such constraints. There appears to be rough consensus among the leading industry players on two accounts. First, th

36、ere is implicit agreement on the set of quality metrics and the measures of engagement 6, 19, 33. Second, there is also a growing realization in this community of the need for a data-driven approach using measurements in-the-wild as opposed to traditional methods of using controlled user studies. Th

37、e key challenge that remains is providing a unified and quantitative understanding of the relationship between the quality metrics and the engagement measures. As we will show in the next section, this turns out to be non-trivial.3. CHALLENGES IN MEASURING QOEIn this section, we use real world measu

38、rements of client viewing sessions from two large content providers, one serving TV episodes and the other providing live sports events, to highlight challenges in QoE measurement. We use industrystandard video quality metrics for our study 6. For concreteness we focus on play time as the measure of

39、 user engagement in this section. A subset of the observations we present have also appeared in other measurement studies (e.g.,12,19,2123,26,31), albeit in other contexts. Our specific contribution lies in highlighting the challenges these raise for developing a unified QoE metric.3.1 Complex relat

40、ionshipsThe relationship between different quality metrics and user engagement are complex. These were extensively studied in 19 and we reconfirm some of these observations. Counter-intuitive effects: Although, intuitively higher rendering quality (frames per second) should lead to higher user engag

41、ement, Dobrian et al. noticed several instances where lower rendering quality led to long play times, especially in the case of live videos 19. This is because of an optimization by the video player to reduce CPU consumption by reducing the frame rate when the video is played in the background. User

42、s may run live videos in the background while focusing on other work, but would be compelled to close the player if the CPU usage was high.Non-monotone effects: Although higher average bitrate should result in higher user engagement, prior work has observed that there is a non-monotonic relationship

43、 between the two 19. This is because CDNs serve content at specific bitrates and the values of average bitrates in between thesestandard bitrates correspond to clients that had to switch bitrates during the session. These clients likely experienced higher buffering and hence the lower engagement.Thr

44、eshold effects: We also observed that the rate of switching has a threshold effect on user engagement. Rates up to 0.5 switches/minute do not have any effect on the user engagement. However, at higher rates, users seem to quit early as shown in Figure 2. Our observation corroborates the user studies

45、 made in 18.3.2 Interaction between metricsNaturally, the various quality metrics are not independent of each other. The interaction and implicit tradeoffs between these metrics also needs careful examination. Next, we highlight some of these interesting tradeoffs: Switching vs buffering: An efficie

46、nt bitrate switching algorithm should pro-actively switch bitrates to avoid buffering events within a session and also let the user experience the best possible video quality based on the current network conditions at the client. However, as shown in Figure 2, high rates of switching annoys users le

47、ading them to quit early.Join time vs bitrate: Although higher bitrate would imply higher quality, it would also imply higher join time since it would take longer time for the player video buffer to fill up sufficiently to start rendering video frames. Figure 3: User viewing pattern of live and VOD

48、videos Figure 4: User interest induced by regional games3.3 ExternalitiesThere are many confounding external factors that affect user engagement and are not captured by the quality metrics. For instance, user-attributes like bandwidth at the client and its variability, and content attributes like ge

49、nre, popularity and age of the content have effects on user engagement. Genre of the content: We observed that live and VOD video sessions experience similar qualitye.g., Figure 3b shows the cdf of buffering ratio (fraction of session time spent in buffering) for live and VOD sessions. However, as s

50、hown in Figure 3a, the distribution of the fraction of video viewed by users over several popular video objects is considerably different. This shows that the viewing pattern of live and VOD sessions are very different.User interest: As observed in Figure 3a, several users tend to sample videos resu

51、lting in many of them quitting early based on interest despite the lack of any quality issues during the session. Another example of how user interest affects user engagement is in the case of regional sporting events.Figure 4a shows the distribution of play time of users watching a regional sportin

52、g event segregated into whether the viewer was from the region or not. On average, local users watch 10 minutes more of the video than non-local users despite the fact that the quality of the video was the same (e.g.,Figure 4b shows the cdf of the buffering ratio in both cases).4. TOWARD A PREDICTIV

53、E MODEL FOR QOE INFERENCEAt a high-level, our goal is to express engagement as a function of the quality metrics, given other factors such as content and user attributes. We want to capture a relationship Engagement = f (QualityMetric), where Engagement can be the video playtime, number of visits to

54、 a website,and each QualityMetric represents observed indices such as buffering ratio, average bitrate. Assuming that this function f is accurate, we can simply use this as our QoE metric to compare two competing solutions. For e.g., a content provider can use observed values of the quality indices

55、for different CDNs or multi-CDN optimizers, and compute the QoE they offer.Figure 5: High-level overview of our proposed approach forderiving a QoE metric via a predictive modelThe foremost question here is whether we can build an accurate model. As we saw in the previous section, there are three hi

56、gh-level challenges in going from low-level quality indices (e.g., buffering, bitrate, bitrate switching, startup latency) to a quantitative and unified QoE metricthere are complex relationships between these metrics and the engagement; there are hidden dependencies among the metrics themselves (e.g

57、, bitrate vs. startup latency); and there confounding effects of user interest, content genre, and user connectivity. We believe that these issues are fundamental and that any video QoE metric must address these challenges.We see two intuitive reasons to be hopeful about the feasibility of building

58、such a predictive model: First, the issue of dependencies and hidden relationships can be handled by choosing a suitable machine learning approach that is expressive enough to capture complex dependencies. Beyond choosing a robust learning approach, the key to success here is the availability of suf

59、ficiently large datasets. Fortunately, many content providers, CDNs, and third-party optimizers and analytic services already have the requisite instrumentation in client- and server-side infrastructure to collect such measurements 6,19. Second, the issue of confounding effects can be tackled by car

60、efully incorporating these domain-specific insights into the learning process. This can be achieved in one of two ways: (1) carefully set up the input data for the machine learning task and (2) explicitly identify such confounding features and let the learning algorithm automatically account for the

61、se factors.Figure 5 presents a high-level overview of our vision for building a unified and quantitative QoE metric. We use measurements of quality and engagement metrics and optionally take into account user actions during video sessions to build a basic prediction model. We further refine the pred

62、iction using domain-specific knowledge of content-induced and userinduced effects.Figure 6: Evaluating different models that predict user engagement from quality metrics5. PRELIMINARY RESULTSIn this section, we show preliminary results to confirm the intuition outlined in the previous section. Our r

63、esults are based on 10-fold cross-validation 30 on around 10 million video sessions spanning a month of viewership.Strawman solutions: As mentioned in Section 4, we setup a learning problem where the objective is to predict the user engagement given the various quality metrics. We use play time to m

64、easure user engagement. We categorize the play time into classes based on the fraction of the video that the user viewed. For example, when the number of classes is set to 5, the model tries to predict if the user viewed 0-20% or 20-40% or 40-60% or 60-80% or 80-100% of the video. By varying the num

65、ber of classes, we change the granularity at which the model predicts user engagement. The quality metrics are also appropriately classified into fine-grained classes.The choice of the learning algorithm is crucial as it should be adept at learning the various complex relationships and interdependencies between the different quality metrics. We employed learning algorithms such as naive Bayes, simple regression and classical binary decision tree from the machine learning literature 30 to learn a predictive model. As shown in Figure 6a, d