Implementing Cue Points Programmatically

Product(s)
Video Cloud
Brightcove Player
Role(s)
Player Developer
Topic(s)
Cue Points

In this topic, you will gain an understanding of cue points with Brightcove Player. You will then see how to programmatically create cue points and handle when a cue point is dispatched.

Overview

Cue points can be set for a video. During video playback, an event is dispatched when each cue point is hit.

Play the following video to see the player display cue point information at pre-roll, mid-roll (5 seconds, 10 seconds, 12 seconds) and post roll times.

 

******** Cue Point Information ********

 

****** End Cue Point Information ******

Play this video and you will see information for four cue points displayed below the player.

 

 

Key concepts

A few concepts must be understood to effectively use cue points in Brightcove Player. These concepts are explained in this section of the document.The following concept must be understood to effectively use cue points in Brightcove Player.

Video Cloud catalog cue points

The first concept to understand has to do with terminology. In Brightcove Player, cue points are stored as text track elements, per the HTML standard. This means that when a Video Cloud video is used, any "Video Cloud style" cue points are converted into text tracks.

These "Video Cloud style" cue points can also be referred to as "catalog" cue points since they are read from the Video Cloud catalog (See the Player Catalog document for more information on the catalog). When this conversion takes place, some of the information in the catalog cue point, like type and cue point time, are converted into the text track.

Catalog cue point structure

The next concept to understand is that there is significant difference between catalog cue points and the HTML standard in structure. The difference is that an HTML cue point can have a duration. This means for every cue point, two cue point change events will be dispatched: one on start of the cue point and one on end of the cue point.

In the conversion process, every catalog cue point is converted into a cue point with identical start and stop times. This means that for every catalog cue point, two cue point events will be dispatched and has to be accounted for in code.

activecues array

AnotherAn important conceptual point to understand when dealing with HTML cue points is the activeCues array. All the cue points are defined in an array. There is another array, activeCues, which holds the cue points which are "active", meaning the playback time is between the cue point's start and stop times.

When using catalog cue points, the start and stop times are identical, so they will be active only for the second defined. Also, with catalog cue points, it is highly unlikely two cue points will be active at the same time since they will not overlap.

Cue point types

In this document you will see cue points with a type property assigned a value. These type values are assigned when creating cue points in the Studio user interface. There are two types of cue points, but both simply supply a string value to the type property. The appropriate value is helpful only if used in custom JavaScript while processing the cue point information. The types are detailed here:

  • Ad - Assigns the string value of AD to the type property
  • Code - Assigns the string value of CODE to the type property

Video Cloud cue points

In this section of the document, you will learn to set catalog cue points and listen for catalog cue point event dispatches.

Setting Video Cloud cue points

Video Cloud cue points can be associated with a video using the Video Cloud Studio, and a number of other ways, detailed in this document: Working with Cue Points in the Media Module.

Acting on Video Cloud cue points - video statically bound to the player

In this section of the document, you will learn to process catalog cue points when the video has been statically bound to the player, meaning the video has been loaded in the player either in Studio or using the Player Management API directly.

To prevent a race condition of trying to process cue points before they are loaded, you need to use the loadedmetadata event to be dispatched before dealing with the cue points. Once the proper text track has been read, use the oncuechange event to listen for cue points events to be dispatched.

The following code shows listening for cue points and displaying the data from the cue point. Note that in this example the video is statically bound to the Player.

  • Line 24: Create a paragraph element as a location in which to inject dynamically created HTML.
  • Lines 31,43: Use the one() method to add an event listener for the loadedmetadata event only once. The event handler function is defined here anonymously.
  • Line 32: Retrieve the TextTracks array using the textTracks() method, then assign the zeroth element, which holds the cue points, to the variable tt. Note that it may be possible in some implementations that the cue points could be in a different array element. See the Find correct track section below for more information.
  • Lines 33,41: Sets the event handler function for when the oncuechangeevent is dispatched.
  • Line 34: Check to be sure that you are getting the first (zeroth array element) cue point dispatch. Without this condition, you would see that each cue point would be acted upon twice. Note that if you are using cue points that have durations that overlap this condition would have to be different.
  • Lines 35-39: Dynamically create HTML using information from the cue point and inject into the HTML page.
  • Line 42: Play the video.
<video-js id="myPlayerID"
data-account="1507807800001"
data-player="6aac508a-741e-41a2-bc7a-5805644cf089"
data-embed="default"
data-video-id="1507781667001"
class="video-js" controls></video-js>

<p id="insertionPoint"></p>
<script src="https://players.brightcove.net/1507807800001/6aac508a-741e-41a2-bc7a-5805644cf089_default/index.min.js"></script>
<script>
  videojs.getPlayer('myPlayerID').ready(function() {
    var player = this;
    player.one("loadedmetadata", function () {
      var tt = player.textTracks()[0];
      tt.oncuechange = function () {
        if (tt.activeCues[0] !== undefined) {
          var dynamicHTML = "id: " + tt.activeCues[0].id + ", ";
          dynamicHTML += "text: " + tt.activeCues[0].text + ", ";
          dynamicHTML += "startTime: " + tt.activeCues[0].startTime + ",  ";
          dynamicHTML += "endTime: " + tt.activeCues[0].endTime;
          document.getElementById("insertionPoint").innerHTML += dynamicHTML + "<br/>";
        }
      }
      player.play();
    });
  });
</script>

Acting on Video Cloud cue points - dynamically load video in the player

In this section of the document, you will learn to process catalog cue points when the video is dynamically loaded into the player using the catalog.getVideo() and catalog.load() methods.

When you use the player catalog to get and load the video, processing cue points is bit easier than when you use a statically bound video, as you do not need to use the loadedmetadata event.

  • Line 76: Create a paragraph element as a location in which to inject dynamically created HTML.
  • Lines 83,97: Use the catalog.getVideo() method to retrieve a video. The callback function is defined here anonymously.
  • Line 85: Use the catalog.load() method to load the video into the player.
  • Line 86: Retrieve the TextTracks array using the textTracks() method, then assign the zeroth element, which holds the cue points, to the variable tt. Note that it may be possible in some implementations that the cue points could be in a different array element. See the Find correct track section below for more information.
  • Lines 87,95: Sets the event handler function for when the oncuechangeevent is dispatched.
  • Line 88: Check to be sure the first (zeroth array element) cue point is defined. Without this condition you would see that each cue point would be acted upon twice, the second time with no defined elements in the activecues array (since the second cue change is for cue point end). Note that if you are using cue points that have durations that overlap this condition would have to be different.
  • Lines 89-93: Dynamically create HTML using information from the cue point and inject into the HTML page.
  • Line 96: Play the video.
<video-js id="myPlayerID"
  data-account="1507807800001"
  data-player="3c59134e-89e8-4864-87e2-3cc871f8c361"
  data-embed="default"
  data-video-id=""
  class="video-js" controls></video-js>

  <p id="insertionPoint"></p>

  <script src="https://players.brightcove.net/1507807800001/3c59134e-89e8-4864-87e2-3cc871f8c361_default/index.min.js"></script>

  <script>
  videojs.getPlayer('myPlayerID').ready(function() {
    var player = this;
    player.catalog.getVideo('1507781667001', function (error, video) {
      //deal with error
      player.catalog.load(video);
      player.one("loadedmetadata", function () {
        var tt = player.textTracks()[0];
        tt.oncuechange = function () {
          if (tt.activeCues[0] !== undefined) {
            var dynamicHTML = "id: " + tt.activeCues[0].id + ", ";
            dynamicHTML += "text: " + tt.activeCues[0].text + ", ";
            dynamicHTML += "startTime: " + tt.activeCues[0].startTime + ",  ";
            dynamicHTML += "endTime: " + tt.activeCues[0].endTime;
            document.getElementById("insertionPoint").innerHTML += dynamicHTML + "<br/>";
          }
        }
      });
      player.play();
    });
  });
</script>

Retrieve all Video Cloud cue point info

As you may have noticed not all Video Cloud cue point information is directly available to you from the activecues array. This is easily remedied by retrieving the desired info from the mediainfo property.

The basic approach of this solution is:

  1. Wait for the loadstart event for the mediainfo property to be populated.
  2. Assign the cue_points array from the mediainfo property to a variable. This variable contains complete Video Cloud cue point information.
  3. On a cue point event, retrieve the corresponding cue point data based on the time property value. This will be done using a helper function that grabs an object out of an array based on a property value in the object.
  4. Use the data from the cue point.

The following image shows the entire cue point array (top-left), the single cue point data collection (top-right), and one property from that single cue point data collection (bottom-right).

cue point data

In the snippet below only the new/changed code from examples above will be explained.

  • Lines 452-463: Include the function that will be used to extract a single cue point data collection from the array of all cue points. Note that you pass it the array of all cue points, the property in which you are searching for a specific value, and lastly the value you are searching for.
  • Line 432: Listens for the loadstart event. When your video starts to load, then the mediainfo property is populated.
  • Line 434: Assign the array of all Video Cloud cue points to a variable.
  • Line 443: In the cue point dispatch event handler, assign the specific cue point's data collection to a variable. This is where the function mentioned in the first bullet point is called. The arguments used are:
    • cuePointAra: The entire collection of Video Cloud cue points.
    • 'time': The property in which to search for a value.
    • tt.activeCues[0].startTime: The start time of the cue point currently being handled in the cue point dispatch event handler.
  • Lines 444-445: Debugging console.log() method calls, which should be removed in production code.
<script>
  videojs.getPlayer('myPlayerID').ready(function() {
    var myPlayer = this,
      cuePointAra = [],
      allCuePointData;
    myPlayer.on('loadstart', function () {
      //console.log('mediainfo', myPlayer.mediainfo);
      cuePointAra = myPlayer.mediainfo.cue_points;
      var tt = myPlayer.textTracks()[0];
      tt.oncuechange = function () {
        if (tt.activeCues[0] !== undefined) {
          var dynamicHTML = "id: " + tt.activeCues[0].id + ", ";
          dynamicHTML += "text: " + tt.activeCues[0].text + ", ";
          dynamicHTML += "startTime: " + tt.activeCues[0].startTime + ",  ";
          dynamicHTML += "endTime: " + tt.activeCues[0].endTime;
          document.getElementById("insertionPoint").innerHTML += dynamicHTML + "<br/>";
          allCuePointData = getSubArray(cuePointAra, 'time', tt.activeCues[0].startTime);
          console.log('cue point data:', allCuePointData);
          console.log('cue point metadata:', allCuePointData[0].metadata);
        }
      }
      myPlayer.play();
      myPlayer.muted(true);
    });

    function getSubArray(targetArray, objProperty, value) {
      var i, totalItems = targetArray.length,
        objFound = false,
        idxArr = [];
      for (i = 0; i < totalItems; i++) {
        if (targetArray[i][objProperty] === value) {
          objFound = true;
          idxArr.push(targetArray[i]);
        }
      }
      return idxArr;
    };
  });
</script>

HTML5 standard cue points

HTML5 standard cue points are stored as track elements in a prescribed format. Good introductory content can be found in the following HTML5 Rocks tutorial: Getting Started With the Track Element. In this section of the document, you will learn the format of WebVTT cue point files and then how to process those cue points.

WebVTT file format for cue points

The WebVTT file format is strictly defined. For cue points the file is composed as follows:

  • The string WebVTT as the first line of the file
  • A single blank line
  • The identifier for a specific cue point
  • The duration in the form 00:00:00.000 --> 00:00:00.000; this is hours:minutes:seconds.milliseconds format and is strictly parsed; numbers must be zero padded if necessary
  • The characters following the duration to the next blank line are place into a text value; if you wish to store multiple, disparate values here JSON works well as it can be easily parsed
  • A single blank line
  • Multiple cue points can be added using the identifier/duration/text/blank line format

The following is a valid WebVTT cue point document that defines two cue points, one with a duration from 2-5 seconds and another with duration from 10-15 seconds:

WEBVTT

multiCell
00:00:02.000 --> 00:00:05.000
{
"title": "Multi-celled organisms",
"description": "Multi-celled organisms have different types of cells that perform specialized functions.",
"src": "multiCell.jpg",
"href": "http://en.wikipedia.org/wiki/Multicellular"
}

insects
00:00:10.000 --> 00:00:15.000
{
"title": "Insects",
"description": "Insects are the most diverse group of animals on the planet with estimates for the total number of current species range from two million to 50 million. The first insects appeared around 400 million years ago, identifiable by a hard exoskeleton, three-part body, six legs, compound eyes and antennae.",
"src": "insects.jpg",
"href": "http://en.wikipedia.org/wiki/Insects"
}

When you play the following video, you will see the cue point WebVTT file shown above processed. At the start of a cue point, you see dynamically created HTML injected into the page, followed by some parsed JSON from the text field. At the end of the cue point the message Cue point duration over is displayed.

******** Cue Point Information ********

 

****** End Cue Point Information ******

Process cue points overview

The basics steps to use HTML5 standard cue points are:

  1. Read in the WebVTT file containing the cue points using a <track> tag nested as a child of the <video-js> tag.
  2. In a loadedmetadata event handler, retrieve the proper text track element.
  3. In the same event handler, configure a second event handler for the oncuechange cue point event.
  4. In the oncuechange event handler, check if the activecues array object has zeroth element defined, if yes, act on the cue point start.
  5. In the oncuechange event handler, check if the activecues array object has zeroth element defined, if no, act on the cue point stop.

Process cue points code

 

  • Line 122: Create a text track by reading in the WebVTT file using the <track> tag.
  • Line 126: Create a paragraph element as a location in which to inject dynamically created HTML.
  • Lines 134,152: Use the one() method to listen only once for the loadedmetadata event. The event handler function is defined here anonymously.
  • Line 135: Assign to a variable the index of the last text track loaded.
  • Line 136: Retrieve the textTracks array using the textTracks() method the assign the last element of the array to a variable using the index determined in the previous step. See the Find correct track section below for variations on this logic.
  • Lines 137,149: Sets the event handler function for when the oncuechangeevent is dispatched.
  • Line 138: Check to be sure the first (zeroth array element) cue point is defined in the activeCues array. This tells you that this is a start cue point event.
  • Lines 139-145: Dynamically create HTML using information from the cue point and inject into the HTML page. This includes parsing JSON and displaying various fields of that JSON.
  • Lines 146-147: Display a note that the cue point end has been reached.
  • Line 150: Play the video.
<video-js id="myPlayerID2"
  data-account="1752604059001"
  data-player="8cfb6785-c4be-43b4-af68-7ca2fe3a6ebf"
  data-embed="default"
  data-video-id="3851380732001"
  class="video-js" controls>

  <track kind="metadata" src="https://solutions.brightcove.com/bcls/tests/player/cuepoints.vtt" />
</video-js>

<p id="insertionPoint2"></p>

<script src="https://players.brightcove.net/1752604059001/8cfb6785-c4be-43b4-af68-7ca2fe3a6ebf_default/index.min.js"></script>

<script type="text/javascript">
  videojs.getPlayer('myPlayerID').ready(function() {
    var myPlayer = this;
    myPlayer.one("loadedmetadata", function () {
      var trackIndex = myPlayer.textTracks().length - 1;
      var tt = myPlayer.textTracks()[trackIndex];
      tt.oncuechange = function () {
        if (tt.activeCues[0] !== undefined) {
          var dynamicHTML = "id: " + tt.activeCues[0].id + ", ";
          dynamicHTML += "text: " + tt.activeCues[0].text + ", ";
          dynamicHTML += "startTime: " + tt.activeCues[0].startTime + ",  ";
          dynamicHTML += "endTime: " + tt.activeCues[0].endTime;
          document.getElementById("insertionPoint2").innerHTML += dynamicHTML + "<br/><br/>";
          jsonData = JSON.parse(tt.activeCues[0].text);
          document.getElementById("insertionPoint2").innerHTML += jsonData.title + ": " + jsonData.description + "<br/><br/>";
        } else {
          document.getElementById("insertionPoint2").innerHTML += "Cue point duration over" + "<br/><br/>";
        }
      } //end oncuechange
      myPlayer.play();
      myPlayer.muted(true);
    }); //end loadedmetadata
  }); //end ready
</script>

Process cue points code

 

  • Line 270: Create a text track by reading in the WebVTT file using the <track> tag.
  • Line 273: Create a paragraph element as a location in which to inject dynamically created HTML.
  • Lines 285,309: Use the one() method to listen only once for the loadedmetadata event. The event handler function is defined here anonymously.
  • Line 286: Assign to a variable the index of the last text track loaded.
  • Line 287: Retrieve the textTracks array using the textTracks() method the assign the last element of the array to a variable using the index determined in the previous step. See the Find correct track section below for variations on this logic.
  • Lines 289,301: Sets the event handler function for when the oncuechangeevent is dispatched.
  • Line 290: Check to be sure the first (zeroth array element) cue point is defined in the activeCues array. This tells you that this is a start cue point event.
  • Lines 291-294: Dynamically create HTML using information from the cue point and inject into the HTML page. This includes parsing JSON and displaying various fields of that JSON.
  • Lines 295-297: Display a note that the cue point end has been reached.
  • Line 302: Play the video.

 

One could assume you could add the WebVTT file programmatically using the addRemoteTextTrack() method. In this case, it would not be reliable since you can experience a race condition where the cue points would not be ready to process before trying to use them. It is safe to add the WebVTT file using the <track> tag as shown.

Find correct track

In a number of places in this document a possible problem could occur if multiple text tracks are associated with a player. An assumption was made that only one text track is associated with the player so this code was used: var tt = myPlayer.textTracks()[0];. Selecting the zeroth array element assumes a single text track associated the player.

Text tracks are not only used for cue points, but other kinds of data also. The kind attribute can contain the following values:

  • subtitles
  • captions
  • descriptions
  • chapters
  • metadata

This means it is very possible multiple text tracks could be associated with the player, and a way is needed to find the correct one for your application logic. The following code loops over the available text tracks until the metadata (cue point) value is found:

<script type="text/javascript">
  videojs.getPlayer('myPlayerID').ready(function() {
    var myPlayer = this,
      allTextTacks,
      attLength,
      tt;
    myPlayer.one("loadedmetadata", function () {
      allTextTacks = myPlayer.textTracks();
      attLength = allTextTacks.length;
      for (var i = 0; i < attLength; i++) {
        if (allTextTacks[i].kind === 'metadata') {
          tt = allTextTacks[i];
          break;
        };
      };
    });
  });
</script>

The logic from previous examples could then be used to process the cue points using the tt variable, which contains the wanted text track.

Programmatic cue points

It is possible to programmatically add cue points. The key method comes from the HTML5 VTTCue interface. You can create cue points use the following syntax:

new VTTCue( startTime, endTime, text )

The logic of processing the cue points is very similar to previous examples, so only the code different from that used above will be detailed.

  • Lines 43,44: Adds a textTrack element to the player using the player's addRemoteTextTrack() method. Specify the kind to be metadata, and a label, in this case Timed Cue Point.

  • Lines 45-48: Wait 10 milliseconds before creating two cue points using the player's addCue() method. The cue points are instantiated using HTML5's VTTCue() constructor.

  • Line 22: Listens for the addtrack event.
  • Line 23: Retrieves all text tracks associated with the player.
  • Line 24: Sets the attLength variable to the number of text tracks. This value will be used in a loop in the next code segment.
  • Lines 25-30: Loop over each text track checking for the label to be equal to the correct value. When found, assign the current text track to a variable and break out of the loop.
<video-js id="myPlayerID"
  data-video-id="4607746980001"
  data-account="1507807800001"
  data-player="default"
  data-embed="default"
  class="video-js"
  width="640" height="360"
  controls></video-js>

<script src="https://players.brightcove.net/1507807800001/default_default/index.min.js"></script>

<p id="insertionPoint"></p>

<script type="text/javascript">
  videojs.getPlayer("myPlayerID").ready(function () {
    var myPlayer = this,
      textTrack = [],
      allTextTacks,
      attLength,
      tt;
    myPlayer.one("loadedmetadata", function () {
      myPlayer.textTracks().addEventListener('addtrack', function () {
        allTextTacks = myPlayer.textTracks();
        attLength = allTextTacks.length;
        for (var i = 0; i < attLength; i++) {
          if (allTextTacks[i].label === 'Timed Cue Point') {
            tt = allTextTacks[i];
            break;
          }
        }
        tt.oncuechange = function () {
          if (tt.activeCues[0] !== undefined) {
            var dynamicHTML = "id: " + tt.activeCues[0].id + ", ";
            dynamicHTML += "text: <strong>" + tt.activeCues[0].text + "</strong>, ";
            dynamicHTML += "startTime: <strong>" + tt.activeCues[0].startTime + "</strong>,  ";
            dynamicHTML += "endTime: <strong>" + tt.activeCues[0].endTime + "</strong>";
            document.getElementById("insertionPoint").innerHTML += dynamicHTML + "<br/><br/>";
          } else {
            document.getElementById("insertionPoint").innerHTML += "Cue point duration over" + "<br/><br/>";
          }
        }; //end oncuechange
      }); // end playing
      textTrack = myPlayer.addRemoteTextTrack({kind: 'metadata', label: 'Timed Cue Point', mode: 'hidden'}, false);
      textTrack.track.mode = 'hidden';
      setTimeout(function(){
        textTrack.track.addCue(new window.VTTCue(2, 5, 'cue point 1 text'));
        textTrack.track.addCue(new window.VTTCue(10, 15, 'cue point 2 text'));
      }, 10);
    }); //end on loadedmetadata
  }); //end ready
</script>

The code generates output as shown in the following screenshot. Note the id values are not supplied a value when using the VTTCue() constructor method.

dynamic cue points