Get Status of Dynamic Ingest Requests (Legacy Ingest)

Product(s)
Video Cloud
Role(s)
API Developer
Task(s)
Add Videos/Assets
Topic(s)
Notifications
API(s)
Dynamic Ingest API

When you use the Dynamic Ingest API to add videos to your Video Cloud account, what you most want to know is when the video has been processed and whether or not renditions were successfully created. This document explains how you can do that using Dynamic Ingest API notifications. We also provide a sample dashboard app that automates the process.

Getting the data

The Dynamic Ingest notifications give you all the information you need to know when your video is ready - you just need to know what to look for...and to define what "ready" means for your systems. This diagram summarizes the workflow:

Ingest Status Workflow
Ingest Status Workflow

Dynamic Ingest Notifications

The Dynamic Ingest notification service sends you notifications for several kinds of events. The three that are most useful for figuring out when the video is "ready" are ones indicating that particular renditions have been created, ones indicating that a manifest has been created, and the one that indicates that all processing is complete. Here are examples of each:

Rendition created notification

{
    "entity": "5002412132001",
    "profileRefId": "ts7",
    "accountId": "57838016001",
    "entityType": "ASSET",
    "videoId": "5002361893001",
    "status": "SUCCESS",
    "version": "1",
    "action": "CREATE",
    "jobId": "bb316631-c58b-4bd4-a686-13c5f7a3a779"
}
Notification for Rendition Created

Note in this example:

  • The videoId value lets you know which video the rendition is for (in case you have multiple ingest jobs running)
  • The profileRefId value is the reference id for the rendition specified in the ingest profile
  • if the status value is "SUCCESS", the rendition was created successfully
  • For a segmented type like HLS or MPEG-DASH, the existence of the rendition does not make it playable - you also need the appropriate manifest (see the next example). MP4 renditons are playable as soon as they are created.

Manifest created notification

{
    "jobId": "31f0b112-9890-4567-adb5-0f4ed1673701",
    "status": "SUCCESS",
    "version": "1",
    "action": "CREATE",
    "entityType": "ASSET",
    "entity": "5002412528001",
    "videoId": "5002361895001",
    "profileRefId": "HlsManifest",
    "accountId": "57838016001"
}
Notification for Manifest Created

Note in this example:

  • The videoId value lets you know which video the rendition is for (in case you have multiple ingest jobs running)
  • The profileRefId value is a special code that tells you that the asset created was an HLS manifest (the other possible values are HdsManifest, DashManifest, and SmoothIsmManifest)
  • For HLS and HDS, one manifest will be created, so you will see one notification. For DASH and SmoothIsm, two manifests are created (one for use in the legacy Media API, the other for the CMS API), so you will see two notifications of this type.
  • If the status value is "SUCCESS", the manifest was created successfully
  • For a segmented type like HLS or MPEG-DASH, there is no definite order for the creation of the renditions and manifest - these renditions are not playable until both are created (or the video has been fully processed - see the next example).

Processing complete notification

{
    "entityType": "TITLE",
    "status": "SUCCESS",
    "accountId": "57838016001",
    "entity": "5002412652001",
    "action": "CREATE",
    "jobId": "3e98b3a0-f624-4f2d-81c1-4e43e1d04a0f",
    "version": "1",
    "videoId": "5002412652001"
}
Notification for Processing Complete

Note in this example:

  • The videoId value lets you know which video the rendition is for (in case you have multiple ingest jobs running)
  • The profileRefId is not included in this notification
  • If the status value is "SUCCESS", the video was processed successfully

To receive notifications, you need to include a "callbacks" field in you Dynamic Ingest API requests, pointing to one or more callback addresses:

{
    "master": {
        "url": "https://s3.amazonaws.com/bucket/mysourcevideo.mp4"
    }, "profile": "high-resolution",
    "callbacks": ["http://host1/path1”, “http://host2/path2”]
}

Sample Dashboard

This section explains how notications can be put together to build a simple dashboard for the Dynamic Ingest API. The handler for notifications parses notifications from the Dynamic Ingest API to identify processing complete notifications. It then adds the video notifications into an array of objects for each video in a JSON file. The dashboard itself is an HTML page that imports the JSON file to get the notification data. It uses the ids to makes a request to the CMS API to get the video metadata. You can view the dashboard here.

All the files for this app, along with instructions for setting it up for your account, are in this repository.

Here is the high-level architecture of the app:

Ingest Dashboad Architecture
Ingest Dashboad Architecture

The app parts

The handler for notifications is built in PHP - it looks for processing complete notifications and adds the video id to an array in a separate JavaScript file:

<?php
// var to log errors, if any
$problem = "No errors";
// var to store current video index
$videoIndex = -1;

// get input data
try {
    $json    = file_get_contents('php://input');
    $decoded = json_decode($json, true);
} catch (Exception $e) {
    $problem = $e->getMessage();
    echo $problem;
}

// get the data file contents and parse them
try {
    $notificationData = file_get_contents('di.json');
    $notificationDataDecoded = json_decode($notificationData, true);
} catch (Exception $e) {
    $problem = $e->getMessage();
    echo $problem;
}


    if (isset($decoded["entityType"])) {
        $entityType = $decoded["entityType"];
        // if the entity type is ASSET or TITLE, add it to notification data array
        if ($entityType == "ASSET" || $entityType == "TITLE") {
            array_push($notificationDataDecoded, $decoded);
        }
        // now we'll replace the contents of di.json with what we have
        file_put_contents('di.json', json_encode($notificationDataDecoded));

    }

echo "Dynamic Ingest callback app is running";
var_dump($notificationData);

?>

JSON file:

The JSON file is initially an empty array ([]) - data is added by the notification handler.

Dashboard

The dashboard includes the HTML and JavaScript to fetch the notification data and additional video data from the CMS API and write the results into a table:

<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>Dynamic Ingest Log</title>
        <style>
            body {
                font-family: sans-serif;
                margin: 5em;
            }
            .hide {
                display: none;
            }
            .show {
                display: block;
            }
            table {
                border-collapse: collapse;
                border: 1px #999999 solid;
            }
            th {
                background-color: #666666;
                color: #f5f5f5;
                padding: .5em;
                font-size: .7em;
            }
            td {
                border: 1px #999999 solid;
                font-size: .7em;
                padding: .5em
            }
            .hidden {
                display: none;
            }
        </style>
    </head>
    <body>
        <h1>Dynamic Ingest Log</h1>
        <h2>Account: Brightcove Learning (57838016001)</h2>
        <p style="width:70%">
            Videos are listed in order of processing completion time, newest to oldest. The reference id (generated by the <a href="./di-tester.html">Dynamic Ingest tester</a>) is a combination of the date/time that the Dynamic Ingest job was initiated and the ingest profile that was used. You can add additional videos using the <a href="./di-tester.html">Dynamic Ingest tester</a>. New videos will appear in this log after processing is complete.
        </p>
        <p>
            <button id="clearLogBtn">Clear the log</button>
        </p>
        <div id="videoLogBlock">
            <table>
                <thead>
                    <tr>
                        <th>Video ID</th>
                        <th>Name</th>
                        <th>Reference ID</th>
                        <th>HLS Manifests Created</th>
                        <th>HLS Renditions Created</th>
                        <th>MP4 Renditions Created</th>
                        <th>Processing Complete</th>
                    </tr>
                </thead>
                <tbody id="logBody"></tbody>
            </table>
            <h4 id="loadingMessage">Loading data, please wait...</h4>
        </div>
        <script>
        var BCLS = ( function (window, document) {
            // to use another account, set the account_id value appropriately
            // the client_id and client_secret will also need to be changed in the proxy
            var my_account_id = 57838016001,
                account_id = my_account_id,
                logBody = document.getElementById('logBody'),
                loadingMessage = document.getElementById('loadingMessage'),
                clearLogBtn = document.getElementById('clearLogBtn'),
                i = 0,
                iMax,
                // set the proxyURL to the location of the proxy app that makes Brightcove API requests
                proxyURL = './brightcove-learning-proxy.php',
                dataFileURL = './di.json',
                videoDataArray = [],
                requestOptions = {},
                currentVideo,
                currentIndex = 0;

                /**
                 * Logging function - safe for IE
                 * @param  {string} context - description of the data
                 * @param  {*} message - the data to be logged by the console
                 * @return {}
                 */
                function bclslog(context, message) {
                    if (window["console"] && console["log"]) {
                      console.log(context, message);
                    }
                    return;
                }

                /**
                 * tests for all the ways a variable might be undefined or not have a value
                 * @param {*} x the variable to test
                 * @return {Boolean} true if variable is defined and has a value
                 */
                function isDefined(x) {
                    if ( x === '' || x === null || x === undefined || x === NaN) {
                        return false;
                    }
                    return true;
                }

                /**
                 * find index of an object in array of objects
                 * based on some property value
                 *
                 * @param {array} targetArray - array to search
                 * @param {string} objProperty - object property to search
                 * @param {string|number} value - value of the property to search for
                 * @return {integer} index of first instance if found, otherwise returns null
                 */
                function findObjectInArray(targetArray, objProperty, value) {
                    var i, totalItems = targetArray.length, objFound = false;
                    for (i = 0; i < totalItems; i++) {
                        if (targetArray[i][objProperty] === value) {
                            objFound = true;
                            return i;
                        }
                    }
                    if (objFound === false) {
                        return null;
                    }
                }

                /**
                 * factory for new video objects
                 * @param {String} videoId the video id
                 * @return {object} the new object
                 */
                function makeVideoDataObject(videoId) {
                    var obj = {};
                    obj.id = videoId;
                    obj.name = '';
                    obj.reference_id = '';
                    obj.hlsManifests = 0;
                    obj.hlsRenditions = 0;
                    obj.mp4Renditions = 0;
                    obj.complete = 'no';
                    return obj;
                }

                /**
                 * processes notification objects
                 * creates a new object in the videoDataArray if it doesn't exist
                 * and updates the videoDataArray object based on the notification
                 * @param {Object} notificationObj the raw notification object
                 */
                function processNotification(notificationObj) {
                    var objIndex, videoObj;
                    // if notification object contains a video id, find the corresponding
                    // object in the videoDataArray or create it if it's not there
                    if (isDefined(notificationObj) && isDefined(notificationObj.videoId)) {
                        objIndex = findObjectInArray(videoDataArray, 'id', notificationObj.videoId);
                        // if not found, create one
                        if (!isDefined(objIndex)) {
                            videoObj = makeVideoDataObject(notificationObj.videoId);
                            videoDataArray.push(videoObj);
                            objIndex = videoDataArray.length - 1;
                        }
                        // now update properties based on what's in the notification
                        if (notificationObj.entityType === 'ASSET') {
                            // if it's a rendition or manifest, there will be a profileRefId
                            if (isDefined(notificationObj.profileRefId)) {
                                // see if it's an HLS manifest
                                if (notificationObj.profileRefId === 'HlsManifest') {
                                    // increment the hls manifest count
                                    videoDataArray[objIndex].hlsManifests++;
                                } else if (notificationObj.profileRefId.charAt(0) === 't') {
                                    // increment the hls rendition count
                                    videoDataArray[objIndex].hlsRenditions++;
                                } else if (notificationObj.profileRefId.charAt(0) === 'm') {
                                    // increment the mp4 rendition count
                                    videoDataArray[objIndex].mp4Renditions++;
                                }
                            }
                        } else if (notificationObj.entityType === 'TITLE') {
                            // overall processing notification - checked for SUCCESS / FAILED
                            if (notificationObj.status === 'SUCCESS') {
                                // mark complete
                                videoDataArray[objIndex].complete = 'yes';
                            } else if (notificationObj.status === 'FAILED') {
                                // mark failed
                                videoDataArray[objIndex].complete = 'failed';
                            }
                        }
                    }
                    return;
                }

                /**
                 * creates the dashboard table body
                 */
                function writeReport() {
                    var j,
                        jMax = videoDataArray.length,
                        item,
                        t;
                    loadingMessage.textContent = 'This page will refresh in 1 minute...';
                    /* just showing HLS and MP4 renditions, because
                     * that's all that will be produced in this account,
                     * but you could modify the notification handler and
                     * this page to handle other formats
                     */
                    for (j = 0; j < jMax; j++) {
                        item = videoDataArray[j];
                        if (item.id !== undefined) {
                            logBody.innerHTML += '<tr><td>' + item.id + '</td><td>' + item.name + '</td><td>' + item.reference_id + '</td><td>' + item.hlsManifests + '</td><td>' + item.hlsRenditions + '</td><td>' + item.mp4Renditions + '</td><td>' + item.complete + '</td></tr>';
                        }
                    }
                    // set timeout for refresh
                    t = window.setTimeout(init, 60000);
                };

                // function to set up the notification data request
                function setJSONRequestOptions() {
                    submitRequest(null, dataFileURL, 'notificationData');
                }

                // function to set up video data request
                function setVideoRequestOptions() {
                    requestOptions = {};
                    requestOptions.url = 'https://cms.api.brightcove.com/v1/accounts/' + account_id + '/videos/' + currentVideo.id;
                    submitRequest(requestOptions, proxyURL, 'video');
                }

                /**
                 * initiates the cms api requests
                 */
                function getVideoInfo() {
                    iMax = videoDataArray.length;
                    if (currentIndex < iMax) {
                        currentVideo = videoDataArray[currentIndex];
                        setVideoRequestOptions();
                    } else {
                        loadingMessage.innerHTML = 'No videos have been ingested - you can add some using the <a href="./di-tester.html">Dynamic Ingest tester</a>';
                    }
                }

                /**
                 * make the cms api requests
                 * @param {Object} options request options
                 * @param (String) url URL to send request to
                 * @param (String) type the request type
                 */
                function submitRequest(options, url, type) {
                    var httpRequest = new XMLHttpRequest(),
                        requestData,
                        responseData,
                        videoDataObject,
                        parsedData,
                        getResponse = function () {
                            try {
                                if (httpRequest.readyState === 4) {
                                  if (httpRequest.status === 200) {
                                    responseData = httpRequest.responseText;
                                    switch (type) {
                                        case 'notificationData':
                                        var k, kMax, dataArray;
                                        dataArray = JSON.parse(responseData);
                                        bclslog('dataArray', dataArray);
                                        // process the notifications
                                        kMax = dataArray.length;
                                        for (k = 0; k < kMax; k++) {
                                            processNotification(dataArray[k]);
                                        }
                                        getVideoInfo();
                                        break;
                                        case 'video':
                                        parsedData = JSON.parse(responseData);
                                        bclslog('parsedData', parsedData);
                                        videoDataArray[currentIndex].reference_id = parsedData.reference_id;
                                        videoDataArray[currentIndex].name = parsedData.name;
                                        currentIndex++;
                                        if (currentIndex < iMax) {
                                            currentVideo = videoDataArray[currentIndex];
                                            setVideoRequestOptions();
                                        } else {
                                            writeReport();
                                        }
                                        break;
                                    }
                                  } else {
                                    bclslog("There was a problem with the request. Request returned " + httpRequest.status);
                                    if (type === 'video') {
                                        setVideoRequestOptions();
                                    } else {
                                        setSourcesRequestOptions();
                                    }
                                  }
                                }
                              }
                              catch(e) {
                                bclslog('Caught Exception: ' + e);
                              }
                        };
                    // notifications data is a special case
                    if (type === 'notificationData') {
                        // set response handler
                        httpRequest.onreadystatechange = getResponse;
                        // open the request
                        httpRequest.open("GET", url);
                        // set headers
                        httpRequest.setRequestHeader("Content-Type", "application/json");
                        // open and send request
                        httpRequest.send();
                    } else {
                        // requests via proxy
                        // set up request data
                        requestData = "url=" + encodeURIComponent(options.url) + "&requestType=GET";
                        // set response handler
                        httpRequest.onreadystatechange = getResponse;
                        // open the request
                        httpRequest.open("POST", url);
                        // set headers
                        httpRequest.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
                        // open and send request
                        httpRequest.send(requestData);
                    }
                };

                // event handlers
                clearLogBtn.addEventListener('click', function () {
                    if (window.confirm('Are you sure? This action cannot be undone!')) {
                        // if your clear-log app resides in another location, change the URL
                        window.location.href = 'clear-log.php';
                    }
                });

                // get things started
                function init() {
                    // clear table and the video data array
                    logBody.innerHTML = "";
                    videoDataArray = [];
                    setJSONRequestOptions();
                }
                // kick off the app
                init();
            })(window, document);
        </script>
    </body>
</html>

Proxy

<?php
/**
 * brightcove-learning-proxy.php - proxy for Brightcove RESTful APIs
 * gets an access token, makes the request, and returns the response
 * Accessing:
 *     URL: https://solutions.brightcove.com/bcls/bcls-proxy/bcsl-proxy.php
 *         (note you should *always* access the proxy via HTTPS)
 *     Method: POST
 *
 * @post {string} url - the URL for the API request
 * @post {string} [requestType=GET] - HTTP method for the request
 * @post {string} [requestBody=null] - JSON data to be sent with write requests
 *
 * @returns {string} $response - JSON response received from the API
 */

// CORS enablement
header("Access-Control-Allow-Origin: *");

// set up request for access token
$data = array();
//
// change the values below to use this proxy with a different account
//
$client_id     = "YOUR_CLIENT_ID_HERE";
$client_secret = "YOUR_CLIENT_SECRET_HERE";
$auth_string   = "{$client_id}:{$client_secret}";
$request       = "https://oauth.brightcove.com/v4/access_token?grant_type=client_credentials";
$ch            = curl_init($request);
curl_setopt_array($ch, array(
        CURLOPT_POST           => TRUE,
        CURLOPT_RETURNTRANSFER => TRUE,
        CURLOPT_SSL_VERIFYPEER => FALSE,
        CURLOPT_USERPWD        => $auth_string,
        CURLOPT_HTTPHEADER     => array(
            'Content-type: application/x-www-form-urlencoded',
        ),
        CURLOPT_POSTFIELDS => $data
    ));
$response = curl_exec($ch);
curl_close($ch);

// Check for errors
if ($response === FALSE) {
    die(curl_error($ch));
}

// Decode the response
$responseData = json_decode($response, TRUE);
$access_token = $responseData["access_token"];

// set up the API call
// get data
if ($_POST["requestBody"]) {
    $data = json_decode($_POST["requestBody"]);
} else {
    $data = array();
}
// get request type or default to GET
if ($_POST["requestType"]) {
    $method = $_POST["requestType"];
} else {
    $method = "GET";
}

// get the URL and authorization info from the form data
$request = $_POST["url"];

//send the http request
$ch = curl_init($request);
curl_setopt_array($ch, array(
        CURLOPT_CUSTOMREQUEST  => $method,
        CURLOPT_RETURNTRANSFER => TRUE,
        CURLOPT_SSL_VERIFYPEER => FALSE,
        CURLOPT_HTTPHEADER     => array(
            'Content-type: application/json',
            "Authorization: Bearer {$access_token}",
        ),
        CURLOPT_POSTFIELDS => json_encode($data)
    ));
$response = curl_exec($ch);
curl_close($ch);

// Check for errors
if ($response === FALSE) {
    echo "Error: "+$response;
    die(curl_error($ch));
}

// Decode the response
// $responseData = json_decode($response, TRUE);
// return the response to the AJAX caller
echo $response;
?>

Clear the log

This simple PHP app just restores the JavaScript file to its original state, clearing out the old video ids:

<?php
    $logFileLocation = "di.json";
    $freshContent = array ();
    $encodedContent = json_encode($freshContent);
    file_put_contents($logFileLocation, $encodedContent);

echo 'Log file cleared - <a href="di-log.html">go back to the dashboard</a>';
?>