Implementing Multiple Audio Tracks Using the APIs

Product(s)
Video Cloud
Role(s)
API Developer
Task(s)
Add Videos/Assets
Topic(s)
Audio
Dynamic Delivery
API(s)
Dynamic Ingest API

This topic explains how to add and manage multiple audio tracks for videos using the Dynamic Ingest and CMS APIs.

Introduction

The ability to associate multiple audio tracks with a single title can be useful in a couple of key scenarios:

  • Playing back the same video in different languages for a broader reach globally 
  • Providing audio with descriptions for the visually challenged

Note that adding audio tracks can also be done in Studio - see Adding Audio Tracks to Videos using the Media Module for more details.

Video formats supported

Multiple audio tracks are supported for HLS V4+ and DASH - with and without DRM protection.

Sample

Below is a simple example of a video with multiple audio tracks.

Setup

The setup for Dynamic Ingest requests is the same, whether you are ingesting a video, images, audio tracks, WebVTT files, or all of these:

Request URL
https://ingest.api.brightcove.com/v1/accounts/{account_id}/videos/{video_id}/ingest-requests
Authentication
Authentication requires an access token passed as a Bearer token in an Authorization header:
Authorization: Bearer {access_token}

To get access tokens, you will need client credentials (see below). For the process of obtaining an access token, see Get Access Tokens.

Note on S3

If your source files will be pulled from a protected S3 bucket, you will need to set a bucket policy to allow Video Cloud to access the files. See Using Dynamic Ingest with S3 for details.

Getting Credentials

To get a client_id and client_secret, you will need to go to the OAuth UI and register this app:

These are the permissions you will need:

<span class=Dynamic Ingest Permissions" src="//learning-services-media.brightcove.com/doc-assets/video-cloud-apis/di-api/di-permissions.png" />
Dynamic Ingest Permissions

You can also get your credentials via CURL or Postman - see:

If you are getting credentials directly from the API, these are the permissions you need:

[
  "video-cloud/video/all",
  "video-cloud/ingest-profiles/profile/read",
  "video-cloud/ingest-profiles/account/read",
  "video-cloud/upload-urls/read"
]

Workflow

There are two primary use cases:

  • Ingest a new video with multiple audio tracks
    New Video Workflow
    New Video Workflow
  • Add multiple audio tracks to an existing video
    Existing Video Workflow
    Existing Video Workflow

We will look at the details of the API requests in a section below.

Audio track metadata

There are some metadata fields attached to audio tracks - some of these are set when you ingest the tracks, while others are created by Video Cloud. Some of these fields are read-only, and others can be updated by you. The metadata fields will be detailed below in the context of the relevant API requests, but two particular ones require some explanation here, as they are crucial in determining how the Brightcove Player will handle the multiple audio tracks.

language

The language field, set for each track during ingestion, specifies the language for the track. This is important if the track is a dub of spoken words in the video. The value of this field will be a two-letter code like en or hi. For a full list of language codes, see the Subtag values in http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry.

variant

The variant field describes the kind of audio track. The possible values, with standard meanings are:

  • main - the main track, usually the one muxed into the video file
  • alternate - an alternate audio track
  • commentary - an audio track that provides commentary on the video track
  • dub - a track containing a dubbed version of spoken words in a different language
  • descriptive - track is descriptive of the video content in some way

Account defaults

You can set account defaults for language and variant to determine what audio track will be treated as the default by the Brightcove Player (the default can also be overridden by updating the track metadata, as we will see in a section below). To set the defaults for your account, Contact Brightcove Support.

Ingest audio tracks

Now we will look at the API calls to ingest audio tracks for the two use cases described earlier.

New video with multiple audio tracks

Create the video object (CMS API)

  1. You can add several items of video metadata when you create the video object, but here we will just add the minimum: a name for the video:
    {
        "name": "YOUR_VIDEO_NAME"
    }
  2. Submit the JSON above (with the placeholder text replaced by your video name) as the request body for POST request to https://cms.api.brightcove.com/v1/accounts/YOUR_ACCOUNT_ID/videos
  3. You will get a lot of video metadata back in the response, but the important piece here is the id (the video id), which you need for the next step.

Ingest the video and audio tracks

Next we will ingest the video and audio tracks (we could add other assets like images and text tracks, but we will keep it simple here). The one thing you may find a little confusing is that audio_tracks appears twice in the JSON for the request body:

  • An audio_tracks object within the master object contains metadata for the audio track included in the video file (if any - this is also referred to as the muxed in audio) - this will include metadata only, without a URL for the audio file, since the audio track is already included in the video file
  • A top-level audio_tracks object describing the additional audio tracks you are ingesting - these will include a URL for audio file, as well as other metadata
  1. The JSON data to be sent in the request body is as follows:
    {
      "master": {
          "url": "https://learning-services-media.brightcove.com/videos/Great_Blue_Heron.mp4",
          "audio_tracks": [
              {
                  "language": "en",
                  "variant": "main"
              }
          ]
      },
      "audio_tracks": {
          "merge_with_existing": true,
          "masters": [
              {
                  "url": "http://learning-services-media.brightcove.com/audio/celtic_lullaby.m4a",
                  "language": "en",
                  "variant": "alternate"
              },
              {
                  "url": "http://learning-services-media.brightcove.com/audio/audio1.m4a",
                  "language": "en",
                  "variant": "commentary"
              }
          ]
      },
        "profile": "BoltIngestProfile",
        "capture-images": true,
        "callbacks": [
            "http://solutions.brightcove.com/bcls/di-api/di-callbacks.php"
        ]
    }
  2. Send the JSON above, substituting your own URLs for the placeholders, and adjusting the language and variant values, in a POST request to https://ingest.api.brightcove.com/v1/accounts/ACCOUNT_ID/videos/ID/ingest-requests (the ID here is the video id returned from request to create the video object)

Add audio tracks to an existing video

To add additional audio tracks to an existing video, the procedure is the same, except that you do not need to make the request to the CMS API to create the video, because it already exists. And in the request to the Dynamic Ingest API, you will not need to include the URL for the video file or the audio_tracks under master providing the metadata for the muxed in audio. You do, however, want to include the metadata for the muxed in audio track in the existing video. So the JSON for the ingest request will look like this:

{
  "audio_tracks": {
      "merge_with_existing": true,
      "masters": [
          {
              "url": "http://learning-services-media.brightcove.com/audio/celtic_lullaby.m4a",
              "language": "en",
              "variant": "alternate"
          },
          {
              "url": "http://learning-services-media.brightcove.com/audio/audio1.m4a",
              "language": "en",
              "variant": "commentary"
          }
      ]
  },
    "profile": "BoltIngestProfile",
    "capture-images": true,
    "callbacks": [
        "http://solutions.brightcove.com/bcls/di-api/di-callbacks.php"
    ]
}

Audio track fields for ingest

Audio Track Fields
Field Type Description
master.audio_tracks Object[] Metadata for the muxed in audio
master.audio_tracks.language String Language code for the muxed in audio from the subtags in http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
master.audio_tracks.variant String The kind of audio track: main|alternate|dub|commentary|descriptive (main would generally be used for the muxed in audio)
audio_tracks Object Information for the additional audio tracks
audio_tracks.merge_with_existing Boolean Whether these tracks should be merged into existing ones or replace them
audio_tracks.masters Object[] Information for the individual audio tracks
audio_tracks.masters.url String Url for the audio track file
audio_tracks.masters.language String Language code for the audio track from the subtags in http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
audio_tracks.masters.variant String The kind of audio track: main|alternate|dub|commentary|descriptive (main would generally be used for the muxed in audio)

Notifications

If you specify one or more callback URLs (as in the sample JSON for the ingest request above), Video Cloud will send a notification for each of the audio tracks you ingest. Notifications will look like this:

{
  "entity": "default/audio128",
  "entityType": "DYNAMIC_RENDITION",
  "version": "1",
  "action": "CREATE",
  "jobId": "0f703adb-0f17-4a35-8395-21c7fcdd2649",
  "videoId": "5298468208001",
  "dynamicRenditionId": "default/audio128",
  "accountId": "1910141565001",
  "status": "SUCCESS",
  "language" : "en",
  "variant" : "alternate"
}

To identify notifications for audio tracks, look for the language and variant fields in the notification. The "action": "CREATE" and "status": "SUCCESS" fields indicate that the track was successfully ingested.

Managing audio tracks

Once you have ingested the audio tracks, you can manage them via the CMS API.

Get all audio track metadata for a video

To retrieve the metadata for all audio tracks associated with a video, make a GET request to:

https://cms.api.brightcove.com/v1/accounts/account_id/videos/video id/audio_tracks

The response will be an array of objects containing metadata for each audio track. See the table of response fields below for details.

[
  {
    "id": "en_alternate",
    "language": "en",
    "variant": "alternate",
    "duration": 86100,
    "encoding_rates": [
      64000,
      96000,
      127000
    ]
  },
  {
    "id": "en_commentary",
    "language": "en",
    "variant": "commentary",
    "duration": 34203,
    "encoding_rates": [
      10000,
      13000,
      15000
    ]
  },
  {
    "id": "en_main",
    "language": "en",
    "variant": "main",
    "duration": 31488,
    "encoding_rates": [
      62000,
      94000,
      125000
    ]
  }
]

You can also view this information in Studio by viewing the video in the Media module:

Audio Track Information in Studio
Audio Track Information in Studio

Get metadata for one audio track

To retrieve the metadata for one audio track associated with a video, make a GET request to:

https://cms.api.brightcove.com/v1/accounts/account_id/videos/video id/audio_tracks/audio_track_id

The response will be an object containing metadata for each audio track. See the table of response fields below for details.

Updating audio track metadata

You can update any writable metadata field for an audio track by making a PATCH request to:

https://cms.api.brightcove.com/v1/accounts/account_id/videos/video id/audio_tracks/audio_track_id

In the request body, include the field(s) you want to modify - for example:

{
    "language": "es",
    "is_default": true
}

Delete an audio track

To delete an audio track, send a DELETE request to:

https://cms.api.brightcove.com/v1/accounts/account_id/videos/video id/audio_tracks/audio_track_id

Note that the success response code may be 202 (Accepted) rather than 204 (No content) because the deletion process is asynchronous and may not be completed immediately.

Audio Track Metadata Fields
Field Type Read-only Description
id String yes Id for the track, formed as language_variant - note that the id may change if these values are changed
language String No Language code for the audio track from the subtags in http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
variant String No The kind of audio track: main|alternate|dub|commentary|descriptive (main would generally be used for the muxed in audio)
duration Number Yes The length of the track in milliseconds
encoding_rates Number[] Yes A list of available encodings of this track, in bps
is_default Boolean No If true, this will be used the default track for playback (overriding any account-level default)

Playback

For information on how the Brightcove web and SDK players handle multiple audio tracks see:

Known issues

Audio masters not stored
  • Video Cloud will not store audio masters
  • If you retranscode the video from the video master, the additional audio tracks will be lost, and must be re-added using the Add audio tracks to an existing video method described above
Audio-only files must be used
Audio tracks must be audio-only files without video tracks
HLSv3, HLS with audio and video in the same segment
  • The playback API will not return HLSv3 manifests
  • All HLS manifests will include de-muxed video/audio
Smooth Streaming
Smooth Streaming URLs will not be available.
Social Distribution
It is not possible to select which audio track will be used for distribution. The track included in the video source (the muxed in audio) will be used, always.
Studio
  • Studio will display information about audio tracks
  • To add audio tracks using Studio, see Multiple Audio Tracks
Ordering of audio tracks beyond the one "default" track
  • You can pick a default audio track with CMS API per title by setting the is_default field to true
  • There is also an account default, which can be set by Support
  • This will only affect the "default" track in HLS manifests
  • No other ordering is possible
Ingesting more than one audio track from a single source
We support one audio track from the source supplied in ".master", and one for each source under ".audio_tracks.masters".
DRM protection on videos that include only audio
As soon as a video track is added, DRM protection will be enabled.
End-user friendly labels
We do not support custom labels for audio tracks. If you require that, you will need to make the change on the client side via the Player API.
In some instances, switching tracks may cause the Brightcove Player to become unstable
  • Track switching before all audio segments have been downloaded
  • When the video is played back using the Silverlight plugin (in versions of IE less than 10, or any version of IE in versions of Window less than 8) - multiple audio tracks are not supported in Silverlight.
  • If the audio and video are different durations, the player will may stop whenever the shorter one runs out.
Video "duration"
The video duration reported by the catalog/Playback API may be incorrect if the audio tracks have different durations.