Using FTP Batch Provisioning

Product
Video Cloud
Applies to Roles
Publisher, Developer
Version
Brightcove 5
Edition
Pro, Enterprise
Created by Grant Goodman.
FTP Batch Provisioning

Learn how to use FTP batch provisioning to add or edit assets without using the Media module in the Video Cloud Studio. This document gives you some background in XML principles and explains how to prepare your assets for upload, create your XML manifest file, and upload your files and XML manifest file to the Video Cloud FTP server.

FTP batch provisioning is available only for Video Cloud Pro and Enterprise customers. If you are interested in upgrading your Video Cloud account, please contact Brightcove for more information.

Before using batch provisioning, be sure to read Best Practices for Using Batch Provisioning.

This document includes the following sections:

Why use FTP batch provisioning?

FTP batch provisioning enables you to upload, update, or delete multiple videos and playlists at once using an XML manifest file and the Video Cloud FTP server. Most actions that you want to perform on your assets and playlists via the Video Cloud Studio Media module can be done using FTP batch provisioning, and vice-versa. You can use FTP batch provisioning to generate videos and playlists, which will appear in your Media Library in the Media module once the upload process is complete. As well, you can upload large numbers of assets or make multiple edits quickly without accessing the Media module. However, when you choose to perform actions on your assets with the FTP batch provisioning feature instead of the Media module, you must specify your upload with precise syntax, tags, and attributes in an XML document (the XML manifest file), rather than working with a graphical user interface.

Often, publishers use FTP batch provisioning rather than the Media module when they need to upload, modify settings for, or delete large amounts of video or playlists based on the needs of a Content Management System (CMS), or to perform initial ingestion of their existing video library into the Video Cloud platform.

Important: If a video or playlist with a particular reference ID already exists in your account, whether you added it using the Media module, the Media API, or a previous FTP batch provisioning XML manifest file, it will be updated with the information in the current XML manifest file.

An overview of the FTP batch provisioning process

The following flow chart explains the process for FTP batch provisioning.
 
FTP batch provisioning flow chart

Comparing FTP batch provisioning and the Media module

You can use the Media module to work with videos and playlists that you added with FTP batch provisioning. However, bear in mind the following:

  • If you make any changes to a video or playlist using FTP batch provisioning, you will overwrite any information you have made earlier using the Media module, even if you didn't change all the information. For example, suppose you have a video already in your account with all attributes already set. If you use batch provisioning to modify the video's short description, but don't supply any other information about the video, the other attributes of the video remain blank are not set. Therefore, if you use batch provisioning to modify existing videos and playlists, be sure to provide all the information about them, not just the attributes you want to change. Or, for another approach, consider using the overlay-update attribute, as described in the topic Modifying Existing Videos with FTP Batch Provisioning.
  • If you set schedule information using FTP batch provisioning, you need to provide start and end times using the Pacific time zone. When you view schedule information in the Media module, these times are automatically translated to the time zone set for your user profile.

Main steps for FTP batch provisioning

To use the Video Cloud FTP batch provisioning feature, follow these steps, each of which is described in more detail below.

  1. Read Before you begin to understand the XML principles and the software that you will need to use.
  2. Prepare your assets for upload.
  3. Create your XML manifest. The XML manifest provides Video Cloud with all the information the service needs to handle the videos and playlists you upload.
  4. Upload your assets to the Video Cloud FTP server.
  5. Upload your XML manifest to the Video Cloud FTP server.

After your upload is complete, you can receive an email notification confirming that your files have been successfully received by Video Cloud, or notifying you of any problems encountered during the transfer. See Troubleshooting for information that can help you identify any problems you run into.

Before you begin

Before you begin, there are a few things you need to know or set up prior to proceeding, such as ensuring that you have a basic understanding of XML, getting FTP access, and installing an XML editor/validator and an FTP client.

Get a basic understanding of XML

To use the Video Cloud FTP batch provisioning system, you should have a basic knowledge of XML principles and experience creating XML documents. If you need a primer on XML, here are a few basic resources:

Get FTP access to the Video Cloud servers

To get an FTP account for uploading videos:

You must have a Video Cloud Studio account with an administrator role to create an FTP user and request FTP access from Brightcove support.

  1. Set up an email account for your FTP user in your organization's domain, such as BrightcoveFTP@mycompany.com. Make sure you can access this email. The email address that you set for the user here will be the user name for logging into the Video Cloud FTP server.
  2. Create a Video Cloud account for your dedicated FTP user with the email address you created in step 1. See Adding a user for the steps of this process. Provide an appropriate first and last name for the user, such as FTP User. On the Video Cloud Users page, set the role for your FTP user to Standard and uncheck access to the Media, Advertising, and Publishing modules. When creating your FTP account, Brightcove Support will remove any role granted to this user, restrict all access to the Video Cloud Studio for this user, and enable the user for FTP uploads only.
  3. Check the FTP user’s email for the new account message from Brightcove and set the password for the FTP user according to instructions within that email. The email and password for the Video Cloud account will be the same credentials for logging into the Video Cloud FTP server.
  4. Add this dedicated FTP user with the identical email, and names to all the Video Cloud accounts for which you use FTP batch provisioning. Again, set the role of this FTP user to Standard and uncheck access to the Media, Ad, and Publishing modules. When creating your FTP account, Brightcove Support will restrict all access to the Video Cloud Studio for this user.
  5. Once you have created the dedicated user for FTP batch provisioning, for all the appropriate Video Cloud accounts, submit a support case requesting FTP access for that user. Be sure to enter your email address in the form, not the FTP user’s email, so you can get the notice when your FTP account is ready. Include the following in the description section of your request:
    • The email address for this dedicated FTP user account.
    • The Video Cloud account Publisher IDs for all the accounts where you have added the dedicated FTP user. You can find the Publisher ID under your name at the top of the Home page for each account of the Video Cloud Studio.
  1. Brightcove Support will reply to you via email when your account is enabled for FTP access. The Support team prioritizes urgent issues above account administration requests such as FTP access. Support will provide you with an FTP account swiftly, but not immediately. When you have received confirmation that your FTP access is ready, access the Video Cloud FTP server at ftp://upload.brightcove.com and log in with the email and password you created for your FTP user.

Even though you may have more than one Video Cloud account with FTP access, each FTP batch manifest can apply to only one Video Cloud account. Indicate which Video Cloud account you are working with by including the Publisher ID in the FTP batch manifest.

You can change the password for your dedicated FTP user by signing out of the Video Cloud Studio, selecting the forgot your password? link at the sign in page, and filling out the recover password form using the FTP user’s email address. Then log into the FTP user’s email, open the password recovery email, and change the password from there.

Get Aspera access

As an alternative, Video Cloud Enterprise publishers can connect to the Video Cloud Aspera servers, which enable secure uploads with remarkably higher throughput, particularly for uploads originating from greater distances. Contact Brightcove support or your account manager to help you with Aspera access.

Install these tools

To use the Video Cloud FTP batch provisioning system, you'll need a few basic software tools:

  • An XML editor to create the XML manifest file that you will upload after you upload your assets. Two free XML editors are XML Marker and XRay; two other popular XML editors are Altova XMLSpy and Stylus Studio. You can use a plain text editor, but an XML editor can help reduce errors in your XML.
  • A utility to validate that your XML is valid and well-formed. Most XML editors validate your XML. You could also use a utility such as W3Cs web based validation service.
  • FTP client software to upload your files and the XML manifest to the Video Cloud server. Some FTP clients that you could use are: WSFTP, AceFTP, and CuteFTP. There is also an FTP client built into Windows Explorer.

Preparing your assets

Use the following steps to prepare your assets for upload to the Video Cloud FTP server. 

  1. Gather the assets you want to upload in a location or system convenient for working with them and uploading them. 
  2. Make sure that each asset is compatible with Video Cloud's requirements for assets of that type.
  3. Ensure compatibility with the automated systems that process uploaded assets.
    • Ensure that your assets do not contain any illegal characters that might interfere with the processing of your files. 
    • File names must only contain English alphanumeric characters only (1-9, A-Z, a-z). 
    • Eliminate the following illegal characters in your file names:
      • Non-English and/or high-ASCII characters, such as: é ñ å
      • Non-printable characters like spaces and tabs. Replace any spaces with underscores. 
      • Certain punctuation characters such as: @ ! * & #. Hyphens and underscores are allowed. 
  4. For each file you will upload, gather the following information, which you will include in your XML manifest file:
    • The Video Cloud asset type.
    • A unique reference ID for the file. Reference IDs are not required for videos you upload using the Media module or Media APIs, but are required for videos you upload or modify using the FTP batch provisioning XML manifest file.
    • Optionally, you can create an MD5 checksum (hash) for each asset file to assist with error checking. See Creating Checksums for FTP Batch Provisioning for more information.
    • Optionally, you can include the file size in bytes for each asset file to assist with error checking. You may need to convert file size from kilobytes or megabytes to bytes. If you use the Mac OS, to get the file size, select the file and press "Apple + i", or right-click and select Get Info.

Creating your XML manifest file

The XML manifest provides Video Cloud with all the information it needs to process your upload and make it available on the Video Cloud servers. A single XML manifest file can describe multiple videos. Use the following steps to create your XML manifest file:

  1. Download the Sample XML Manifest for FTP Batch Provisioning, which you can use as a starting point for creating your own XML manifest file. This XML manifest file contains the most common use case for FTP batch provisioning, which is to upload videos using the XML manifest file.
  2. In your XML editor, open the sample XML manifest file or the one that you have created and modify it based on the actions you need it to perform. See FTP Batch Provisioning: Reference for the XML Manifest, which contains the required specifications and syntax for the XML tags and attributes you must include in your file, as well as specifications for the different actions you can perform with the file.
  3. Validate your XML manifest file. In FTP Batch Provisioning: Reference for the XML Manifest, you can download a Document Type Definition (DTD) to validate your document.

How many videos should go in a manifest?

For best results, don't upload too many videos or other assets in a single XML manifest. A single batch upload and its corresponding manifest should include no more than 5 GB of assets, or 100 assets, whichever is reached first. If you have single assets larger than 5 GB, upload each of them in its own manifest.

Final checks

When you have finished preparing your XML manifest file, perform one final check to ensure that you are ready to begin uploading your assets and XML manifest file. Use this checklist as a guide:

  • Are my assets properly named, without spaces or illegal characters?
  • Is my XML manifest file well-formed and valid XML, free of any illegal characters? Run your manifest file through an XML validation utility.
  • Are my asset's filenames correctly specified in my XML manifest file, and do they match the assets Im about to upload?
  • Does the number of assets on disk match the number of assets in my XML manifest file?
  • Important: If a video or playlist with a particular reference ID already exists in your account, whether you added it using the Media module, the Media APIs, or a previous FTP batch provisioning XML manifest, it will be updated with the information in the current XML manifest file.

Uploading your assets and XML manifest

After you have prepared your assets, created the XML manifest that describes the assets, and checked everything over, you are ready to upload your assets and the XML manifest file.

  1. To access the Video Cloud FTP server, go to ftp://upload.brightcove.com and log in with the credentials of that dedicated user. If you do not have a Video Cloud FTP account, request that a support representative grant you access to the Video Cloud FTP servers.
  2. Upload all of the assets listed in your XML manifest file.
    • As your upload progresses, you will notice your files disappear from the FTP server upload directory soon after they are uploaded to the FTP server. This is normal files are secured to a staging area until the manifest is uploaded.
    • Do not upload the XML manifest file until all of your files have been transferred successfully.
  3. Upload your XML manifest file. 
    • It is important to upload all of your asset files before you upload your XML manifest; otherwise, the Video Cloud server will start to process your XML manifest file. This will generate errors when the server can't locate asset files that are specified in the XML manifest file, but have not yet been uploaded to the Video Cloud FTP server.
    • You have 48 hours to upload the XML manifest file after assets have completed uploading.
  4. When Video Cloud finishes processing your manifest, you will receive an email confirming that the files detailed in the manifest have been successfully received by Video Cloud, or notifying you of any problems encountered during the transfer. You can also receive a notification callback posted to a URL you provide in your XML manifest file. Note: you will receive an email confirming success only if you set the report-success="true" attribute in your XML manifest file. For information about the possible causes of problems or error messages, see FTP Batch Provisioning: Troubleshooting.
  5. Sign into the Video Cloud Studio Media module and edit videos and playlists that you have uploaded using FTP batch provisioning.
    Important note: Assets can take up to one hour to fully propagate on the network before they are available for preview. Wait for one hour before trying to view your assets in the Video Cloud Studio Media module.

Related topics

In addition to this document, you can read more documentation to help you understand and use the Video Cloud FTP batch provisioning feature.

Read the following documents to get foundation knowledge of FTP batch provisioning:

The following documents explain the different ways you can use FTP batch provisioning:

 

Tags
batch, FTP, upload