Smart Summaries: Teaching AI to Pick Key Scenes from Scripts

Table of Links

Part 1

Part 2

Part 3

Part 4

Part 5

Part 6

4. Scene Saliency Classification Model

Using the set of generated silver-standard labels for scene saliency, we train a neural network-based classification model to predict scene saliency. We formulate this task as a sequence labeling task where the model takes a sequence of scenes M = {S1, S2, ..., SN} as input and predicts a sequence of binary labels Y = {y1, y2, ..., yN } denoting whether a scene is salient.

The model consists of two components, as shown in Figure 1. The first component is a scene encoder which computes scene representations by concatenating the sentences in the scene and encodes them using a pre-trained language model. Next, to learn contextual scene representation across the whole movie, we further encode the scene embeddings generated by the scene encoder using a transformer (Vaswani et al., 2017) block (L layers stacked), with unmasked self-attention initialized with random weights (Liu and Lapata, 2019). To preserve the sequence of the scenes, we add positional encodings to scene representations obtained from the first component. The final contextualized representation of the scenes is then used to classify whether scenes are salient or not. The model is trained for binary sequence labeling using the binary cross-entropy loss.

4.1 Dataset

To train the saliency model, we used the ScriptBase corpus (Gorinski and Lapata, 2015) that contains preprocessed scripts of movies with Wikipedia summaries. We removed the movies used in our gold-standard MENSA dataset from Scriptbase from the training set. This resulted in a training set containing 824 movie scripts, for which we generated silver-standard scene saliency labels using the model of Mirza et al. (2021), as previously discussed. We randomly split our gold-standard scene saliency dataset of 100 movies, using half of it for validation and the other half for testing.

4.2 Baselines

Majority Class: We used predicting the majority class as a simple baseline for classification. The dataset is highly imbalanced, with non-salient being the majority class.

Supervised Bi-LSTM: For a supervised baseline, we used a bi-directional LSTM (Bi-LSTM) to learn contextual representation for the classification of scene saliency. Again, we computed scene representations by concatenating the sentences and encoding them using a pre-trained language model.

Note that the alignment model of Mirza et al. (2021) cannot be used as a baseline for saliency classification: it requires summaries to align to movie scripts at test time. In a summarization scenario, no summaries are available at test time.

4.3 Implementation Details

Our Scene Saliency Model and baseline models employed RoBERTa-large as the pre-trained scene encoder. Representation of a scene is computed using the first token’s last hidden state of the model. The movie encoder transformer block has 10 layers with 16 heads and a feedforward hidden size of 2048. As the binary scene labels in the dataset are highly imbalanced, we used weighted binary cross entropy. We employed AdamW with β1 = 0.9, β2 = 0.999 as our optimizer. The learning rate was fixed at 5e-5. For the baseline TextRank model, we performed a grid search for hyperparameters and used λ1 = 0.7, λ2 = 0.3, and K = 15% of move length. For Bi-LSTM, we used hidden dimension of size 512 followed by a fully connected layer.

4.4 Results

The results of our saliency classification model and the baselines are summarized in Table 4. We report macro-averaged precision (P), recall (R), and F1 score for each model, as the labels are highly imbalanced given that only a limited number of scenes in each movie are salient. Our model outperforms the baselines and achieves 68.38, 68.13, and 68.01 on precision, recall, and F1. The results show that the majority baseline performance is equivalent to random guessing (macro-average). The unsupervised TextRank model has higher precision, recall, and F1 than the majority baseline, which indicates that it is able to correctly predict some scenes as salient based on the centrality score. Also, the high value of λ1 (see Section 4.3) signifies that the backward-looking context is more important than forward-looking context for computing scene importance. The transformer-based scene saliency model achieves better performance than other baselines, indicating the effectiveness of transformer layers in learning the context across scene representations, which is helpful in classifying scene saliency. We also found that a higher number of layers worked better for the transformer, which indicates that more layers help in capturing complex relationships in the input. See Appendix C for k-fold cross-validation on the test set.