注意

点击这里下载完整的示例代码

使用 Wav2Vec2 进行语音识别¶

作者：Moto Hira

本教程展示了如何使用 wav2vec 2.0 的预训练模型 [论文] 进行语音识别。

概述¶

语音识别的过程如下所示。

从音频波形中提取声学特征
逐帧估计声学特征的类别
从类别概率序列生成假设

Torchaudio 提供了对预训练权重和相关信息（如期望的采样率和类别标签）的便捷访问。它们被捆绑在一起，可在 torchaudio.pipelines 模块下找到。

准备¶

import torch
import torchaudio

print(torch.__version__)
print(torchaudio.__version__)

torch.random.manual_seed(0)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print(device)

2.10.0.dev20251013+cu126
2.8.0a0+1d65bbe
cuda

import IPython
import matplotlib.pyplot as plt
from torchaudio.utils import _download_asset

SPEECH_FILE = _download_asset("tutorial-assets/Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav")

100.0%

创建管道¶

首先，我们将创建一个执行特征提取和分类的 Wav2Vec2 模型。

Torchaudio 中有两种类型的 Wav2Vec2 预训练权重可用。一种是为 ASR 任务进行了微调的，另一种是没有微调的。

Wav2Vec2 (和 HuBERT) 模型以自监督方式进行训练。它们首先仅使用音频进行表示学习，然后使用附加标签为特定任务进行微调。

未经微调的预训练权重也可以为其他下游任务进行微调，但本教程不涵盖此内容。

这里我们将使用 torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H。

在 torchaudio.pipelines 中有多个预训练模型可用。请查看文档以了解它们是如何训练的详细信息。

bundle 对象提供了实例化模型和其他信息的接口。采样率和类别标签如下所示。

bundle = torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H

print("Sample Rate:", bundle.sample_rate)

print("Labels:", bundle.get_labels())

Sample Rate: 16000
Labels: ('-', '|', 'E', 'T', 'A', 'O', 'N', 'I', 'H', 'S', 'R', 'D', 'L', 'U', 'M', 'W', 'C', 'F', 'G', 'Y', 'P', 'B', 'V', 'K', "'", 'X', 'J', 'Q', 'Z')

模型可以按如下方式构建。此过程将自动获取预训练权重并将其加载到模型中。

model = bundle.get_model().to(device)

print(model.__class__)

Downloading: "https://download.pytorch.org/torchaudio/models/wav2vec2_fairseq_base_ls960_asr_ls960.pth" to /root/.cache/torch/hub/checkpoints/wav2vec2_fairseq_base_ls960_asr_ls960.pth

0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
0%
1%
1%
1%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
9%
0%
0%
1%
1%
1%
2%
2%
2%
3%
3%
3%
4%
4%
4%
5%
5%
5%
6%
6%
6%
7%
7%
7%
8%
8%
8%
9%
9%
0%
0%
0%
<class 'torchaudio.models.wav2vec2.model.Wav2Vec2Model'>

加载数据¶

我们将使用来自 VOiCES 数据集的语音数据，该数据集是在 Creative Commos BY 4.0 许可下发布的。

IPython.display.Audio(SPEECH_FILE)

要加载数据，我们使用 torchaudio.load()。

如果采样率与管道期望的不同，我们可以使用 torchaudio.functional.resample() 进行重采样。

注意

torchaudio.functional.resample() 也适用于 CUDA 张量。
当对同一组采样率执行多次重采样时，使用 torchaudio.transforms.Resample 可能会提高性能。

waveform, sample_rate = torchaudio.load(SPEECH_FILE)
waveform = waveform.to(device)

if sample_rate != bundle.sample_rate:
    waveform = torchaudio.functional.resample(waveform, sample_rate, bundle.sample_rate)

提取声学特征¶

下一步是从音频中提取声学特征。

注意

为 ASR 任务微调的 Wav2Vec2 模型可以一步完成特征提取和分类，但为了本教程的方便，我们也在此展示如何执行特征提取。

with torch.inference_mode():
    features, _ = model.extract_features(waveform)

返回的特征是张量列表。每个张量都是 Transformer 层的输出。

fig, ax = plt.subplots(len(features), 1, figsize=(16, 4.3 * len(features)))
for i, feats in enumerate(features):
    ax[i].imshow(feats[0].cpu(), interpolation="nearest")
    ax[i].set_title(f"Feature from transformer layer {i+1}")
    ax[i].set_xlabel("Feature dimension")
    ax[i].set_ylabel("Frame (time-axis)")
fig.tight_layout()

Feature from transformer layer 1, Feature from transformer layer 2, Feature from transformer layer 3, Feature from transformer layer 4, Feature from transformer layer 5, Feature from transformer layer 6, Feature from transformer layer 7, Feature from transformer layer 8, Feature from transformer layer 9, Feature from transformer layer 10, Feature from transformer layer 11, Feature from transformer layer 12

特征分类¶

提取声学特征后，下一步是将它们分类到一组类别中。

Wav2Vec2 模型提供了一个方法来一步完成特征提取和分类。

with torch.inference_mode():
    emission, _ = model(waveform)

输出是以 logits 的形式。不是以概率的形式。

让我们可视化一下。

plt.imshow(emission[0].cpu().T, interpolation="nearest")
plt.title("Classification result")
plt.xlabel("Frame (time-axis)")
plt.ylabel("Class")
plt.tight_layout()
print("Class labels:", bundle.get_labels())

Class labels: ('-', '|', 'E', 'T', 'A', 'O', 'N', 'I', 'H', 'S', 'R', 'D', 'L', 'U', 'M', 'W', 'C', 'F', 'G', 'Y', 'P', 'B', 'V', 'K', "'", 'X', 'J', 'Q', 'Z')

我们可以看到，在时间线上存在一些明确的标签指示。

生成字幕¶

从标签概率序列中，我们现在想生成字幕。生成假设的过程通常称为“解码”。

解码比简单分类更复杂，因为某个时间步的解码会受到周围观测值的影响。

例如，考虑像 night 和 knight 这样的词。即使它们的先验概率分布不同（在典型的对话中，night 比 knight 出现得更频繁），为了准确地生成字幕，例如 a knight with a sword，解码过程必须推迟最终决定，直到它看到足够多的上下文。

已经提出了许多解码技术，它们需要外部资源，例如单词字典和语言模型。

在本教程中，为了简单起见，我们将执行贪婪解码，它不依赖于这些外部组件，而只是在每个时间步选择最佳假设。因此，不使用上下文信息，只能生成一个字幕。

我们首先定义贪婪解码算法。

class GreedyCTCDecoder(torch.nn.Module):
    def __init__(self, labels, blank=0):
        super().__init__()
        self.labels = labels
        self.blank = blank

    def forward(self, emission: torch.Tensor) -> str:
        """Given a sequence emission over labels, get the best path string
        Args:
          emission (Tensor): Logit tensors. Shape `[num_seq, num_label]`.

        Returns:
          str: The resulting transcript
        """
        indices = torch.argmax(emission, dim=-1)  # [num_seq,]
        indices = torch.unique_consecutive(indices, dim=-1)
        indices = [i for i in indices if i != self.blank]
        return "".join([self.labels[i] for i in indices])

现在创建解码器对象并解码字幕。

decoder = GreedyCTCDecoder(labels=bundle.get_labels())
transcript = decoder(emission[0])

让我们检查结果并再次收听音频。

print(transcript)
IPython.display.Audio(SPEECH_FILE)

I|HAD|THAT|CURIOSITY|BESIDE|ME|AT|THIS|MOMENT|

ASR 模型使用一种称为连接时序分类 (CTC) 的损失函数进行了微调。CTC 损失的详细信息在此处进行了解释。在 CTC 中，空白标记 (ϵ) 是一个特殊标记，表示前一个符号的重复。在解码过程中，这些标记会被简单地忽略。

结论¶

在本教程中，我们研究了如何使用 Wav2Vec2ASRBundle 来执行声学特征提取和语音识别。构建模型和获取发射信号只需要两行代码。

model = torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H.get_model()
emission = model(waveforms, ...)

脚本总运行时间： ( 0 分 5.806 秒)

由 Sphinx-Gallery 生成的画廊