Hi icefall team,
I have a question about FLOPs and RTF profiling for Conformer-CTC models in icefall.
I found PR #1093:
#1093
which adds a FLOPs profiler and supports profiling Zipformer and Conformer encoders. In that PR, FLOPs are reported for several encoder architectures using a 30-second input of shape `(1, 3000, 80)`:
Conformer encoder, (pruned_transducer_stateless) params: 83.49 M, flops: 221.62 G
Reworked Conformer encoder, (pruned_transducer_stateless4) params: 77.36 M, flops: 149.51 G
Old Zipformer encoder (pruned_transducer_stateless7), 69.14 M, flops: 96.65 G
Zipformer encoder (zipformer), params: 64.26 M, flops: 63.29 G
As far as I understand, these reported numbers are for the encoder only.
For a Conformer-CTC model, what is the recommended convention in icefall for computing and reporting FLOPs?
Should I report only the Conformer encoder FLOPs, or should I also include the CTC projection/output layer? If the recipe uses an attention decoder or rescoring module, should that part also be included in the reported FLOPs?
I understand that a pure CTC model does not have a prediction network and joiner like an RNN-T/Transducer model. My goal is to compare the computational cost of Conformer-CTC models with Transducer/Zipformer models fairly, so I would like to know which components icefall recommends including.
I also have a related question about RTF, i.e. real-time factor.
Do the existing decoding/evaluation scripts in icefall already compute and report RTF for Conformer-CTC models?
If yes, what is the recommended convention for reporting it? Should RTF be measured for the full decoding script, including feature extraction and decoding/search, or only for the acoustic model forward pass?
If RTF is not currently reported automatically, is there a recommended way to measure it for fair comparison across Conformer-CTC, Transducer, and Zipformer models?
Also, is there an existing example script for profiling FLOPs and/or measuring RTF for conformer_ctc models, or should I adapt the profiler from PR #1093 manually?
Thanks in advance for your guidance.
Hi icefall team,
I have a question about FLOPs and RTF profiling for Conformer-CTC models in icefall.
I found PR #1093:
#1093
As far as I understand, these reported numbers are for the encoder only.
For a Conformer-CTC model, what is the recommended convention in icefall for computing and reporting FLOPs?
Should I report only the Conformer encoder FLOPs, or should I also include the CTC projection/output layer? If the recipe uses an attention decoder or rescoring module, should that part also be included in the reported FLOPs?
I understand that a pure CTC model does not have a prediction network and joiner like an RNN-T/Transducer model. My goal is to compare the computational cost of Conformer-CTC models with Transducer/Zipformer models fairly, so I would like to know which components icefall recommends including.
I also have a related question about RTF, i.e. real-time factor.
Do the existing decoding/evaluation scripts in icefall already compute and report RTF for Conformer-CTC models?
If yes, what is the recommended convention for reporting it? Should RTF be measured for the full decoding script, including feature extraction and decoding/search, or only for the acoustic model forward pass?
If RTF is not currently reported automatically, is there a recommended way to measure it for fair comparison across Conformer-CTC, Transducer, and Zipformer models?
Also, is there an existing example script for profiling FLOPs and/or measuring RTF for conformer_ctc models, or should I adapt the profiler from PR #1093 manually?
Thanks in advance for your guidance.