Journal: International Conference on Computational Linguistics
Languages: All Languages
Programming languages: Python
Project website: https://github.com/lukecq1231/generalized-pooling
Pooling is an essential component of a wide variety of sentence representation and embedding models. This paper explores generalized pooling methods to enhance sentence embedding. We propose vector-based multi-head attention that includes the widely used max pooling, mean pooling, and scalar self-attention as special cases. The model benefits from properly designed penalization terms to reduce redundancy in multi-head attention.