Melody-based retrieval in audio collections

Searching audio collections using high-level musical descriptors is a difficult problem, due to the lack of reliable methods for extracting melody, harmony, rhythm, and other such descriptors from unstructured audio signals. Our goal is to develop an approach to melody-based retrieval in audio collections that would supports audio, as well as symbolic queries.

Retrieval algorithm

Our algorithm is based on a melodic mid-level representation and locality sensitive hashing. It supports audio, as well as symbolic queries and ranks results according to melodic similarity to the query. We introduce a beat-synchronous mid-level melodic representation consisting of salient melodic lines, which are extracted from the analysed audio signal. We propose the use of a two-dimensional shift-invariant transform to extract shift-invariant melodic fragments from the melodic representation and demonstrate how such fragments can be indexed and stored in a song database. An efficient search algorithm based on locality-sensitive hashing is used to perform retrieval according to similarity of melodic fragments. On the cover song detection task, good results are achieved for audio, as well as for symbolic queries, while fast retrieval performance makes the proposed system suitable for retrieval in large databases.

For more details, see:

  • [PDF] M. Marolt, "A mid-level representation for melody-based retrieval in audio collections," IEEE transactions on multimedia, vol. 10, iss. 8, pp. 1617-1625, 2008.
    [Bibtex] 
    @article{6908756,
author={Matija Marolt},
year={2008},
pages={1617-1625},
volume={10},
title={A mid-level representation for melody-based retrieval in audio collections},
journal={IEEE transactions on multimedia},
number={8},
}
  • [PDF] M. Marolt, "Performing query-by-melody on audio collections," in 154th Meeting of the Acoustical Society of America, 27-29 November 2007, New Orleans, 2007, pp. 1-5.
    [Bibtex] 
    @conference{6241108,
author={Matija Marolt},
year={2007},
pages={1-5},
title={Performing query-by-melody on audio collections},
booktitle={154th Meeting of the Acoustical Society of America, 27-29 November 2007, New Orleans},
}
  • [PDF] M. Marolt, "A mid-level melody-based representation for calculating audio similarity," in ISMIR 2006 : proceedings, 2006, pp. 280-285.
    [Bibtex] 
    @conference{5554004,
author={Matija Marolt},
year={2006},
pages={280-285},
title={A mid-level melody-based representation for calculating audio similarity},
booktitle={ISMIR 2006 : proceedings},
}
Mid-level representation

Mid-level song representation for retrieval