Pyramid evaluation method for summarization

The pyramid method is a manual method for summarization evaluation and it was developed in an attempt to address a key problem in summarization---namely the fact that different humans choose different content when writing summaries. In DUC 2001 to 2004, the manual evaluation was based on comparison with a single human-written model and a lot of the information of evaluated summaries (both human and automic), was marked as "related to the topic, but not directly expressed in the model summary". The pyramid method addresses the problem by using multiple human summaries to create a gold-standard and by expoiting the frequency of information in the human summaries in order to assign importance to different facts.

A more complete description of the approach can be found in Ani Nenkova and Rebecca Passonneau Evaluating Content Selection in Summarization: the Pyramid Method NAACL-HLT 2004.

The pyramid gold-standard is based on a comparison between human-written summaries in terms of Summary Content Units (SCUs). A description of content units and their use in annotation ca be found HERE.