1. Sentence selection and document structure

• Goal: robustness, high compression
• Tracking of discourse structure
• Three step process:
  
  
  
  
  

  
  
  

  summary:	human written
  abstract:	machine generated
  excerpt:	collection of sentences

  
  

• Excerpts are not end results

• Emphasis: metalinguistic (rhetorical)
  information in sentences

• [Kupiec, Pedersen, Chen 1995]:
  A trainable document summarizer

2. Learning the contribution of heuristics

[Kupiec et al. 1995]'s gold standards (target extracts): alignable summary sentences.

Alignment of summary sentences
and document sentences:

Source of abstracts: professional abstractors
Alignment rate: 79%

3. Training and evaluation

Classifier:

Trained from corpus: P(F_j) and P (F_j element S). Independence assumption

Crossvalidation results (precision/recall):

Heuristics	Indiv	Cumul
Location	33%	33%
Cue Phrases	29%	42%
Sentence Length	24%	44%
tf*idf	20%	42%
tf*idf + capitalization	20%	42%
Baseline	2c24%

improvement of 74% over baseline
improvement of 29% over best single method

4. Our data

• 124 academic papers about computational linguistics
• Total: 24,083 sentences. Avg. document length: 210 sent. (Kupiec et al.: 86 sent.)
• Source: The Computation and Language E-Print Archive, 1994-96
• Wide range of subdomains:
  • argumentative (theoretical linguistics)
  • implementational
  • empirical (corpus studies)
  • experimental (psycholinguistics)
• Summaries written by authors. Avg. length: 4.7 sentences (Kupiec et al.: 3 sent.)

5. Target extracts and heuristics

Whole training set:




Division into three sub training sets:

compression = #sentences in target extract / # sentences in document

Target extract type	Compression
A	1.2%
B	3.2%
AB	4.4%
Kupiec et al.	3.0%

Heuristics:
Cue Phrases:	+3, +2, +1, 0, -1
Location:	p, m, 0
Sentence length:	short or long
tf*idf:	good or bad score
Title:	good or bad score

6. Cue phrases

• 202 triggers and 1691 cue phrases
• fixed phrases

2 we         argue        we argued
1 we         argue        we have argued
1 we         argue        we have argued that
1 we         argue        we will argue
1 I          argue        what I have argued is
1 we         argue        what we have argued is

3 article    article      ^this article
3 article    article      in this article

1 attempt    attempt      is an attempt to
2 I          attempt      I attempt to
2 I          attempt      I have attempted
2 I          attempt      I have attempted to
2 our        attempt      our work attempts
2 paper      attempt      the present paper is an attempt
2 paper      attempt      this paper is an attempt to

7. First experiment

Is Kupiec et al.'s methodology useful for our problem? YES

Heuristics	Indiv	Cumul
Cue Phrases	55.2%	55.2%
Location	32.1%	65.3%
Sentence Length	28.9%	66.3%
tf*idf	17.1%	66.5%
Title	21.7%	68.4%
Baseline	2c30.1%

improvement of 126% over baseline
improvement of 17% over best single method

7. Second experiment

Difference between gold standards in training

Distribution of features almost identical
between gold standards

8. Third experiment

Is there a data sparseness problem?

No data sparseness problem:

Distribution of features almost identical between 24,000 and 8,000 sentence training sets

9. Influence of precision on recall

11. Conclusions

• Three step robust abstracting system
• Emphasis: tracking discourse structure
• First step: sentence extraction

  
  
  
  

• Kupiec et al.'s methodology can be applied to
  -- different data: author-written summaries
  -- different task: excerpts not end results
• Methodology robust, computationally inexpensive;
  acquisition of training material can be automated

  
  
  
  

• No significant difference in training between
  human chosen and aligned training material
• Train on aligned sentences