Tech Reports

Tech Report kmi-00-14 Abstract

Feature Reduction for Document Clustering and Classification
Techreport ID: kmi-00-14
Date: 2000
Author(s): Stefan Rüger and Susan Gauch

Often users receive search results which contain a wide range of documents, only some of which are relevant to their information needs. To address this problem, ever more systems not only locate information for users, but also organise that information on their behalf. We look at two main automatic approaches to information organisation: interactive clustering of search results and pre-categorising documents to provide hierarchical browsing structures. To be feasible in real world applications, both of these approaches require accurate yet efficient algorithms. Yet, both suffer from the curse of dimensionality - documents are typically represented by hundreds or thousands of words (features) which must be analysed and processed during clustering or classification. In this paper, we discuss feature reduction techniques and their application to document clustering and classification, showing that feature reduction improves efficiency as well as accuracy. We validate these algorithms using human relevance assignments and categorisation.

Publication(s):

DTR 2000/8, Department of Computing, Imperial College London

Future Internet

Knowledge
Management Multimedia &
Information Systems Narrative
Hypermedia New Media Systems Semantic Web &
Knowledge Services

Future Internet is...

With over a billion users, today's Internet is arguably the most successful human artifact ever created. The Internet's physical infrastructure, software, and content now play an integral part of the lives of everyone on the planet, whether they interact with it directly or not. Now nearing its fifth decade, the Internet has shown remarkable resilience and flexibility in the face of ever increasing numbers of users, data volume, and changing usage patterns, but faces growing challenges in meetings the needs of our knowledge society. Globally, many major initiatives are underway to address the need for more scientific research, physical infrastructure investment, better education, and better utilisation of the Internet. Within Japan, USA and Europe major new initiatives have begun in the area.

To succeed the Future Internet will need to address a number of cross-cutting challenges including:

Scalability in the face of peer-to-peer traffic, decentralisation, and increased openness

Trust when government, medical, financial, personal data are increasingly trusted to the cloud, and middleware will increasingly use dynamic service selection