eTRIMS - E-Training for Interpreting Images of Man-Made Scenes
 
eTRIMS
     
project results:

Semi-supervised incremental learning of hierarchical appearance models

Susanne Wenzel, Wolfgang Förstner
Department of Photogrammetry
University of Bonn
June, 2008

Due to the diversity of buildings, it is not possible to teach a recognition system once using a limited amount of training data. A training set that grasp the whole variety of building types would be of tremendous size and therefore it would not be realistic. Thus the goal of this work is to develop a system, which can improve its capabilities of learning continuously that is the recognition system must be capable of continually updating its learned models.
Furthermore we assume that learning man-made objects needs to be linked to notions defined by humans, therefore is supervised to some extent. Hence we need a system that is weakly semi-supervised in the initial stage and that becomes more robust and autonomous with every new image.

To reduce the amount of user interaction in the initial stage we want to exploit the fact that building parts often show some degree of symmetry.
Therefor we address the following problem: Given one example of an object and given the prior knowledge that it is repeatedly present in the same image we can learn its object class appearance. This provides a tool to detect other objects of the same type in other images with minimal need of user interaction.
Additionally, we build up an object class hierarchy with minimal amount of user interactions, hence that is managed by the users notion of the world. As we want to increase our knowledge about classes appearances with every new image, we want to propose models that are capable for incremental learning.

To achieve this goal, we need four procedures:
  • a detector for finding new instances
  • a prototype generator for finding new class candidates in an unsupervised manner,
  • a classifier for the envisaged classes and finally
  • an GUI for the interaction between the system and the supervisor.
The internal representation of the classes needs to be updatable incrementally.

Getting prototypes is based on a clustering procedure. Therefore we start with a single example and perform a recursive search procedure based on simple correlation to get a set of similar objects.
The clustering within this set of similar image patches is based on a reduced similarity graph, where all objects that are connected are assumed to belong to the same class.

For every cluster the user has to decide whether it is an interesting cluster or not and thus he builds up the classhierarchy.

Results for getting prototypes for a single facade, given only one example. Left: found image patches which were named by the teacher. Right: The according class hierarchy.

The representatives of every class are simply their mean images. We use these prototypes to detect probably new instances in new images. Our classifiers consists on a combined LDAaPCA subspace method. And we train a MAP classifier within this subspace, with a reject option depending on the probability ratio.
For further images the system automatically detect and classifies new instances using already learned models. The teacher has to supervise this step, that is he has to decide for uncertain classifications and has to fix miss-classifications. After this the system updates prototypes and classifiers. And the process starts again.

The will use this method to explore different curricula for incremental learning. That is, we will test different ways to present examples to the system.

References
[1] Susanne Wenzel, Wolfgang Förstner (2008).
Semi-supervised incremental learning of hierarchical appearance models. To appear in: 21st Congress of the International Society for Photogrammetry and Remote Sensing (ISPRS), Beijing, China, 2008. pdf
[2] Susanne Wenzel, Wolfgang Förstner (2008).
Semi-supervised discovering of prototypes for generating hierarchical appearance models. eTRIMS deliverable D3.2. May 2008.
[3] Susanne Wenzel, Wolfgang Förstner (2008).
Semi-supervised incremental learning of hierarchical appearance models. eTRIMS deliverable D3.4. May 2008.