Learning a Probabilistic Partonomy from Annotated Images

Partonomies play a central role in image interpretation especially when viewed as a priori scene knowledge. As partonomies may be quite complex we developed a method to automatically derive a partonomy from annotated images. Such annotations consist of region-category pairs. As the regions may be only sketched and the categories may be used in different contexts, one has to expect the resulting partonomy to be uncertain. We use the conditional probabilities of a pixel to belong to a certain category given it belongs to another category, derive the tree structure of the regions in an image and finally by aggregating instances of the same category achieve an aggregation hierarchy together with the aggregated conditional probabilities. This contextual information may be used to partition given categories into subtypes or use it to transform the representation into a directed acyclic graph. The outcome is a graph that defines an irreflexive partial order of the set of categories, possibly appearing in different contexts. We show results on images of building facades and demonstrate the potential of the method.