Exploring the set of abstract images

This project is being developed in collaboration with Luis Alvarez, Universidad de Las Palmas Gran Canaria. Have also contributed Bruno Galerne, Yann Gousseau and Agustin Salgado de la Nuez. Its goal is to develop the mathematical and computation theory of abstract images, and to apply it to the development of a generative theory of decorative art.

You can try on line our abstract shape and texture generators and see many experimental results at jmandla.com

Before explaining the project, I'll show several images obtained without drawing, by a computer program making random choices and generating automatically the image. The only input of the program is a small set of silhouettes (cat, swan,...) and a color image that serves as color palette. This means that when I show a series of images, it has been successively and randomly generated. There is, however, another human intervention: the images have been sieved out, obviously by esthetic criteria that are personal. I therefore indicate as a percentage the approximate proportion of images that have been selected in the series produced by the computer.

Figure 1: Variations, painting in a cat silhouette, random texture, black and white palette

In Figure 1 I show a cat silhouette with random textures painted into it; the texture adapts to the containing shape, thus following the gestalt law that the "whole conditions the parts". Figure 2 is another illustration of the same painting principle, applied to a swan silhouette and with varying color palette.

Figure 2: Variations, painting in a swan silhouette, random textures, three different color palettes

The color palette is an input image used to select the color in the random generation. Figure 3 shows a color palette followed by random images respecting this palette.

Figure 3: abstract variations on a Van Gogh palette. Left, the palette, right, two random images.

A last preliminary illustration (Figure 4) shows the influence of the (random) choice of the painting technique on the final result. Starting from simple shapes and a palette, the random images can combine abstract painting techniques with exclusion, inclusion, transparency, tesselation, perspective, among other effects.

Figure 4: abstract variations with a palette also chosen randomly. The second palette is black and white. The third synthesis shows random Mandalas obtained with a variant of our image synthesis software, also used for designing scarfs (see Figure 12). The fourth image uses exclusively a random tessellation technique and the last one a transparency technique.

Perspectives on image sciences

Figure 5: Exemplar based texture sampling. Left: Portilla-Simoncelli’s method applied to a textile image. Right : Galerne et al. method applied to sea water and gravel on abeach.

The science of images is by nature interdisciplinary. It covers endeavours and disciplines as diverse as abstract art, decorative art, computer graphics, geometric measure theory, stochastic geometry, image processing, computer vision, gestalt psychology, and psychophysics. Using very diverse terminologies and techniques, all of these disciplines address the perceptual content of images and the ways to create or analyze them. The challenge of our project is to link all of these aspects in a mathematical and computational theory. The envisaged tools are mathematical models linked to algorithms and software permitting to explore and expand by random synthesis the set of abstract images.

This page argues that all of the above mentioned image formation and image interpretation theories rely on on few generative principles that can be geometrically formalized and simulated by random simulation. This means that one can design and visualize very general classes of images by random sampling. Thus the project aims at producing:

-an extension of geometric measure theory describing the structure of planar sets and func- tions;

-a general synthesis theory for (perceptually discriminable) artificial and natural textures;

-new sorts of abstract images expanding the technical possibilities of abstract digital art;

-a generative theory of decorative art.

Figure 6: Left (in grey): three images in Sobolev spaces images with a Fourier coefficient decay implying that they belong respectively to the Hilbert and Sobolev spaces L2, H1/2 and H1. Right: random emulation of a wood image by the Heeger Bergen model. As for Besov spaces it is characterized by a first order statisticsi of its wavelet coefficients.

Function spaces and texture synthesis

Image processing and computer graphics have revealed wide gaps in the mathematical classification of functions based on their geometric properties, evenfor 2D functions. Our mathematical function spaces have been designed to express variational principles modeling solids and liquids. They are associatedwith an energy given by some space integral. Its boundedness characterizes the elements of the function space, and it is a really rough characterization.To realize this by using computer graphics, one can generate random elements of each function space and see how they look. For example the spot noisealgorithm can synthesize a wide class of micro-textures including random elements of any Sobolev space. Fig. 5 shows on the left three generic elements ofthree Sobolev spaces. On the right, the same algorithm shows two natural textures that can be also modeled by fixing much more precisely the decay oftheir Fourier coefficients moduli: waves and gravel. Similarly, texture synthesis models based on histograms of wavelet coefficients incidentally permit tosimulate all elements of Besov spaces. They manage describing certain more complex textures24 as illustrated on the right of Figure 6 with a woodsimulation.In short, some observed textures in nature are modeled by a variant of Sobolev or Besov space where the profile of the Fourier or waveletcoefficients is fixed. But how many such different textures are possible? This question makes no sense unless we give a discrimination criterion. It is an easypsychophysical experiment to check that all textures generated by spot noise with identical Fourier coefficient moduli are visually indiscriminable. (Thisexperiment can be performed online on any image with the IPOL paper25.)

There is a still more general version of the dead leaves model involving transparency42. Random tessellations are another way to simulate planar texturesby simple subdivision processes43. See Figure 7, which also shows a fractal generated by iterated function system44.

These mathematical attempts are each focused on a single generation technique. The question of generating abstract images and drawings andexploring their perception was investigated almost simultaneously and in strikingly similar terms by abstract painters and Gestalt psychologists at thebeginning of the past century. Gestalt theory45 postulates that our perception proceeds by grouping the local percepts present in an image into moreglobal entities, the Gestalts. The Gestalt grouping laws46 are simple, geometric, and can be simulated numerically: proximity, symmetry, same color, sameshape, same orientation, good continuation, periodicity. Strikingly, these grouping laws are the implicit generative principles of decorative art (Figure 13),which also proceeds by combining simple shapes recursively. Kanizsa’s books actually focus on human perception of the two main painting techniques,occlusion and transparency47.

According to Bela Julesz, two textures are called indiscriminable if they cannot be discriminated from each other by a short examination, such as twopieces of the same textile. For example all white noise images with a same mean and variance are indiscriminable, and are considered “the same”texture26. This psychophysical theory has become computational thanks to computer graphics. Portilla and Simoncelli have invented a texture synthesisalgorithm27 convincingly achieving Julesz’s program: it shows that a wide range of examples of natural and artificial textures are perceptuallycharacterized (and can be approximately synthesized) by about 700 statistical wavelet moments and wavelet coefficient correlations (Figure 5, left). Twotextures characterized by the same parameters are indeed indiscriminable.

Figure 7: Basic mathematical processes generating abstract textures: a scaling dead leaves model a random tessellation, a fractal (iterated function system).

Structure theorems for 2D images and sets?

We have examined briefly functional spaces as image models and noticed that they could be viewed by image synthesis techniques and would boil down torather specific texture models. A generic Sobolev function indeed looks like a cloudy sky with no geometric structure (Figure 6). But in most images ofthe world, we do see a lot of geometric content!

Yet little mathematical effort has been dedicated to the geometric structure of 2D shapes and images in general. Only two complete mathematical theoriesexist: Besicovitch28 characterized the geometric structure of subsets of the plane with finite length. This theory was extended to arbitrary dimensions byMarstrand and Mattila29. This theory does not yield information on the geometry of “visible” measurable sets, with codimension 0. Ironically, theinteresting sets in Besicovitch’s theory are invisible, having zero measure projections. The visible sets of the plane have positive Lebesgue measure. Theycan be approximated by closed or open sets with arbitrary precision. Thus, for perceptual purposes, we could limit ourselves to classifying the geometricstructures of closed and of open sets.

Mathematical morphology is probably the first attempt to do so theoretically and computationally. The analysis of photographs of geological samples ledMatheron and his school30 to invent operators analyzing closed and open sets and characterizing their “granulometry”. These operators are extendable toimages, which can be described up to a contrast change by the stack of their upper level sets. In 2D, one can also describe the topographic map and aMorse theory for functions with minimal regularity31. The morphological approach is more general but a close parent of the theory of functions withbounded variation32. Being the only theory accounting for a geometric structure in images, it was used very early in image processing33. The level setsof a BV image have finite perimeter and have been thor- oughly described34. Yet this does not account for the natural existence of open sets withintrinsically infinite perimeter, like irrigation networks. Such irrigating sets or networks, which are essential in anatomy and geology, are dense connectedopen sets with arbitrarily small measure. They arise as solutions of optimal transport problems35.

24 Heeger, D. J., and J. R. Bergen. “Pyramid-based texture analysis/synthesis.”, Proc. ACM, 1995.

25 Galerne, B., Gousseau, Y., and Morel, J. M. (2011). Microtexture synthesis by phase randomization. Image Processing on Line.

26 Julesz, B. (1981). Textons, the elements of texture perception, and their interactions. Nature, 290(5802).

27 Portilla, J., and Simoncelli, E. P. (2000). A parametric texture model based on joint statistics of complex wavelet

coefficients. IJCV, 40(1), 49-70.

28 Besicovitch, A. S. “On the fundamental geometrical properties of linearly measurable plane sets of points (III).”

Mathematische Annalen 116.1 (1939): 349-357.

29 Mattila, P. (1999). Geometry of sets and measures in Euclidean spaces: fractals and rectifiability (No. 44). CUP.

30 Serra, J. (1982). Image analysis and mathematical morphology, v. 1. Academic press.

31 Caselles, V., and Monasse, P. (2010). Geometric description of images as topographic maps, Springer.

This fast description of the state of affairs in the geometry of 2D functions points out the necessity of a complete classification of closed and open sets, andof images modeled as upper semi-continuous functions. The example to imitate for such a theory is of course the analysis of BV functions. Yet a moregeneral structure theory for images should include a structural description of closed sets and perhaps a structure decomposition of them into Cantorianfibrillary parts and fully disconnected sets with Cantorian structure, or ”granulometries”, while non trivial open sets would be irrigation networks.36.

Figure 10: Random textures similar to natural textures (zoom-in on the .pdf for detail).

Figure 11: Abstract but understandable random textures (zoom-in for detail).

Figure 12: Five scarf models obtained by simple symmetrization of random textures.

Figure 13: Three fragments illustrating the common structure of abstract painting (Kandinsky) and decorative art (Klimt, a Kurdi carpet).

Figure 14: Digital art with abstract art techniques. Georg Nees 1965-1968; Michael Noll’s 1964 “Computer composition with lines”, a random pastiche of Mondrian’s1917 “Composition with lines” (Fig. 8); John Maeda’s “Florada, a computational study”. Technique: dead leaves models.