Image Processing on Line

This short report discusses the development of the online journal IPOL (Image Processing on Line), which seems to be the first journal publishing reviewed software associated with each paper describing an algorithm, and providing each article with an online execution permitting immediate testing by readers.

1 Main scientific and technical achievements

Progress towards the establishment of a full state of the art in each of the main sections of the journal

The IPOL journal is currently divided into 19 sections, each containing an average of 5 articles and preprints. We tried to identify and publish first the algorithms representing each a very substantial portion of the state of the art. When choosing the methods to include, we follow two criteria. First, the most efficient algorithms must be present, and second all theories and methods that were proposed for a given problem should be represented, even though they are not (or no longer) considered the best.

The sections: Color and Contrast (11 articles), Denoising (17 articles), Demosaicking (6 articles), interpolation (4 articles), Optical flow (6 articles) and Vision through turbulence (3 articles) cover most of their respective state of the art, both in terms of best performance and of description of the main mathematical techniques. The other sections (3D, Blur, Computational Photography, Geometry, Infrared, Learning and Detection, Inpainting, Image Comparison, PDE, Stereovision, Texture) contain each at least three fundamental algorithms. The sections are listed in alphabetical order, but can also be grouped according to the application purpose.

The sections: Blur, Calibration, Denoising, Color and Contrast, Demosaicking and interpolation handle the main issues raised by the treatment of a raw image arriving at a camera, including the calibration of the camera itself. The 3D and Stereovision sections address the main issues of 3D reconstruction from a set of images. Other sections focus on analysis methods to extract information: detection of segments, faces, comparison of two images with invariant viewpoint and lighting, texture analysis, texture synthesis, restora- tion of missing parts (inpainting), motion analysis (optical flow). The last sections address classical or new acquisition methods (computational photography, infrared).

Editing online: each method requires a special setup . Editing on line a paper publishing an image processing method implies the invention of an adequate editing set-up, often including auxiliary algorithms to rprepare the data uploaded by users or to display the results.

Image denoising (17 papers) IPOL publishes the state of the art in image denoising.The papers analyze and finalize the often incomplete algorithms reportedin the literature and permit the first online objective comparison. These results are contained in the denoising section of the IPOL website. For each algorithm, users are requested to upload noise free images, to chose the noise level, which is added by the facility. In continuation the denoised image, the bias of the method, the extracted noise, and the PSNR are calculated and displayed.

To summarize the content and impact of this section, we can mention:

• A complete state of the art of denoising giving the best noise reduction algorithms based on the different theories: wavelets, neural networks, sparserepresentations, Bayesian methods, variational methods. The direct comparison of the algorithms corrects what was thought to be an established hierarchy of themethods.

• The“Noise Clinic”, first blind denoising algorithm to process any image, estimate its

noise without further information and denoise it correctly.

• The first generic noise model adaptable to most processed images (used for the noise clinic).

The publication of this state of the art has led the company DxO Labs to commercialize their PRIME algorithm, a direct adaptation of the Nonlocal Bayes algorithmpublished at IPOL.

• All the best noise image estimation algorithms are in this section and permit to estimate automatically the noise in any digital image.

Stereo Vision (6 methods) The stereo vision category at IPOL contains six fundamental algorithms, and three others are in preparation. The sophisticated editing of these methods included a first automatic stereorectification step. In that way all algorithms can be tried immediately on any image. A workshop paper presents the result of the design and implementation of the use of the Pleiades stereovision satellites. Through IPOL, the team has made available online the firstcomplete processing chain that takes a pair or triplet of satellite images and delivers a 3D digital elevation model of the ground

(http://dev.ipol.im/~carlo/ipol_demo/workshop_s2p/.) This facilitates the direct use of the Pleiades or Worldview Earth stereo vision satellites.

Texture analysis and synthesis (6 methods) The Texture section of IPOL gives the best examplar based texture synthesis algorithm and the best cartoon + texture papers.

Optical flow (7 methods) The sophisticated editing of the best optical flow methods is made in a multiscale form, and can therefore be tried on any pair of images.

2 Technical issues overcome through the development of IPOL

Most of these issues were resolved by Nicolas Limare (now IPOL technical director) as part of his thesis dedicated to reproducible research in image processing.

Reference programming language Some rules were needed to precise how algorithms can be implemented and how they should be evaluated by reviewers. The editorial board prepared a set of software guidelines with the list of authorized programming languages and libraries and basic criteria of software quality expected bythe journal. The first programming language was set to C/C++ for the performances these languages allow and their standardization and reliability, and common andstable scientific libraries are allowed. The submissions in Matlab are now also allowed.

Online publication It was necessary to design a web interface managing all online articles. Its first classic role was to allow conventional access requests to thearticle (.pdf) and to the associated code. But the main software design questions were raised by the online interface of the algorithmic demos, applicable on anyimage submitted by the users. This interface also stores the results of every user demo in the archives of each algorithm. This program managing all articles wasdesigned in 2009 by Nicolas Limare and revised in 2013, before being placed in the hands of other team members.

It was necessary to design a web interface adapted to each paper and its particular demo. This framework is a Python web server handling inputs and generating web pages in result of every user interaction with a demo: selecting or updating a file, changing a parameter of the algorithm, choosing a part of an image, a point,a line, etc.

The editorial question was then: how to manage the editorial burden of designing a different web service for each article? For the first articles it was designed byone person (Nicolas Limare), who then provided guidelines for the subsequent articles. Fortunately, it became soon clear that the number of types of articlesand online demos was limited. This is the basis for the next step, an automated demo generation from a limited catalog of standard interactions and displayfeatures.

Quality of articles and a “natural selection”

IPOL articles must be impeccable: every proposed algorithm has to be verifiable online on any given image selected by readers. Each algorithm has to bedescribed in detail by the authors, to check its conformity to the corresponding computer program. The online execution of each algorithm must take no morethan 30 seconds to spare the patience of readers (this often requires a parallelization). Finally the results of the algorithm need to be useful, interesting andcompetitive.

The vast majority of the algorithms published in image processing and computer vision journals are far from meeting our publication criteria. Most failed IPOLprojects have aborted when the authors realized that the algorithm described was incomplete, did not give all the results described on paper, was not feasible in areasonable time, or only worked on a certain type of data but not on “any” image. For example, many algorithms for stereo vision presuppose a pair of fronto-parallel cameras. This is certainly a convenient simplification, but it makes their algorithm a “proof of concept” algorithm, unsuited for any practical application onan actual stereo pair. Other authors with a potentially suitable algorithm were put off by the work of cleaning up and editing of their code to achieve a publishablecode. It quickly became apparent that the IPOL journal would not reject a significant proportion of submissions, since most non-viable projects fail before theyeven get submitted.

3 Lessons from the IPOL project

Criticisms to IPOL Authors and publishers have criticized IPOL for:

i. the excessive effort required to arrive at an executable article (rigor, acceleration, running on any data) and to describethe code and deliver perfectly readable code;

ii. the excessive length of the peer review (reviewers are slow and very exigent);

iii. the large number of objects to be published with the article: .pdf, source code, viable demo running on a Python editor ;

iv. the journal being less than four years old did not yet receive an official impact factor. IPOL has already more than 400citations on Google scholar for an average existence of two years for its articles (2011, 2012, 2013, 2014). The two years impact factor, calculable only on the2011 and 2012 papers, is higher than 10;

v. to some authors it is frustrating to work so hard to elucidate algorithms designed by others. Even when thosealgorithms are inventive and original, they are generally incomplete and require considerable research effort. Nevertheless, the symbolic benefit of this work goesback in good part to the originators.

Authors and publishers praised IPOL for:

vi. the immediate impact of their publications, reflected in the archive of experiments showing strong interest and use of thedemos for technical issues far beyond the inter- ests of the image processing community;

vii. the promise that this impact will continue due to the very existence of the demo;

viii. the fact that authors publishing these articles gain a tangible industrial and academic credibility, which greatly facilitates theobtention of research funding. As we observed, research funding agencies funding (ERC, ONR, ANR, DGA, CNES, FUI, ...) increased significantly their stake in theteam because of the reproducibility of this research.

4 Conclusion

Conceptually, the main IPOL innovation is that the publication of algorithms forces researchers to write a very compact, fully reproducible pseudo-code and enablesa universal disclosure of algorithms. Do not forget that even the best-written code is not easily readable and reusable, if only because of its necessary prolixity: nounderstatement or ellipse is possible when programming. A pseudo-code can be shorter by one or two orders of magnitude, while remaining unambiguous for amathematician programmer. In sum, the mathematical definition of functions is often much shorter than their program. While the computer pro- gramm isperishable, the pseudo code is not. Programming languages, program libraries, and operating systems change. This actually causes recurrent maintenanceproblems. But the pseudo-code remains. The pseudo-code of IPOL articles occupies between one and four pages maximum per article. It can be read in a few hoursand reprogrammed in any language in reasonable time. Thus it will appear as the main production of the journal. Noticeably, several code published at IPOL hasbeen licensed to several companies, in particular to Datalogic and to Decorwiser.

5 What is next? Outline of a new way of doing research

The incremental research More than 5,000 articles are published per year (counting newspapers and major conferences) on computer vision and imageprocessing. Approximately 100,000 articles have been published on the subject in the last thirty years. Most of these articles describe one or more algorithms.Nevertheless, some hundred methods would cover most of computer vision and image processing tasks and goals. For each of these operations there may be at mostthree or four variants or different competitive approaches. It follows that the state of the art of the subject might be eventually described rigorously with some 400 algorithms. Thus, a yearly production of some 40 articles at IPOL might exhaust the state of the art, old and new in ten years. The current pace of IPOL (20 articles/year) is not sufficient, but is still a sizeable contribution.

Does that mean that if the new publishing method proposed by IPOL succeeds, it will seal the end of image processing? Not at all. Indeed, it should encouragenew types of articles focusing on incremental research, where established and published algorithms (eg at IPOL) will be revised, combined, improved to achievemore sophisticated results and applications. This virtuous cycle, which exists in other disciplines such as the analysis of the genome or the numerical analysisof partial differential equations, has not yet been kick-started in image processing. Experiments that took years in a biological lab ten years ago sometimes take minutes nowadays because each new lab technique is soon industrialized. This should by all means happen in all disciplines producing algorithms, where it isparticularly easy to make finalized research available to all. IPOL simply demonstrates that it is technically possible and even easy to do so, and that it boosts the teams that adopt this work method.