Discriminative sensing

Recently I have red a very interesting paper [1] written by Keith Lewis about discriminative sensing. In this post I have selected most interesting ideas (from my point of view) as well as to share my thoughts.

Both natural and artificial vision systems have many points in common; for example, three different photo receptors for red, green, and blue bands of visible light or ability to process multi element scenes. But natural vision systems have significant advantage of pre-processing of images before processing and understanding by a visual cortex in a brain:

In general the biological imaging sensor takes a minimalist approach to sensing its environment, whereas current optical engineering approaches follow a ``brute'' force solution...[1]

This is a very serious problem: in case of in-vehicle systems, which requires real-time image processing, such ``brute'' force solutions are inefficient. Once you need to process images fast, you have to use powerful computers, parallel image processing algorithms, and symmetric multiprocessor methods. All of such ``brute force'' solutions led to increase of energy consumption of in-vehicle systems, requires more batteries, and eventually increasing size and weight of such dinosaurs-alike devices.

Further, the only one sensor is used in many cases of unmanned systems construction. Images produced by such sensor are tend to be redundant, excessively vast, and hence it is difficult to process them fast.

In the biological world, most organisms have an abundant and diverse assortment of peripheral sensors, both across and within sensory modalities.iMultiple sensors offer many functional advantages in relation to the perception and response to environmental signals....[1]

So I convinced that the next generation of imaging techniques and devices should use ideas and methods from natural vision systems. Indeed, it is sometimes useful to take lessons from the Nature as from an engineer with multi-billion years of experience.

Bio-inspiration

I have always been fascinating with insects - such small creatures that can distinguish and understand clogs and make decisions in complicated situations - more or less intellectually. For example,

...the fly has compound eyes, ...as well as the requisite neural processing cortex, all within an extremely small host organism. Its compound eyes provide the basis for sensing rapid movements across a wide field of view, and as such provide the basis of a very effective threat detection system...[1]

That's the main idea, I presume: only relevant objects are registered by small compound eyes and then understood by neural processing cortex of the fly. Interesting enough that there are much more complex vision systems exists even than human vision:

...a more complex vision architecture is found in the mantis shrimp[2]. Receptors in different regions of its eye are anatomically diverse and incorporate unusual structural features, not seen in other compound eyes. Structures are provided for analysis of the spectral and polarisation properties of light, and include more photoreceptor classes for analysis of ultraviolet light, color, and polarization than occur in any other known visual system...This implies that the visual cortex must be associated with some significant processing capability if the objective is to generate an image of its environment...[1]

In contrast, artificial systems are far less intelligent than ants or flies. For our unmanned systems, it is required to register all the scene at once, without understanding or even preprocessing it. Then using on-board computer systems, unmanned devices process such a huge stream of images by pixel-by-pixel strategy, without understanding of what kind of signals are relevant.

Although both natural and artificial vision systems use the same idea of tri-chromatic photoreceptor, the result differs dramatically. While animals are very good in recognition of preys or threats, artificial systems such as correlators and expert systems are relatively bad in making decisions. Primitive artificial YES-NO logic is not so flexible as natural neural networks based fuzzy sets of rules and growing experience of dealing with threats.

Such situation is very like a history of human's attempts of flight: for a long time people tried to get off the ground like birds. The success have came only after understanding the idea of flight.

Beyond a Nyquist's limit

As an example of non-trivial yet elegant approach, coded aperture systems are remarkable. Such idea can be applied both for visible [3] and IR [4] band. As it has been truly stated that such technique

...provides significant advantage in improving signal-to-noise ratio at the detector, without compromising the other benefits of the coded aperture technique. Radiation from any point in the scene is still spread across several hundred elements, but this is also sufficient to simplify the signal processing required to decode the image...[1]

It is noteworthy that analogous techniques such as ``wavefront coding''[5,6] and ``pupil engineering''[7,8] are applied in various optical systems, too. Application of such paradigms allows creating unique devices that combine both high optical parallelism and flexibility of digital algorithms of images processing.

It is clear that there is a little way to go yet before such computational imaging systems can be fielded on a practical basis...[1]

Moreover, such computational imaging systems are already here, in practical applications! Devices that are based on such techniques are used in security systems [9], tomography [10], aberrations correction [11,12] in optical systems, in depth of field improving [13], an so on.

It is curious that coded aperture approach can be found even in natural vision systems such as snakes vision [14]. These sensory organs enable the snake to successfully strike prey items even in total darkness or following the disruption of other sensory systems. Although the image that is formed on the pit membrane has a very low quality, the information that is needed to reconstruct the original temperature distribution in space is still available. Mathematical model that allows the original heat distribution to be reconstructed from the low-quality image on the membrane is reported in [15].

Instead of conclusion

There are no doubts that more and more approaches from natural vision systems will be used in artificial imaging systems. Hence the more we know about animals' eyes, the better we can design our artificial vision systems. I presume that in the near future, many of us are going to be constant readers of biological scientific journals...

Bibliography

1

Keith Lewis.
Discriminative sensing techniques.
Proc. of SPIE, Vol. 7113:71130C-10, 2008.

2

Cronin, T. W. and Marshall, J.
Parallel processing and image analysis in the eyes of mantis shrimps.
Biol. Bulletin, 200:177, 2001.

3

Slinger, C., Eismann, M., Gordon, N., Lewis, K., McDonald, G., McNie, M., Payne, D., Ridley, K., Strens, M., de Villiers G., and Wilson R.
An investigation of the potential for the use of a high resolution adaptive coded aperture system in the mid-wave infrared.
In Proc. SPIE 6714, 671408, 2007.

4

Slinger, C., Dyer, G., Gordon, N., McNie, M., Payne, D., Ridley, K., Todd, M., de Villiers, G., Watson, P., Wilson, R., Clark, T., Jaska, E., Eismann, M., Meola, J., and Rogers, S.
Adaptive coded aperture imaging in the infrared: towards a practical implementation.
In Proc. SPIE Annual Meeting, 2008.

5

J. van der Gracht, E.R. Dowski, M. Taylor, and D. Deaver.
New paradigm for imaging systems.
Optics Letters, Vol. 21, No 13:919-921, July 1, 1996.

6

Jr. Edward R. Dowski and Gregory E. Johnson.
Wavefront coding: a modern method of achieving high-performance and/or low-cost imaging systems.
In Proc. SPIE, Current Developments in Optical Design and Optical Engineering VIII, volume 3779, pages 137-145, 1999.

7

R. J. Plemmons, M. Horvath, E. Leonhardt, V. P. Pauca, S. Prasad, S. B. Robinson, H. Setty, T. C. Torgersen, J. van der Gracht, E. Dowski, R. Narayanswamy, and P. E. X. Silveira.
Computational imaging systems for iris recognition.
In Proc. SPIE, Advanced Signal Processing Algorithms, Architectures, and Implementations XIV, volume 5559, pages 346-357, 2004.

8

Sudhakar Prasad, Todd C. Torgersen, Victor P. Pauca, Robert J. Plemmons, and Joseph van der Gracht.
Engineering the pupil phase to improve image quality.
In Proc. SPIE, Visual Information Processing XII, volume 5108, pages 1-12, 2003.

9

Songcan Lai and Mark A. Neifeld.
Digital wavefront reconstruction and its application to image encryption.
Optics Communications, 178:283-289, 2000.

10

David J. Brady Daniel L. Marks, Ronald A. Stack.
Three-dimensional tomography using a cubic-phase plate extended depth-of-field system.
Opt. Letters No 4, 24:253-255, 1999.

11

H. Wans, E.R. Dowski, and W.T. Cathey.
Aberration invariant optical/digital incoherent systems.
Applied Optics, Vol. 37, No. 23:5359-5367, August 10, 1998.

12

Sara C. Tucker, W. Thomas Cathey, and Edward R. Dowski, Jr.
Extended depth of field and aberration control for inexpensive digital microscope systems.
Optics Express, Vol. 4, No. 11:467-474, 24 May 1999.

13

Daniel L. Barton, Jeremy A. Walraven, Edward R. Dowski Jr., Rainer Danz, Andreas Faulstich, and Bernd Faltermeier.
Wavefront coded imaging systems for MEMS analysis.
Proc. of ISTFA, pages 295-303, 2002.

14

Andreas B. Sichert, Paul Friedel, and J. Leo van Hemmen.
Snake's perspective on heat: Reconstruction of input using an imperfect detection system.
PHYSICAL REVIEW LETTERS, PRL 97:068105-1-4, 2006.

15

Andreas B. Sichert, Paul Friedel, and J. Leo van Hemmen.
Modelling imaging performance of snake infrared sense.
In Proceedings of the 13th Congress of the Societas Europaea Herpetologica. pp. 219-223; M. Vences, J. Kohler, T. Ziegler, W. Bohme (eds): Herpetologia Bonnensis II., 2006.

High dynamic range imagery

The dynamic range (DR) of modern solid-state photo sensors is generally not wide enough to image natural and some artificial scenes. This is especially the case for CMOS image sensors, since their read noise and DSNU are typically larger than CCDs. For reference, standard CMOS image sensors have a DR of 40-60 dB, CCDs around 60-70 dB; special CMOS imagers that employ continuous-time logarithmic readout achieve DR up to 140 dB, but they suffer from the loss of contrast and an increase of noise [1,2,3]. In contrast, the human vision system exhibits an enormous optical dynamic range of about 200 dB from the scotopic threshold to the glare limit [4]. Such capability is also required in many imaging applications.

To overcome DR limitations of photo sensors, several approaches were presented. Such approaches can be divided to hardware approaches (creating new architectures of sensors), software approaches (using colour light filters or multi-exposure of the same CMOS/CCD sensors), and hybrid hardware-software approaches (using spatial modulators, combining sensors with new architecture and multi-exposure technologies).

Hardware methods of HDR vision

Besides Active Pixel Sensor [5,6], many new sensor's architectures were presented recently. For example, a CMOS imager [2] that automatically selects a single optimum integration time and readout gain out of a variety of available integration times and gains individually for each pixel. Some CMOS approaches published in the open literature that rely on discrete-time integration either employ multiple sampling [7], sometimes combined with non-linear analogue-to-digital (A/D) conversion [8], or use non-linear integration [9]. An interesting approach is to design CMOS imagers with purely local on-chip brightness adoption [10].

Another way of hardware HDR imaging is to use ``smart sensors'' approach. ``Smart sensors'' have augmented photo-sensors with local processing for tasks such as edge detection, motion sensing and tracking. Mead's silicon retina and adaptive retina [11] chips were among the first to mimic vertebrate retina computations, and inspired many later efforts [12]. For example, in Mitsubishi's artificial retina [13] each photodetector's sensitivity was controllably modulated by others nearby to avoid saturation and aid in fast edge detection. In [14], a novel solid state image sensor is described where each pixel on the device includes a computational element that measures the time it takes to attain full potential well capacity.

It is also worth noting the approach to HDR imaging that uses a custom detector [15,16] where each detector cell includes two sensing elements (potential wells) of different sizes (and hence sensitivities). A general purpose logarithmic camera suitable for applications from family photographs to robotic vision to tracking and surveillance is presented [17].

Thus it can be concluded that hardware HDR solutions are compact and integrated (there is no need to use powerful computers for image processing), and ability to receive very wide dynamic range of images. The only disadvantage is the price of such photosensors: many of them are state-of-the-art devices and it is sometimes difficult to provide HDR-sensors with large amount of elements (5 Mpix and more).

Software methods of HDR vision

Many high dynamic range (HDR) photography methods were proposed that merge multiple images with different exposure settings [18,19,20]. Nayar et al. [21] has proposed a suite of HDR techniques that included spatially-varying exposures and adaptive pixel attenuation, and micro-mirror arrays to re-aim and modulate incident light on each pixel sensor [22]. Numerous reconstruction and tone mapping methods [23,20,18] were proposed for digital HDR photography. At each exposure setting, a different range of intensities can be measured reliably. Fusing the data from all the exposures [20,18] results in a single high dynamic range (HDR) image.

It is worth noting a simple and efficient method of obtaining HDR images from conventional photo sensor using Bayer colour filters array - Spatially Varying Pixel Exposures [24,25]. The idea is to assign different (fixed) exposures to neighbouring pixels on the image detector. When a pixel is saturated in the acquired image it is likely to have a neighbour that is not, and when a pixel produces zero brightness it is likely to have a neighbour that produces non-zero brightness.

Hence software methods are tend to be less expensive because it is possible to use conventional photosensors. For HDR images obtaining it is necessary only to develop special software; the disadvantages of such approaches are lower DR images and necessity to use external computers for image processing. But for portable machine vision systems software HDR methods can be the only one possibility.

Hybrid hard'n'soft HDR approaches

Most of hybrid hard'n'soft HDR methods use the combination of conventional CMOS or CCD photo sensor and light modulators (either LCD or light filters). For example, Adaptive Dynamic Range (ADR) concept was introduced in [21,26] were LCD ligth modulators as spatial filters were used. Such ADR concept is suitable not only for stil images, but for video sequences, too.

Simplest approach is to use multiple image detectors: beam splitters [27] are used to generate multiple copies of the optical image of the scene. Each copy is measured by an image detector whose exposure is preset by using an optical attenuator or by adjusting the exposure time of the detector.

Another approach is mosaicking with Spatially Varying Filter. Recently, the concept of generalized mosaicking [28,29], was introduced where a spatially varying neutral density filter is rigidly attached to the camera. When this imaging system is rotated, each scene point is observed under different exposures.

Hybrid methods of HDR imaging are tend to use software HDR methods, conventional photosensors, and external optical devices to control input scene's lightness. Advantages of hybrid methods are inexpensiveness and ability to receive wider DR images than pure software methods. But machine vision devices that use hybrid HDR approach are more cumbersome and hence may not be suitable for compact applications such as in-vehicle systems.

Bibliography

1

J. Huppertz, R. Hauschild, B. J. Hosticka, T. Kneip, S. Mller, and M.Schwarz.
Fast CMOS imaging with high dynamic range.
In Proc. Workshop Charge Coupled Devices & Advanced Image Sensors, Bruges, Belgium, pp. R7-1-R7-4., June 1997.

2

Michael Schanz, Christian Nitta, Arndt Bumann, Bedrich J. Hosticka, and Reiner K. Wertheimer.
A high-dynamic-range CMOS image sensor for automotive applications.
IEEE Journal of Solid-State Circuits, Vol. 35, No. 7:932-938, July 2000.

3

M. Schanz, W. Brockherde, R. Hauschild, B. J. Hosticka, and M.Schwarz.
Smart CMOS image sensor arrays.
IEEE Trans. Electron Devices, 44:1699-1705, Oct. 1997.

4

T. N. Cornsweet.
Visual Reception.
New York, NY: Academic, 1970.

5

O. Yadid-Pecht and A. Belenky.
In-pixel autoexposure CMOS APS.
IEEE Journal of Solid-State Circuits, vol. 38, no. 8:1425-1428, August 2003.

6

Orly Yadid-Pecht.
Active pixel sensor (aps) design - from pixels to systems.
Lectures.

7

O. Yadid-Pecht and E. Fossum.
Wide intrascene dynamic range CMOS APS using digital sampling.
IEEE Trans. Electron Devices, 44:1721-1723, Oct. 1997.

8

B. Fowler D. Yang, A. El Gamal and H. Tian.
A CMOS image sensor with ultrawide dynamic range floating-point pixel-level adc.
IEEE J. Solid-State Circuits, 34:1821-1834, Dec. 1999.

9

K. Brehmer S. J. Decker, R. D. McGrath and C. G. Sodini.
A cmos imaging array with wide dynamic range pixels and column-parallel digital output.
IEEE J. Solid-State Circuits, 33:2081-2091, Dec. 1998.

10

R. Hauschild, M. Hillebrand, B. J. Hosticka, J. Huppertz, T. Kneip, and M. Schwarz.
A cmos image sensor with local brighness adaption and high intrascene dynamic range.
In Proc. Eur. Solid-State Circuit Conf. (ESSCIRC'98), The Hague, the Netherlands, pp. 308-311, Sept. 22-24, 1998.

11

C. Mead.
Analog VLSI implementation of neural systems, chapter Adaptive Retina, pages 239-246.
Kluwer, 1989.

12

A. Moini.
Vision chips or seeing silicon, 1997.

13

E. Funatsu et al.
An artificial retina chip with a 256x256 array of n-mos variable sensitivity photodetector cells.
Proc. SPIE. Machine Vision App., Arch., and Sys. Int. IV, 2597:283-291, 1995.

14

V. Brajovic and T. Kanade.
A sorting image sensor: An example of massively parallel intensity-to-time processing for low-latency computational sensors.
In Proc. of IEEE Conference on Robotics and Automation, pages 1638-1643, April 1996.

15

R. J. Handy.
High dynamic range ccd detector/imager.
Technical report, U.S. Patent 4623928, November 1986.

16

M. Inuiya M. Konishi, M. Tsugita and K. Masukane.
Video camera, imaging method using video camera, method of operating video camera, image processing apparatus and method, and solid-state electronic imaging device.
Technical report, U.S. Patent 5420635, May 1995.

17

J. Tumblin, A. Agrawal, and R. Raskar.
Why i want a gradient camera.
In Proc. of IEEE CVPR, 2005.

18

S. Mann and R. Picard.
Being ``undigital'' with digital cameras: Extending dynamic range by combining differently exposed pictures.
In Proc. IST`s 48th Annual Conf., pages 422-428, 1995.

19

S. Winder S. B. Kang, M. Uyttendaele and R. Szeliski.
High dynamic range video.
ACM Trans. on Graphics, 22(3):319-325, July 2003.

20

P. E. Debevec and J. Malik.
Recovering high dynamic range radiance maps from photographs.
SIGGRAPH , 1997.

21

S.K. Nayar and V. Branzoi.
Adaptive dynamic range imaging: Optical control of pixel exposures over space and time.
IEEE International Conference on Computer Vision, Vol.2:1168-1175, Oct, 2003.

22

V. Branzoi S. K. Nayar and T. Boult.
Programmable imaging using a digital micromirror array.
CVPR., 1:436-443, 2004.

23

M.D. Grossberg and S.K. Nayar.
Determining the camera response from images: What is knowable?
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, No 11:1455-1467, 2003.

24

S. K. Nayar and T. Mitsunaga.
High dynamic range imaging: Spatially varying pixel exposures.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1:pp.472-479, June, 2000.

25

Srinivasa G. Narasimhan and Shree K. Nayar.
Enhancing resolution along multiple imaging dimensions using assorted pixels.
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, No. 4:pp. 518-530, April 2005.

26

Hidetoshi Mannami, Ryusuke Sagawa, Yasuhiro Mukaigawa, Tomio Echigo, and Yasushi Yagi.
High dynamic range camera using reflective liquid crystal.
In Proc. IEEE International Conference on Computer Vision Rio de Janeiro, October 14-20, 2007.

27

K. Saito.
Electronic image pickup device.
Technical report, Japanese Patent 07-254965, February 1995.

28

Y.Y. Schechner and S.K. Nayar.
Generalized Mosaicing: High Dynamic Range in a Wide Field of View.
International Journal on Computer Vision, 53(3):245-267, Jul 2003.

29

M. Aggarwal and N. Ahuja.
High dynamic range panoramic imaging.
In Proc. of International Conference on Computer Vision (ICCV), 1:2-9, 2001.

A scrutiny investigations in PixeLink's CMOS cameras: rolling shutter and all around

Active Pixel Sensors (text from [1])

A sensor with an active amplifier within each pixel was proposed [2]. Figure 1 shows the general architecture of an APS array and the principal pixel structure.

Figure 1: General architecture of an APS array (picture from work [1]).

The pixels used in these sensors can be divided into three types: photodiodes, photogates and pinned photodiodes [1].

Photodiode APS

The photodiode APS was described by Noble [2] and has been under investigation by Andoh [3]. A novel technique for random access and electronic shuttering with this type of pixel was proposed by Yadid-Pecht [4].

The basic photodiode APS employs a photodiode and a readout circuit of three transistors: a photodiode reset transistor (Reset), a row select transistor (RS) and a source-follower transistor (SF). The scheme of this pixel is shown in Figure 2.

Figure 2: Basic photodiode APS pixel (picture from work [1]).

Generally, pixel operation can be divided into two main stages, reset and phototransduction.

(a) The reset stage. During this stage, the photodiode capacitance is charged to a reset voltage by turning on the Reset transistor. This reset voltage is read out to one of sample-and-hold (S/H) in a correlated double sampling (CDS) circuit [5]. The CDS circuit, usually located at the bottom of each column, subtracts the signal pixel value from the reset value. Its main purpose is to eliminate fixed pattern noise caused by random variations in the threshold voltage of the reset and pixel amplifier transistors, variations in the photodetector geometry and variations in the dark current [1].

(b) The phototransduction stage. During this stage, the photodiode capacitor is discharged through a constant integration time at a rate approximately proportional to the incident illumination. Therefore, a bright pixel produces a low analogue signal voltage and a background pixel gives a high signal voltage. This voltage is read out to the second S/H of the CDS by enabling the row select transistor of the pixel. The CDS outputs the difference between the reset voltage level and the photovoltage level [1].

Because the readout of all pixels cannot be performed in parallel, a rolling readout technique is applied.

Readout from photodiode APS

All the pixels in each row are reset and read out in parallel, but the different rows are processed sequentially. Figure 3 shows the time dependence of the rolling readout principle.

Figure 3: Rolling readout principle of the photodiode APS (picture from work [1]).

A given row is accessed only once during the frame time (Tframe). The actual pixel operation sequence is in three steps: the accumulated signal value of the previous frame is read out, the pixel is reset, and the reset value is read out to the CDS. Thus, the CDS circuit actually subtracts the signal pixel value from the reset value of the next frame. Because CDS is not truly correlated without frame memory, the read noise is limited by the reset noise on the photodiode [1]. After the signals and resets of all pixels in the row are read out to S/H, the outputs of all CDS circuits are sequentially read out using X-addressing circuitry, as shown in Figure 2.

Other types are global shutter and fast-reset shutter, but such things are out of scope of this note.

Rolling shutter

In electronic shuttering, each pixel transfers its collected signal into a light-shielded storage region. Not of all CMOS imagers are capable of true global shuttering. Simpler pixel designs, typically with three transistors (3T), can only offer a rolling shutter [6]. Each row will represent the object at a different point of time, and because the object is moving, it will be at different point in space.

More sophisticated CMOS devices (4T and 5T pixels) can be designed with global shuttering and exposure control (EC) [6].

Typically, the rows of pixels in the image sensor are reset in sequence, starting at the top of the image and proceeding row by row to the bottom. When this reset process has moved some distance down the image, the readout process begins: rows of pixels are read out in sequence, starting at the top of the image and proceeding row by row to the bottom in exactly the same fashion and at the same speed as the reset process [7].

The time delay between a row being reset and a row being read is the integration time. By varying the amount of time between when the reset sweeps past a row and when the readout of the row takes place, the integration time (hence, the exposure) can be controlled. In the rolling shutter, the integration time can be varied from a single line (reset followed by read in the next line) up to a full frame time (reset reaches the bottom of the image before reading starts at the top) or more [7].

With a Rolling Shutter, only a few rows of pixels are exposed at one time. The camera builds a frame by reading out the most exposed row of pixels, starting exposure of the next unexposed row down in the ROI, then repeating the process on the next most exposed row and continuing until the frame is complete. After the bottom row of the ROI starts its exposure, the process ``rolls'' to the top row of the ROI to begin exposure of the next frame's pixels [8].

The row read-out rate is constant, so the longer the exposure setting, the greater the number of rows being exposed at a given time. Rows are added to the exposed area one at a time. The more time that a row spends being integrated, the greater the electrical charge built up in the row's pixels and the brighter the output pixels will be [8]. As each fully exposed row is read out, another row is added to the set of rows being integrated (see Fig. ).

Figure 4: Rolling shutter in work (picture from work [8]).

If there is a requirement of shooting with photoflash, there must be succeed some conditions. The operation of a photoflash with a CMOS imager [7] operating in rolling shutter mode is as follows:

1. The integration time of the imager is adjusted so that all the pixels are integrating simultaneously for the duration of the photoflash;
2. The reset process progresses through the image row by row until the entire imager is reset;
3. The photoflash is fired;
4. The imager is read out row by row until the entire imager is read out.

The net exposure in this mode will result from integrating both ambient light and the light from the photoflash. As previously mentioned, to obtain the best image quality, the ambient light level should probably be significantly below the minimum light level at which the photoflash can be used, so that the photoflash contributes a significant portion of the exposure illumination. Depending on the speed at which the reset and readout processes can take place, the minimum exposure time to use with photoflash may be sufficiently long to allow image blur due to camera or subject motion during the exposure. To the extent that the exposure light is provided by the short duration photoflash, this blur will be minimized.

Bibliography

1

Orly Yadid-Pecht.
Active pixel sensor (aps) design - from pixels to systems.
Lectures.

2

P. Noble.
Self-scanned image detector arrays.
IEEE Trans. Electron Devices, ED-15:202, 1968.

3

J. Yamazaki M Sagawara Y. Fujita K. Mitani Y. Matuzawa K. Miyata F. Andoh, K. Taketoshi and S. Araki.
A 250,000 pixel image sensor with FET amplification at each pixel for high speed television cameras.
IEEE ISSCC, pages 212-213, 1990.

4

R. Ginosar O. Yadid-Pecht and Y. Diamand.
A random access photodiode array for intelligent image capture.
IEEE J. Solid-State Circuits, SC-26:1116-1122, 1991.

5

J. Hynecek.
Theoretical analysis and optimization of CDS signal processing method for CCD image sensors.
IEEE Trans. Nucl. Sci., vol.39:2497-2507, Nov. 1992.

6

DALSA corp.
Electronic shuttering for high speed cmos machine vision applications.
Technical report, DALSA Corporation, Waterloo, Ontario, Canada, 2005.

7

David Rohr.
Shutter operations for ccd and cmos image sensors.
Technical report, Kodak, IMAGE SENSOR SOLUTIONS, 2002.

8

PixeLink.
Pixelink product documentation.
Technical report, PixeLink, December, 2007.

Optical encryption attacks to Double Random Phase Encryption

Double Random Phase Encryption (DRPE) technique has been criticised recently for poor security and low cryptography resistance because of its linearity. Recently has the security of DRP started to be thoroughly analysed and a few weaknesses were reported [1,2,3].

Double Random Phase Encryption technique

As it shortly described in [4], the image to be encrypted P is immediately followed by a first random phase mask, which is the first key X. Both the image and the mask are located in the object focal plane of a first lens (see Fig. 1).



In the image focal plane of this lens is therefore obtained the Fourier transform (FT) of the product $ P\cdot X$ . This product is then multiplied by another random phase mask that is the second key Y. Lastly, another FT is performed by a second lens to return to the spatial domain. Since the last FT does not add anything to the security of the system, we will perform all our analyses in the Fourier plane. The ciphered image C is then:

$\displaystyle C = Y \cdot \mathcal{F}(P\cdot X)$ (1)

where F stands for the Fourier transform operation. In most of the paper, we will assume that P is a grey-level image.

Attacks to the DRPE

Several attacks are proposed against the double random phase encryption scheme. Of source, as it mentioned in Javidi's article [4], brute force attack is useless due to huge amount of keys to be tested.

Reducing the number of combinations

More wise attack is the use of approximate version of the phase mask, especially to binary phase mask. Binarization of the phase mask reduces possible combinations dramatically. Of course, the fewer phase levels, the more noise is introduced in the reconstructed image.

In order to reduce the combinations of decryption keys further it is advisable [1,5] to decode with partial window of second key Y.

Plain-text attacks

The main idea of the plain-attack is to compromise an encryption system by specific known images. In Javidi's paper [4] is mentioned Dirac's delta function, uniform (spatially constant) image.

These attacks are demonstrated on computer-generated ciphered images, and the article [4] gives a comprehensive survey of attacks to DRPE. The scheme is shown to be resistant against brute force attacks but susceptible to chosen and known plaintext attacks. A technique to recover the exact keys with only two known plain images is described. Comparison of such technique to other attacks proposed in the literature is provided.

To sum up, with at most three chosen plain-ciphered image pairs, it is possible to recover the two encryption keys and break the system. But it is only theoretical review, no experimental works were provided. Also, there is no quantitative analysis of decryptability: only ``fuzzy'' visual estimations are presented (like ``the image is still recognizable'' [4] on page 6).

Personal remarks

In other terms, the plain image is entirely black except for a single pixel. It can be argued that such a plain image can look suspicious to the authorized user that is to encrypt it.

Why such image is suspicious? There is an example: you are going to encrypt an image printed on a paper. You are attaching the printed piece of paper on a pin upon the input scene and illuminating the input scene. A bright reflection from the input scene gives you an exact encryption key.

Related works: a little survey

As an example of cryptographical analysis and optical encryption cryptoresistance testing, Nauton's article is interesting [6], and iterative attempt to decrypt DRPE images is proposed in [7]. Another successful attempt to crack DRPE encryption method is reported in [7]. More detailed analysis of phase encoding's attack and quantization influence is covered in [8].

Moreover, Nauton published plain-text cryptographic attack method in [2]. The Fourier plane encryption algorithm is subjected to a known-plaintext attack, - he writes in article. So that Fourier plane encryption algorithm is found to be susceptible to a known-plaintext heuristic attack. Nauton applied a SA algorithm [9] to find a phase mask which would approximately decrypt the ciphertext. He successfully decrypted DRPE-coded $ 64\times 64$ image.

Bibliography

1
A. Carnicer, M. Montes-Usategui, S. Arcos, and I. Juvells.
Vulnerability to chosen-cyphertext attacks of optical encryption schemes based on double random phase keys.
Optics Letters, 30:1644-1646, 2005.
2
Thomas J. Naughton Unnikrishnan Gopinathan, David S. Monaghan and John T. Sheridan.
A known-plaintext heuristic attack on the fourier plane encryption algorithm.
Optics Express, Vol. 14, No. 8:3181-3186, 2006.
3
X. Peng, P. Zhang, H. Wei, and B. Yu.
Known-plaintext attack on optical encryption based on double random phase keys.
Optics Letters, 31:1044-1046, 2006.
4
Yann Frauel, Albertina Castro, Thomas J. Naughton, and Bahram Javidi.
Resistance of the double random phase encryption against various attacks.
Optics Express, Vol. 15, No. 16:10253-10265, 6 August 2007.
5
X. Peng, H. Wei, and P. Zhang.
Chosen-plaintext attack on lensless double-random phase encoding in the fresnel domain.
Optics Letters, 31:3261-3263, 2006.
6
David S. Monaghan, Unnikrishnan Gopinathan, Thomas J. Naughton, and John T. Sheridan.
Key-space analysis of double random phase encryption technique.
Applied Optics, Vol. 46, No. 26:6641-6647, 10 September 2007.
7
Guohai Situ, Unnikrishnan Gopinathan, David S. Monaghan, and John T. Sheridan.
Cryptanalysis of optical security systems with significant output images.
Applied Optics, Vol. 46, No. 22:5257-5262, 1 August 2007.
8
David S. Monaghan, Guohai Situ, Unnikrishnan Gopinathan, Thomas J. Naughton, and John T. Sheridan.
Role of phase key in the double random phase encoding technique: an error analysis.
Applied Optics, Vol. 47, No. 21:3808-3816, 20 July 2008.
9
S. Kirkpatrick, C.D. Gellatt, and M.P. Vecchi.
Optimization by simulated annealing.
Science, 220:771-680, 1983.