Books on Kalman Filter

Here is small note about the Kalman and Particle filters. The list and comment reflect only my humble point of view: I'm an engineer who does not particularly like terse theoretical-orientated volumes.

Optimal State Estimate by Dan Simon
A very comprehensive book and, which is more important, it uses a bottom-up approach that enables readers to master the material rather quickly. Dan Simon's book is the best and the most suitable for self-study. Explanations are concise and straightforward. This book relates control theory elegantly. The author uses well laid out algorithmic approaches, suitable for programming, and examples to explain the details and show the complexities in action.
You will find all estimation topics in one book: Kalman filter, Unscented Kalman (UKF), Extended Kalman (EKF) and a very good explanation of Particle filtering (PF).

Moreover, the author has a website with a lot of MATLAB code for the examples used in book. This is a great help in terms of understanding of the subject, as most of examples are very insightful.


Introduction to Random Signals and Applied Kalman Filtering by Brown and Hwang
The book is easy to read and easy to follow: it starts from the very detailed explanation of the background needed for the Kalman filter. Obviously, the authors had an extensive teaching experience. The explanation of the Discrete Kalman Filter is one of the best I ever found. The extended KF and some implementation issues (UDU filter, sequential estimation) are not covered as well as other topics.

One should admit, however, that starting from the Chapter 6, the authors evidently exhausted and the text started to be more and more terse. The chapter about Kalman smoothing I could hardly translate to human language - better to use Dan Simon book.



Advanced Kalman Filtering, Least Squares and Modelling: A practical Handbook by Bruce Gibbs
The book is very terse in terms of explanations, and one should read it only as a reference, with a good background in Kalman filtering. Advanced topics, such as Particle Filter and Unscented Kalman, are covered in a very short manner and not very insightful. This is a handbook after all and not the book for the self-study. However, the book provides a lot of in-depth information and insight into various areas not found elsewhere.
This will be most useful for somebody with a strong mathematical background, particularly in linear algebra, who is looking for a comprehensive understanding and the best solution for a particular application.
Overall: a good reference, but not very helpful in terms of explanation.

Notes on the structure of the scientific article

Here are some notes about the structure of the scientific papers and some helpfull tips. Basically, any scientific paper can be divided on:

• abstract
• introduction
• methods (mathematical formulation)
• results
• discussion
  
• conclusion
• acknowledgements
  

Title
The title should be short and unambiguous, yet be an adequate description of the work. A general rule-of-thumb is that the title should contain the key words describing the work presented. Remember that the title becomes the basis for most on-line computer searches - if your title is insufficient, few people will find or read your paper.


Abstract
Once you have the completed abstract, check to make sure that the information in the abstract completely agrees with what is written in the paper. Confirm that all the information appearing the abstract actually appears in the body of the paper.

A well-prepared abstract enables the reader to identify the basic content of a document quickly and accurately. The abstract concisely states the principal objectives and scope of the investigation where these are not obvious from the title. The abstract must be concise; most journals specify a length, typically not exceeding 250 words. If you can convey the essential details of the paper in 100 words, do not use 200.


Introduction
The introduction defines the subject and must: outline the scientific objectives for the research performed and give the reader sufficient background to understand the rest of the report.

A good introduction will answer several questions:

1. Why was this study performed? Answers to this question may be derived from observations of nature or from the literature.

2. What knowledge already exists about this subject? That is review of the literature, showing the historical development of an idea and including the confirmations, conflicts, and gaps in existing knowledge.

3. What is the specific purpose of the study? The specific hypotheses and experimental design pertinent to investigating the topic should be described.

4. What is the Novelty of this paper? An important function of the introduction is to establish the significance of your current work: Why was there a need to conduct the study? Having introduced the pertinent literature and demonstrated the need for the current study, you should state clearly the scope and objectives.

Fundamental questions to answer here include:

• Do your results provide answers to your testable hypotheses? If so, how do you interpret your findings?
• Do your findings agree with what others have shown? If not, do they suggest an alternative explanation or perhaps a unforseen design flaw in your experiment (or theirs?)
• Given your conclusions, what is our new understanding of the problem you investigated and outlined in the Introduction?
• If warranted, what would be the next step in your study, e.g., what experiments would you do next?

Methods
Materials and methods used in the experiments should be reported. What equipment was used, what is the mathematics that describes the process?
However, it is still necessary to describe special pieces of equipment and the general theory of the assays used. This can usually be done in a short paragraph, possibly along with a drawing of the experimental apparatus.


Results

1. All scientific names (genus and species) must be italicized. (Underlining indicates italics in a typed paper.)
2. Use the metric system of measurements. Abbreviations of units are used without a following period.
3. Be aware that the word data is plural while datum is singular. This affects the choice of a correct verb. The word species is used both as a singular and as a plural.
4. Numbers should be written as numerals when they are greater than ten or when they are associated with measurements; for example, 6 mm or 2 g but two explanations of six factors. When one list includes numbers over and under ten, all numbers in the list may be expressed as numerals; for example, 17 sunfish, 13 bass, and 2 trout. Never start a sentence with numerals. Spell all numbers beginning sentences.
5. Be sure to divide paragraphs correctly and to use starting and ending sentences that indicate the purpose of the paragraph. A report or a section of a report should not be one long paragraph.
6. Every sentence must have a subject and a verb.
7. Avoid using the first person, I or we, in writing. Keep your writing impersonal, in the third person. Instead of saying, "We weighed the frogs and put them in a glass jar," write, "The frogs were weighed and put in a glass jar."
8. Avoid the use of slang and the overuse of contractions.
9. Be consistent in the use of tense throughout a paragraph--do not switch between past and present. It is best to use past tense.
10. Be sure that pronouns refer to antecedents. For example, in the statement, "Sometimes cecropia caterpillars are in cherry trees but they are hard to find," does "they" refer to caterpillars or trees?

After writing a report, read it over, watching especially for lack of precision and for ambiguity. Each sentence should present a clear message.



Conclusion
Introductions and conclusions can be the most difficult parts of papers to write. While the body is often easier to write, it needs a frame around it. An introduction and conclusion frame your thoughts and bridge your ideas for the reader.

Your conclusion should make your readers glad they read your paper. Your conclusion gives your reader something to take away that will help them see things differently or appreciate your topic in personally relevant ways. It can suggest broader implications that will not only interest your reader, but also enrich your reader's life in some way.

Generally, three questions in the conclusion should be addressed:

1. What the problem was addressed?
2. What are results?
3. So what?

Play the "So What" Game. If you're stuck and feel like your conclusion isn't saying anything new or interesting, ask a friend to read it with you. Whenever you make a statement from your conclusion, ask the friend to say, "So what?" or "Why should anybody care?" Then ponder that question and answer it. Here's how it might go:

You: Basically, I'm just saying that education was important to Douglass.

Friend: So what?

You: Well, it was important because it was a key to him feeling like a free and equal citizen.

Friend: Why should anybody care?

You: That's important because plantation owners tried to keep slaves from being educated so that they could maintain control. When Douglass obtained an education, he undermined that control personally.

You can also use this strategy on your own, asking yourself "So What?" as you develop your ideas or your draft.

Synthesize, don't summarize. Include a brief summary of the paper's main points, but don't simply repeat things that were in your paper. Instead, show your reader how the points you made and the support and examples you used fit together.

Propose a course of action, a solution to an issue, or questions for further study. This can redirect your reader's thought process and help her to apply your info and ideas to her own life or to see the broader implications.

Looking to the future: Looking to the future can emphasize the importance of your paper or redirect the readers' thought process. It may help them apply the new information to their lives or see things more globally.

Books about Adaptive Optics

Here is a small list of books on Adaptive optics. There are not many of them since the subject is quite specific and requires wide and diverce background.

Adaptive Optics for Astronomical Telescopes (Oxford Series in Optical \& Imaging Sciences) by John Hardy
Hardy was a pioneer in adaptive optics and in the 1970s he built the first system capable of compensating the turbulence of a large astronomical telescope at visible wavelengths.

The book is very comprehensive, with an excellent bibliography and outstanding illustrations. The text is informative and consistent, with strong points in atmosphere turbulence and deformable mirrors. As a minor issues I must mention that the control part of adaptive optics is covered less deeply, but enough for the first-time reader. Despite of its age, the book gives all necessary background to enter to adaptive optics field.

The book by Hardy is by far The Best book in adaptive optics.

John W. Hardy, Adaptive optics for astronomical telescopes, Oxford University Press, USA, 1998.


Numerical Simulation of Optical Wave Propagation With Examples in MATLAB, by Jason Schmidt
The book presents the latest advances in numerical simulations of optical wave propagations in turbulent media. The book is clearly written and abundant of excellent examples in MATLAB giving to the reader a lot of step-by-step introductions as well as understanding of the waves propagation. The writing style is very engaging.

However, the Chapter 9 is slightly denser than others (I think it could be split on two different chapters). The operators notations used in Chapter 6 sometimes are more difficult to follow than conventional expressions. But those are minor issues that do not affect the material of the book.

Overall, the material of the book and the MATLAB code present a solid basis for the numerical simulations. Carefully selected bibliography of the book allows to use it as an excellent reference.


Jason D. Schmidt, Numerical Simulation of Optical Wave Propagation, With Examples in Matlab, Society of Photo-Optical Instrumentation Engineers, 2010.



Adaptive Optics in Astronomy by Francois Roddier
This is a very good example of how one should NOT write a book. This is not even a book but just a draft of several conference proceedings meshed together. There is no transition between chapters, the writing skills of different authors are very different and it is quite annoying.

For instance, in chapter about numerical simulations you will find NOTHING about how to really simulate the AO systems: no sinlge formula or plot. Chapter about ``theoretical'' background is written in a manner that there actually is a solid theory behind it - and most formulas are starting with numercial coefficient, which is improssible to get theoretically.

The second part of book is just a outdate garbage: stories of how the author built telescopes, with unnecessary detailed information that is useless now.

Don't waste your time on it.


Adaptive optics handbook by R.Tyson
The following books is by R. Tyson. Personally, I think that reading books from Tyson is mostly waste of time. He is like bakery that produces books as they are cakes. Very few useful information. More pity is that Tyson is a non-stop-backery than stamps and stamps books, one worse than other.

Linear Algebra: small survey of books

It is sometimes useful to repeat some fundamentals of mathematics, especially when it has not been used for a long time. For the study of coherence estimation in complex-valued data I had to review and refresh my Linear Algebra skills. For that I needed to study several topics and the result of it will be this small survey of books, handbooks and articles that helped me in such challenging study.

General-purpose books

First of all, one need a good general-purpose books for initial refresh of Linear Algebra. I think that Gilbert Strang's book [1] is among the best for systematic study. Although sometimes (especially in the appendices) book became too verbose (lots of exotic examples) or too brief (the reader must guess of how to actually calculated many types of decompositions), Strang's book is a good starting point anyway. It covers all basic topics except quadratic forms; very short introduction to SVD and Pseudo-Inverse. It is noticeable that Strang's video lectures are sometimes way better than his own book.

The book ``Linear Algebra'' written by J. Hefferon [2] is great for start and initial understanding of Linear Algebra. Clearly written and with lost of examples, it is a must-read book to quickly remember some base material of Linear Algebra. Moreover, there are some applications after each chapter. The textbook [2] is useful as a teaching material because a lot of examples and deductive approach of material's explaining. Examples in this book are very insightful (least-squares, crystals, economic examples and so on). Coverage stretches from basic to eigenvalue decomposition that is not enough (no SVD, Pseudo-Inverse and Quadratic forms at all).

Specialised books

One of the most necessary books for me is Matrix Algebra from a statistical perspective [3] written by D. Harville. The book contains most of necessary topics for Linear Algebra applications from the statistical and engineering point of view. For instance, the book contains insightful chapters about matrix differentiating (Chapter 14) that is very helpful.

The book written by Hoffman and Kunze [4] is a good starting point of studying bilinear and quadratic forms. The book of Marcus and Ming [5] is a good reference for matrix inequalities and other special topics of algebra. A very good book [6] written by Gene H. Golub contains interesting topics and discussion of Schur decomposition and Pseudoinverse.

For several advanced topics and pure mathematical proofs, I want to mention Radjendra Bhatia's Matrix Analysis [7] book. The questions such as Spectral variations of Normal matrices and Majorization were helpful for me.

Articles and notes

Many properties of product and sum of pseudoinverse matrix are described in articles. One particular article that discuss the properties of product of pseudoinverse that is (AB)^+ =A^+B^+ is Taussky's article [8].

A very good introduction to differentiating of matrices is written in small ``Notes on Matrix Calculus'' [9] by Paul L. Fackler from North Carolina State University. That is one os the most easy, clear and bright introduction to matrix differentiating that I could found.

The small book about Toeplitz and Circulant matrices with very good introduction is written by [10] (can be download from the Internet). This book allows to use the necessary properties of Toeplitz matrices in applied science without digging in pure mathematical folios. Writing style is clear and shiny with reasonable amount of examples in statistics and signal processing.

And of course, one should definitely read Schur's original paper (available in digital form) about Schur decomposition [11].

Reference books and Handbooks

A truly great reference is Leslie Hogben's handbook [12] that contains the most of material of Linear Algebra. Very concentrated material, with numerous links to other books and articles, ``Handbook of Linear Algebra'' is indispensable on reference and quick recalling some additional properties and relations of Algebra's objects.

Although the book by Horn& Johnson [13] ``Matrix Analysis'' appears in practically any reference sources, it is not an easy reading material. One should not read it from front to back, but rather selected topics. The material is well organised but is very dense: Horn&Johnson book is rather handbook than a textbook.

Useful and helpful handbook that contains many inequalities and interesting properties of Linear Algebra's objects is The Matrix Cookbook [14] written by K. B. Petersen and M. S. Pedersen. Although there are mentions the the Cookbook contains many mistakes and inaccuracies, it is useful and may be utilised as quick reference. For instance, it contains short but bright description of the matrix differentiating.

Bibliography

1

Gilber Strang.
Linear Algebra and its Applications.
Thomson Learning, 1988, 3^d Edition.

2

Jim Hefferon.
Linear Algebra.
2000.

3

D.A. Harville.
Matrix algebra from a statistician's perspective.
Springer Verlag, 2008.

4

K. Hoffman and R. Kunze.
Linear Algebra.
Prentice-Hall, Englewood Cliffs, NJ, 1971.

5

M. Marcus and H. Minc.
A survey of matrix theory and matrix inequalities.
Allyn and Bacon, Boston, 1964.

6

G.H. Golub and C.F. Van Loan.
Matrix computations, 1996.

7

R. Bhatia.
Matrix analysis.
Springer Verlag, 1997.

8

O. Taussky.
Commutativity in finite matrices.
American Mathematical Monthly, 64(4):229-235, 1957.

9

Paul L. Fackler.
Notes on matrix calculus.
North Carolina State University, 2005.

10

R.M. Gray.
Toeplitz and circulant matrices: A review.
2006.

11

I. Schur.
On the characteristic roots of a linear substitution with an application to the theory of integral equations.
Math. Ann, 66:488-510, 1909.

12

L. Hogben.
Handbook of linear algebra.
CRC Press, 2007.

13

Roger A. Horn and Charles R. Johnson.
Matrix Analysis.
Cambridge University Press, 1985.

14

K.B. Petersen and M.S. Pedersen.
The Matrix Cookbook.
Technical University of Denmark, 2008.
20081110.

EMVA1288 Standard

The European Machine Vision Association Standard 1288 is designed for unification of description digital photocameras and photosensors' characteristics. This is an open standard that provides a framework for estimation of camera's characteristics. The EMVA1288 standard is build modularly and contains all necessary recommendations and comprehensive mathematical formulation of how to measure and estimate photosensor's characteristics.

Every module of EMVA1288 Standard consists of mathematical model, the experimental setup, calculation steps and recommendations of how to publish the results of measuring. Currently (version 2.01A) there are two modules for the EMVA1288 standard: Module 1 "Characterizing the Image Quality and Sensitivity" and Module 2 "Linearity and Linearity Error".

In the Module 1 "Characterizing the Image Quality and Sensitivity of Machine Vision Cameras and Sensors", the procedure of how to characterize the temporal and spatial noise of a camera and it's sensitivity to light is described.

In the Module 2 "Linearity and Linearity Error" is described the method of estimation of area and linescan sensors/cameras for which the output signal is expected to be directly proportional to the impinging photon flux (exposure). Although this module is optional, it may be useful for estimation of the real dynamic range of the photo sensor.

The EMVA1288 Standard was re-typesetted in LaTeX format as the more appropriate format for scientific use. The latest LaTeX version of the EMVA1288 Standard can be downloaded from these mirrors:

• mirror1
• mirror2

As for concluding remark, I can additionally say that EMVA1288 Standard is useful not only for machine-vision cameras but for consumer-grade cameras, too. RAW data from the consumer-grade cameras, after appropriate conversion by such software as dcraw, can be used for characterisation of consumer-grade camera as a measuring device.

Small survey of Objective Image Quality metrics

PDF version of this post is here

The importance of objective quality metric methods cannot be underestimated: such methods are being used in automated images restoration algorithms, for comparison of images compression algorithms and so on. Quality metrics are graphically presented in Fig. 1.

All proposed quality metrics can divided to two general classes¹: subjective and objective [2].

Subjective evaluation of images quality is oriented on Human Vision System (HVS). As it was mentioned in [3], the best way to assess the quality of an image is perhaps to look at it because human eyes are the ultimate receivers in most image processing environments. The subjective quality measurement Mean Opinion Score (MOS) has been used for many years.

Objective metrics include Mean Squared Error (MSE), or $L_p$-norm [4,5], and measures that are mimicking the HVS such as [6,7,8,9,10,11]. In particular, it is well known that a large number of neurons in the primary visual cortex are tuned to visual stimuli with specific spatial locations, frequencies, and orientations. Images quality metrics that incorporate perceptual quality measures by considering human visual system (HVS) were proposed in [12,13,14,15,16]. Image quality measure (IQM) that computes image quality based on the 2-D spatial frequency power spectrum of an image was proposed in [10]. But still such metrics have poor performance in real applications and widely criticized for not correlating well with perceived quality measurement [3].

As a promising techniques for images quality measure, Universal Quality Index [17,3], Structural SIMilarity index [18,19], and Multidimensional Quality Measure Using SVD [1] are worth to be mentioned ².

Figure 1: Types of images quality metrics.

So there are three objective methods of images' quality estimation to be discussed below: the UQI, the SSIM, and MQMuSVD. Brief information about main ideas of those metrics is given. But first of all, let me render homage to a mean squared error (MSE) metric.

A Good-Old MSE

Considering that $x={x_i | i = 1,2,\dots N}$ and $y={x_i | i = 1,2,\dots N}$ are two images, where N is the number of image's pixels, the MSE between these images is:
Of course, there is more general and well-suitable formulation of MSE for images processing given by Fienup [5]:

(2)

where

(3)

Such NRMSE metrics allows to estimate quality of images especially in various applications of digital deconvolution techniques. Although Eq. 2 is better than pure MSE, the NRMSE metric have been criticizing a lot.

As it was written in remarkable paper [19], the MSE is used commonly for many reasons. The MSE is simple, parameter-free, and easy to compute. Moreover, the MSE has clear physical meaning as the energy of the error signal. Such an energy measure is preserved after any orthogonal linear transformation, such as Fourier transform. The MSE is widely used in optimization tasks and in deconvolution problem [21,22,23]. Finally, competing algorithms have most often been compared using the MSE or Peak SNR ratio.

But problems arising when one is trying to predict human perception of image fidelity and quality using MSE. As it was shown in [19], the MSE is very similar despite the differences in image's distortions. That is why there were many attempts to overcome MSE's limitations and find a new images quality metrics. Some of them are briefly discussed below.

Multidimensional Quality Measure Using SVD

The new metric of images quality called ``Multidimensional Quality Measure Using SVD'' was proposed in [1]. The main idea is that every real matrix A can be decomposed into a product of 3 matrices A = USV<sup>T</sup>, where U and V are orthogonal matrices, U<sup>T</sup>U = I, V<sup>T</sup>V = I, and $S = diag (s_1, s_2, \dots)$. The diagonal entries of S are called the singular values of A, the columns of U are called the left singular vectors of A, and the columns of V are called the right singular vectors of A. This decomposition is known as the Singular Value Decomposition (SVD) of A [24]. If the SVD is applied to the full images, we obtain a global measure whereas if a smaller block is used, we compute the local error in that block:
$s_i$ are the singular values of the original block, $\hat{s}_i$ are the singular values of the distorted block, and N is the block size. If the image size is $K$, we have $(K/N) \times (K/N)$ blocks. The set of distances, when displayed in a graph, represents a ``distortion map''.

A universal image quality index (UQI)

As a more promising new paradigm of images quality measurements, a universal image quality index was proposed in [17]. This images quality metric is based on the following idea:

The main function of the human eyes is to extract structural information from the viewing field, and the human visual system is highly adapted for this purpose. Therefore, a measurement of structural distortion should be a good approximation of perceived image distortion.

The key point of the new philosophy is the switch from error measurement to structural distortion measurement. So the problem is how to define and quantify structural distortions. First, let's define a necessary mathematics [17] for original image X and test image Y . The universal quality index can be written as [3]:

(5)

where

The first component is the linear correlation coefficient between x and y, i.e., this is a measure of loss of correlation. The second component measures how close the mean values are between x and y, i.e., luminance distortion. The third component measures how similar the variances of the signals are, i.e., contrast distortion.

UQI quality measurement method is applied to local regions using sliding window approach. For overall quality index to be obtained, average value of local quality indexes $Q_i$ must be calculated:

(6)

As it mentioned in [17], the average quality index UQI coincides with the mean subjective ranks of observers. That gives to researchers a very powerful tool for images' quality estimation.

Structural SIMilarity (SSIM) index

The Structural Similarity index (SSIM) that is proposed in [18] is a generalized form of a Universal Quality Index [17]. As above, $x$ and $y$ are discrete non-negative signals; $\mu_x$, $\sigma_{x}^2$, and $\sigma_{xy}$ are the mean value of $x$, the variance of $x$, and the covariance of $x$ and $y$, respectively. According to [18] the luminance, contrast, and structure comparison measures were given as follows:

			(7)
			(8)
			(9)

where $C_1$, $C_2$ and $C_3$ are small constants given by $C_1 = (K_1\cdot L)^2$ ; $C_2 = (K_2 \cdot L)^2$ and $C_3 = C_2/2$. Here $L$ is the dynamic range of the pixel values, and $K_1 \ll 1$ and $K_2 \ll 1$ are two scalar constants. The general form of the Structural SIMilarity (SSIM) index between signal x and y is defined as:

(10)

where $\alpha, \beta, \; \text{and} \; \gamma$ are parameters to define the relative importance of the three components [18]. If $\alpha= \beta= \gamma =1$, the resulting SSIM index is given by:

(11)

SSIM is maximal when two images are coinciding (i.e., SSIM is <=1 ). The universal image quality index proposed in [17] corresponds to the case of $C_1 = C_2 = 0$ , therefore is a special case of Eq. (11).

A drawback of the basic SSIM index is its sensitivity to relative translations, scalings and rotations of images [18]. To handle such situations, a waveletdomain version of SSIM, called the complex wavelet SSIM (CW-SSIM) index was developed [25]. The CWSSIM index is also inspired by the fact that local phase contains more structural information than magnitude in natural images [26], while rigid translations of image structures leads to consistent phase shifts.

Despite its simplicity, the SSIM index performs remarkably well [18] across a wide variety of image and distortion types as has been shown in intensive human studies [27].

Instead of conclusion

As it was said in [18], ``we hope to inspire signal processing engineers to rethink whether the MSE is truly the criterion of choice in their own theories and applications, and whether it is time to look for alternatives.'' And I think that such articles provide a great deal of precious information for making decision to give away the MSE.


Useful links:
A very good and brief survey of images quality metrics, with links to MATLAB examples. Zhou Wang's page with huge amount of articles and MATLAB source code for UQI and SSIM. Another useful link for HDR images quality metrics.

Bibliography

1

Aleksandr Shnayderman, Alexander Gusev, and Ahmet M. Eskicioglu.
A multidimensional image quality measure using singular value decomposition.
In Image Quality and System Performance. Edited by Miyake, Yoichi; Rasmussen, D. Rene. Proceedings of the SPIE, Volume 5294, pp. 82-92, 2003.

2

A. M. Eskicioglu and P. S. Fisher.
A survey of image quality measures for gray scale image compression.
In Proceedings of 1993 Space and Earth Science Data Compression Workshop, pp. 49-61, Snowbird, UT, April 2, 1993.

3

Ligang Lu Zhou Wang, Alan C. Bovik.
Why is image quality assessment so difficult?
In In: Proceedings of the ICASSP'02, vol. 4, pp. IV-3313-IV-3316., 2002.

4

W. K. Pratt.
Digital Image Processing.
John Wiley and Sons, Inc., USA, 1978.

5

J.R. Fienup.
Invariant error metrics for image reconstruction.
Applied Optics, No 32, 36:8352-57, 1997.

6

J. L. Mannos and D. J. Sakrison.
The effects of a visual fidelity criterion on the encoding of images,.
IEEE Transactions on Information Theory, Vol. 20, No. 4:525-536, July 1974.

7

J. O. Limb.
Distortion criteria of the human viewer.
IEEE Transactions on Systems, Man, and Cybernetics, Vol. 9, No. 12:778-793, December 1979.

8

H. Marmolin.
Subjective mse measures.
IEEE Transactions on Systems, Man, and Cybernetics, Vol. 16, No. 3:486-489, May/June 1986.

9

J. A. Saghri, P. S. Cheatham, and A. Habibi.
Image quality measure based on a human visual system model.
Optical Engineering, Vol. 28, No. 7:813-818, July 1989.

10

B. N. Norman and H. B. Brian.
Objective image quality measure derived from digital image power spectra.
Optical Engineering, 31(4):813-825, 1992.

11

A.A. Webster, C. T. Jones, M. H. Pinson, S. D. Voran, and S. Wolf.
An objective video quality assessment system based on human perception.
In Proceedings of SPIE, Vol. 1913, 1993.

12

T. N. Pappas and R. J. Safranek.
in book ``Handbook of Image and Video Processing'' (A.Bovik, ed.), chapter Perceptual criteria for image quality evaluation.
Academic Press, May 2000.

13

B. Girod.
in book Digital Images and Human Vision (A. B. Watson, ed.), chapter What's wrong with mean-squared error, pages 207-220.
the MIT press, 1993.

14

S. Daly.
The visible difference predictor: An algorithm for the assessment of image fidelity.
In in Proceedings of SPIE, vol. 1616, pp. 2-15, 1992.

15

A. B. Watson, J. Hu, and J. F. III. McGowan.
Digital video quality metric based on human vision.
Journal of Electronic Imaging, vol. 10, no. 1:20-29, 2001.

16

J.-B. Martens and L. Meesters.
Image dissimilarity.
Signal Processing, vol. 70:155-176, Nov. 1998.

17

Z. Wang and A.C. Bovik.
A universal image quality index.
IEEE Signal Processing Letters, vol. 9, no. 3:81-84, Mar. 2002.

18

Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.
Image quality assessment: From error visibility to structural similarity.
IEEE Transactions on Image Processing, vol. 13, no. 4:600-612, Apr. 2004.

19

Zhou Wang and Alan C. Bovik.
Mean squared error: Love it or leave it?
IEEE Signal Processing Magazine, 98:98-117, January 2009.

20

D.M. Chandler and S.S. Hemami.
Vsnr: A wavelet-based visual signal-to-noise ratio for natural images.
IEEE Transactions on Image Processing, vol. 16, no. 9:2284-2298, Sept. 2007.

21

Wiener N.
The extrapolation, interpolation and smoothing of stationary time series.
New York: Wiley, page 163р, 1949.

22

J.R. Fienup.
Refined wiener-helstrom image reconstruction.
Annual Meeting of the Optical Society of America, Long Beach, CA, October 18, 2001.

23

James R. Fienup, Douglas K. Griffith, L. Harrington, A. M. Kowalczyk, Jason J. Miller, and James A. Mooney.
Comparison of reconstruction algorithms for images from sparse-aperture systems.
In Proc. SPIE, Image Reconstruction from Incomplete Data II, volume 4792, pages 1-8, 2002.

24

D. Kahaner, C. Moler, and S. Nash.
Numerical Methods and Software.
Prentice-Hall, Inc., 1989.

25

Z. Wang and E.P. Simoncelli.
Translation insensitive image similarity in complex wavelet domain.
In Proceedings of IEEE International Conference of Acoustics, Speech, and Signal Processing, pp. 573-576., Mar. 2005.

26

T.S. Huang, J.W. Burdett, and A.G. Deczky.
The importance of phase in image processing filters.
IEEE Transactions on Acoustic, Speech, and Signal Processing, vol. 23, no. 6:529-542, Dec. 1975.

27

H.R. Sheikh, M.F. Sabir, and A.C. Bovik.
A statistical evaluation of recent full reference image quality assessment algorithms.
IEEE Transactions on Image Processing, vol. 15, no. 11:3449-3451, Nov. 2006.

Discriminative sensing

Recently I have red a very interesting paper [1] written by Keith Lewis about discriminative sensing. In this post I have selected most interesting ideas (from my point of view) as well as to share my thoughts.

Both natural and artificial vision systems have many points in common; for example, three different photo receptors for red, green, and blue bands of visible light or ability to process multi element scenes. But natural vision systems have significant advantage of pre-processing of images before processing and understanding by a visual cortex in a brain:

In general the biological imaging sensor takes a minimalist approach to sensing its environment, whereas current optical engineering approaches follow a ``brute'' force solution...[1]

This is a very serious problem: in case of in-vehicle systems, which requires real-time image processing, such ``brute'' force solutions are inefficient. Once you need to process images fast, you have to use powerful computers, parallel image processing algorithms, and symmetric multiprocessor methods. All of such ``brute force'' solutions led to increase of energy consumption of in-vehicle systems, requires more batteries, and eventually increasing size and weight of such dinosaurs-alike devices.

Further, the only one sensor is used in many cases of unmanned systems construction. Images produced by such sensor are tend to be redundant, excessively vast, and hence it is difficult to process them fast.

In the biological world, most organisms have an abundant and diverse assortment of peripheral sensors, both across and within sensory modalities.iMultiple sensors offer many functional advantages in relation to the perception and response to environmental signals....[1]

So I convinced that the next generation of imaging techniques and devices should use ideas and methods from natural vision systems. Indeed, it is sometimes useful to take lessons from the Nature as from an engineer with multi-billion years of experience.

Bio-inspiration

I have always been fascinating with insects - such small creatures that can distinguish and understand clogs and make decisions in complicated situations - more or less intellectually. For example,

...the fly has compound eyes, ...as well as the requisite neural processing cortex, all within an extremely small host organism. Its compound eyes provide the basis for sensing rapid movements across a wide field of view, and as such provide the basis of a very effective threat detection system...[1]

That's the main idea, I presume: only relevant objects are registered by small compound eyes and then understood by neural processing cortex of the fly. Interesting enough that there are much more complex vision systems exists even than human vision:

...a more complex vision architecture is found in the mantis shrimp[2]. Receptors in different regions of its eye are anatomically diverse and incorporate unusual structural features, not seen in other compound eyes. Structures are provided for analysis of the spectral and polarisation properties of light, and include more photoreceptor classes for analysis of ultraviolet light, color, and polarization than occur in any other known visual system...This implies that the visual cortex must be associated with some significant processing capability if the objective is to generate an image of its environment...[1]

In contrast, artificial systems are far less intelligent than ants or flies. For our unmanned systems, it is required to register all the scene at once, without understanding or even preprocessing it. Then using on-board computer systems, unmanned devices process such a huge stream of images by pixel-by-pixel strategy, without understanding of what kind of signals are relevant.

Although both natural and artificial vision systems use the same idea of tri-chromatic photoreceptor, the result differs dramatically. While animals are very good in recognition of preys or threats, artificial systems such as correlators and expert systems are relatively bad in making decisions. Primitive artificial YES-NO logic is not so flexible as natural neural networks based fuzzy sets of rules and growing experience of dealing with threats.

Such situation is very like a history of human's attempts of flight: for a long time people tried to get off the ground like birds. The success have came only after understanding the idea of flight.

Beyond a Nyquist's limit

As an example of non-trivial yet elegant approach, coded aperture systems are remarkable. Such idea can be applied both for visible [3] and IR [4] band. As it has been truly stated that such technique

...provides significant advantage in improving signal-to-noise ratio at the detector, without compromising the other benefits of the coded aperture technique. Radiation from any point in the scene is still spread across several hundred elements, but this is also sufficient to simplify the signal processing required to decode the image...[1]

It is noteworthy that analogous techniques such as ``wavefront coding''[5,6] and ``pupil engineering''[7,8] are applied in various optical systems, too. Application of such paradigms allows creating unique devices that combine both high optical parallelism and flexibility of digital algorithms of images processing.

It is clear that there is a little way to go yet before such computational imaging systems can be fielded on a practical basis...[1]

Moreover, such computational imaging systems are already here, in practical applications! Devices that are based on such techniques are used in security systems [9], tomography [10], aberrations correction [11,12] in optical systems, in depth of field improving [13], an so on.

It is curious that coded aperture approach can be found even in natural vision systems such as snakes vision [14]. These sensory organs enable the snake to successfully strike prey items even in total darkness or following the disruption of other sensory systems. Although the image that is formed on the pit membrane has a very low quality, the information that is needed to reconstruct the original temperature distribution in space is still available. Mathematical model that allows the original heat distribution to be reconstructed from the low-quality image on the membrane is reported in [15].

Instead of conclusion

There are no doubts that more and more approaches from natural vision systems will be used in artificial imaging systems. Hence the more we know about animals' eyes, the better we can design our artificial vision systems. I presume that in the near future, many of us are going to be constant readers of biological scientific journals...

Bibliography

1

Keith Lewis.
Discriminative sensing techniques.
Proc. of SPIE, Vol. 7113:71130C-10, 2008.

2

Cronin, T. W. and Marshall, J.
Parallel processing and image analysis in the eyes of mantis shrimps.
Biol. Bulletin, 200:177, 2001.

3

Slinger, C., Eismann, M., Gordon, N., Lewis, K., McDonald, G., McNie, M., Payne, D., Ridley, K., Strens, M., de Villiers G., and Wilson R.
An investigation of the potential for the use of a high resolution adaptive coded aperture system in the mid-wave infrared.
In Proc. SPIE 6714, 671408, 2007.

4

Slinger, C., Dyer, G., Gordon, N., McNie, M., Payne, D., Ridley, K., Todd, M., de Villiers, G., Watson, P., Wilson, R., Clark, T., Jaska, E., Eismann, M., Meola, J., and Rogers, S.
Adaptive coded aperture imaging in the infrared: towards a practical implementation.
In Proc. SPIE Annual Meeting, 2008.

5

J. van der Gracht, E.R. Dowski, M. Taylor, and D. Deaver.
New paradigm for imaging systems.
Optics Letters, Vol. 21, No 13:919-921, July 1, 1996.

6

Jr. Edward R. Dowski and Gregory E. Johnson.
Wavefront coding: a modern method of achieving high-performance and/or low-cost imaging systems.
In Proc. SPIE, Current Developments in Optical Design and Optical Engineering VIII, volume 3779, pages 137-145, 1999.

7

R. J. Plemmons, M. Horvath, E. Leonhardt, V. P. Pauca, S. Prasad, S. B. Robinson, H. Setty, T. C. Torgersen, J. van der Gracht, E. Dowski, R. Narayanswamy, and P. E. X. Silveira.
Computational imaging systems for iris recognition.
In Proc. SPIE, Advanced Signal Processing Algorithms, Architectures, and Implementations XIV, volume 5559, pages 346-357, 2004.

8

Sudhakar Prasad, Todd C. Torgersen, Victor P. Pauca, Robert J. Plemmons, and Joseph van der Gracht.
Engineering the pupil phase to improve image quality.
In Proc. SPIE, Visual Information Processing XII, volume 5108, pages 1-12, 2003.

9

Songcan Lai and Mark A. Neifeld.
Digital wavefront reconstruction and its application to image encryption.
Optics Communications, 178:283-289, 2000.

10

David J. Brady Daniel L. Marks, Ronald A. Stack.
Three-dimensional tomography using a cubic-phase plate extended depth-of-field system.
Opt. Letters No 4, 24:253-255, 1999.

11

H. Wans, E.R. Dowski, and W.T. Cathey.
Aberration invariant optical/digital incoherent systems.
Applied Optics, Vol. 37, No. 23:5359-5367, August 10, 1998.

12

Sara C. Tucker, W. Thomas Cathey, and Edward R. Dowski, Jr.
Extended depth of field and aberration control for inexpensive digital microscope systems.
Optics Express, Vol. 4, No. 11:467-474, 24 May 1999.

13

Daniel L. Barton, Jeremy A. Walraven, Edward R. Dowski Jr., Rainer Danz, Andreas Faulstich, and Bernd Faltermeier.
Wavefront coded imaging systems for MEMS analysis.
Proc. of ISTFA, pages 295-303, 2002.

14

Andreas B. Sichert, Paul Friedel, and J. Leo van Hemmen.
Snake's perspective on heat: Reconstruction of input using an imperfect detection system.
PHYSICAL REVIEW LETTERS, PRL 97:068105-1-4, 2006.

15

Andreas B. Sichert, Paul Friedel, and J. Leo van Hemmen.
Modelling imaging performance of snake infrared sense.
In Proceedings of the 13th Congress of the Societas Europaea Herpetologica. pp. 219-223; M. Vences, J. Kohler, T. Ziegler, W. Bohme (eds): Herpetologia Bonnensis II., 2006.

Optical-digital encryption systems

As an alternative approach of encryption techniques, optical-digital system approach is worth of noting. Such systems introduce optically a known distortion in a registered image. Then such introduced distortion can be compensated digitally.

All of such encryption systems can be divided in three category: optical, digital, and hybrid optical-digital systems. A brief survey of them is presented further.

Digital encryption systems

Digital encryption systems are performing all operations in numerical form in computer [1] so there is no need of optical systems to be created. There are several systems that are mostly digital such as Virtual Optics system [2] and virtual-optical-holography (VOH) [3] that are characterized by high cryptography resistance. Other digital systems that are worth noting are using fractional wavelet transform [4] or fractional Fourier transform [5].

One of the most widespread digital encryption technique is virtual-optical imaging scheme, VOIS [2] that simulates an optical imaging system in a computer model. As a computation model, Fresnel approach is used; encryption parameters are the wavelength of coherent ``virtual'' light LAMBDA , the distance between the image to be encoded and ``lens'' d0 , focal distance of the ``lens'' f , and distance between the ``lens'' and observing plane d_i (see Fig. 1).

Figure 1: The key diagram of the Virtual Optics.

Fourier-spectrum of an image to be encoded is multiplied by Fourier-spectrum of coding mask, so one need to know exact parameters of LAMBDA, d0 , d_i , and f in order to decrypt the image.

Optical encryption systems

In optical encryption techniques are utilized high speed and parallelism of optical images processing. As encryption keys, diffractive optical elements (DOEs) are used that are synthesized and outputted to ferroelectric [6] or LCD-modulators [7]. For example, such systems use lensless approach [8], 4-f based systems [9], fractional Fourier transform [10,11,12] or systems based on double random phase mask [13]. Apart from these systems, toroidal zone plates encryption systems is worth mention [14].

The most widespread approach in optical encryption systems is double random phase mask [13,15,16,17]. As it shortly described in [18], the image to be encrypted P is immediately followed by a first random phase mask, which is the first key X . Both the image and the mask are located in the object focal plane of a first lens (see Fig. 2).

Figure 2: DRPE coding scheme

In the image focal plane of this lens is therefore obtained the Fourier transform (FT) of the product P*X. This product is then multiplied by another random phase mask that is the second key Y. Lastly, another FT is performed by a second lens to return to the spatial domain. Since the last FT does not add anything to the security of the system, we will perform all our analyses in the Fourier plane. The ciphered image C is then:

C = Y * FT(P*X)

(1)

where F stands for the Fourier transform operation. In most of the paper, we will assume that P is a grey-level image.

Such systems allow obtaining encrypted images that are characterized by high cryptography resistance. Images are being encrypted in a very short time because of parallel optical processing is performed. Complexity of optical key diagram and expensiveness are drawbacks of such systems. Moreover, several vulnerabilities of double random phase mask were reported recently [19,20,21].

Hybrid optical-digital systems

Hybrid optical-digital systems allow to combine advantages of optical processing (high speed and parallelism) and digital processing (flexibility of digital image processing methods). Application of digital methods in optical coding allow to reduce weight and cost of devices.

As a most widespread optical-digital paradigms, ``wavefront coding'' [22] and ``pupil engineering'' [23] are worth mention. Systems based on these paradigms are used in enhancing depth of field in microscopic imaging [24], aberrations compensation [25], depth-of-field improvement in MEMS-systems [26], and enhancing of tomography images [27].

Coding diffractive element (DOE) is introduced in the imaging scheme of such devices; hence optical convolution of input object and point spread function (PSF) of the DOE is performed optically. As a result, the image registered is blurred but a blur is the same across the image. Digital images deconvolution is performed in order to reconstruct the image and compensate introduced distortion. An example of hybrid optical-digital device is shown in Fig. 3.

Figure 3: Hybrid optical-digital imaging system: a photo sensor and a kinoform (DOE).

Hybrid optical-digital systems based on ``wavefront coding'' and ``pupil engineering'' paradigms can be used not only for encryption but for depth-of-field enhancing, too.

Such systems are advantageous because of their inexpensiveness, flexibility, and reliability (may be used not only for data encryption). But after image decryption, visual quality of the image is degraded slightly that is a disadvantage.

Bibliography

1

Ari Y. Benbasat.
A survey of current optical security techniques.
Technical report, MIT Media Lab, Prepared for Prof. Cardinal Warde 6.637 Spring 1999 Research Project, April 15, 1999.

2

Xiang Peng, Zhiyong Cui, and Tieniu Tan.
Information encryption with virtual-optics imaging system.
Optics Communications, 212(4-6):235-245, November 2002.

3

Xiang Peng and Peng Zhang.
Security of virtual-optics-based cryptosystem.
Optik, 117:525-531, 2006.

4

Linfei Chen and Daomu Zhao.
Optical image encryption based on fractional wavelet transform.
Optics Communications, 254(4-6):361-367, October 2005.

5

B.M. Hennelly and J.T. Sheridan.
Image encryption and the fractional fourier transform.
Optik - International Journal for Light and Electron Optics, 114(6):251-265, 2003.

6

G. Unnikrishnan, M. Pohit, and K. Singh.
A polarization encoded optical encryption system using ferroelectric spatial light modulator.
Optics Communications, 185(1-3):25-31, November 2000.

7

Chau-Jern Cheng and Mao-Ling Chen.
Polarization encoding for optical encryption using twisted nematic liquid crystal spatial light modulators.
Optics Communications, 237(1-3):45-52, July 2004.

8

Guohai Situ and Jingjuan Zhang.
A lensless optical security system based on computer-generated phase only masks.
Optics Communications, 232(1-6):115-122, March 2004.

9

Xiaogang Wang, Daomu Zhao, and Linfei Chen.
Image encryption based on extended fractional fourier transform and digital holography technique.
Optics Communications, 260(2):449-453, April 2006.

10

Naveen Kumar Nishchal, Joby Joseph, and Kehar Singh.
Fully phase-encrypted memory using cascaded extended fractional fourier transform.
Optics and Lasers in Engineering, 42(2):141-151, August 2004.

11

Naveen Kumar Nishchal, Joby Joseph, and Kehar Singh.
Securing information using fractional fourier transform in digital holography.
Optics Communications, 235(4-6):253-259, May 2004.

12

Banghe Zhu and Shutian Liu.
Optical image encryption based on the generalized fractional convolution operation.
Optics Communications, 195(5-6):371-381, August 2001.

13

Steven C. Verrall Enrique Tajahuerce, Osamu Matoba and Bahram Javidi.
Optoelectronic information encryption with phase-shifting interferometry.
Applied Optics, Vol. 39, No. 14:2313-2320, 10 May 2000.

14

John Fredy Barrera, Rodrigo Henao, and Roberto Torroba.
Optical encryption method using toroidal zone plates.
Optics Communications, 248(1-3):35-40, April 2005.

15

Bahram Javidi Pedro Andres Enrique Tajahuerce, Jesus Lancis.
Optical security and encryption with totally incoherent light.
Optics Letters, Vol. 26, No. 10:678-680, May 15, 2001.

16

Enrique Tajahuerce and Bahram Javidi.
Encrypting three-dimensional information with digital holography.
Applied Optics, Vol. 39, No. 35:6595-6601, 10 December 2000.

17

Enrique Tajahuerce Bahram Javidi.
Three-dimensional object recognition by use of digital holography.
Optics Letters, Vol. 25, No. 9:610-612, May 1, 2000.

18

Yann Frauel, Albertina Castro, Thomas J. Naughton, and Bahram Javidi.
Resistance of the double random phase encryption against various attacks.
Optics Express, Vol. 15, No. 16:10253-10265, 6 August 2007.

19

A. Carnicer, M. Montes-Usategui, S. Arcos, and I. Juvells.
Vulnerability to chosen-cyphertext attacks of optical encryption schemes based on double random phase keys.
Optics Letters, 30:1644-1646, 2005.

20

Thomas J. Naughton Unnikrishnan Gopinathan, David S. Monaghan and John T. Sheridan.
A known-plaintext heuristic attack on the fourier plane encryption algorithm.
Optics Express, Vol. 14, No. 8:3181-3186, 2006.

21

X. Peng, P. Zhang, H. Wei, and B. Yu.
Known-plaintext attack on optical encryption based on double random phase keys.
Optics Letters, 31:1044-1046, 2006.

22

W.T. Cathey and E.R. Dowski.
New paradigm for imaging systems.
Applied Optics, 41:6080-6092, 2002.

23

Sudhakar Prasad, Todd C. Torgersen, Victor P. Pauca, Robert J. Plemmons, and Joseph van der Gracht.
Engineering the pupil phase to improve image quality.
In Proc. SPIE, Visual Information Processing XII, volume 5108, pages 1-12, 2003.

24

M. R. Fetterman P. Potuluri and D. J. Brady.
High depth of field microscopic imaging using an interferometric camera.
Optics Express, Vol. 8, No. 11:624-630, 21 May 2001.

25

Sara C. Tucker, W. Thomas Cathey, and Edward R. Dowski, Jr.
Extended depth of field and aberration control for inexpensive digital microscope systems.
Optics Express, Vol. 4, No. 11:467-474, 24 May 1999.

26

Daniel L. Barton, Jeremy A. Walraven, Edward R. Dowski Jr., Rainer Danz, Andreas Faulstich, and Bernd Faltermeier.
Wavefront coded imaging systems for MEMS analysis.
Proc. of ISTFA, pages 295-303, 2002.

27

David J. Brady Daniel L. Marks, Ronald A. Stack.
Three-dimensional tomography using a cubic-phase plate extended depth-of-field system.
Opt. Letters No 4, 24:253-255, 1999.