MATLAB vs Octave Benchmark

The benchmark is performed by a script that is a part of AORTA simulator. For the time-consumption measure, standard tic-toc combination is used. For the memory consumption, UNIX command is executed during the computation to obtain actual memory consumption.

The hardware used was Dell Latitude E5100 laptop with Intel Celeron 2.2GHz processor, 2Gb DDR RAM, and ATA-100 160Gb HDD. The test was run in a system's terminal (graphical X Window system was shut down) so the interference with other software was negligible.

Since the simulator uses Fast Fourier Transform (FFT) and matrix operations heavily, the benchmark consists of those operations. In order to gain maximum performance, all matrices were power of two: from 2x2 up to 8192x8192 .

Those numbers, of course, are not absolute and change from version to version. But the according to our results, the tendency remains the same: Octave tends to be faster on Fourier transform than MATLAB but consumes more memory.

Comparison of computation time

The basic matrix matrix operations, such as Gauss and point-wise multiplication as well as summation, the performance of MATLAB and Octave are approximately the same. The differences are in computational time of Fourier transform. As one can see from the Fig 1.1, Octave outperforms MATLAB on FFT transforms on any matrix sizes.

a) b) Figure 1.1: Median computation time versus matrix size for: a) FFT function and for b) FFTSHIFT function.

For example, on typical size of matrices in adaptive optics simulations (thousands by thousands of elements), Octave is almost 3 times faster than MATLAB with similar memory consumption. However for 8192x8192 matrix, the ``memory exhausted or requested size too large for range of Octave's index type'' message appeared and there is no datapoint for such a large matrix. It is also notable that on Windows and MATLAB v2009 an attempt to calculate huge matrices (more than 5000-6000 elements in each direction) results to the same ``out of memory'' error message.

Hence, except matrices that are bigger than 8192x8192, GNU/Octave is almost 3 times faster than MATLAB in FFT.

Comparison of memory consumption

Memory consumption for Octave is generally larger than for MATLAB as it is apparent from the Fig. 1.2. For small matrices (less than 2048x2048) it can be 2.5-3 times more memory, and for large matrices ( and more) that is typically 10-20%. Consequently, on the available hardware with 2Gb RAM the calculation of FFT of matrix 8192x8192 elements is not possible.

Figure 1.2: Median memory consumption versus matrix size for: a) FFT function and for b) FFTSHIFT function.

a) b)

Although Octave consumes considerably more memory than MATLAB while performing FFT and thus may run out of memory on huge matrices.