MPI Halo test fails under intel MPI during clean up phase

[TomMelt]

In #1098 , I fix CMake so that the MPI tests actually run.

They all technically pass but testHaloExchangeCB_MPI3 will show failed status e.g.,

Test output

$ ctest
Test project /home/tdm39/nextsimdg/build-mpi
    Start 1: testHaloExchangeCB_MPI3
1/6 Test #1: testHaloExchangeCB_MPI3 ..........***Failed    4.10 sec
    Start 2: testParaGrid_MPI2
2/6 Test #2: testParaGrid_MPI2 ................   Passed    2.94 sec
    Start 3: testRectGrid_MPI3
3/6 Test #3: testRectGrid_MPI3 ................   Passed    1.97 sec
    Start 4: testModelMetadataCB_MPI3
4/6 Test #4: testModelMetadataCB_MPI3 .........   Passed    1.66 sec
    Start 5: testModelMetadataPB_MPI3
5/6 Test #5: testModelMetadataPB_MPI3 .........   Passed    2.68 sec
    Start 6: testConfigOutput_MPI2
6/6 Test #6: testConfigOutput_MPI2 ............   Passed   14.18 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) =  27.58 sec

The following tests FAILED:
          1 - testHaloExchangeCB_MPI3 (Failed)
Errors while running CTest
Output from these tests are in: /home/tdm39/nextsimdg/build-mpi/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

Interestingly, the testHaloExchangeCB_MPI3 "fails". The test actually runs, but the MPI clean up fails:

testHaloExchangeCB_MPI3 output

$ ctest --rerun-failed --output-on-failure
Test project /home/tdm39/nextsimdg/build-mpi
    Start 1: testHaloExchangeCB_MPI3
1/1 Test #1: testHaloExchangeCB_MPI3 ..........***Failed    0.99 sec
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
[doctest] doctest version is "2.4.11"
[doctest] run with "--help" for options
===============================================================================
[doctest] test cases:   5 |   5 passed | 0 failed | 0 skipped
[doctest] assertions: 927 | 927 passed | 0 failed |
[doctest] Status: SUCCESS!
===============================================================================
[doctest] assertions on all processes:   2949 |   2949 passed |      0 failed |
===============================================================================
[doctest] Status: SUCCESS!
malloc_consolidate(): invalid chunk size
malloc_consolidate(): unaligned fastbin chunk detected

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 2924112 RUNNING AT cpu-q-565
=   KILLED BY SIGNAL: 6 (Aborted)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 2 PID 2924113 RUNNING AT cpu-q-565
=   KILLED BY SIGNAL: 6 (Aborted)
===================================================================================


0% tests passed, 1 tests failed out of 1

Total Test time (real) =   1.02 sec

The following tests FAILED:
          1 - testHaloExchangeCB_MPI3 (Failed)
Errors while running CTest

I haven't look into, but perhaps there is an issue with the 1-side RMA MPI calls. This is the main different between this test and the other MPI tests.

Unfortunately it will be hard to debug because Intel MPI has a bug that breaks mdb. I would recommend trying to use Intels gdb which has some MPI support.