MPI Message Size (PIC)

In Runko’s PIC module, the functions \(pack_all_particles()\) and \(pack_outgoing_particles()\) (defined in \(pic/particle.c++\)) are responsible for packaging particles leaving an old Tile to be passed to a new Tile via an MPI Message. If there are lots of particles leaving the tile, then the performance can take a hit from constant resizing of the arrays. In that case, you can try and increase the \(first_message_size\) in \(particle.h\). An optimal simulation would always send majority of the particles via the first message and whatever remains (fluctuations) via the extra message.

Note

Note: the MPI_ERR_TRUNCATE bug described here should be outdated. It is left here in case it resurfaces.

If too many particles are leaving a Tile, then this can cause an \(MPI_ERR_TRUNCATE\) (error code 15). This usually occurs if the simulation conditions are very extreme with high particle numbers, and thus exceed the maxiumum that Runko can handle (equal to \(first_message_size+extra_message_size\) in \(particle.h\), by default this is 4096+4096 particles). You can estimate the maximum number of particles leaving a cell with

\(#outgoing_particles = #number_of_cells_in_tile_border * cfl * #species * parrticles_per_cell * factor\),

where \(factor\) is a fudge factor which quantifies the maximum expected overdensity in your simulation. If your estimate for \(#outgoing_particles\) exceeds the default value of \(first_message_size\) in \(pic/particle.h\), then this may cause MPI errors. To fix this you must increase the value of \(extra_message_size\) in the same file in order to prevent any errors.