%===============================================================================
\section{Parallel example problems}\label{s:ex_parallel}
%===============================================================================

\subsection{A nonstiff example: cvAdvDiff\_non\_p}\label{ss:cvAdvDiff_p}

This problem begins with a simple diffusion-advection equation
for $u=u(t,x)$
\begin{equation}
\frac{\partial u}{\partial t}=\frac{\partial ^{2}u}{\partial x^{2}}
   + 0.5\frac{\partial u}{\partial x}  \label{PDE1}
\end{equation}
for $0 \leq t \leq 5, ~~ 0\leq x \leq 2$, and subject to homogeneous
Dirichlet boundary conditions and initial values given by
\begin{eqnarray}
u(t,0) &=& 0 ~,~~~~u(t,2) = 0 ~, \label{BCIC1} \\
u(0,x) &=& x(2-x)e^{2x} ~. \nonumber
\end{eqnarray}
A system of \id{MX} ODEs is obtained by discretizing the $x$-axis with \id{MX}+2
grid points and replacing the first and second order spatial derivatives
with their central difference approximations. Since the value of $u$ is
constant at the two endpoints, the semi-discrete equations for those points
can be eliminated.  With $u_{i}$ as the approximation to $u(t,x_{i})$,
$x_{i} = i(\Delta x)$, and $\Delta x = 2/($\id{MX}$+1)$, the resulting system of
ODEs, $\dot{u} = f(t,u)$, can now be written:
\begin{equation}
\dot{u}_i=\frac{u_{i+1}-2u_{i}+u_{i-1}}{(\Delta x)^{2}}
  + 0.5 \frac{u_{i+1}-u_{i-1}}{2(\Delta x)} ~. \label{cvAdvDiff_pode}
\end{equation}
This equation holds for $i=1,2,\ldots ,$ \id{MX}, with the understanding
that $u_{0} = u_{\id{MX}+1}=0.$

In the parallel processing environment, we may think of the several
processors as being laid out on a straight line with each processor to
compute its contiguous subset of the solution vector.  Consequently
the computation of the right hand side of Eq. (\ref{cvAdvDiff_pode}) requires
that each interior processor must pass the first component of its block of
the solution vector to its left-hand neighbor, acquire the last component of
that neighbor's block, pass the last component of its block of the solution
vector to its right-hand neighbor, and acquire the first component of that
neighbor's block. If the processor is the first ($0$th) or last processor,
then communication to the left or right (respectively) is not required.

This problem uses the Adams (non-stiff) integration formula and fixed-point
nonlinear solver.  It is unrealistically simple, but serves to illustrate
use of the parallel version of CVODE.

The \id{cvAdvDiff\_non\_p.c} file begins with \id{\#include} declarations for
various required header files, including lines for
\id{nvector\_parallel} to access the parallel \id{N\_Vector} type and related
macros, and for \id{mpi.h} to access MPI types and constants. Following
that are definitions of problem constants and a data block for communication
with the \id{f} routine.  That block includes the number of PEs, the index
of the local PE, and the MPI communicator.

The \id{main} program begins with MPI calls to initialize MPI and to set
multi-processor environment parameters \id{npes} (number of PEs) and
\id{my\_pe} (local PE index).  The local vector length is set according
to \id{npes} and the problem size \id{NEQ} (which may or may not be
multiple of \id{npes}).  The value \id{my\_base} is the base value for
computing global indices (from 1 to \id{NEQ}) for the local vectors.
The solution vector \id{u} is created with a call to \id{N\_VNew\_Parallel}
and loaded with a call to \id{SetIC}.  The calls to \id{CVodeCreate},
\id{CVodeInit}, and \id{CVodeSStolerances} specify a {\cvode} solution with
the nonstiff method and scalar tolerances.  The call to \id{CVodeSetUserdata}
insures that the pointer \id{data} is passed to the \id{f} routine whenever it
is called.
A heading is printed (if on processor 0).  In a loop over \id{tout} values,
\id{CVode} is called, and the return value checked for errors.  The
max-norm of the solution and the total number of time steps so far
are printed at each output point.  Finally, some statistical counters are
printed, memory is freed, and MPI is finalized.

The \id{SetIC} routine uses the last two arguments passed to it to compute
the set of global indices (\id{my\_base}+1 to \id{my\_base+my\_length})
corresponding to the local part of the solution vector \id{u}, and then to
load the corresponding initial values.  The \id{PrintFinalStats} routine
uses \id{CVodeGet***} calls to get various counters, and then prints these.
The counters are: \id{nst} (number of steps), \id{nfe} (number of
\id{f} evaluations), \id{nni} (number of nonlinear iterations),
\id{netf} (number of error test failures), and \id{ncfn} (number of
nonlinear convergence failures).  This routine is suitable for general use
with {\cvode} applications to nonstiff problems.

The \id{f} function is an implementation of Eq. (\ref{cvAdvDiff_pode}), but preceded
by communication operations appropriate for the parallel setting.
It copies the local vector \id{u} into a larger array \id{z}, shifted by 1
to allow for the storage of immediate neighbor components.  The first and last
components of \id{u} are sent to neighboring processors with \id{MPI\_Send} calls,
and the immediate neighbor solution values are received from the neighbor
processors with \id{MPI\_Recv} calls, except that zero is loaded into \id{z[0]}
or \id{z[my\_length+1]} instead if at the actual boundary.  Then the central
difference expressions are easily formed from the \id{z} array, and loaded into
the data array of the \id{udot} vector.

The \id{cvAdvDiff\_non\_p.c} file includes a routine \id{check\_flag} that checks the
return values from calls in \id{main}.  This routine was written to be used
by any parallel {\sundials} application.

The output below is for \id{cvAdvDiff\_non\_p} with \id{MX} = 10 and four processors.
Varying the number of processors will alter the output, only because
of roundoff-level differences in various vector operations.  The fairly
high value of \id{ncfn} indicates that this problem is on the borderline
of being stiff.

%%
\includeOutput{cvAdvDiff\_non\_p}{../../examples/cvode/parallel/cvAdvDiff_non_p.out}
%%

%-------------------------------------------------------------------------------

\subsection{A user preconditioner example: cvDiurnal\_kry\_p}\label{ss:cvDiurnal_p}

As an example of using {\cvode} with the Krylov linear solver
{\sunlinsolspgmr}, {\cvls} linear solver interface, and
the parallel MPI {\nvecp} module, we describe a test problem based on
the system PDEs given above for the \id{cvDiurnal\_kry} example.
As before, we discretize the PDE system with central differencing, to
obtain an ODE system $\dot{u} = f(t,u)$ representing (\ref{cvDiurnalpde}).
But in this case, the discrete solution vector is distributed over
many processors.  Specifically, we may think of the processors as
being laid out in a rectangle, and each processor being assigned a
subgrid of size \id{MXSUB}$\times$\id{MYSUB} of the $x-y$ grid. If
there are \id{NPEX} processors in the $x$ direction and \id{NPEY}
processors in the $y$ direction, then the overall grid size is
\id{MX}$\times$\id{MY} with \id{MX}$=$\id{NPEX}$\times$\id{MXSUB} and
\id{MY}$=$\id{NPEY}$\times$\id{MYSUB}, and the size of the ODE system is
$2\cdot$\id{MX}$\cdot$\id{MY}.

To compute $f$ in this setting, the processors pass and receive
information as follows.  The solution components for the bottom row of
grid points in the current processor are passed to the processor below
it and the solution for the top row of grid points is received from
the processor below the current processor. The solution for the top
row of grid points for the current processor is sent to the processor
above the current processor, while the solution for the bottom row of
grid points is received from that processor by the current
processor. Similarly the solution for the first column of grid points
is sent from the current processor to the processor to its left and
the last column of grid points is received from that processor by the
current processor. The communication for the solution at the right
edge of the processor is similar. If this is the last processor in a
particular direction, then message passing and receiving are bypassed
for that direction.

This code is intended to provide a more realistic example than that in
\id{cvAdvDiff\_non\_p}, and to provide a template for a stiff ODE system
arising from a PDE system. The solution method is BDF with Newton
iteration and {\spgmr}. The left preconditioner is the block-diagonal
part of the Newton matrix, with $2 \times 2$ blocks, and the
corresponding diagonal blocks of the Jacobian are saved each time the
preconditioner is generated, for re-use later under certain conditions.

The organization of the \id{cvDiurnal\_kry\_p} program deserves some comments. The
right-hand side routine \id{f} calls two other routines: \id{ucomm}, which
carries out inter-processor communication; and \id{fcalc}, which operates on
local data only and contains the actual calculation of $f(t,u)$. The
\id{ucomm} function in turn calls three routines which do, respectively,
non-blocking receive operations, blocking send operations, and
receive-waiting. All three use MPI, and transmit data from the local \id{u}
vector into a local working array \id{uext}, an extended copy of \id{u}.
The \id{fcalc} function copies \id{u} into \id{uext}, so that the
calculation of $f(t,u)$ can be done conveniently by operations on
\id{uext} only.  Most other features of \id{cvDiurnal\_kry\_p.c} are the same as
in \id{cvDiurnal\_kry.c}, except for extra logic involved with distributed
vectors.

The following is a sample output from \id{cvDiurnal\_kry\_p}, for four processors
(in a $2 \times 2$ array) with a $5 \times 5$ subgrid on each.
The output will vary slightly if the number of processors is changed.

%%
\includeOutput{cvDiurnal\_kry\_p}{../../examples/cvode/parallel/cvDiurnal_kry_p.out}
%%

%-------------------------------------------------------------------------------

\subsection{A CVBBDPRE preconditioner example: cvDiurnal\_kry\_bbd\_p}\label{ss:cvDiurnal_bbd_p}

In this example, \id{cvDiurnal\_kry\_bbd\_p}, we solve the same
problem as in \id{cvDiurnal\_kry\_p}
above, but instead of supplying the preconditioner, we use the {\cvbbdpre} module,
which generates and uses a band-block-diagonal preconditioner.  The
half-bandwidths of the Jacobian block on each processor are both equal to
$2\cdot$\id{MXSUB}, and that is the value supplied as \id{mudq} and \id{mldq}
in the call to \id{CVBBDPrecInit}.  But in order to reduce storage and computation
costs for preconditioning, we supply the values \id{mukeep} = \id{mlkeep} = 2
(= \id{NVARS}) as the half-bandwidths of the retained band matrix blocks.
This means that the Jacobian elements are computed with a difference quotient
scheme using the true bandwidth of the block, but only a narrow band matrix
(bandwidth 5) is kept as the preconditioner.

As in \id{cvDiurnal\_kry\_p.c}, the \id{f} routine in \id{cvDiurnal\_kry\_bbd\_p.c} simply calls a
communication routine, \id{fucomm}, and then a strictly computational routine,
\id{flocal}.  However, the call to \id{CVBBDPrecInit} specifies the pair of
routines to be called as \id{ucomm} and \id{flocal}, where \id{ucomm} is NULL.
This is because each call by the solver to \id{ucomm} is
preceded by a call to \id{f} with the same \id{(t,u)} arguments, and therefore the
communication needed for \id{flocal} in the solver's calls to it have already been
done.

In \id{cvDiurnal\_kry\_bbd\_p.c}, the problem is solved twice --- first with preconditioning
on the left, and then on the right.  Thus prior to the second solution, calls
are made to reset the initial values (\id{SetInitialProfiles}), the main solver
memory (\id{CVodeReInit}), the {\cvbbdpre} memory (\id{CVBBDPrecReInit}),
as well as the preconditioner type (\id{SUNLinSol\_SPGMRSetPrecType}).

Sample output from \id{cvDiurnal\_kry\_bbd\_p} follows, again using $5 \times 5$ subgrids
on a $2 \times 2$ processor grid.  The performance of the preconditioner,
as measured by the number of Krylov iterations per Newton iteration,
\id{nli/nni}, is very close to that of \id{cvDiurnal\_kry\_p} when preconditioning is on
the left, but slightly poorer when it is on the right.

%%
\includeOutput{cvDiurnal\_kry\_bbd\_p}{../../examples/cvode/parallel/cvDiurnal_kry_bbd_p.out}
%%

%-------------------------------------------------------------------------------
