%===============================================================================
\section{{\petsc} example problems}\label{s:ex_petsc}
%===============================================================================

\subsection{A nonstiff example: idaHeat2D\_kry\_petsc}\label{ss:idaHeat2D_kry_petsc}

This example is the same as the one in \ref{ss:idaHeat2D_p}, except 
it uses {\petsc} vector instead of {\sundials} native parallel vector 
implementation. The output of the two examples is identical. In the following, 
we will describe only the implementation differences between the two. 

Before {\petsc} functions can be called, the library needs to be initialized. 
In this example we use initialization without arguments:
\begin{verbatim}
  PetscInitializeNoArguments(); 
\end{verbatim}
Alternatively, a call that takes {\petsc} command line arguments could be used.
At the end of the program, \id{PetscFinalize()} is called  to clean up any 
objects that {\petsc} may have created automatically. 
We use {\petsc} data management library (DM) to create 2D grid and set 
the partitioning. In our implementation we follow Example 15 from 
{\petsc} Time Stepping component (TS) documentation \cite{petsc-web-page}. 
We store a pointer to thus created {\petsc} distributed array object 
in user defined structure \id{data}.
\begin{verbatim}
  ierr = DMDACreate2d(comm, 
                      DM_BOUNDARY_NONE,  /* NONE, PERIODIC, GHOSTED */
                      DM_BOUNDARY_NONE,
                      DMDA_STENCIL_STAR, /* STAR, BOX */
                      MX,                
                      MY,                
                      NPEX,              
                      NPEY,              
                      1,                 /* degrees of freedom per node */
                      1,                 /* stencil width */
                      NULL,              
                      NULL,              
                      &(data->da));
\end{verbatim}
This call will create $M_X \times M_Y$ grid on MPI communicator \verb|comm|
with Dirichlet boundary conditions, using 5-point star stencil. Once 
the distributed array is created, we create {\petsc} vector by calling:
\begin{verbatim}
  ierr = DMCreateGlobalVector(data->da, &uvec);
\end{verbatim}
Template vector \id{uu} is created as a wrapper around {\petsc} vector 
\verb|uvec| using \verb|N_VMake_petsc| constructor. All other vectors are 
created by cloning the template to ensure the same partitioning and 2D data 
mapping is used everywhere. One should note that the template vector does 
not own the underlying {\petsc} vector, and it is user's responsibility to 
delete it after the template vector is destroyed.

To use {\petsc} vector wrapper in user supplied functions such as 
\verb|resHeat|, one needs first to extract {\petsc} vector with 
\verb|N_VGetVector_petsc|, and then use {\petsc} methods to access 
vector elements. Providing {\petsc} tutorial is beyond the scope of 
this document, and interested reader should consult \cite{petsc-user-ref}. 
Instead, we provide a brief description of functions used in this example.
\begin{itemize}
\item \ID{PetscFunctionBeginUser;}
  \par First executable line of user supplied {\petsc} function. It should 
  precede any other {\petsc} call in the user supplied function.
  
\item \ID{DMGetLocalVector(da,\&localU)}
  \par Allocates a local vector \id{localU} with space for ghost values, based 
  on partitioning in distributed array \id{da}. Vector \id{localU} is an object 
  equivalent to array \id{uext} in function \id{reslocal} in example
  in Section \ref{ss:idaHeat2D_kry_petsc}.
  
\item \ID{DMDAGetInfo(da,...,\&Mx, \&My,...)}
  \par Function to get information about data array \id{da}. In this example
  it is used only to get the grid size $M_X \times M_Y$.
  
\item \ID{DMGlobalToLocalBegin(da, U, INSERT\_VALUES, localU)}
  \par Moves data (including ghosts) from the global vector \id{U} to the 
  local vector \id{localU}. 
  
\item \ID{DMGlobalToLocalEnd(da, U, INSERT\_VALUES, localU)}
  \par Barrier for \id{DMGlobalToLocalBegin(...)}.
  
\item \ID{DMDAVecGetArray(da, F, \&f)}
  \par Gets a handle to data array \id{f} that shares data with vector \id{F} and
  is indexed using global dimensions from distributed array object \id{da}. This is 
  logically collective call.
  
\item \ID{DMDAVecGetArrayRead(da, U, \&u)}
  \par Gets a handle to data array \id{u} that shares data with vector \id{U} and
  is indexed using global dimensions from distributed array object \id{da}. 
  This is \emph{not} a collective call. Elements of the data array \id{u} are 
  accessed by indexing \id{u[i][j]}, where $i \in {0,\ldots,M_X}$ and 
  $j \in {0,\ldots,M_Y}$ are global mesh indices.
  
\item \ID{DMDAGetCorners(da, \&xs, \&ys, NULL, \&xm, \&ym, NULL)}
  \par Gets boundaries of grid defined in distributed array object \id{da}.
  Returns the global indices of the lower left corner ($x_s$, $y_s$), 
  and size of the local region $x_m \times y_m$, excluding ghost points. 
  
\item \ID{DMDAVecRestoreArray(da, F, \&f)}
  \par ``Restores'' array \id{f}. This function needs to be called after 
  reading/writing to \id{f} is done. Similar holds for functions 
  \id{DMDAVecRestoreArrayRead} and \id{DMRestoreLocalVector}.
  
\item \ID{PetscFunctionReturn(0)}
  \par This function should be used instead of \id{return} call in user
  supplied {\petsc} functions. It is used for error handling.
  
\end{itemize}

Using {\petsc} library when dealing with a structured grid problem like this 
allows one to use global indices when implementing the model and thus 
separate the model from the parallelization scheme. Also, note that {\petsc} 
functions used here replace private functions \id{rescomm}, \id{reslocal}, 
\id{BSend}, \id{BRecvPost}, \id{BRecvWait} and \id{InitUserData} from the 
\id{idaHeat2D\_kry\_p} example in Section \ref{ss:idaHeat2D_p}, and 
therefore simplify the implementation.


\paragraph{\bf Notes} 
           
\begin{itemize}
                                        
\item
  {\warn}At this point interfaces to {\petsc} solvers and preconditioners are 
  not available. They will be added in subsequent {\sundials} releases. 

\end{itemize}



%-------------------------------------------------------------------------------

