<?xml version="1.0"?>
<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook V4.1.2//EN"
  "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">

<!-- $Id$ -->

<refentry id="gfpcopy.1">

<refentryinfo><date>27 Aug 2015</date></refentryinfo>

<refmeta>
<refentrytitle>gfpcopy</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo>Gfarm</refmiscinfo>
</refmeta>

<refnamediv id="name">
<refname>gfpcopy</refname>
<refpurpose>copy Gfarm files in parallel</refpurpose>
</refnamediv>

<refsynopsisdiv id="synopsis">
<cmdsynopsis sepchar=" ">
  <command moreinfo="none">gfpcopy</command>
  <arg choice="opt" rep="norepeat">-nqvdpPU</arg>
  <arg choice="opt" rep="norepeat">-X <replaceable>regexp</replaceable></arg>
  <arg choice="opt" rep="norepeat">-S <replaceable>source-domainname</replaceable></arg>
  <arg choice="opt" rep="norepeat">-h <replaceable>source-hostfile</replaceable></arg>
  <arg choice="opt" rep="norepeat">-D <replaceable>destination-domainname</replaceable></arg>
  <arg choice="opt" rep="norepeat">-H <replaceable>destination-hostfile</replaceable></arg>
  <arg choice="opt" rep="norepeat">-j <replaceable>num-of-processes</replaceable></arg>
  <arg choice="opt" rep="norepeat">-J <replaceable>num-of-processes</replaceable></arg>
  <arg choice="opt" rep="norepeat">-M <replaceable>limit-byte</replaceable></arg>
  <arg choice="opt" rep="norepeat">-z <replaceable>minimum-byte</replaceable></arg>
  <arg choice="opt" rep="norepeat">-Z <replaceable>maximum-byte</replaceable></arg>
  <arg choice="opt" rep="norepeat">-w <replaceable>way-of-scheduling</replaceable></arg>
  <arg choice="opt" rep="norepeat">-W <replaceable>kilobytes-for-threshold</replaceable></arg>
  <arg choice="opt" rep="norepeat">-s <replaceable>KB/s-to-simulate</replaceable></arg>
  <arg choice="opt" rep="norepeat">-F <replaceable>num-for-readahead</replaceable></arg>
  <arg choice="opt" rep="norepeat">-b <replaceable>bufsize</replaceable></arg>
  <arg choice="opt" rep="norepeat">-f</arg>
  <arg choice="opt" rep="norepeat">-e</arg>
  <arg choice="opt" rep="norepeat">-k</arg>
  <arg choice="plain" rep="norepeat"><replaceable>source-path</replaceable></arg>
  <arg choice="plain" rep="norepeat"><replaceable>destination-path</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>

<!-- body begins here -->

<refsect1 id="description"><title>DESCRIPTION</title>

<para>
<command moreinfo="none">gfpcopy</command> copies files in parallel.
</para>
<para>
When the <parameter moreinfo="none">source-path</parameter> is
a directory, files under the directory will be copied recursively.
</para>
<para>
When the <parameter moreinfo="none">destination-path</parameter> does
not exist, the directory is created.
When the <parameter moreinfo="none">destination-path</parameter>
exists, a directory of the same name as
the <parameter moreinfo="none">source-path</parameter> is created
under the <parameter moreinfo="none">destination-path</parameter>.
</para>
<para>
A set of source/destination hosts can be specified by a domain name
and/or a hostlist file.  When both a domain name and a hostlist file
are specified, a set of hosts is determined by both conditions.  When a
set of source hosts is specified, only files stored on the source
hosts are copied.
</para>
<para>
<command moreinfo="none">gfpcopy</command> also retrieves the directory
entries in parallel.
</para>
</refsect1>

<refsect1 id="source-path"><title>SOURCE PATH</title>
<para>
<parameter moreinfo="none">source-path</parameter> must be one of the following formats.
Files on HPSS cannot be copied.
</para>

<variablelist>

<varlistentry>
<term><parameter moreinfo="none">path-name</parameter></term>
<listitem>
<para>
is a relative path or an absolute path of a local file system.  When the path
is a mount point on gfarm2fs, files are copied without passing through
the gfarm2fs.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><parameter moreinfo="none">gfarm:...</parameter></term>
<listitem>
<para>
is a Gfarm URL.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><parameter moreinfo="none">file:...</parameter></term>
<listitem>
<para>
is an URL of a local file system.
</para>
</listitem>
</varlistentry>

</variablelist>
</refsect1>

<refsect1 id="source-path"><title>DESTINATION PATH</title>
<para>
<parameter moreinfo="none">destination-path</parameter>
must be one of the following formats.
</para>

<variablelist>

<varlistentry>
<term><parameter moreinfo="none">path-name</parameter></term>
<listitem>
<para>
is a relative path or an absolute path of a local file system.  When the path
is a mount point on gfarm2fs, files are copied without passing through
the gfarm2fs.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><parameter moreinfo="none">gfarm:...</parameter></term>
<listitem>
<para>
is a Gfarm URL of a directory.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><parameter moreinfo="none">file:...</parameter></term>
<listitem>
<para>
is an URL of a directory on a local file system.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><parameter moreinfo="none">hpss:...</parameter></term>
<listitem>
<para>
is an URL of a directory on HPSS.
If the same directory as
the <parameter moreinfo="none">source-path</parameter> exists under
this directory, the <parameter moreinfo="none">source-path</parameter>
cannot be copied.
The differential copy is not supported for HPSS.
Relative path such as "hpss:". and "hpss:dir" can be specified.
</para>
</listitem>
</varlistentry>

</variablelist>
</refsect1>

<refsect1 id="options"><title>GFPCOPY OPTIONS</title>
<para>
These are options only for <command moreinfo="none">gfpcopy</command>.
</para>
<variablelist>

<varlistentry>
<term><option>-b</option> <parameter moreinfo="none">bufsize</parameter></term>
<listitem>
<para>
Specifies the buffer size in bytes to copy.
</para>
<para>
The default value is 64 KiB (64 * 1024).
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-f</option></term>
<listitem>
<para>
With the -f option, existing files will be overwritten when
the size is different or the modification time (mtime) is different from the
source file.
</para>
<para>
Without the -f option, existing files will be overwritten when
they are older than the corresponding source files in the
modification time.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-e</option></term>
<listitem>
<para>
Skips existing files in order to execute gfpcopy
simultaneously.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><option>-k</option></term>
<listitem>
<para>
Does not copy symbolic links.
</para>
</listitem>
</varlistentry>

</variablelist>
</refsect1>

<refsect1 id="common-options"><title>COMMON OPTIONS</title>
<para>
There are several common options
with <command moreinfo="none">gfprep</command>.  (see
man <command moreinfo="none">gfprep</command>)
</para>
</refsect1>

<refsect1 id="examples"><title>EXAMPLES</title>
<para>
To copy files under the directory recursively.
</para>
<literallayout format="linespecific" class="normal">
$ gfpcopy gfarm:///dir file:///tmp/dir
</literallayout>
<literallayout format="linespecific" class="normal">
$ gfpcopy file:///tmp/dir gfarm:///dir
</literallayout>
<literallayout format="linespecific" class="normal">
$ gfpcopy gfarm:///dir1 gfarm:///dir2
</literallayout>
<literallayout format="linespecific" class="normal">
$ gfpcopy gfarm:///dir hpss:///tmp/dir
</literallayout>
<literallayout format="linespecific" class="normal">
$ cd /mnt/gfarm2fs
$ gfpcopy dir /tmp/dir
</literallayout>

 <para>
To copy a file.
</para>
<literallayout format="linespecific" class="normal">
$ gfpcopy gfarm:///dir/file file:///dir
</literallayout>
<literallayout format="linespecific" class="normal">
$ cd /mnt/gfarm2fs
$ gfpcopy file /tmp/dir
</literallayout>
</refsect1>

<refsect1 id="notes"><title>NOTES</title>
<para>
To retrieve the directory entries efficiently, it is better to execute
<command moreinfo="none">gfpcopy</command> command near the metadata
server.  When you need to
execute <command moreinfo="none">gfpcopy</command> command far from
the metadata server, increase the parallelism by the -j and -J
options.
</para>
</refsect1>

<refsect1 id="see-also"><title>SEE ALSO</title>
<para>
  <citerefentry>
  <refentrytitle>gfprep</refentrytitle><manvolnum>1</manvolnum>
  </citerefentry>,
  <citerefentry>
  <refentrytitle>gfreg</refentrytitle><manvolnum>1</manvolnum>
  </citerefentry>,
  <citerefentry>
  <refentrytitle>gfexport</refentrytitle><manvolnum>1</manvolnum>
  </citerefentry>,
  <citerefentry>
  <refentrytitle>gfarm2.conf</refentrytitle><manvolnum>5</manvolnum>
  </citerefentry>
</para>
</refsect1>

</refentry>
