=pod

=head1 NAME

README for dta-tokwrap - programs, scripts, and perl modules for DTA XML corpus tokenization

=cut

##======================================================================
=pod

=head1 DESCRIPTION

This package contains various utilities for
tokenization of DTA "base-format" XML documents.
see L</INSTALLATION> for requirements and installation instructions,
see L</USAGE> for a brief introduction to the high-level command-line interface,
and
see L</TOOLS> for an overview of the individual tools included in this distribution.

=cut


##======================================================================
=pod

=head1 INSTALLATION

=cut

##--------------------------------------------------------------
=pod

=head2 Requirements

=cut

##~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=pod

=head3 C Libraries

=over 4

=item expat

tested version(s): 1.95.8, 2.0.1

=item libxml2

tested version(s): 2.7.3, 2.7.8

=item libxslt

tested version(s): 1.1.24, 1.1.26

=back

=cut

##~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=pod

=head3 Perl Modules

See F<DTA-TokWrap/README.txt> for a full list of required
perl modules.

=cut

##~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=pod

=head3 Development Tools

=over 4

=item C compiler

tested version(s): gcc / linux: v4.3.3, 4.4.6

=item GNU flex (development only)

tested version(s): 2.5.33, 2.5.35

Only needed if you plan on making changes to the lexer sources.

=item GNU autoconf (SVN only)

tested version(s): 2.61, 2.67

Required for building from SVN sources.

=item GNU automake (SVN only)

tested version(s): 1.9.6, 1.11.1

Required for building from SVN sources.

=back

=cut

##--------------------------------------------------------------
=pod

=head2 Building from SVN

To build this package from SVN sources, you must first run the shell command:

 bash$ sh ./autoreconf.sh

from the distribution root directory B<BEFORE> running F<./configure>.
Building from SVN sources requires additional development tools to present
on the build system.  Then, follow the instructions in L</"Building from Source">.

=cut

##--------------------------------------------------------------
=pod

=head2 Building from Source

To build and install the entire package, issue the following commands to the shell:

 bash$ cd dta-tokwrap-0.01   # (or wherever you unpacked this distribution)
 bash$ sh ./configure        # configure the package
 bash$ make                  # build the package
 bash$ make install          # install the package on your system

More details on the top-level installation process can be found in
the file F<INSTALL> in the distribution root directory.

More details on building and installing the DTA::TokWrap perl module included in this distribution
can be found in the F<perlmodinstall(1)> manpage.

=cut

##======================================================================
=pod

=head1 USAGE

The perl program L<dta-tokwrap.perl|/dta-tokwrap.perl> installed from the F<DTA-TokWrap/>
distribution subdirectory provides a flexible high-level command-line interface
to the tokenization of DTA XML documents.

=cut

##--------------------------------------------------------------
=pod

=head2 Input Format

The L<dta-tokwrap.perl|dta-tokwrap.perl> script takes as its input DTA "base-format" XML files,
which are simply (TEI-conformant) UTF-8 encoded XML files with one C<E<lt>cE<gt>>
element per character:

=over 4

=item *

the document B<MUST> be encoded in UTF-8,

=item *

all text nodes to be tokenized should be descendants of a C<E<lt>textE<gt>> element,
and may optionally be immediate daughters of a C<E<lt>cE<gt>> element
(XPath C<//text//text()|//text//c/text()>). C<E<lt>cE<gt>> elements may not be nested.

Prior to dta-tokwrap v0.38, C<E<lt>cE<gt>> elements were required.

=back

=cut

##--------------------------------------------------------------
=pod

=head2 Example: Tokenizing a single XML file

Assume we wish to tokenize a single DTA "base-format" XML file F<doc1.xml>.
Issue the following command to the shell:

 bash$ dta-tokwrap.perl doc1.xml

... This will create the following output files:

=over 4

=item F<doc1.t.xml>

"Master" tokenizer output file encoding sentence boundaries, token boundaries,
and tokenizer-provided token analyses.  Source for various stand-off annotation formats.
This format can also be passed directly to and from the L<DTA::CAB(3pm)|DTA::CAB>
analysis suite using the L<DTA::CAB::Format::XmlNative(3pm)|DTA::CAB::Format::XmlNative>
formatter class.

=back

=cut

##--------------------------------------------------------------
=pod

=head2 Example: Tokenizing multiple XML files

Assume we wish to tokenize a corpus of three DTA "base-format" XML files
F<doc1.xml>, F<doc2.xml>, and F<doc3.xml>.
This is as easy as:

 bash$ dta-tokwrap.perl doc1.xml doc2.xml doc3.xml

For each input document specified on the command line,
master output files and stand-off annotation files will be created.

See L<"the dta-tokwrap.perl manpage"|dta-tokwrap.perl> for more details.

=head2 Example: Tracing execution progess

Assume we wish to tokenize a large corpus of XML input files F<doc*.xml>,
and would like to have some feedback on the progress of the
tokenization process.
Try:

 bash$ dta-tokwrap.perl -verbose=1 doc*.xml

or:

 bash$ dta-tokwrap.perl -verbose=2 doc*.xml

or even:

 bash$ dta-tokwrap.perl -traceAll doc*.xml

=cut

##--------------------------------------------------------------
=pod

=head2 Example: From TEI to TCF and Back

Assume we have a TEI-like document F<doc.tei.xml> which we want
to encode as TCF to the file F<doc.tei.tcf>, using only whitespace tokenizer "hints", but
not actually tokenizing the document yet. This can be accomplished by:

 $ dta-tokwrap.perl -t=tei2tcf -weak-hints doc1.tei.xml

If the output should instead be written to STDOUT, just call:

 $ dta-tokwrap.perl -t=tei2tcf -weak-hints -dO=tcffile=- doc1.tei.xml

Assume that the resulting TCF document has undergone further processing
(e.g. via L<WebLicht|http://weblicht.sfs.uni-tuebingen.de/weblichtwiki/index.php/Main_Page>)
to produce an annotated TCF document C<doc.out.tcf>.

selected TCF layers (in particular the C<tokens> and C<sentences> layers) can be spliced back into the TEI document as
F<doc.out.xml> by calling:

 $ dta-tokwrap.perl -t=tcf2tei doc.out.tcf -dO=tcffile=doc.out.tcf -dO=tcfcwsfile=doc.out.xml

=cut

##======================================================================
=pod

=head1 TOOLS

This section provides a brief overview of the individual tools included
in the dta-tokwrap distribution.

=cut

##--------------------------------------------------------------
=pod

=head2 Perl Scripts & Programs

The perl scripts and programs included with this distribution are installed
by default in F</usr/local/bin> and/or wherever your perl installs
scripts by default (e.g. in C<`perl -MConfig -e 'print $Config{installsitescript}'`>).

=over 4

=item dta-tokwrap.perl

Top-level wrapper script for document tokenization
using the L<DTA::TokWrap|DTA::TokWrap> perl API.

=item dtatw-add-c.perl

Script to insert C<E<lt>cE<gt>> elements and/or
C<xml:id> attributes for such elements into an
XML document which does not yet contain them.
Guaranteed not to clobber any existing //c IDs.

//c/@xml:id attributes are generated by a simple document-global counter
("c1", "c2", ..., "c65536").

See L<"the dtatw-add-c.perl manpage"|dtatw-add-c.perl> for more details.

=item dtatw-cids2local.perl

Script to convert C<//c/@xml:id> attributes to page-local encoding.
Never really used.

See L<"the dtatw-cids2local.perl manpage"|dtatw-cids2local.perl> for more details.

=item dtatw-add-ws.perl

Script to splice C<E<lt>sE<gt>> and C<E<lt>wE<gt>> elements encoded from a standoff (.t.xml or .u.xml) XML file
into the I<original> "base-format" (.chr.xml) file, producing a .cws.xml file.
A tad too generous with partial word segments, due to strict adjacency and boundary criteria.

In earlier versions of dta-tokwrap, this functionality was split between the scripts
C<dtatw-add-w.perl> and C<dtatw-add-s.perl>, which required only an I<id-compatible>
base-format (.chr.xml) file as the splice target.  As of dta-tokwrap v0.35, the splice target
base-format file must be I<original> source file itself, since the current implementation
uses byte offsets to perform the splice.

See L<"the dtatw-add-ws.perl manpage"|dtatw-add-ws.perl> for more details.

=item dtatw-splice.perl

Script to splice generic standoff attributes and/or content into a base file;
useful e.g. for merging flat DTA::CAB standoff analyses into TEI-structured
*.cws.xml files.

See L<"the dtatw-splice.perl manpage"|dtatw-splice.perl> for more details.

=item dtatw-get-ddc-attrs.perl

Script to insert DDC-relevant attributes extracted from a base file into a *.t.xml file,
producing a pre-DDC XML format file (by convention *.ddc.t.xml, a subset of the *.t.xml
format).

See L<"the dtatw-get-ddc-attrs.perl manpage"|dtatw-get-ddc-attrs.perl> for more details.

=item dtatw-get-header.perl

Simple script to extract a single header element from an XML file (e.g. for later
inclusion in a DDC XML format file).

See L<"the dtatw-get-header.perl manpage"|dtatw-get-header.perl> for more details.

See L<"the dtatw-get-header.perl manpage"|dtatw-get-header.perl> for more details.

=item dtatw-pn2p.perl

Script to conver insert E<lt>pE<gt>...E<lt>/pE<gt> wrappers for C<//s/@pn> key attributes
in "flat" *.t.xml files.

=item dtatw-xml2ddc.perl

Script to convert *.ddc.t.xml files and optional headers to DDC-XML format.

See L<"the dtatw-xml2ddc.perl manpage"|dtatw-xml2ddc.perl> for more details.

=item dtatw-t-check.perl

Simple script to check consistency of tokenizer output (*.t) offset + length
fields with input (*.txt) file.

=item dtatw-add-c.perl

Script to add C<E<lt>cE<gt>> elements to an XML document which does
not already contain them.
Not really useful as of dta-tokwrap v0.38.

=item dtatw-rm-c.perl

Script to remove C<E<lt>cE<gt>> elements from an XML document.
Regex hack, fast but not exceedingly robust, use with caution.
See also L</"dtatw-rm-c.xsl">

=item dtatw-rm-w.perl

Fast regex hack to remove C<E<lt>wE<gt>> elements from an XML document

=item dtatw-rm-s.perl

Fast regex hack to remove C<E<lt>sE<gt>> elements from an XML document.

=item dtatw-rm-lb.perl

Script to remove C<E<lt>lbE<gt>> (line-break) elements from an XML document,
replacing them with newlines.
Regex hack, fast but not robust, use with caution.
See also L</"dtatw-rm-lb.xsl">

=item dtatw-lb-encode.perl

Encodes newlines under //text//text() in an XML document as C<E<lt>lbE<gt>> (line-break) elements
using high-level file heuristics only.
Regex hack, fast but not robust, use with caution.
See also L</"dtatw-ensure-lb.perl">, L</"dtatw-add-lb.xsl">, L</"dtatw-rm-lb.perl">.

=item dtatw-ensure-lb.perl

Script to ensure that all //text//text() newlines in an XML document are explicitly encoded
with C<E<lt>lbE<gt>> (line-break) elements, using optional file-, element-,
and line-level heuristics.
Robust but slow, since it actually parses XML input documents.
See also L</"dtatw-lb-encode.perl">, L</"dtatw-add-lb.xsl">, L</"dtatw-rm-lb.perl">.

=item dtatw-tt-dictapply.perl

Script to apply a type-"dictionary" in one-word-per-line (.tt) format to a
token corpus in one-word-per-line (.tt) format.  Especially useful together with
standard UNIX utilities such as cut, grep, sort, and uniq.

=item dtatw-cabtt2xml.perl

Script to convert DTA::CAB::Format::TT (one-word-per-line with variable analysis
fields identified by conventional prefixes) files to expanded .t.xml format used
by dta-tokwrap.  The expanded format should be identical to that used by the
DTA::CAB::Format::Xml class.  See also L<dtatw-txml2tt.xsl>.

=item file-substr.perl

Script to extract a portion of a file,
specified by byte offset and length.
Useful for debugging index files created by other tools.

=back

=cut

##--------------------------------------------------------------
=pod

=head2 GNU make build system template

The distribution directory F<make/> contains a "template"
for using GNU F<make> to
organizing the conversion of large corpora with
the dta-tokwrap utilities.  This is useful because:

=over 4

=item *

F<make>'s intuitive, easy-to-read syntax provides a
wonderful vehicle for user-defined configuration files,
obviating the need to remember the names of all 64
(at last count)
C<dta-tokwrap.perl|/dta-tokwrap.perl> options,

=item *

F<make>
is very good at tracking complex dependencies of the sort
that exist between the various temporary files generated
by the dta-tokwrap utilities,

=item *

F<make>
jobs can be made "robust" simply by adding a C<-k>
(C<--keep-going>) to the command-line,
and

=item *

last but certainly not least,
F<make>
has built-in support for parallelization of complex
tasks by means of the C<-j N> (C<--jobs=N>) option,
allowing us to take advantage of multiprocessor systems.

=back

By default, the contents of the distribution F<make/>
subdirectory are installed to F</usr/local/share/dta-tokwrap/make/>.
See the comments at the top of F<make/User.mak> for instructions.

=cut


##--------------------------------------------------------------
=pod

=head2 Perl Modules

=over 4

=item L<DTA::TokWrap|DTA::TokWrap>

Top-level tokenization-wrapper module, used by L<dta-tokwrap.perl|dta-tokwrap.perl>.

=item L<DTA::TokWrap::Document|DTA::TokWrap::Document>

Object-oriented wrapper for documents to be processed.

=item L<DTA::TokWrap::Processor|DTA::TokWrap::Processor>

Abstract base class for elementary document-processing operations.

=back

See the L<DTA::TokWrap::Intro(3pm)|DTA::TokWrap::Intro> manpage for more details
on included modules, APIs, calling conventions, etc.

=cut


##--------------------------------------------------------------
=pod

=head2 XSL stylesheets

The XSL stylesheets included with this distribution are installed
by default in F</usr/local/share/dta-tokwrap/stylesheets>.

=over 4

=item dtatw-add-lb.xsl

Replaces newlines with C<E<lt>lb/E<gt>> elements in input document.

=item dtatw-assign-cids.xsl

Assigns missing C<//c/@xml:id> attributes using the XSL C<generate-id()> function.

=item dtatw-rm-c.xsl

Removes C<E<lt>cE<gt>> elements from the input document.
Slow but robust.

=item dtatw-rm-lb.xsl

Replaces C<E<lt>lb/E<gt>> elements with newlines.

=item dtatw-txml2tt.xsl

Converts "master" tokenized XML output format (F<*.t.xml>) to
TAB-separated one-word-per-line format
(F<*.mr.t>
aka F<*.t>
aka F<*.tt>
aka "tt"
aka "CSV"
aka DTA::CAB::Format::TT
aka "TnT"
aka "TreeTagger"
aka "vertical"
aka "moot-native"
aka ...).
See the F<mootfiles(5)> manpage for basic format details, and
see the top of the XSL script for some influential transformation parameters.

=back

=cut


##--------------------------------------------------------------
=pod

=head2 C Programs

Several C programs are included with the distribution.
These are used by the L<dta-tokwrap.perl|dta-tokwrap.perl> script
to perform various intermediate document processing operations,
and should not need to be called by the user directly.

B<Caveat Scriptor>: The following programs are meant for
internal use by the C<DTA::TokWrap> modules only, and their
names, calling conventions, and very presence is subject to change
without notice.

=over 4

=item dtatw-mkindex

Splits input document F<doc.xml>
into
a "character index" F<doc.cx> (CSV),
a "structural index" F<doc.sx> (XML),
and a
"text index" F<doc.tx> (UTF-8 text).

=item dtatw-rm-namespaces

Removes namespaces from any XML document by
renaming "C<xmlns>" attributes to "C<xmlns_>"
and "C<xmlns:*>" attributes to "C<xmlns_*>".
Useful because XSL's namespace handling is annoyingly slow and ugly.

=item dtatw-tokenize-dummy

Dummy C<flex> tokenizer.  Useful for testing.

=item dtatw-txml2sxml

Converts "master" tokenized XML output format (F<*.t.xml>) to
sentence-level stand-off XML format (F<*.s.xml>).

=item dtatw-txml2wxml

Converts "master" tokenized XML output format (F<*.t.xml>) to
token-level stand-off XML format (F<*.w.xml>).

=item dtatw-txml2axml

Converts "master" tokenized XML output format (F<*.t.xml>) to
token-analysis-level stand-off XML format (F<*.a.xml>).

=back

=cut

##======================================================================
=pod

=head1 SEE ALSO

perl(1).

=head1 AUTHOR

Bryan Jurish E<lt>moocow@cpan.orgE<gt>

=head1 COPYRIGHT AND LICENSE

Copyright (C) 2009-2018 by Bryan Jurish

This package is free software. Redistribution and modification
of C portions of this package are subject to the terms of the
version 3 or greater of the GNU Lesser General Public License; see the
files COPYING and COPYING.LESSER which came with the distribution for details.

Redistribution and/or modification of the Perl portions of this package
are subject to the same terms as Perl itself, either Perl version 5.24.1 or,
at your option, any later version of Perl 5 you may have available.

=cut