Apache Ant

Ant is an installation tool similar to make build that creates a directory structure and fills it with files as directed by a special buildfile called build.xml. This file is written in XML and structured around a sequence of targets defined using the XML <target> tag.

This is not a full primer on Ant; it's only enough to understand how the e-Lab deployment scripts work. The deploying developer will invoke quarkcat/deploy-from-svn, which in turn calls quarkcat/internal-deploy-from-svn, which is what invokes Ant and its build.xml file.

Targets are like waypoints along the build process. When you invoke Ant while supplying the name of a target, Ant will open build.xml and execute the file up to that target. A targets defined in a build.xml file must specify any other targets it depends on; when completing a target, Ant completes these subtargets first.

For our deployment script, build.xml is found in the root directory of the repository, or quarkcat/sw/i2u2svn/build.xml.

The script internal-deploy-from-svn invokes Ant with the target "deploy-all-clean". From the build file, this target is defined as

    <target name="deploy-all-clean" depends="distclean, deploy-all">
    </target>

Note that there is no content inside this particular set of <target> tags. Essentially, this target is simply shorthand for "complete the distclean target, then complete the deploy-all target" (subtargets are always completed in the order listed). In brief, distclean erases the existing website installation, while deploy-all re-installs it from the fresh local copy of the repository that internal-deploy-from-svn checked out.

The complete chain of target dependencies in the buildfile is shown below:

<nicer image to be added later>

                                       / clean-jars
deploy-all-clean --> / distclean  -->  | clean-classes
                     \ deploy-all      \ clean-web
                            |
                            |
                            |          / deploy-cosmic    \
                            |          | deploy-cms       |
                            | -------> | deploy-cms-tb    | --> do-deploy-* --> / build-*          -->   build-common
                                       | deploy-ligo      |                     \ do-deploy-common --> / build-common
                                       | deploy-embedded  /                                            \ clean-jars
                                       | ---------------
                                       \ deploy-i2u2

where "*" represents each of cosmic, cms, cms-tb, ligo, or embedded, as appropriate. That is, deploy-cosmic depends on do-deploy-cosmic, which depends on build-cosmic, etc. The target deploy-i2u2 has no dependencies and is the last target to be executed during a typical deployment.

The first target to be completed, distclean, removes files from the live website directories within quarkcat/sw/tomcat/webapps/elab/ (recall that files served to the client are within tomcat/webapps/ROOT/).

• clean-jars deletes the Java archive .jar files in elab/WEB-INF/lib/
• clean-classes removes the Java .class files in elab/WEB-INF/classes/
• clean-web removes .html and .jsp files in the elab/ directory itself.

After this cleaning, the next fully-executed target invoked by internal-deploy-from-svn is build-common.

target build-common

The full text of the build-common target is

<target name="build-common">
    <mkdir dir="common/build"/>
    <delete quiet="true">
        <fileset dir="common/build"
            includes="**/*.class"/>
    </delete>
    <javac srcdir="common/src/java" destdir="common/build" debug="true">
        <classpath refid="cp-common"/>
        <compilerarg value="-deprecation"/>
    </javac>
    <!-- the properties file -->
    <copy todir="common/build">
        <fileset dir="common" includes="elab.properties"/>
        <filterset>
            <filter token="container.home" value="${container.home}"/>
            <filter token="webapps" value="${webapps}"/>
            <filter token="elab.name" value="${elab.name}"/>
            <filter token="vds.home" value="${vds.home}"/>
            <filter token="portal.home" value="${portal.home}"/>
        </filterset>
    </copy>
</target>

There are three main components of this task:

1. Prepare the common/build/ directory
2. Compile .java files within the common/src/java/ directory into .class files within common/build/
3. Copy configuration files into common/build/

1. Prepare common/build/

<mkdir dir="common/build"/>
<delete quiet="true">
    <fileset dir="common/build"
        includes="**/*.class"/>
</delete>

<mkdir dir="common/build"/> creates the directory common/build/. Relative paths are interpreted with respect to wherever build.xml is, so this will be quarkcat/sw/i2u2svn/common/build/. The directory common/build/ is not included in the repository, so it will not exist the first time you deploy from a fresh working copy; it will typically already exist afterwards, though. The <mkdir> task does not overwrite an existing directory, so if common/build/ does already exist, the <delete> tag clears away unwanted files so that we can write fresh ones. The <fileset> tag defines the set of files to be deleted, in this case everything within common/build/ that is a .class file.

The quiet="true" attribute of the <delete> tag keeps these deletions from being written to the terminal output. The double-asterisk ** syntax is an Ant pattern-matching wildcard that can stand for any number of directories at once (the usual * can stand for only one directory name at a time). Thus, **/.class matches any .class file within the specified root directory of the fileset, no matter how many directories deep it is. In other words, this construction makes the delete recursive within common/build/.

2. Compile .java source files into common/build/

<javac srcdir="common/src/java" destdir="common/build" debug="true">
    <classpath refid="cp-common"/>
    <compilerarg value="-deprecation"/>
</javac>

The <javac> task compiles a Java source tree. The srcdir attribute gives the source directory, the location of the Java files to be compiled. The destdir attribute is the destination directory, the place to store the output .class files. From inspection, you can see that the directory structure under srcdir="common/src/java" mirrors that of destdir="common/build"; the <javac> task recursively scans for and compiles each .java file within common/src/java/ and places the output .class file in the corresponding location in common/build/. Only if a .java file has no corresponding .class file, or if the .java file is newer than its output .class file, will <javac> compile that source.

Compare source:

quarkcat/sw/i2u2svn/common/src/java/
                                 |--be/telio/mediastore/ui/upload/MonitoredDiskFileItem.java
                                 |                                ...
                                 |--gov/fnal/elab/<lots of subdirectories and .java files>
                                 |--org/
                                     |--gryphyn/common/catalog/replica/ElabRC.java
                                     |--mindrot/BCrypt.java

with destination:

quarkcat/sw/i2u2svn/common/build/
                              |--be/telio/mediastore/ui/upload/MonitoredDiskFileItem.class
                              |                                ...
                              |--gov/fnal/elab/<lots of subdirectories and .class files>
                              |--org/
                                  |--gryphyn/common/catalog/replica/ElabRC$OneElementSet$1.class
                                  |                                 ElabRC$OneElementSet.class
                                  |                                 ElabRC.class
                                  |--mindrot/BCrypt.class

Note that, by far, the bulk of the action is within the gov/fnal/elab/ directory, representing the gov.fnal.elab Java package. I don't know what the deal is with the ElabRC$*.class files within the common/build/org/gryphyn/ subdirectory. Perhaps remnants of an earlier experiment?

The debug="true" option specifies that debugging information should be printed to the terminal, based on the value of debuglevel, which does not appear to be set in build.xml. Presumably it can be passed in as an argument when the buildfile is invoked outside of internal-deploy-from-svn.

The tag <classpath refid="cp-common"/> establishes "cp-common" as the class path for this particular Java compilation, as if one had used the -classpath flag when invoking javac from the command line. The parameter cp-common itself is defined earlier in build.xml:

<path id="cp-common">
    <fileset dir="common/lib">
        <include name="**/*.jar"/>
    </fileset>
    <fileset dir="${container.home}/${container.libs}">
        <include name="**/*.jar"/>
    </fileset>
    <pathelement location="common/build"/>
    <pathelement location="common/resources/classes"/>
</path>

The file quarkcat/sw/i2u2svn/build.properties defines ${container.home} = quarkcat/sw/tomcat/ (i.e., the Tomcat installation directory) and ${container.libs} = lib. Thus, the <fileset> tasks define all .jar files within the quarkcat/sw/i2u2svn/common/lib/ and quarkcat/sw/tomcat/lib/ directories as part of the path cp-common. In addition, the <pathelement> tasks add the specific directories common/build/ and common/resources/classes/ identified by the location attribute. The Java class path for build-common’s <javac> task is therefore the four directories

quarkcat/sw/
         |--i2u2svn/common/
         |              |--build/
         |              |--lib/
         |              |--resources/classes/
         |--tomcat/lib/

or sw/i2u2svn/common/build/ : sw/i2u2svn/common/lib : sw/i2u2svn/common/resources/classes : sw/tomcat/lib (though not necessarily in that order). Note that tomcat/lib/ is outside the repository, whose root directory is i2u2svn/. These are standard Tomcat .jar files included with the installation (i.e., you'll never need to edit them or worry about them).

Back to build-common: <compilerarg> gives values to pass to the Java compiler, as if provided at the command line. The one here, -deprecation, tells <javac> to provide descriptions when deprecated members or classes are invoked.

3. Copy config files into common/build/

<copy todir="common/build">
    <fileset dir="common" includes="elab.properties"/>
    <filterset>
        <filter token="container.home" value="${container.home}"/>
        <filter token="webapps" value="${webapps}"/>
        <filter token="elab.name" value="${elab.name}"/>
        <filter token="vds.home" value="${vds.home}"/>
        <filter token="portal.home" value="${portal.home}"/>
    </filterset>
</copy>

[NB: This section is problematic. As written, it appears to accomplish several different varieties of nothing.]

After compilation, build-common copies files to the common/build/ directory, as given by todir="common/build". The files to be copied are given by the <fileset> task, which specifies i2u2svn/common/ as its root and includes any file named elab.properties within this root as an element of the fileset. Since the include attribute is not recursive, this copies only the file common/elab.properties into the directory common/build. However, the file common/elab.properties does not exist in the repo. Furthermore, the destination i2u2svn/common/build/ directory includes no elab.properties file, before or after the execution of build.xml. This element of the buildfile copies nothing in both theory and in practice.

[There are the files quarkcat/sw/i2u2svn/common/resources/classes/elab.properties and quarkcat/sw/local-settings/common/resources/classes/elab.properties. The latter does not exist in the repository, only in local environments. Changes to it, however, are reflected in quarkcat/sw/i2u2svn/common/resources/classes/elab.properties after deployment. Be aware of this and figure out how it works.]

As part of the <copy> task, the buildfile defines a set of <filter> tags that alter the contents of the copied files while in transit. It works like this:

In the source file, any string that you intend to be changed is called a token and should be bracketed with "@" symbols: @token.example@. During copying, the <copy> task looks for token attributes defined within <filter> tags and replaces any tokens that match them with the corresponding value attributes. In the above, for example, any copied files that contain the text "@container.home@" will have that string replaced with whatever ${container.home} evaluates to when the copies are written to common/build/ (${container.home} is defined in i2u2svn/build.properties; mine, for instance, is container.home=/Users/jgriffith/quarkcat/sw/tomcat8). The original files are not altered.

The tokens given here (i.e., @container.home@) never appear anywhere within the quarkcat/ folder, so I can't figure out what purpose these serve. In particular, none of them appear in any of the elab.properties files that appear in the installation, and only "elab.name" and "vds.home" appear in uncommented form in them (and even then without "@" token delimiters).

target build-<elab>

When build-common completes, Ant returns to the sequence of build-<elab> targets build-cosmic, build-cms, build-cms-tb, build-ligo and build-embedded ("cms-tb" is the CMS test beam, which is no longer an active e-Lab, since CMS has real beam to work with now. I have no idea what "embedded" is, though). Since all of these targets are highly similar, build-cosmic suffices as an example for all of them.

Note that when the chain of targets hits deploy-<elab>, the invocation of do-deploy-<elab> is not simply a depends attribute, but a more elaborate <antcall> task that functions much the same as depends, but which allows the e-lab name to be passed as a parameter, like so:

    <target name="deploy-cosmic">
        <antcall target="do-deploy-cosmic">
            <param name="elab.name" value="cosmic"/>
        </antcall>
    </target>

The variable elab.name will appear later in the chain.

The full code for example target build-cosmic is

<target name="build-cosmic" depends="build-common" description="--> builds COSMIC">
    <mkdir dir="cosmic/build"/>
    <javac srcdir="cosmic/src/java" destdir="cosmic/build" debug="true">
        <include name="**/*.java"/>
        <classpath refid="cp-common"/>
        <classpath refid="cp-cosmic"/>
        <compilerarg value="-deprecation"/>
    </javac>
    <copy todir="common/build">
        <fileset dir="cosmic" includes="elab.properties.cosmic"/>
        <filterset>
            <filter token="container.home" value="${container.home}"/>
            <filter token="webapps" value="${webapps}"/>
            <filter token="elab.name" value="${elab.name}"/>
            <filter token="vds.home" value="${vds.home}"/>
            <filter token="portal.home" value="${portal.home}"/>
        </filterset>
    </copy>
</target>

First, the target creates the directory quarkcat/sw/i2u2svn/cosmic/build/. Then it compiles .java files from cosmic/src/java/ into cosmic/build/.

Compare source:

quarkcat/sw/i2u2svn/cosmic/src/java/gov/fnal/elab/
                                               |--cosmic/<.java files and subdirectories with .java files>
                                               |--usermanagement/CosmicElabUserManagementProvider.java                                
                                                              |--impl/CosmicDatabaseUserManagementProvider.java

with destination:

quarkcat/sw/i2u2svn/cosmic/build/gov/fnal/elab/
                                            |--cosmic/<.class files and subdirectories with .class files>
                                            |--usermanagement/CosmicElabUserManagementProvider.class                                
                                                           |--impl/CosmicDatabaseUserManagementProvider.class

[Note that cosmic/build/gov/fnal/elab/cosmic/ also contains classes with $ in the filename that don't correspond to source files in cosmic/src/java/gov/fnal/elab/cosmic/]

Classpaths are defined as cp-common and cp-cosmic. We saw cp-common before when it was used by build-common, but cp-cosmic looks like

    <path id="cp-cosmic">
        <fileset dir="cosmic">
            <include name="lib/**/*.jar"/>
        </fileset>
        <pathelement location="cp-cosmic/build"/>
    </path>

This is problematic for several reasons.

First, it defines a <fileset> of all .jar files within cosmic/lib/ (recursive) as part of the path. There is no cosmic/lib/ in the repository. Second, the <pathelement> location cp-cosmic/build does not exist; there is no cp-cosmic/ directory. If this is a typo and cosmic/build/ is the intended path element, this wouldn't make sense because cosmic/build/ does not exist until build-cosmic creates it immediately prior in the deployment process; obviously there won't be any class files there for <javac> to find. Thus, the classpath cp-cosmic is void, as defined.

Finally, the file cosmic/elab.properties.cosmic is copied into common/build/ with filters as noted. Again, the file cosmic/elab.properties.cosmic does not exist to be copied. The only files by this name in the repository are in the obsolete directories

config/www13/home/quarkcat/sw/local-settings/cosmic/resources/classes/elab.properties.cosmic
config/www17/home/quarkcat/sw/local-settings/cosmic/resources/classes/elab.properties.cosmic
config/www18/home/quarkcat/sw/local-settings/cosmic/resources/classes/elab.properties.cosmic

These three files have the nearly identical content

data.dir = /disks/i2u2/cosmic/data         ## 17 and 18 only 
data.dir = /disks/i2u2-dev/cosmic/data     ## 13 only

#override the default user management provider
provider.usermanagement = gov.fnal.elab.usermanagement.impl.CosmicDatabaseUserManagementProvider
analysis.runtime.estimator.set = gov.fnal.elab.cosmic.estimation.CosmicAnalysisRunTimeEstimatorSet

target do-deploy-common

With target build-cosmic complete, the buildfile executes target do-deploy-common. This target depends on build-common, which was completed as a prerequisite to build-cosmic (and to build-<elab> generally) and on clean-jars, which was completed as a prerequisite of distclean.

    <target name="do-deploy-common" depends="build-common, clean-jars">
        <mkdir dir="${container.home}/${webapps}/elab"/>
        <mkdir dir="${container.home}/${webapps}/elab/capture" />
        <mkdir dir="${container.home}/${webapps}/elab/capture/img" />
        <mkdir dir="${container.home}/${webapps}/elab/WEB-INF"/>
        <mkdir dir="${container.home}/${webapps}/elab/WEB-INF/classes"/>
        <mkdir dir="${container.home}/${webapps}/elab/WEB-INF/lib"/>
        <mkdir dir="${container.home}/${webapps}/elab/${elab.name}"/>
        
        <copy todir="${container.home}/${webapps}/elab/capture">
            <fileset dir="capture" includes="*" />
        </copy>
        <copy todir="${container.home}/${webapps}/elab/WEB-INF/lib">
            <fileset dir="common/lib" includes="**/*.jar"/>
        </copy>
        <copy todir="${container.home}/${webapps}/elab/WEB-INF">
            <fileset dir="common/resources" includes="**/*.*"/>
        </copy>
        <copy todir="${container.home}/${webapps}/elab/WEB-INF/classes">
            <fileset dir="common/build" includes="**/*.*" excludes="elab.properties"/>
        </copy>
        <concat destfile="${container.home}/${webapps}/elab/WEB-INF/elab.properties" force="no">
            <fileset dir="common/build" includes="elab.properties*"/>
        </concat>
        <copy todir="${container.home}/${webapps}/elab/${elab.name}">
            <fileset dir="common/src/jsp" includes="**/*.*"/>
        </copy>
    </target>

First, this target creates a set of new directories. With container.home = /Users/jgriffith/quarkcat/sw/tomcat8, webapps = webapps, and elab.name=cosmic (for this example), the directories

quarkcat/sw/tomcat8/webapps/elab/
quarkcat/sw/tomcat8/webapps/elab/capture/
quarkcat/sw/tomcat8/webapps/elab/capture/img/
quarkcat/sw/tomcat8/webapps/elab/WEB-INF/
quarkcat/sw/tomcat8/webapps/elab/WEB-INF/classes/
quarkcat/sw/tomcat8/webapps/elab/WEB-INF/lib/
quarkcat/sw/tomcat8/webapps/elab/cosmic/

(individual installations mayl differ on container.home, of course).

Note that unlike the build/ directories, these are located outside of the repository structure of i2u2svn/ and in the Tomcat directory, where the live files are stored.

Next (and last), the target copies files according to

source	what	destination
capture/	everything in capture/	elab/capture/
common/build/	everything except common/build/elab.properties	elab/WEB-INF/classes/
common/resources/	everything	elab/WEB-INF/
common/src/jsp	everything	elab/cosmic/
common/lib/	all .jar files (recursive)	elab/WEB-INF/lib/

Note that there are no such files as common/build/elab.properties*.

In addition, there's one <concat> task

       <concat destfile="${container.home}/${webapps}/elab/WEB-INF/elab.properties" force="no">
          <fileset dir="common/build" includes="elab.properties*"/>
       </concat>

that takes all files of the form common/build/elab.properties* (say, elab.properties and elab.properties.cosmic, which - again - don't exist) and concatenates them into a single file tomcat8/webapps/elab/WEB-INF/elab.properties. In the event that this destination file exists and is newer than any of the source files, the force="no" attribute prevents this more up-to-date file from being overwritten.

Also note that in the post-deployment environments of my localhost (dvt4) and i2u2-prod, the destination file tomcat8/webapps/elab/WEB-INF/elab.properties does not exist.

target do-deploy-<elab>

From do-deploy-common, the chain of <elab> deployment returns to the e-Lab-specific do-deploy-<elab>, or do-deploy-cosmic in the current example.

    <target name="do-deploy-cosmic" depends="build-cosmic, do-deploy-common">
        <mkdir dir="${container.home}/${webapps}/elab/${elab.name}"/>
    
        <copy todir="${container.home}/${webapps}/elab/WEB-INF">
            <fileset dir="cosmic" includes="lib/**/*.jar"/>
        </copy>
        <mkdir dir="cosmic/resources"/>
        <copy todir="${container.home}/${webapps}/elab/WEB-INF">
            <fileset dir="cosmic/resources" includes="**/*.*"/>
        </copy>
        <copy todir="${container.home}/${webapps}/elab/WEB-INF/classes">
            <fileset dir="cosmic/build" includes="**/*.*"/>
        </copy>
        <copy todir="${container.home}/${webapps}/elab/${elab.name}" overwrite="true">
            <fileset dir="cosmic/src/jsp" includes="**/*.*"/>
        </copy>
        <exec executable="ln" logError="true">
            <arg value="-s"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/graphics"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/jsp"/>
        </exec>
        <exec executable="ln" logError="true">
            <arg value="-s"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/users"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/jsp"/>
        </exec>
        <exec executable="ln" logError="true">
            <arg value="-s"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/graphics"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/references"/>
        </exec>
        <exec executable="ln" logError="true">
            <arg value="-s"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/graphics"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/test"/>
        </exec>
    </target>

gadgegggfdsg

This target begins by making the directory tomcat8/webapps/elab/cosmic. It then copies all .jar files from within cosmic/lib/ (recursive) into tomcat8/webapps/elab/WEB-INF/.

Then, it makes the directory cosmic/resources/

Example: deploy-cosmic

For example, here's the first subtarget, deploy-cosmic:

    <target name="deploy-cosmic">
        <antcall target="do-deploy-cosmic">
            <param name="elab.name" value="cosmic"/>
        </antcall>
    </target>
    
    <target name="do-deploy-cosmic" depends="build-cosmic, do-deploy-common">
        <mkdir dir="${container.home}/${webapps}/elab/${elab.name}"/>
    
        <copy todir="${container.home}/${webapps}/elab/WEB-INF">
            <fileset dir="cosmic" includes="lib/**/*.jar"/>
        </copy>
        <mkdir dir="cosmic/resources"/>
        <copy todir="${container.home}/${webapps}/elab/WEB-INF">
            <fileset dir="cosmic/resources" includes="**/*.*"/>
        </copy>
        <copy todir="${container.home}/${webapps}/elab/WEB-INF/classes">
            <fileset dir="cosmic/build" includes="**/*.*"/>
        </copy>
        <copy todir="${container.home}/${webapps}/elab/${elab.name}" overwrite="true">
            <fileset dir="cosmic/src/jsp" includes="**/*.*"/>
        </copy>
        <exec executable="ln" logError="true">
            <arg value="-s"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/graphics"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/jsp"/>
        </exec>
        <exec executable="ln" logError="true">
            <arg value="-s"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/users"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/jsp"/>
        </exec>
        <exec executable="ln" logError="true">
            <arg value="-s"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/graphics"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/references"/>
        </exec>
        <exec executable="ln" logError="true">
            <arg value="-s"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/graphics"/>
            <arg value="${container.home}/${webapps}/elab/${elab.name}/test"/>
        </exec>
    </target>

The target deploy-cosmic itself simply performs an antcall task, which is a way of passing off to a subtarget that allows you to include parameters (the depends keyword of the <target> tag doesn't allow parameter passing). Inside the <antcall> tag we define the parameter name-value pair "elab.name" and "cosmic". When the subtarget do-deploy-cosmic executes, it will replace all instances of elab.name with the string "cosmic".

do-deploy-cosmic will, in order, complete the subtarget build-cosmic, complete the subtarget do-deploy-common, and then execute the tasks within its <target> tags.

-- Main.JoelG - 2016-05-03

Comments