Annually Release Procedures

This page describes an annually release procedures for Lexical tools package with new set of lexicon data.

I. lvg${YEAR} baseline (without new data)

  1. Prepare lvg${YEAR} baseline
    • Copy lvg${PREV_YEAR} to lvg${YEAR}
      > cp -rp $LVG/lvg${PREV_YEAR} $LVG/lvg${YEAR}
    • Change ${PREV_YEAR} to ${YEAR} in build.xml files under $LVG, $LVG/examples, and $LVG/install
    • Change ${PREV_YEAR} to ${YEAR} in $LVG/overview.html
    • Update $LVG/data/config/lvg.properties, lvg.properties.TEMPLATE

      => Try to build with

      > ant release
      (should be OK to build)

  2. Update Java source code
    • Modify prolog of java files to
      • Remove all SCRs-XX from history tag
      • Modify V-${PREV_YEAR} from version tag
      shell> cd ${LVG}/Components/BaselineCode/bin
      shell> ModifyLvgJavaCode
      > YYYY (${YEAR})
      > 1
      > y
      

      => move sources.new to sources

      shell> rm -rf sources
      shell> mv sources.new sources
      

      => go to ${LVG}/${SOURCES}

      shell> fgrep SCR */*.java
      => no SCR should exist in the source code
      

  3. Update Database for baseline

    => reload old data to database and update the DB_NAME to ${YEAR}
    Please refer to Update DB session

    • Update ${YEAR} in $LVG_DIR/loadDb/sources/gov/nih/nlm/nls/lvg/loadDb/HSqlDb/db.cfg
    • ${LVG_DIR}/loadDb/bin/2.LoadDb ${YEAR}
    • add readonly=true in $LVG_DIR/data/HSqlDb/lvg${YEAR}.properties

  4. Update Installation program for baseline

    => update installation programs:
    Please refer to Update Installation Program session

    • Update ${year} in installation script under ${LVG}.${YEAR}/install/bin/
    • Update ${YEAR} in ${LVG_DIR}/install/sources/gov/nih/nlm/nls/lvg/install/Setup/Param.java

  5. Build/install the baseline

    => build and test to make sure the result is same as last release

    > cd ${LVG}
    > ant clean
    > ant release
    > cd ..
    > gtar -czvf lvg${YEAR}.tgz lvg${YEAR}
    > install to ${PROJECT}
    > The installation should be successful
    

  6. Test Baseline: build the unit test suite

    Please see Regression test on Flows Unit pages. The result on unit test should be the same as last year because neither function or data are changed at this point.

II. Upgrade all software components

  1. Update DB
    • Update HyperSonic JDBC Driver:
      • Download latest version of:
      • unzip it (use unzip)
        • HyperSql Db

          copy 3 files to ${LVG}/lib/jdbcDrivers/HSqlDb/

          • ${HSQLDB}/lib/hsqldb.jar
          • ${HSQLDB}/doc/hsqldb_lic.txt
          • h${HSQLDB}/doc/hypersonic_lic.txt
    • Compile:
      • > cd ${LVG_DIR}/loadDb
      • > ant clean
      • > ant cleanJar
      • > ant
    • Load previous data into latest HyperSonic Sql Db:
      • > vi ${LVG_DIR}/loadDb/sources/gov/nih/nlm/nls/lvg/loadDb/HSqlDb/db.cfg
        • DB_NAME = lvg${YEAR}
        • LVG_DIR = /export/home/chlu/Development/LVG/lvg${YEAR}
      • > mv ${LVG_DIR}/data/HSqlDb ${LVG_DIR}/data/HSqlDb.${PRE_YEAR}
      • > cd ${LVG_DIR}/loadDb/bin
      • > ./2.LoadDb ${YEAR}
        => Source codes might need to be modified (on SQL State and Vendor Error code) in case of dropping non-exist table if HsqlDb is upgraded with new SQL/JDBC standard
        DbBase.ExecuteDdl(String query)
        ...
        if((e.getSQLState().equals("25006"))
        && (e.getErrorCode() == -3706))
        ...
        
      • add readonly=true in ${LVG_DIR}/data/HSqlDb/lvg${YEAR}.properties
        => This is a must to make HSqlDb multi-threads
    • Test:
      • > cd ${LVG_DIR}/loadDb/bin
      • > ./3.TestDb ${YEAR}

  2. Update ICU4J
    • Download the latest version of icu4j-xx.jar
    • Put icu4j-xx.jar to ${LVG}/lib/unicode/
    • Update icu.jar in ${LVG}/build.xml

  3. Update JRE
    • Download latest version of: Java (JRE) to ${LVG_DIR}/bin/jreDist/:
      • jre-XuXX-linux-i586.tar.gz (32-bit)
      • jre-XuXX-linux-x64.tar.gz (64-bit)
      • jre-XuXX-windows-i586.exe (32-bit)
      • jre-XuXX-windows-x64.exe (64-bit)
        => Need to download from 64-bit PC, then change .com to .exe

    • Install the lastest JDK to /usr/local/Applications/Java/
      • set JAVA_HOME in ~/.cshrc

      • /usr/bin/java -> link to the real java
      • /usr/bin/javac -> link to the real javac

      • /etc/alternatives/java -> link to the real java
      • /etc/alternatives/javac -> link to the real javac

      • source ~/.cshrc

  4. Update Installation Program
    • Change program parameters:
      • > cd ${LVG_DIR}/install/sources/gov/nih/nlm/nls/lvg/install/Setup
      • Update all data members in Param.java
      • Update all scripts in install/bin
        • Update lvg${YEAR}
        • Update Java version
    • Compile:
      • > cd ${LVG_DIR}/install
      • > ant clean
      • > ant cleanJar
      • > ant

      • > cd ${LVG_DIR}
      • > ant dist
      • > ant release
    • Test:
      • > cd $LVG_DIR
      • > ./install/bin/install_test.sh

      • Change JAVA=java for all *_test
      • Change name for all *_test to *
        • > mv lgt_test lgt
        • > mv luiNorm_test luiNorm
        • > mv lvg_test lvg
        • > mv norm_test norm
        • > mv toAscii_test toAscii
        • > mv wordInd_test wordInd
        • > mv fields_test fields

      Pack and perform unit test again to complete the baseline

      > cd ${LVG}
      > ant clean
      > ant release
      > pack lvg${YEAR}.tgz
      > install to ${PROJECT}
      > perform Unit test
      => Update config file: lvg.properties.hsql
      > Make a copy of this baseline
      

III. Update functions: Complete SCRs for ${YEAR} release

  • Update ${YEAR} in ${LVG_DIR}/Lib/GlobalBehavior.java
  • Complete all SCRs

IV. Update data: Integrate with New Lexicon Data

V. Update other software components

  • Update $LVG_DIR/sources/gov/nih/nlm/nls/lvg/Tools/GuiTool
  • Update $LVG_DIR/examples
    • update ${YEAR} in ${LVG_DIR}/examples/bin/testExample
    • Add API examples if new tool APIs are added

  • Update lvg web site and documents:
    • apiDoc: should be done in the build process
    • Web Pages:
      • ${LVG_WEB}/web/index.html
      • ${LVG_WEB}/web/download.html
      • ${LVG_WEB}/web/fag.html
      • ${LVG_WEB}/web/release/index.html
      • ${LVG_WEB}/web/release/${YEAR}.html
    • userDoc (${LVG_WEB}/docs/userDoc):
      • ${USER_DOC}/install/releaseNotes.html
      • ${USER_DOC}/install/install.html
      • ${USER_DOC}/install/config.html
      • ${USER_DOC}/install/mySql.html
      • ${USER_DOC}/install/installManual.html

      => This is good enough for the internal release to OCCS if there is a schedule issue
      Bellows can be done after internal release, but must complete before official public release. However, we try to complete this before internal release in the past.

    • designDoc:
      • ${LVG_WEB}/docs/designDoc/LifeCycle/deploy/release/index.html (updates new processes)
      • ${LVG_WEB}/docs/designDoc/UDF/derivations (copy from ${DERIVATION_DIR})
        • prefixList.html
        • PD-Examples
        • PD-Exceptions

        • suffixDRules.html
        • SD-Examples
        • SD-Exceptions
      • ${LVG_WEB}/docs/designDoc/LifeCycle/test (use scripts in ${LVG}/PostProc/TestExamples/bin)
      • ${LVG_WEB}/docs/designDoc/UDF/flow/*.html (update flow examples)
      • ${LVG_WEB}/docs/designDoc/LifeCycle/requirement/*.html (update option examples)
    • SD-Rules:
      • If new SD-Rules are evaluated, please follow SD-Rules Evaluation to get the optimized SD-Rules
      • Theoretically, the optimized SD-Rules set should be updated annually
      • Re-generate SD-Rule trie for Lvg after updating the optimized set
        • Input files:
          • ./data/${YEAR}/data/suffixD.no.data (for exceptions)
          • ./data/${YEAR}/dataR/sdRules.stats.out (for optimized set)
            => use the optimized out for ./data/${YEAR}/dataR/sdRules.stats.out exist

        • shell>cd ${SUFFIX_D}/bin
          shell>GetSdRule 2015
          8
        • Output: ${SUFFID_D}/data/${YEAR}/dataR/dm.rul.${YEAR}.${GOOD_RULE_NO}
        • Copy to ${LVG}/data/rules/dm.rul
          => To test it, the stem must be >= 4
          shell> ./bin/lvg -f:d -kd:3 -p -m
          > 1234se
          > 1234se|1234zation|128|1|d|1|RULE|se$|verb|base|zation$|noun|base|
          => use -kd:3 to include results from SD-Rules
          => 1234 is the stem (>= 4)

VI. Test Procedures

  • ASCII test:
    make sure no non-ASCII characters in the output of norm and luiNorm:
    shell> ${TEST}/LVG/ComponentTest/Ascii/bin
    shell> TestAsciiOnNorm ${YEAR}
    6
    
    => The word count of all four *.2.out fiels should be 0
    

  • Unit test:
    => ${TEST}/LVG/UnitTest/bin/

  • Test Results to Doc:
    Deploy test result to web site document
    shell> cd /export/home/lu/Development/LVG/PostProc/TestExamples/bin
    shell> cd /export/home/lu/Development/LVG/PostProc/UnicodeTables/bin
    

  • Platform test:
    release test on different platforms
    • Linux
      • 64 bit
      • 32 bit => lexdevvm32:/${PLATFORM_TEST}/32Bit/lvg${YEAR}/
    • PC

  • Example test:
    Run all programs under ${LVG_DIR}/examples

  • Above tests should be completed for the internal release for OCSS. This is the best scenario for internal test. The following steps are needed for the final official public release.

  • Performance test on norm and luiNorm:
    • Preparation:
      shell> cd ${TEST}/LVG/PerformanceTest/data/
      shell> mkdir ${YEAR}
      shell> cd ${YEAR}
      shell> mkdir Input
      shell> ln -sf ../../Input/test.in test.in
      shell> ln -sf ../../Input/test.in.100 test.in.100
      shell> ln -sf ../../Input/test.in.10000 test.in.10000
      shell> ln -sf /export/home/lu/Development/Lexicon/data/2007/outputs/lexiconTerms.data lexiconTerms.data
      shell> ln -sf /export/home/lu/Development/Meta/data/2006/meta.data meta.data
      shell> cp -rp ../../${YEAR}/Input/script .
      shell> edit * (change ${YEAR-1} to ${YEAR})
      
    • Run Test:
      shell> cd ${TEST}/LVG/PerformanceTest/bin/
      shell> TestAll
      ${YEAR}
      
  • Change "\t" to "    " for all Java code under $LVG_DIR/sources/gov/nih/nlm/nls/lvg/
    shell> cd ${LVG}/Components/BaselineCode/ModifyLvgJavaCode
    

VII. Lvg Compile & Pack

  • replace indents with 4 spaces in the source code

  • > cd $LVG_DIR
  • > ant clean
  • > ant
  • > ant release

  • pack: gtar -czvf lvg${YEAR}.tgz lvg${YEAR}
  • unpack: gtar -xzvf lvg${YEAR}.tgz

VIII. Update Web Tools

  • Copy $WEB_LVG/${PREV_YEAR} to $WEB_LVG/${YEAR} ...