Annually Release Procedures

This page describes an annually release procedures for Lexical tools package with new set of lexicon data.

  1. Prepare lvg${YEAR} baseline
    • Copy lvg${PREV_YEAR} to lvg${YEAR}
      > cp -rp $LVG/lvg${PREV_YEAR} $LVG/lvg${YEAR}
    • Change ${PREV_YEAR} to ${YEAR} in build.xml files under $LVG, $LVG/examples, and $LVG/install
    • Change ${PREV_YEAR} to ${YEAR} in $LVG/overview.html
    • Update $LVG/data/config/lvg.properties, lvg.properties.TEMPLATE

      => Try to build with

      > ant release
      (should be OK to build)

  2. Update Java source code
    • Modify prolog of java files to
      • Remove all SCRs-XX from history tag
      • Modify V-${PREV_YEAR} from version tag
      		shell> cd ${LVG}/Components/BaselineCode/bin
      		shell> ModifyLvgJavaCode
      		> YYYY (${YEAR})
      		> 1
      		> y
      		

      => move sources.new to sources

      		shell> rm -rf sources
      		shell> mv sources.new sources
      		

      => go to ${LVG}/${SOURCES}

      		shell> fgrep SCR */*.java
      		=> no SCR should exist in the source code
      		

  3. Update Database for baseline

    => reload old data to database and update the DB_NAME to ${YEAR}
    Please refer to Update DB session

    • Update ${YEAR} in $LVG_DIR/loadDb/sources/gov/nih/nlm/nls/lvg/loadDb/HSqlDb/db.cfg
    • ${LVG_DIR}/loadDb/bin/2.LoadDb ${YEAR}
    • add readonly=true in $LVG_DIR/data/HSqlDb/lvg${YEAR}.properties

  4. Update Installation program for baseline

    => update installation programs:
    Please refer to Update Installation Program session

    • Update ${year} in installation script under ${LVG}.${YEAR}/install/bin/
    • Update ${YEAR} in ${LVG_DIR}/install/sources/gov/nih/nlm/nls/lvg/install/Setup/Param.java

  5. Build the baseline

    => build and test to make sure the result is same as last release

    		> cd ${LVG}
    		> ant clean
    		> ant release
    		> cd ..
    		> gtar -czvf lvg${YEAR}.tgz lvg${YEAR}
    		> install to ${PROJECT}
    		> The installation should be successful
    		

  6. Build the unit test suite

    Please see Regression test on Flows Unit pages

  7. Update DB
    • Update HyperSonic JDBC Driver:
      • Download latest version of:
      • unzip it (use unzip)
        • HyperSql Db

          copy 3 files to ${LVG}/lib/jdbcDrivers/HSqlDb/

          • ${HSQLDB}/lib/hsqldb.jar
          • ${HSQLDB}/doc/hsqldb_lic.txt
          • h${HSQLDB}/doc/hypersonic_lic.txt
    • Compile:
      • > cd $LVG_DIR/loadDb
      • > ant clean
      • > ant cleanJar
      • > ant
    • Load previous data into latest HyperSonic Sql Db:
      • > vi $LVG_DIR/loadDb/sources/gov/nih/nlm/nls/lvg/loadDb/HSqlDb/db.cfg
        • DB_NAME = lvg${YEAR}
        • LVG_DIR = /export/home/lu/Development/LVG/lvg${YEAR}
      • > mv ${LVG_DIR}/data/HSqlDb ${LVG_DIR}/data/HSqlDb.${PRE_YEAR}
      • > cd $LVG_DIR/loadDb/bin
      • > ./2.LoadDb ${YEAR}
        => Source codes might need to be modified (on SQL State and Vendor Error code) in case of dropping non-exist table if HsqlDb is upgraded with new SQL/JDBC standard
        		DbBase.ExecuteDdl(String query)
        		...
        		if((e.getSQLState().equals("25006"))
        		&& (e.getErrorCode() == -3706))
        		...
        		
      • add readonly=true in $LVG_DIR/data/HSqlDb/lvg${YEAR}.properties
        => This is a must to make HSqlDb multi-threads
    • Test:
      • > cd $LVG_DIR/loadDb/bin
      • > ./3.TestDb ${YEAR}

  8. Update ICU4J
    • Donwload the latest version of icu4j-xx.jar
    • Put icu4j-xx.jar to ${LVG}/lib/unicode/
    • Update icu.jar in ${LVG}/build.xml

  9. Update JRE
    • Download latest version of: Java (JRE): Windows (offline installation), Solaris (32-bit), Solaris x 86, Linux (self-extracting file)
    • JDK and install to /usr/local/Applications/Java/
      • set JAVA_HOME in ~/.cshrc
      • /usr/bin/java -> link to the real java
      • /usr/bin/javac -> link to the real javac
      • source ~/.cshrc

  10. Update Installation Program
    • Change program parameters:
      • > cd ${LVG_DIR}/install/sources/gov/nih/nlm/nls/lvg/install/Setup
      • Update all data members in Param.java
      • Update all scripts in install/bin
        • Update lvg${YEAR}
        • Update Java version
    • Compile:
      • > cd ${LVG_DIR}/install
      • > ant clean
      • > ant cleanJar
      • > ant

      • > cd ${LVG_DIR}
      • > ant dist
    • Test:
      • > cd $LVG_DIR
      • > ./install/bin/install_test.sh

      • Change JAVA=java for all *_test
      • Change name for all *_test to *
        • > mv lgt_test lgt
        • > mv luiNorm_test luiNorm
        • > mv lvg_test lvg
        • > mv norm_test norm
        • > mv toAscii_test toAscii
        • > mv wordInd_test wordInd
        • > mv fields_test fields

      Pack and perform unit test again to complete the baseline

      	> cd ${LVG}
      	> ant clean
      	> ant release
      	> pack lvg${YEAR}.tgz
      	> install to ${PROJECT}
      	> perform Unit test
      	=> Update config file: lvg.properties.hsql
      	> Make a copy of this baseline
      	

  11. Complete SCRs for ${YEAR} release

  12. Integrate with New Lexicon Data

  13. Update other software components
    • Update $LVG_DIR/sources/gov/nih/nlm/nls/lvg/Tools/GuiTool
    • Update $LVG_DIR/examples
      • update ${YEAR} in ${LVG_DIR}/examples/bin/testExample
      • Add API examples if new tool APIs are added

    • Update documents:
      • apiDoc: should be done in the build process
      • designDoc:
        • ${LVG_WEB}/docs/designDoc/LifeCycle/deploy/release
        • ${LVG_WEB}/docs/designDoc/UDF/flow
        • ${LVG_WEB}/docs/designDoc/LifeCycle/requirement
        • ${LVG_WEB}/docs/designDoc/LifeCycle/test (use ${LVG}/PostProc/TestExamples)
      • userDoc:
        • ${LVG_WEB}/docs/userDoc
    • Update lvg web site
    • Update lvg web tools

  14. Test Procedures
    • ASCII test: make sure no non-ASCII characters in the output of norm and luiNorm:
      shell> ${TEST}/LVG/ComponentTest/Ascii/bin
      shell> TestAsciiOnNorm ${YEAR}
      6
      
      => The word count of all four *.2.out fiels should be 0
      

    • Unit test:
      => ${TEST}/LVG/UnitTest/bin/

    • Deploy test result to web site document
      shell> cd /export/home/lu/Development/LVG/PostProc/TestExamples/bin
      shell> cd /export/home/lu/Development/LVG/PostProc/UnicodeTables/bin
      
    • Release test on different platforms
      • Linux
      • PC

    • This is the Internal release for OCSS.

    • Change "\t" to "    " for all Java code under $LVG_DIR/sources/gov/nih/nlm/nls/lvg/
      shell> cd ${LVG}/Components/BaselineCode/ModifyLvgJavaCode
      
    • Performance test on norm and luiNorm:
      • Preparation:
        shell> cd ${TEST}/LVG/PerformanceTest/data/
        shell> mkdir ${YEAR}
        shell> cd ${YEAR}
        shell> mkdir Input
        shell> ln -sf ../../Input/test.in test.in
        shell> ln -sf ../../Input/test.in.100 test.in.100
        shell> ln -sf ../../Input/test.in.10000 test.in.10000
        shell> ln -sf /export/home/lu/Development/Lexicon/data/2007/outputs/lexiconTerms.data lexiconTerms.data
        shell> ln -sf /export/home/lu/Development/Meta/data/2006/meta.data meta.data
        shell> cp -rp ../../${YEAR}/Input/script .
        shell> edit * (change ${YEAR-1} to ${YEAR})
        
      • Run Test:
        shell> cd ${TEST}/LVG/PerformanceTest/bin/
        shell> TestAll
        	${YEAR}
        

  15. Lvg Compile & Pack
    • > cd $LVG_DIR
    • > ant clean
    • > ant
    • > ant release

    • pack: gtar -czvf lvg${YEAR}.tgz lvg${YEAR}
    • unpack: gtar -xzvf lvg${YEAR}.tgz

  16. Update Web Tools
    • Copy $WEB_LVG/${PREV_YEAR} to $WEB_LVG/${YEAR} ...