Annual DataBase Update Procedures

This page describes annual release procedures for updating database with new set of THE SPECIALIST LEXICON data in lexAccess tools. This is the only procedure needed after release 2013+.

  1. LA PreProc
    • Generate Db table files (UTF-8)
      shell> cd ${LA_PRE_PROC}/PreDatabase/bin
      shell> 1.GenerateDbFiles ${YEAR}
      --------------------------------------
      Which Program ?
      --------------------------------------
      1) Copy LEXICON
      2) Copy inflVars.data
      3) Generate lexicon.data
      4) Generate eui.data
      ----------
      5) Copy above 4 files to LexAccessDbData2013+
      => Go to LexAccessDbData2013+ to load files to DB tables
      ----------
      6) Copy Db tables from LexAccessDbData2013+ to laData
      7) Back Db tables from laData to Backup/Releases/LA/laData
      ----------
      10) All (default: Step 1- 5)
      ----------
      

      Steps 1-5: 4 files are copied and generated:
      File nameAction
      LEXICON copy ${LEX_DIR}/data/${YEAR}/tables.frozen/LEXICON to ${LA_DIR}/data/tables/LEXICON
      inflVars.data copy ${LEX_DIR}/data/${YEAR}/tables.frozen/inflVars.data to ${LA_DIR}/data/tables/inflVars.data
      lexicon.data Generate ${LA_DIR}/data/tables/lexicon.data (from LEXICON)
      shell> ${LA}/PreDatabase/bin/ModifyLexicon -f:d -i:LEXICON -o:lexicon.data
      eui.data Generate eui.data. The total number of EUI should be equals to total lexical records in LEXICON because this file include deleted records if using lexicon.data from LB.

    • Check Max. length of fields in Lexicon.data with database tables
      shell> cd ${LA_PRE_PROC}/PreDatabase/bin
      shell> 2.AnalyzeDbFiles ${YEAR}
      

  2. LoadDb Program (After Step-5)
    • Update db.cfg
      • update ${lexAccessDbData2013+}/loadDb/sources/gov/nih/nlm/nls/lexAccess/loadDb/HSqlDb/db.cfg
        • DB_NAME=lexAccess${YEAR}
        • DB_DATA_DIR=data.${YEAR}/HSqlDb

    • Setup directory
      shell> cd ${lexAccessDbData2013+}/loadDb/bin
      shell> 1.Setup ${YEAR}
      
      • mkdir ${LA_DB_DATA_DIR}/data.${YEAR}
      • ln -s ${LA_DB_DATA_DIR}/data.${YEAR} ${LA_DB_DATA_DIR}/data
      • Rename tables to tables.utf8 in ${lexAccess2013+}/data.${YEAR}/
      => Use this script to create Db data directory. The database name and username are defined in the cfg.data file.

    • Load data into HSqlDb (skip MySql)

      Manully link tables

      shell> cd ${lexAccessDbData2013+}/data.${YEAR}
      shell> ln -sf ./tables.utf8 tables
      

      shell> cd ${lexAccessDbData2013+}/loadDb/bin
      shell> 2.LoadDb
      1
      3
      
      • Update DB:
        add "readonly=true" in *.properties after tables are created/loaded

    • Check loaded data in Db
      shell> cd ${lexAccessDbData2013+}/loadDb/bin
      shell> 3.TestDb
      
      => The total number of lexicon does not includes deleted records.

      => Manully move HSqlDb to HSqlDb.utf8 (${lexAccessDbData2013+}/data.${YEAR}/

  3. laData and backup
    shell> cd ${LA_PRE_PROC}/PreDatabase/bin
    shell> 1.GenerateDbFiles ${YEAR}
    --------------------------------------
    Which Program ?
    --------------------------------------
    ...
    ----------
    6) Copy Db tables from LexAccessDbData2013+ to laData
    7) Back Db tables from laData to Backup/Releases/LA/laData
    ----------
    
    • laData
      shell> cd ${lexAccess2013+}/data.${YEAR}
      shell> mv HSqlDb to HSqlDb.utf8
      shell> 1.GenerateDbFiles ${YEAR}
      6
      
      => Copy -rp ${lexAccess2013+}/data.${YEAR}/HSqlDb.utf8 to ${LA_DATA}/HSqlDb.${YEAR}

    • Backup
      shell> 1.GenerateDbFiles ${YEAR}
      7
      
      => Copy -rp ${LA_DATA}/HSqlDb.${YEAR} ${BACKUP}Releases/LA/laData/.
      => gtar -czvf HSqlDb.${YEAR}.tgz HSqlDb.${YEAR}

      => Manully mv HSqlDb.${YEAR}* to ${BACKUP}Releases/LA/laData/HSqlDb.2.3.2/.

  4. Generate pure ASCII Database (for MetaMap Project)
    • generate and copy 4 ASCII files to ${LA_DIR}/data/tables.ascii
      shell> cd ${LA_PRE_DIR}/bin
      shell> 3.GenerateAsciiDbFiles ${YEAR}
      1
      2
      3
      4
      5
      
    • LoadDb Program (silimar to Step in UTF-8):
      • update DB_NAME=lexAccess${YEAR}ASCII in ${lexAccess2013+}/loadDb/sources/gov/nih/nlm/nls/lexAccess/loadDb/HSqlDb/db.cfg
      • shell> cd ${lexAccessDbData2013+}/data.${YEAR}
        shell> ln -s ./tables.ascii tables
        shell> cd ${lexAccessDbData2013+}/loadDb/bin
        shell> 2.LoadDb
        	1
        	3
        
      • change "readonly=true" in *.properties after tables are loaded
      • Rename HSqlDb to HSqbDb.ascii in ${LA_DIR}/data/
    • laData (silimar to Step in UTF-8):
    • Backup (silimar to Step in UTF-8):

  5. Add new Database to ${PROJECT}/lexAccess
    • LexAccess2016 & lagt:
      • Link ${LA_DIR}/data/HSqlDb.${YEAR} to ${laData}/HSqlDb.${YEAR}
      • Link ${LA_DIR}/data/HSqlDb.${YEAR}ASCII to ${laData}/HSqlDb.${YEAR}ASCII
      • Update config file: ${LA_DIR}/data/config/lexAccess.properties.${YEAR}
        • Update DB_DIR=HSqlDb.{YEAR}/ for default Db
        • Update LATEST_VERSION=${YEAR} for the lastest version
        • Update DB_NAME=lexAccess${YEAR} for default Db table
      • ln -sf ./lexAccess.properties.${YEAR} lexAccess.properties
      • Update ./bin/lexAccess.{YEAR}
        • shell>cd ${LA_DIR}/bin
        • shell>cp -p lexAccess.${PREV_YEAR} lexAccess.${YEAR}
        • shell>update ./data/config/lexAccess.properties.${YEAR} in lexAccess.${YEAR}
    • WebLexAccess.2016:
      • Update ${WEB_LA_DIR}/WEB_INF/lexiconFile
        • ${YEAR}
        • ${YEAR}ASCII
      • restart the web appliation server

  6. go through unit test

  7. Update WebSite
    • Update ${YEAR}, LexRecord, INflVars, Max.EUI, etc. in ${LA_WEB}/web/download.html
    • add Db files from ${BACKUP} to ${LA_WEB}/release/laData
      • HSqlDb.${YEAR}.tgz
      • HSqlDb.${YEAR}ASCII.tgz

  8. Update Year
    • Update ${YEAR} on ${WEB_LEXACCESS}/2013/Home/banner*.html (no need for 2013+)