Annual DataBase Update Procedures

This page describes annual release procedures for updating database with new set of THE SPECIALIST LEXICON data in lexAccess tools. This is the only procedure needed after release 2013+.

1. LA PreProc
   • Generate Db table files (UTF-8)

     shell> cd ${LA_PRE_PROC}/PreDatabase/bin
     shell> 1.GenerateDbFiles ${YEAR}
     --------------------------------------
     Which Program ?
     --------------------------------------
     1) Copy LEXICON
     2) Copy inflVars.data
     3) Generate lexicon.data
     4) Generate eui.data
     ----------
     5) Copy above 4 files to LexAccessDbData2013+
     => Go to LexAccessDbData2013+ to load files to DB tables
     ----------
     6) Copy Db tables from LexAccessDbData2013+ to laData
     7) Back Db tables from laData to Backup/Releases/LA/laData
     ----------
     10) All (default: Step 1- 5)
     ----------
     

     
     Steps 1-5: 4 files are copied and generated:

     File name	Action
     LEXICON	copy ${LEX_DIR}/data/${YEAR}/tables.frozen/LEXICON to ${LA_DIR}/data/tables/LEXICON
     inflVars.data	copy ${LEX_DIR}/data/${YEAR}/tables.frozen/inflVars.data to ${LA_DIR}/data/tables/inflVars.data
     lexicon.data	Generate ${LA_DIR}/data/tables/lexicon.data (from LEXICON) shell> ${LA}/PreDatabase/bin/ModifyLexicon -f:d -i:LEXICON -o:lexicon.data
     eui.data	Generate eui.data. The total number of EUI should be equals to total lexical records in LEXICON because this file include deleted records if using lexicon.data from LB.

   • Check Max. length of fields in Lexicon.data with database tables

     shell> cd ${LA_PRE_PROC}/PreDatabase/bin
     shell> 2.AnalyzeDbFiles ${YEAR}
     

2. LoadDb Program (After Step-5)
   • Update db.cfg
     • update ${lexAccessDbData2013+}/loadDb/sources/gov/nih/nlm/nls/lexAccess/loadDb/HSqlDb/db.cfg
       • DB_NAME=lexAccess${YEAR}
       • DB_DATA_DIR=data.${YEAR}/HSqlDb

   • Setup directory

     shell> cd ${lexAccessDbData2013+}/loadDb/bin
     shell> 1.Setup ${YEAR}
     

     • mkdir ${LA_DB_DATA_DIR}/data.${YEAR}
     • ln -s ${LA_DB_DATA_DIR}/data.${YEAR} ${LA_DB_DATA_DIR}/data
     • Rename tables to tables.utf8 in ${lexAccess2013+}/data.${YEAR}/
     => Use this script to create Db data directory. The database name and username are defined in the cfg.data file.

   • Load data into HSqlDb (skip MySql)

     Manully link tables

     shell> cd ${lexAccessDbData2013+}/data.${YEAR}
     shell> ln -sf ./tables.utf8 tables
     

     shell> cd ${lexAccessDbData2013+}/loadDb/bin
     shell> 2.LoadDb
     1
     3
     

     • Update DB:
       add "readonly=true" in *.properties after tables are created/loaded

   • Check loaded data in Db

     shell> cd ${lexAccessDbData2013+}/loadDb/bin
     shell> 3.TestDb
     

     => The total number of lexicon does not includes deleted records.

     => Manully move HSqlDb to HSqlDb.utf8 (${lexAccessDbData2013+}/data.${YEAR}/

3. laData and backup

   shell> cd ${LA_PRE_PROC}/PreDatabase/bin
   shell> 1.GenerateDbFiles ${YEAR}
   --------------------------------------
   Which Program ?
   --------------------------------------
   ...
   ----------
   6) Copy Db tables from LexAccessDbData2013+ to laData
   7) Back Db tables from laData to Backup/Releases/LA/laData
   ----------
   

   • laData

     shell> cd ${lexAccess2013+}/data.${YEAR}
     shell> mv HSqlDb to HSqlDb.utf8
     shell> 1.GenerateDbFiles ${YEAR}
     6
     

     => Copy -rp ${lexAccess2013+}/data.${YEAR}/HSqlDb.utf8 to ${LA_DATA}/HSqlDb.${YEAR}

   • Backup

     shell> 1.GenerateDbFiles ${YEAR}
     7
     

     => Copy -rp ${LA_DATA}/HSqlDb.${YEAR} ${BACKUP}Releases/LA/laData/.
     => gtar -czvf HSqlDb.${YEAR}.tgz HSqlDb.${YEAR}

     => Manully mv HSqlDb.${YEAR}* to ${BACKUP}Releases/LA/laData/HSqlDb.2.3.2/.

4. Generate pure ASCII Database (for MetaMap Project)
   • generate and copy 4 ASCII files to ${LA_DIR}/data/tables.ascii

     shell> cd ${LA_PRE_DIR}/bin
     shell> 3.GenerateAsciiDbFiles ${YEAR}
     1
     2
     3
     4
     5
     

   • LoadDb Program (silimar to Step in UTF-8):
     • update DB_NAME=lexAccess${YEAR}ASCII in ${lexAccess2013+}/loadDb/sources/gov/nih/nlm/nls/lexAccess/loadDb/HSqlDb/db.cfg

     • shell> cd ${lexAccessDbData2013+}/data.${YEAR}
       shell> ln -s ./tables.ascii tables
       shell> cd ${lexAccessDbData2013+}/loadDb/bin
       shell> 2.LoadDb
       	1
       	3
       

     • change "readonly=true" in *.properties after tables are loaded
     • Rename HSqlDb to HSqbDb.ascii in ${LA_DIR}/data/
   • laData (silimar to Step in UTF-8):
   • Backup (silimar to Step in UTF-8):

5. Add new Database to ${PROJECT}/lexAccess
   • LexAccess2016 & lagt:
     • Link ${LA_DIR}/data/HSqlDb.${YEAR} to ${laData}/HSqlDb.${YEAR}
     • Link ${LA_DIR}/data/HSqlDb.${YEAR}ASCII to ${laData}/HSqlDb.${YEAR}ASCII
     • Update config file: ${LA_DIR}/data/config/lexAccess.properties.${YEAR}
       • Update DB_DIR=HSqlDb.{YEAR}/ for default Db
       • Update LATEST_VERSION=${YEAR} for the lastest version
       • Update DB_NAME=lexAccess${YEAR} for default Db table
     • ln -sf ./lexAccess.properties.${YEAR} lexAccess.properties
     • Update ./bin/lexAccess.{YEAR}
       • shell>cd ${LA_DIR}/bin
       • shell>cp -p lexAccess.${PREV_YEAR} lexAccess.${YEAR}
       • shell>update ./data/config/lexAccess.properties.${YEAR} in lexAccess.${YEAR}
   • WebLexAccess.2016:
     • Update ${WEB_LA_DIR}/WEB_INF/lexiconFile
       • ${YEAR}
       • ${YEAR}ASCII
     • restart the web appliation server

6. go through unit test

7. Update WebSite
   • Update ${YEAR}, LexRecord, INflVars, Max.EUI, etc. in ${LA_WEB}/web/download.html
   • add Db files from ${BACKUP} to ${LA_WEB}/release/laData
     • HSqlDb.${YEAR}.tgz
     • HSqlDb.${YEAR}ASCII.tgz

8. Update Year
   • Update ${YEAR} on ${WEB_LEXACCESS}/2013/Home/banner*.html (no need for 2013+)