Tools to Generate LEXICON

This page include the tools under ${LEXICON}/bin. Some other tools from ${LEXBUILD}/tools/ are used during the LEXICON generation to check Cross-reference, Irreg base, TradeMark, etc. are not included here.

I. FinalizeLexicon

  • This tool modifies the LEXICON.freeze to LEXICON.release. It removes annotation, signature & fixes incompliant non-ASCII characters.
  • input: LEXICON.freeze
  • output: LEXICON.release
  • To run:
    shell> FinalizeLexicon <year>
  • Some manual modification might be needed after this. Such as trim extra empty space, correct typo in non-ASCII Unicode, etc..

II. ValidateLexicon

  • This tool validates syntax and Eui of LEXICON.release
  • input: LEXICON.release
  • To run:
    shell> ValidateLexicon <year>
  • Check if there error messages on the screen

III. GenerateTables

  • This tool automatic generates Lexicon tables
  • input: LEXICON.release
  • outputs: table files under tables directory
  • To run:
    shell> GenerateTables <year>
  • Check error messages on the screen and modify LEXICON.release accordingly. Then, repeat the procedures.

IV. ReleaseLexicon

  • This tool generates and copies Lexicon files to release directory
  • To run:
    shell> ReleaseLexicon <year>
  • Make sure the lvg release is the latest
  • Make sure the lvg/data/tables/derivation.data is updated because it is used as ./LEX_DB/DM.DB
  • Go through every single file to ensure the file are correct

V. Other Tools

  • Location: "${LEXICON}/bin"

    Program NamesUsageShort DescriptionNotes
    CompareTablesTableTest <year> Test all generated tablesRun it at the end to compare data
    GenerateLexiconTermsGenerateLexiconTerms <year>Generate all unique terms in LEXICON 
    GenerateStatisticsGenerateStatistics <year>Generate statistics for a given LEXICONInput: ${LEXICON}/data/${YEAR}/data/LEXICON.release
    GetUniqueEuiGetUniqueEui <field number> <inFile> <outFile> Generate unique EUI from a specified field in a file