Performance Test

I. Software Quality Assurance

The performance test suite on lexAccess was set up since 2007 version. JavaDb is added and test its performance in 2008. Due to the slow performance, Java DB is dropped in 2009. This test suite tests on:

  • Performance
  • Quality assurance

Below are the detail descriptions on lexAccess Performance test:

I. Test Procedures

  • create directory ${TEST}/LA/PerformanceTest/data/${YEAR} 
    shell> cd ${TEST}/LA/PerformanceTest/data
shell> mkdir 2012
  • link 5 files from $TEST/LVG/PerformanceTest/data/${YEAR}/Input
    • test.in.100
    • test.in.10000
    • test.in
    • lexiconTerms.data
    • meta.data 
    shell> cd ${TEST}/LA/PerformanceTest/data/2012
shell> mkdir Input
shell> cd Input
shell> ln -sf /export/home/lu/Test/LVG/PerformanceTest/data/2012/Input/test.in.100 test.in.100
shell> ln -sf /export/home/lu/Test/LVG/PerformanceTest/data/2012/Input/test.in.10000 test.in.10000
shell> ln -sf /export/home/lu/Test/LVG/PerformanceTest/data/2012/Input/test.in test.in
shell> ln -sf /export/home/lu/Test/LVG/PerformanceTest/data/2012/Input/lexiconTerms.data lexiconTerms.data 
shell> ln -sf /export/home/lu/Test/LVG/PerformanceTest/data/2012/Input/meta.data meta.data
  • update top LA directory in the config file. lexAccess.property.hsql
  • update ${YEAR} in ${TEST}/LA/PerformanceTest/data/${YEAR}/Input/script/testCaseAll
  • update ${TEST}/LA/PerformanceTest/bin/PerformanceTest (if needed)
  • update ${TEST}/LA/PerformanceTest/bin/testAll (if needed)
  • run ${TEST}/LVG/PerformanceTest/bin/testAll or
  • run ${TEST}/LVG/PerformanceTest/bin/PerformanceTest (just for 1 DB, HSqlDb)

    The test can be performed on software versions and year of data.

II. Quality assurance

Compare the results for all above 5 files for different database, such as HSqlDb and JavaDb. The results of these two databases are under:

  • ${TEST}/LA/PerformanceTest/data/${YEAR}/Output_HSql
  • ${TEST}/LA/PerformanceTest/data/${YEAR}/Output_JavaDb

III. Performance Test results

ReleaseTest Platform10000 RecordsTest: 1462202 Records
(2002 Metathesaurus)		Meta: 4461783 records
(2007 Metathesaurus)
2007
  • Red Hat 2.4.9-11, Pentium 3.4 GHz, 2 Gb Ram
  • HSqlDb 1.8.0.4
  • Java 1.5.0.7
  • 2.571u 0.130s 0:02.72 99.2%
  • 138.254u 1.406s 2:19.64 100.0%
  • 248.153u 2.950s 4:11.11 99.9%
2008
  • Red Hat 2.4.9-11, Pentium 3.4 GHz, 2 Gb Ram
  • HSqlDb 1.8.0.7
  • Java 1.5.0.11
  • HSqlDb: 2.706u 0.264s 0:03.01 98.3%
  • JavaDb: 51.643u 0.496s 0:52.22 99.8%
  • HSqlDb: 156.052u 15.516s 2:52.44 99.4%
  • JavaDb: 7404.864u 48.967s 2:06:26.01 98.2%
  • HSqlDb: 256.782u 7.796s 4:24.62 99.9%
  • JavaDb: 23877.130u 127.545s 6:53:30.46 96.7%
2009
  • Red Hat 4, 2.6.9-78, Intel-Xeon 3.8 GHz, 4 Gb Ram
  • HSqlDb 1.8.0.10
  • Java 1.6.0.06
  • 2.703u 0.273s 0:02.66 111.6%
  • 149.917u 18.233s 2:47.71 100.2%
  • 231.700u 8.118s 3:59.36 100.1%
2010
  • Red Hat 5.4, 2.6.18-164.6.1.el5, Intel-Xeon 5200MHz, 4 Gb Ram
  • HSqlDb 1.8.0.10
  • Java 1.6.0.14
  • 3.992u 0.158s 0:02.48 166.9%
  • 54.367u 7.904s 0:58.82 105.8%
  • 84.191u 3.828s 1:25.36 103.1%
2011
  • Red Hat 5.5, 2.6.18-194.8.1.el5, Intel-Xeon 3333MHz, 4 Gb Ram
  • HSqlDb 2.0.0
  • Java 1.6.0.21
  • 5.849u 0.131s 0:03.12 191.3%
  • 47.596u 2.718s 0:44.66 112.6%
  • 109.171u 3.011s 1:46.38 105.4%
2012
  • Red Hat 5.7, 2.6.18-274.7.1.el5, Intel-Xeon 3333MHz, 4 Gb Ram
  • HSqlDb 2.2.5
  • Java 1.6.0.26
  • 5.483u 0.136s 0:03.20 175.3%
  • 48.678u 2.668s 0:50.61 101.4%
  • 111.946u 3.054s 1:49.86 104.6%
2013
  • lexdev: Red Hat 6.7 (Santiago) CPUE5506@2.13GHz, 48 Gb Ram
  • HSqlDb 2.2.8
  • Java 1.7.0.67
  • 10.141u 0.448s 0:02.95 358.6%
  • 88.905u 4.102s 0:58.50 158.9%
  • 162.479u 7.090s 2:08.15 132.3%
2016
  • lexdev: Red Hat 6.7 (Santiago) CPUE5506@2.13GHz, 48 Gb Ram
  • HSqlDb 2.3.2
  • Java 1.8.0.45
  • 9.929u 0.492s 0:03.02 344.7%
  • 77.737u 4.595s 1:07.05 122.7%
  • 161.607u 7.121s 2:10.97 128.8%

From the results, we conclude:

  • The outputs of HSqlDb and JavaDb are the same
  • The performance of HSqlDb is about 1~2 magnitude faster than JavaDb