Overall Architecture

• Separation between frontend (experiment setup) and backend (experiment execution, possibly multiple machines)
• Backend: XCheck 0.2.0
• Frontend: Custom JSP codes to maintain document+query repository, create experiments, query experiment results
• Repository with data and metadata on engines, datasets, actual documents, queries (currently only partially exposed at the GUI level)
• Experiment data collection with XML experiment setup files, HTML run reports, XML result files and data plots
• Synchronization for experiment setup and experiment results between frontend and backends

Engine Setup

• Latest version available at publication time
• C++ engines: compilation with default optimization settings, 64 bit mode
• Default C++ provided with Redhat Enterprise 5.3: GCC 4.1.2
• Exception: Sedna - build is broken on 64 bit architecture ("checking for best nursery size"),precompiled binary version used
• Java: Precompiled JAR files
• Sun Java 1.6_06, run in 64 bit mode
• No additional tuning, out-of-the-box setup
• Adapter files adapted/developed as required
• Timing information extracted from engines where possible
• Document loading at run of each query set in the setup phase

Experiment Setup

• Currently one backend node
• Limit execution of a single query to 10 minutes - might be increased if necessary
• 4 repetitions of each query, last 3 measured (default setting of XCheck)
• Data store on first disk, input files on second disk/NFS

People

• Overall design, current maintainance: Peter Fischer
• Implementation of benchmarking environment, integration of engines, queries and data sets: Anand Singh
• Implementation of TPC-X queries: David Graf

Contact us