* Bigdata　配置

* Load performance

• Allie upload
• PDBJ upload
• Uniprot upload
• DDBJ upload

* Sparql query performance

• Allie query performance
• PDBJ query performance
• Uniprot query performance
• DDBJ query performance

Bigdata 配置

The journal in Bigdata (please refer to  http://sourceforge.net/apps/mediawiki/bigdata/index.php?title=StandaloneGuide for details.)

The WORM (Write-Once, Read-Many) is the traditional log-structured append only journal. It was designed for very fast write rates and is used to buffer writes for scale-out. This is a good choice for immortal databases where people want access to ALL history. Scaling is to several billions of triples.

The RW store (Read-Write) supports recycling of allocation slots on the backing file. It may be used as a time-bounded version of an immortal database where history is aged off of the database over time. This is a good choice for standalone workloads where updates are continuously arriving and older database states may be released. The RW store is also less sensitive to data skew because it can reuse B+Tree node and leaf revisions within a commit group on large data set loads. Scaling should be better than the WORM for standalone and could reach to 10B+ triples. The default property file is attachment:RWStore.properties .

In the test we modified the following two important parameters:

com.bigdata.btree.writeRetentionQueue.capacity=500000
com.bigdata.rdf.sail.BigdataSail.bufferCapacity=1000000

Load Performance

Approach 1:

Upload data from Bigdata sparql point(NanoSparqlServer?). Post the data every 10000 lines. Please refer to attachment:upload.pl for details.

Approach 2:

Allie upload

Approach 1: 26hours

Approach 2: 5.89hours when Setting JVM GC : 6.75hours

Approach 3: 2.61 hours

PDBJ upload

Result:

Uniprot upload

DDBJ upload

Sparql query performance

Allie query performance

PDBJ query performance

Uniprot query performance

DDBJ query performance