| バージョン 4 (更新者: wu, 12 年 前) |
|---|
OwlimSE 配置
JVMSetting: -Xmx55G -Xms30G -XX:+UseG1GC -XX:+TieredCompilation -Druleset=empty -Dentity-index-size=1147483647 -Dcache-memory=16645m -Dtuple-index-memory=15G -DenablePredicateList=false -DftsIndexPolicy=never -Dbuild-pcsot=false -Dbuild-ptsoc=false -Djournaling=true -Drepository-type=file-repository -Dentity-id-size=32
More information please refer to http://owlim.ontotext.com/display/OWLIMv43/OWLIM-SE+Configuration
Load Performance
Approach 1: 'load' command in the Sesame console application, for files including less than one billion triples.
Owlim showed that they can not load a billion statements with Owlim in a large file with a load command.
Please refer to http://owlim.ontotext.com/display/OWLIMv40/OWLIM-SE+Administrative+Tasks: In general RDF data can be loaded into a given Sesame repository using the 'load' command in the Sesame console application or directly through the workbench web application. However, neither of these approaches will work when using a very large number of triples, e.g. a billion statements. A common solution would be to convert the RDF data into a line-based RDF format (e.g. N-triples) and then split it into many smaller files (e.g. using the linux command 'split'). This would allow each file to be uploaded separately using either the console or workbench applications.
Approach 2:
The idea is from uniprot, which uses owlim as an library as follows:
Basically They have one specific loader program, where there is one java thread that reads the triples into a blocking queue. Then a different number of threads take triples from that queue and insert the data into OWLIM-se (or any other sesame API compatible triplestore). Normally one inserting thread per owlim file-repository fragment. The inserter treads use transactions that commit every half a million statements. The basic is to add statements not files.
final org.openrdf.model.Statement sesameStatement = getSesameStatement(object);
//Takes one from the blocking queue filled by the other thread
connection.add(sesameStatement, graph);
and every millionth statement , do connection.commit();
(Please refer to https://github.com/JervenBolleman/sesame-loader/ for details)
Allie upload
PDBJ upload
Uniprot upload
DDBJ upload
