Input:$input output:$output tol:0.01 resetProp:0.15 version:edge-cut batchSize:1000 psPartitionNum:10 dataPartitionNum:10 In a situation where resources are really tight, try to increase the number of partitions! For example, a 10 billion edge set is about 160G in size, and a 20G * 20 configuration is sufficient. If the memory is tight, 1 times is also acceptable, but relatively slower. Spark的资源配置:The product of num-executors and executor-memory is the total configuration memory of executors, and it is best to store twice the input data.For PageRank, the calculation formula of the model size is: number of nodes * 3 * 4 Byte, according to which you can estimate the size of ps memory that needs to be configured under Graph input of different sizes In order to ensure that Angel does not hang, you need to configure memory about twice the size of the model. Angel PS number and memory: The product of ps.instance and ps.memory is the total configuration memory of ps.useBalancePartition:whether use balance partition, the default is false.storageLevel: storage level( details), the default is MEMORY_ONLY.batchSize:pull the results in batches when saving the node rank value, batchSize is the size of the batch, the default is 1000.version:edge network cutting method(edge-cut:cut by edge,vertex-cut:cut by vertex), the default is edge-cut.isWeight:whether the edge is weighted, the default is false.
0 Comments
Leave a Reply. |