Quantcast
Channel: ZRJ
Viewing all articles
Browse latest Browse all 120

spark 算子理解和存储方式

$
0
0

对 combineByKey 的理解,参看,http://luojinping.com/2016/01/22/%E5%88%9D%E5%AD%A6Spark/

combineByKey应用举例
求均值
val rdd = sc.textFile(“气象数据”)
val rdd2 = rdd.map(x=>x.split(” “)).map(x => (x(0).substring(“从年月日中提取年月”),x(1).toInt))
val createCombiner = (k: String, v: Int)=> {
(v,1)
}
val mergeValue = (c:(Int, Int), v:Int) => {
(c._1 + v, c._2 + 1)
}
val mergeCombiners = (c1:(Int,Int),c2:(Int,Int))=>{
(c1._1 + c2._1, c1._2 + c2._2)
}
val vdd3 = vdd2.combineByKey(
createCombiner,
mergeValue,
mergeCombiners
)
rdd3.foreach(x=>println(x._1 + “: average tempreture is ” + x._2._1/x._2._2)

spark 和 hbase 的结合看这里,Spark算子:RDD行动Action操作(7)–saveAsNewAPIHadoopFile、saveAsNewAPIHadoopDatasetSpark算子:RDD行动Action操作(6)–saveAsHadoopFile、saveAsHadoopDatasetSparkSQL读取HBase数据,通过自定义外部数据源

另外,这一个系列文章也不错,标签:spark算子

这也不错,还保持着持续更新,lujinhong

各种算子的解释也看这里,样例有助于理解,spark常用transformation和action.html


Viewing all articles
Browse latest Browse all 120

Trending Articles


Girasoles para colorear


mayabang Quotes, Torpe Quotes, tanga Quotes


Tagalog Quotes About Crush – Tagalog Love Quotes


OFW quotes : Pinoy Tagalog Quotes


Long Distance Relationship Tagalog Love Quotes


Tagalog Quotes To Move on and More Love Love Love Quotes


5 Tagalog Relationship Rules


Best Crush Tagalog Quotes And Sayings 2017


Re:Mutton Pies (lleechef)


FORECLOSURE OF REAL ESTATE MORTGAGE


Sapos para colorear


tagalog love Quotes – Tiwala Quotes


Break up Quotes Tagalog Love Quote – Broken Hearted Quotes Tagalog


Patama Quotes : Tagalog Inspirational Quotes


Pamatay na Banat and Mga Patama Love Quotes


Tagalog Long Distance Relationship Love Quotes


BARKADA TAGALOG QUOTES


“BAHAY KUBO HUGOT”


Vimeo 10.7.0 by Vimeo.com, Inc.


Vimeo 10.7.1 by Vimeo.com, Inc.