开发者

why mappartition does not see my val - SCALA/SPARK?

开发者 https://www.devze.com 2022-12-07 17:10 出处:网络
I define val like this : val config = Config(args) val product_type = config.product_type thenI send product_type as "AA"

I define val like this :

  val config = Config(args) 
  val product_type = config.product_type 

then I send product_type as "AA"

and my code is this :

  val scores = df.mapPartitions(iterator => {
    val inputStream =
      if(product_type == "AA" ) {
        getClass().getClassLoader().getResourceAsStream("my_aa.hdf5")
      }
      else {
        getClass().getClassLoader().getResourceAsStream("my_bb.hdf5")
      } 
    val multiLayerNetwork: MultiLayerNetwork = KerasModelImport.importKerasSequentialModelAndWeights(inputStream, false)
    val wrapped: ParallelInference = new ParallelInference.Builder(multiLayerNetwork).build()
    val res = iterator.map(row => {
      wrapped.output(row).toDoubleVector
    })
    res
  })

But my inputStream equals "my_bb.hdf5" which is not correct. This value comes from else statement开发者_开发知识库. So why my product_type variable cant read in mappartition?

I print my product_type value before code and I checked it , it is : "AA"

0

精彩评论

暂无评论...
验证码 换一张
取 消