Check out example codes for "org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type". It will help you in understanding the concepts better.

Code Example 1

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")


Learn ReactJs, React Native from