By NetanelRabinowitz


2017-03-19 13:50:23 8 Comments

At the Spark 2.1 docs it's mentioned that

Spark runs on Java 7+, Python 2.6+/3.4+ and R 3.1+. For the Scala API, Spark 2.1.0 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).

at the Scala 2.12 release news it's also mentioned that:

Although Scala 2.11 and 2.12 are mostly source compatible to facilitate cross-building, they are not binary compatible. This allows us to keep improving the Scala compiler and standard library.

But when I build an uber jar (using Scala 2.12) and run it on Spark 2.1. every thing work just fine.

and I know its not any official source but at the 47 degree blog they mentioned that Spark 2.1 does support Scala 2.12.

How can one explain those (conflicts?) pieces of information ?

2 comments

@user7735456 2017-03-19 14:22:09

Spark does not support Scala 2.12. You can follow SPARK-14220 (Build and test Spark against Scala 2.12) to get up to date status.

update: Spark 2.4 added an experimental Scala 2.12 support.

@jjj 2017-03-19 15:15:38

could have been added as comment

@Maziyar 2018-11-20 11:52:55

Spark 2.4 now supports Scala 2.12 as an experimentally.

@George Hawkins 2019-04-07 14:11:19

2.12 support is no longer experimental - it's now GA - see Spark 2.4.1 release notes.

@denvercoder9 2017-09-22 18:08:46

To add to the answer, I believe it is a typo https://spark.apache.org/releases/spark-release-2-0-0.html has no mention of scala 2.12.

Also, if we look at timings Scala 2.12 was not released untill November 2016 and Spark 2.0.0 was released on July 2016.

References: https://spark.apache.org/news/index.html

www.scala-lang.org/news/2.12.0/

Related Questions

Sponsored Content

2 Answered Questions

[SOLVED] Submit job to spark with conflicting jackson dependencies?

1 Answered Questions

Cross-build scala API for various spark version

1 Answered Questions

spark-submit with Mahout error on cluster mode (Scala/java)

0 Answered Questions

Cant find class in uber jar

1 Answered Questions

Spark Streaming with Kakfa

  • 2017-07-17 17:18:59
  • Abhishek Modak
  • 137 View
  • 0 Score
  • 1 Answer
  • Tags:   scala

1 Answered Questions

[SOLVED] Unresolved dependency with xgboost library in scala

0 Answered Questions

spark's example can not build in intellij

6 Answered Questions

[SOLVED] Scala vs. Groovy vs. Clojure

18 Answered Questions

1 Answered Questions

[SOLVED] Run Spark in standalone mode with Scala 2.11?

Sponsored Content