刚开始学习Spark遇到一个新手很容易遇到的问题,记录一下
Error:scalac: Scala compiler JARs not found (module SparkSql): C:\Users\***\.m2\repository\org\scala-lang\scala-compiler\2.11.8\scala-compiler-2.11.8.jar问题原因,项目未添加Scala类库…
HDP3使用apache版Spark2.4查询hive时报错:
Caused by: java.lang.RuntimeException: org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configurationat org.apache.hadoop.hive.ql.session…
WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
下面配置解决上述警告
方式1 1) 在HDFS上创建/spark-yarn/jars目录,并且将spark下的jar包上传到该目录 hadoop fs -mkdir -p /spark-yarn…