Skip to content

Spark related parameters

To import data, you must set parameters for Spark. This table lists some generally-used parameters. For more Spark-related parameters, see Apache Spark documentation. For more information, see the examples.

Parameters Default Data type Required? Description Spark Writer string No Specifies the name of the Spark Driver Program.
spark.driver.cores 1 int No Specifies the number of cores to use for the driver process, only in cluster mode.
spark.driver.maxResultSize 1G string No Specifies the limit of the total size of serialized results of all partitions for each Spark action (e.g. collect) in bytes. Should be at least 1M, or 0 for unlimited.
spark.cores.max None int No When the driver program runs on a standalone deployed cluster or a Mesos cluster in "coarse-grained" sharing mode, the maximum amount of CPU cores to request for the application from across the cluster (not from each machine). If not set, the default will be spark.deploy.defaultCores on the standalone cluster manager of Spark, or infinite (all available cores) on Mesos.

Last update: April 8, 2021