Hello Apache Spark | Installing & Setup for Development and learning on MacOSX

Install Scala, Apache Spark was written in Scala. I am unsure if this is needed but I installed it anyways.

brew install scala

Install Apache Spark

brew install apache-spark

To run Spark shell.

# Scala
Spark-shell

# Python
pyspark

Adding some paths for shell to ease usage. I use OhMyZsh follow your shells directives.

vi ~/.zshrc

export SPARK_VERSION=3.0.1
export SPARK_HOME="/usr/local/Cellar/apache-spark/$SPARK_VERSION/libexec"
export PYTHONPATH="/usr/local/Cellar/apache-spark/$SPARK_VERSION/libexec/python/:$PYTHONP$"

alias sparkpath="cd $SPARK_HOME/sbin"

Reload

source ~/.zshrc

We did some of this to make it easier to run some of Sparks dashboards.

In one terminal run Spark in another run:

sparkpath
./start-all.sh

You should now be able to bring up

http://localhost:8080 and http://localhost:4040

This should get you on your way to learning and playing with Spark.