Flash Sale:75% Off Hosting + Free DomainEnds in13h47m14sView Plans
Hostvento logoHostvento
Get Started

How to install Apache Spark on Ubuntu 18.04

This guide shows you how to install Apache Spark on an Ubuntu server and run a first test. It needs root access, so it fits VPS and dedicated servers.

How-To Guides2 min read15 steps
Tip: Ubuntu 18.04 is old. Newer Ubuntu versions use almost the same steps.

What is Apache Spark?

Spark is a tool that processes large amounts of data quickly. It splits big jobs into small pieces and works on them at the same time. Spark runs on Java, so you install Java first.

Steps

  1. Log in to your server over SSH as root or a user with sudo rights.
  2. Update your package list.
sudo apt update

This refreshes the list of software you can install.

  1. Install Java and the download tool wget.
sudo apt install -y default-jdk wget

JDK is the Java Development Kit. It lets your server run Java programs.

  1. Check Java works.
java -version
  1. Go to the Apache Spark download page on the official Spark website. Pick a release and copy the link to the .tgz file. Use that link in the next command. The link below is only an example of the shape.
wget https://downloads.apache.org/spark/spark-VERSION/spark-VERSION-bin-hadoop3.tgz

This downloads the Spark package. Replace VERSION with the real version number.

  1. Unpack the file.
tar xvf spark-VERSION-bin-hadoop3.tgz
  1. Move it to a permanent place.
sudo mv spark-VERSION-bin-hadoop3 /opt/spark
  1. Tell your system where Spark lives. Add these lines to the end of the ~/.bashrc file.
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

These are environment variables. They are named settings your system remembers.

  1. Load the new settings.
source ~/.bashrc

Test Spark

  1. Start the Spark shell with spark-shell.
  2. Wait for the scala> prompt.
  3. Type spark.version and press Enter. It prints the version.
  4. Leave the shell with :quit.

Start Spark as a service (optional)

  1. Run start-master.sh to start the main Spark process.
  2. Run start-worker.sh spark://YOUR-SERVER-NAME:7077. Use the address that start-master prints in its log. On older Spark versions the script is called start-slave.sh.

The master's web page uses port 8080. Do not open that port to the whole internet. It has no password by default.

Quick recap

  • Install Java first.
  • Download Spark from the official site and unpack it to /opt/spark.
  • Set SPARK_HOME and PATH in ~/.bashrc.
  • Test with spark-shell.