How to Install Spark on Ubuntu

Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.

In this article, we will cover the installation procedure of Apache Spark on the Ubuntu operating system.

Prerequisites

This guide assumes that you are using Ubuntu and Hadoop 2.7 is installed in your system.

System requirements

Prerequisites

Before installing Spark ensure that you have installed Java8 in your Ubuntu Machine. If not installed, please follow below process to install java8 in your Ubuntu System.

a. Install java8 using below command.

sudo apt-get install oracle-java8-installer

Above command creates java-8-oracle Directory in /usr/lib/jvm/ directory in your machine. It looks like below

Now we need to configure the JAVA_HOME path in .bashrc file.

.bashrc file executes whenever we open the terminal.

b. Configure JAVA_HOME and PATH  in .bashrc file and save. To edit/modify .bashrc file, use below command.

Then press i(for insert) -> then Enter below line at the bottom of the file.

Below is the screen shot of that.

Then Press Esc -> wq! (For save the changes) -> Enter.

c. Now test Java installed properly or not by checking the version of Java. Below command should show the java version.

java -version

Below is the screenshot

Go to the below official download page of Apache Spark and choose the latest release. For the package type, choose ‘Pre-built for Apache Hadoop’.

https://spark.apache.org/downloads.html

The page will look like below

Or You can use a direct link to download.

https://www.apache.org/dyn/closer.lua/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz

Create a directory called spark under /usr/ directory. Use below command to create spark directory

Above command asks password to create spark directory under the /usr directory, you can give the password. Then check spark directory is created or not in the /usr directory using below command

It should give the below results with ‘spark’ directory

Go to /usr/spark directory. Use below command to go spark directory.

cd /usr/spark

Download spark2.3.3 in spark directory using below command

wget https://www.apache.org/dyn/closer.lua/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz

If use ll or ls command, you can see spark-2.4.0-bin-hadoop2.7.tgz in spark directory.

Then extract spark-2.4.0-bin-hadoop2.7.tgz using below command.

sudo tar xvzf spark-2.4.0-bin-hadoop2.7

Now spark-2.4.0-bin-hadoop2.7.tgz file is extracted as spark-2.4.0-bin-hadoop2.7

Check whether it extracted or not using ll command. It should give the below results.

Configure SPARK_HOME path in the .bashrc file by following below steps.

Go to the home directory using below command

Open the .bashrc file using below command

Now we will configure SPARK_HOME and PATH

press i for insert the enter SPARK_HOME and PATH  like below

SPARK_HOME=/usr/spark/spark-2.4.0-bin-hadoop2.7

PATH=$PATH:$SPARK_HOME/bin

It looks like below

Then save and exit by entering below commands.

Press Esc -> wq! -> Enter

Now we can verify spark is successfully installed in our Ubuntu Machine or not. To verify use below command then enter.

spark-shell 

Above command should show below screen

Now we have successfully installed spark on Ubuntu System. Let’s create RDD and Dataframe then we will end up.

a. We can create RDD in 3 ways, we will use one way to create RDD.

Define any list then parallelize it. It will create RDD. Below are the codes. Copy paste it one by one on the command line.

Above will create RDD.

b. Now we will create a Data frame from RDD. Follow the below steps to create Dataframe.

Above code will create Dataframe with num as a column.

To display the data in Dataframe use below command

Below is the screenshot of the above code.

You can follow the below steps to uninstall spark on Windows 10.

To remove SPARK_HOME variable from the .bashrc please follow below steps

Go to the home directory. To go to home directory use below command.

Open .bashrc file. To open .bashrc file use below command.

Press i for edit/delete SPARK_HOME from .bashrc file. Then find SPARK_HOME the delete SPARK_HOME=/usr/spark/spark-2.4.0-bin-hadoop2.7 line from .bashrc file and save. To do follow below commands

We will also delete downloaded and extracted spark installers from the system. Please do follow below command.

Above command will delete spark directory from the system.

Open Command Line Interface then type spark-shell,  then press enter, now we get an error.

Now we can confirm that Spark is successfully uninstalled from the Ubuntu System. You can also learn more about Apache Spark and Scala here.

Research & References of How to Install Spark on Ubuntu|A&C Accounting And Tax Services
Source

Share on Facebook

Tweet

Follow us