I used to use jupyter develop R or python application. I found spark can be developed in jupyter and I try to use jupyter to build spark application.
My environment is AWS EC2 ubuntu 14.04.

jupyrt_python

jupyrt_scala

  • Add spark kernel
    • wget http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.6.tgz
    • tar -xvf  spark-2.0.0-bin-hadoop2.6.tgz
    • export SPARK_HOME=”/home/ubuntu/jupyter_spark/spark-2.0.0-bin-hadoop2.6″
    • cd /home/ubuntu/anaconda3
    • git clone https://github.com/apache/incubator-toree.git
    • cd incubator-toree
    • export APACHE_SPARK_VERSION=2.0.0
    • Need sbt
    • Setting JAVA_HOME 
    • make build
      • see “success” means success
    • make dist
    • There is a run.sh in /home/ubuntu/anaconda3/incubator-toree/dist/toree/bin.
    • cd /root/.ipython/kernel/
    • mkdir spark
    • vim kernel.json

json

  • Check jupyter kernel contain spark

spark

  • Start to use jupyter develop spark application

final

Reference

http://spark.apache.org/downloads.html

http://m.blog.csdn.net/article/details?id=51006075