Friday, November 17, 2017

Using ipython in pyspark

Here is the link for ipython installation: http://www.dbaglobe.com/2017/11/install-ipython-on-centos7-redhat-el-7.html


If you use Spark < 1.2 you can simply execute bin/pyspark with an environmental variable IPYTHON=1.

IPYTHON=1 /usr/bin/pyspark


or

export IPYTHON=1
/usr/bin/pyspark

While above will still work on the Spark 1.2 and above recommended way to set Python environment for these versions is PYSPARK_DRIVER_PYTHON


PYSPARK_DRIVER_PYTHON=ipython /usr/bin/pyspark

or

export PYSPARK_DRIVER_PYTHON=ipython
/usr/bin/pyspark

image