Customizing pyspark app start up script

A nice feature of pyspark applications is that you don’t have to use spark-submit manually. Instead, when you instantiate a SparkContext instance, it will take care of running spark-submit ... pyspark-shell for you. Sometimes, however, you may want to customize that launch a bit. For instance, it is useful to tell spark-submit to include specific libraries in the classpath; which is in and of itself a pretty cool feature because you can provide Maven coordinates. ... Read more