Spark standalone configuration having multiple executors
WalkingDeadFan
I'm trying to setup a standalone Spark 2.0 server to process an analytics function in parallel. To do this I want to have a single worker with multiple executors.
I'm using :
- Standalone Spark 2.0
- 8 Cores
- 24gig RAM
- windows server 2008
- pyspark (although this appears unrelated)
This is just for pure proof of concept purposes but I want to have 8 executors, one per each core.
I've tried to follow the other threads on this topic but for some reason it's not working for me. IE: Spark Standalone Number Executors/Cores Control
My configuration is as follows:
conf\spark-defaults.conf
spark.cores.max = 8
spark.executor.cores = 1
I have tried to also change my spark-env.sh file to no avail. Instead what is happening is that it shows that my 1 worker only has 1 executor on it. As you can see below, it still shows the standalone with 1 executor with 8 cores to it.
zero323
I believe you mixed up local and standalone modes:
- Local mode is a development tool where all processes are executed inside a single JVM. Application is started in a local mode by setting master to
local,local[*]orlocal[n].spark.executor.coresandspark.executor.coresare not applicable in the local mode because there is only one embedded executor. Standalone mode requires a standalone Spark cluster. It requires a master node (can be started using
SPARK_HOME/sbin/start-master.shscript) and at least one worker node (can be started usingSPARK_HOME/sbin/start-slave.shscript).SparkConfshould use master node address to create (spark://host:port).
Collected from the Internet
Please contact [email protected] to delete if infringement.
edited at
Related
TOP Ranking
- 1
pump.io port in URL
- 2
How to import an asset in swift using Bundle.main.path() in a react-native native module
- 3
Failed to listen on localhost:8000 (reason: Cannot assign requested address)
- 4
Double spacing in rmarkdown pdf
- 5
SQL Server : need add a dot before two last character
- 6
C++ 16 bit grayscale gradient image from 2D array
- 7
JMeter: Why get error when try to save test plan
- 8
JWT gives JsonWebTokenError "invalid token"
- 9
How to make thrown errors visible outside of a Promise?
- 10
How to tell if iOS Today Widget is being updated in the background?
- 11
Calling Doctrine clear() with an argument is deprecated
- 12
Capybara Selenium Chrome opens About Google Chrome
- 13
How to update azerothcore-wotlk docker container
- 14
Adding Ripple Effect to RecyclerView item
- 15
mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on '127.0.0.1:3306' (111 Connection refused)
- 16
Error while applying filter on dataframe - PySpark
- 17
Unable to add slack to bluemix project
- 18
MyPy fails dataclass argument with optional list of objects type
- 19
How can I validate and parse phone numbers to extract their country calling code and area code?
- 20
Single Sign-On in Spring by using SAML Extension and Shibboleth
- 21
python how to create many-to-many of lists inside one list
Comments