Weblogic server takes long time to start. Performance tuning of Weblogic server startup.
Cause:
It is observed on some Linux boxes that WebLogic server
startup takes several minutes and hangs for a while. Similar behavior happens during the domain
creation, when the security information gets populated.
It has also been observed WLS Admin Console slowness due to
low entropy.
Linux has two devices to provide random data at any
time: /dev/random and /dev/urandom. Both ways should be secure
enough to use them in generating PGP keys, ssh challenges, and other applications
where secure random numbers are required. Starting on kernel 2.6, default
entropy is 4096 bits and problem arises when the entropy available on the
system is minimum (around 100 bits or less).
The main difference between those two devices is that /dev/random
runs out of random bits and makes you wait for more to be accumulated. Note
that on some systems, it can block for a long time waiting for new
user-generated entropy to be entered into the system.
In terms of the outcome, /dev/random is categorized as a high quality entropy device if we compare it with /dev/urandom. The latter uses the entropy pool as long as it is available, but falls back on pseudo random numeric algorithms when depleted.
In terms of the outcome, /dev/random is categorized as a high quality entropy device if we compare it with /dev/urandom. The latter uses the entropy pool as long as it is available, but falls back on pseudo random numeric algorithms when depleted.
Why a system could be
running out of entropy?
You have to consider that an Operating System performs
cryptographic operations frequently (on ssh challenges, https connections,
etc.) so the /dev/random pool gets consumed quite quickly. OS also
expects to feed that pool with I/O operations coming from disk, network, mouse
or keyboard but that situation does not happen as quickly. This is a common
pattern on virtualized environments or headless boxes.
Is important to mention that Java uses /dev/random by
default as entropy generator device.
How to verify if you are encountering this issue?
- Check the
default system entropy.
$ cat
/proc/sys/kernel/random/poolsize
4096
4096
- Check the
available entropy.
$ cat
/proc/sys/kernel/random/entropy_avail
125
125
On previous example, entropy is too low.
- Monitor the
current entropy of the system by using the following command:
$ for i in $(seq 500); do cat
/proc/sys/kernel/random/entropy_avail ; sleep 5; done
- Start a
WebLogic server instance. You should see that entropy decreases or stalls.
Solution:
a) WebLogic Server Scope
i.
Edit the Weblogic startup script ($DOMAIN_HOME/bin/startWebLogic.sh)
ii. Add the following to the JAVA_OPTIONS variable: -Djava.security.egd=file:/dev/./urandom
iii. Save the file.
iv. Set the domain environment. ($DOMAIN_HOME/bin/setDomainEnv.sh)
v. Start WebLogic instances.
ii. Add the following to the JAVA_OPTIONS variable: -Djava.security.egd=file:/dev/./urandom
iii. Save the file.
iv. Set the domain environment. ($DOMAIN_HOME/bin/setDomainEnv.sh)
v. Start WebLogic instances.
b) JDK Scope
i. Edit the Java Security Properties file ($JAVA_HOME/jre/lib/security/java.security)
ii. The securerandom.source property specifies the source of seed data for secure random. If that property points to /dev/random, set it as one of the options listed below.
securerandom.source=file:/dev/./urandom
i. Edit the Java Security Properties file ($JAVA_HOME/jre/lib/security/java.security)
ii. The securerandom.source property specifies the source of seed data for secure random. If that property points to /dev/random, set it as one of the options listed below.
securerandom.source=file:/dev/./urandom
securerandom.source=file:/dev/urandom
iii.
Save changes and start the WebLogic Server instances.
Temporary solution :
i.
Override the JAVA_OPTIONS environment variable before starting WebLogic Server
via shell scripts.
$ export JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.security.egd=file:/dev/./urandom"
$ export JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.security.egd=file:/dev/./urandom"
ii.
Start WebLogic instances.
Happy Learning!
