Goals

Today, only a single JVM can exist and run behind a ZSO. This JVM is started when a request comes in, and it is stopped whenever one of the recycle parameters are reached.

This has a couple of performance characteristics that are undesirable:

• While the JVM is recycling, no requests can be processed.
• Only a single JVM can be used for a single application, limiting scaling on multiple processors / cores

With this design, we are hoping to improve performance by allowing >1 JVM behind a single ZSO. This will allow requests to continue to be serviced when 1 jvm is recycling.

Challenges

There are a couple of challenges that must be met by this design

• The ZSO should stay simple, and should not look at the data contained in the communication. It should continue to only start JVMs when required because a request is pending, and should not look at the traffic on the wire.
• All JVMs should answer through a single ZSO, on a single port. This, combined with the first requirement, mean session affinity is out, and the jvm that answers the request will be random from which ever one does the accept for that request.
• The GC zones that have persistent data in them must be accessible by all of the JVMs -- and the data has to be consistent across the application.
• The solution must require little or no extra configuration by the user to enable
• The solution shouldn't require the opening up of lots of additional TCPIP ports by the application.

Proposal

ZSO

The ZSO will be watch the connection queue of the server socket. When the queue depth reaches certain levels, then a new JVM will be launched behind the ZSO. It will be handed the same ServerSocket as the first JVM. Both JVMs will accept connections and service requests. The one that handles any specific request will be random.

GC behavior

For any set of JVMs behind a ZSO, they must somehow share a single GC view for those zones that require it (/app, /storage, /user) There are different ways this could be handled -- file sharing, memcached, object grid. Here is the proposal:

For each set of JVMs, 1 JVM will be designated the "Master". For any GC action (get, put) to a zone that that requires consistency across the entire application, the app will see if its JVM is currently the "Master" -- if it is not, then the GC action will be forwarded to the JVM that is the current Master. The result of the action (which may be cached locally) will be returned to the application.

This allows a single JVM in the set of JVMs to manage the stores for the zone -- the saving and loading of GC. No file locking or complicated 3rd party processes / configuration will be required. While only 1 JVM at a time will be considered the "Master", all JVMs will be able to service requests.

JVM to JVM communication / IPC

The standard way for 2 JVMs to communicate is via Sockets. But in this case, it is not possible: one JVM can not communicate with another specific JVM through the HTTP port that is already opened -- because it is random which JVM will actually service the request. Also, our design guidelines state we don't want to open up additional network ports, or mess with extra configuration (each app would have to have their own port as to not conflict with other apps.

Instead, we will use Named Pipes to communicate between the Master JVM (holding the real persistent GC zones) and the slave JVMs (for which, any persistent GC zone action is forwarded to the Master JVM.

The named pipes library that has been created emulates sockets communication, with ServerPipe = ServerSocket and Pipe = Socket. The ServerPipe can bind(), meaning only 1 can exist. Only instead of a tcpip listen port -- it binds to a Name and directory location.

JVM startup protocol

The protocol for accessing a persistent GC zone will be the following:

• At JVM start up, the JVM will attempt to create and bind the Master JVM ServerPipe.
• if Bind succeeds, it is Master.
• If JVM is Master, then on persistent zone activation, it will load zone. all zones work as normal with single JVM.
• if JVM is Master, then a server thread will be started to handle requests on the ServerPipe. The requests will be from slave JVMs, for access to persistent GC zones.
• The remote calls into the Master GC will behave as if they were coming from the local app -- they are really no different
• On recycle or JVM shutdown, the Master JVM will persist the current GC zones, then close the ServerPipe
• if Bind fails, then JVM is a slave
• If JVM is slave, than any action on a persistent GC zone will be redirected via a Pipe connection to the Master JVM GC container.
• If GC action to Master fails (because the Master has shutdown) then the JVM will loop attempting: Become Master, by binding to ServerPipe, or connect to other JVM that has been able to bind. Which ever JVM wins, that JVM will load the GC contents from filesystem, then continue processing as Master JVM.

-- gregsmit - 17 Jun 2008