Cut the Clutter : Inside Batch Apex

If you're even remotely familiar with the force.com development practices, you know what apex batches are what they are used for. Running resource-intensive processes on a large volume of data is non-negotiable for a businesses and Salesforce understands this to it's core, which is why it has deployed this powerful asynchronous framework that gets the job done with state-of-the art efficiency.

In this article, i'll try to skip the boring theoretical definitions (there's a ton of them, a google search away!). The goal is to be able to visualise what runs behind the scenes and how the batch framework works.

The chase

You already know, an apex batch class needs to implement the Database.Batchable interface.

If you also don't already know what an interface is, they are like a haunted house which contains body-less souls (interface methods). These souls (methods) do not have a body, are lifeless (empty) by definition, and any one who implements the interface (buys the haunted house, say) needs to necessarily provide a body(definition) to these souls!

Database.Batchable offers three interface methods and we'll talk about them at length, using visual aids. We will, although, keep our discussion brief for the finish method, given how excessively boring it is 🤷🏽♂️ :

Start method

public (Database.QueryLocator) start(Database.BatchableContext bc) {
 
final string QUERY = 'SELECT (FIELDS) FROM (OBJECT) WHERE (IDK)';
return Database.getQueryLocator(QUERY);

}

Start method is first in the line to get executed in a batch run. It's responsible to load the relevant data from the database, based on the query provided. It only runs once in a batch run, and returns a Database.QueryLocator. Think of this QueryLocator as a Bookmark, for now, while the data returned by our SOQL is the book where it's present.

It's imperative to understand the elegant mechanism which the start method uses to retrieve data in large volumes. Suppose, our query is bound to return 50,000 records. Start method does not simply retrieve the entire lot in one go. In fact, it retrieves data in smaller manageable portions, called chunks, and the size of a single chunk is called the retrieveChunkSize.

Note: Number of records returned in a chunk cannot exceed the determined retrieveChunkSize .

But how does system determine this retrieveChunkSize? Well, remember the batch-size? the value of the retrieveChunkSize depends on your batch size in a rather peculiar fashion. There are only three possible values for retrieveChunkSize: 100, 400, and 2000. Which retrieveChunkSize will your batch run use? Here is how it's calculated:

0 < BatchSize <100 : RetrieveChunk size = 100

101 < BatchSize <400 : RetrieveChunk size = 400

401 < BatchSize <2000 : RetrieveChunk size = 2000

In other words and for example, if your start method query is supposed to fetch 50,000 records and your batch size is 1000, the RetrieveChunk size is 2000, which basically means that the data will be delivered to the batch in a total of 25 chunks of 2000 records each. Like how someone would payback a loan of 50,000 they took from a bank, in 25 instalments of 2000 rupees each! (Excuse the interests paid)

Our beloved Database.QueryLocator is the one who makes this possible. Remember when we assumed it to be a bookmark? Well, it quite literally is one! A Database.QueryLocator class instance basically comprise of two things:

• iterator() : Returns a QueryLocatorIterator (The bookmark)
• getQuery() : Returns the exact query you passed in the start method.

To load the first chunk of records, say in our example, the first 2000 records will be brought to the start method. A new Database.QueryLocator object is created in the system.

Database.QueryLocator dbql = new Database.QueryLocator();

An SOQL query is made to retrieve the first 2000 records, and after picking up these records, QueryLocator's iterator is initialised with the location of the 2001st record, so the system remembers from where it would need to pick records for the next chunk, exactly like a bookmark!

Take a look at this illustration:

If you can see past my despairing drawing skills, you'll be able to visualise how Database.QueryLocator works in principle. The iterator method keeps a thumbtack over the location next to the last record of the previous chunk. The batch framework uses a method called queryMore(Database.QueryLocator dbql) to retrieve the subsequent chunks of records.

The queryMore method receives a QueryLocator object from the batch framework to know exactly from where it needs to start collecting the records for the next chunk (using dbql.iterator()) and what exact query needs to be run (using dbql.getQuery()) to get those records!

Take a moment to visualise this too:

Now that we know how records are being retrieved from the database in chunks by the start method, let's see what it does with these chunks.

The Execute method

Start methods sends a single chunk to the execute method, to run the execute logic over it. But execute method has its commitments, it cannot process records more than the specified batch size, in one go! In our example, start method offers the first chunk of 2000 records to the execute method. Execute method further divides this chunk into two sub-chunks of 1000 records each, since our Batch Size is 1000.

These sub-chunks created by the execute method are called executeChunks (rightly so), and the size of each of these chunks is always equals or less than the Batch size, but never more!

Now to my favourite interview question of all time: How many times does the execute method run in a batch execution? We can answer this now, for once and for all: it's the total number of executeChunks created by execute method, from all the retrieveChunks it received from the start method. Here's the simple calculation of our example:

• Number of records: 50,000
• Batch size: 1000
• retrieveChunk Size: 2000 [ Batch size between 401 - 2000, retrieveChunk size: 2000 ]
• Number of retrieveChunks created by start method: 50,000 / 2000 = 25

With batch size = 1000, execute method needs to run twice to consume 1 retrieveChunk of 2000 records. As a result, in our example, the execute method runs for a total of 50 times! [ 2 times for every retrieveChunk x 25 retrieveChunks]. This is also the total number of executeChunks created by execute method.

Note: Every execute method run shows up as a separate job on the Apex Job page.

Which means, there'll be 50 apex jobs queued up in the apex job page for our example batch execution. Try to visualise now, how a single retrieveChunk in our example would get executed:

The Finish Method

Last and also the least interesting, the finish method is like a wrap-it-up method that runs once, at the end, when all the Apex Jobs created by the execute method gets executed. You can do many post-processing activities like sending success/failure reports, batch run acknowledgement to admins, or to even run another batch (chaining batches).

Inference

Asynchronous Apex is a highly sophisticated framework that allows operations over millions of records, while taking advantage of relaxed governor limits. You can exploit the capabilities of this framework to the full, if you understand the way Salesforce get these processes handled in it's backyard!

Quick question, to validate our understanding : If a query is supposed to return 15,000 records, and the batchSize is 75, how many times would the execute method run? Comment your answers down in the comment section.

And again, thanks for reading! 🥂