[java] Is there a common Java utility to break a list into batches?

I wrote myself a utility to break a list into batches of given size. I just wanted to know if there is already any apache commons util for this.

public static <T> List<List<T>> getBatches(List<T> collection,int batchSize){
    int i = 0;
    List<List<T>> batches = new ArrayList<List<T>>();
    while(i<collection.size()){
        int nextInc = Math.min(collection.size()-i,batchSize);
        List<T> batch = collection.subList(i,i+nextInc);
        batches.add(batch);
        i = i + nextInc;
    }

    return batches;
}

Please let me know if there any existing utility already for the same.

With Java 9 you can use IntStream.iterate() with hasNext condition. So you can simplify the code of your method to this:

public static <T> List<List<T>> getBatches(List<T> collection, int batchSize) {
    return IntStream.iterate(0, i -> i < collection.size(), i -> i + batchSize)
            .mapToObj(i -> collection.subList(i, Math.min(i + batchSize, collection.size())))
            .collect(Collectors.toList());
}

Using {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, the result of getBatches(numbers, 4) will be:

[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9]]

import com.google.common.collect.Lists;

List<List<T>> batches = Lists.partition(List<T>,batchSize)

Use Lists.partition(List,batchSize). You need to import Lists from google common package (com.google.common.collect.Lists)

It will return List of List<T> with and the size of every element equal to your batchSize.

Using various cheats from the web, I came to this solution:

int[] count = new int[1];
final int CHUNK_SIZE = 500;
Map<Integer, List<Long>> chunkedUsers = users.stream().collect( Collectors.groupingBy( 
    user -> {
        count[0]++;
        return Math.floorDiv( count[0], CHUNK_SIZE );
    } )
);

We use count to mimic a normal collection index.
Then, we group the collection elements in buckets, using the algebraic quotient as bucket number.
The final map contains as key the bucket number, as value the bucket itself.

You can then easily do an operation on each of the buckets with:

chunkedUsers.values().forEach( ... );

Here an example:

final AtomicInteger counter = new AtomicInteger();
final int partitionSize=3;
final List<Object> list=new ArrayList<>();
            list.add("A");
            list.add("B");
            list.add("C");
            list.add("D");
            list.add("E");
       
        
final Collection<List<Object>> subLists=list.stream().collect(Collectors.groupingBy
                (it->counter.getAndIncrement() / partitionSize))
                .values();
        System.out.println(subLists);

Input: [A, B, C, D, E]

Output: [[A, B, C], [D, E]]

You can find examples here: https://e.printstacktrace.blog/divide-a-list-to-lists-of-n-size-in-Java-8/

In case you want to produce a Java-8 stream of batches, you can try the following code:

public static <T> Stream<List<T>> batches(List<T> source, int length) {
    if (length <= 0)
        throw new IllegalArgumentException("length = " + length);
    int size = source.size();
    if (size <= 0)
        return Stream.empty();
    int fullChunks = (size - 1) / length;
    return IntStream.range(0, fullChunks + 1).mapToObj(
        n -> source.subList(n * length, n == fullChunks ? size : (n + 1) * length));
}

public static void main(String[] args) {
    List<Integer> list = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14);

    System.out.println("By 3:");
    batches(list, 3).forEach(System.out::println);
    
    System.out.println("By 4:");
    batches(list, 4).forEach(System.out::println);
}

Output:

By 3:
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
[10, 11, 12]
[13, 14]
By 4:
[1, 2, 3, 4]
[5, 6, 7, 8]
[9, 10, 11, 12]
[13, 14]

List<T> batch = collection.subList(i,i+nextInc);
->
List<T> batch = collection.subList(i, i = i + nextInc);

You can use below code to get the batch of list.

Iterable<List<T>> batchIds = Iterables.partition(list, batchSize);

You need to import Google Guava library to use above code.

The following example demonstrates chunking of a List:

package de.thomasdarimont.labs;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class SplitIntoChunks {

    public static void main(String[] args) {

        List<Integer> ints = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11);

        List<List<Integer>> chunks = chunk(ints, 4);

        System.out.printf("Ints:   %s%n", ints);
        System.out.printf("Chunks: %s%n", chunks);
    }

    public static <T> List<List<T>> chunk(List<T> input, int chunkSize) {

        int inputSize = input.size();
        int chunkCount = (int) Math.ceil(inputSize / (double) chunkSize);

        Map<Integer, List<T>> map = new HashMap<>(chunkCount);
        List<List<T>> chunks = new ArrayList<>(chunkCount);

        for (int i = 0; i < inputSize; i++) {

            map.computeIfAbsent(i / chunkSize, (ignore) -> {

                List<T> chunk = new ArrayList<>();
                chunks.add(chunk);
                return chunk;

            }).add(input.get(i));
        }

        return chunks;
    }
}

Output:

Ints:   [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Chunks: [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11]]

Use Apache Commons ListUtils.partition.

org.apache.commons.collections4.ListUtils.partition(final List<T> list, final int size)

Here is a simple solution for Java 8+:

public static <T> Collection<List<T>> prepareChunks(List<T> inputList, int chunkSize) {
    AtomicInteger counter = new AtomicInteger();
    return inputList.stream().collect(Collectors.groupingBy(it -> counter.getAndIncrement() / chunkSize)).values();
}

I came up with this one:

private static <T> List<List<T>> partition(Collection<T> members, int maxSize)
{
    List<List<T>> res = new ArrayList<>();

    List<T> internal = new ArrayList<>();

    for (T member : members)
    {
        internal.add(member);

        if (internal.size() == maxSize)
        {
            res.add(internal);
            internal = new ArrayList<>();
        }
    }
    if (internal.isEmpty() == false)
    {
        res.add(internal);
    }
    return res;
}

Another approach to solve this, question:

public class CollectionUtils {

    /**
    * Splits the collection into lists with given batch size
    * @param collection to split in to batches
    * @param batchsize size of the batch
    * @param <T> it maintains the input type to output type
    * @return nested list
    */
    public static <T> List<List<T>> makeBatch(Collection<T> collection, int batchsize) {

        List<List<T>> totalArrayList = new ArrayList<>();
        List<T> tempItems = new ArrayList<>();

        Iterator<T> iterator = collection.iterator();

        for (int i = 0; i < collection.size(); i++) {
            tempItems.add(iterator.next());
            if ((i+1) % batchsize == 0) {
                totalArrayList.add(tempItems);
                tempItems = new ArrayList<>();
            }
        }

        if (tempItems.size() > 0) {
            totalArrayList.add(tempItems);
        }

        return totalArrayList;
    }

}

Another approach is to use Collectors.groupingBy of indices and then map the grouped indices to the actual elements:

    final List<Integer> numbers = range(1, 12)
            .boxed()
            .collect(toList());
    System.out.println(numbers);

    final List<List<Integer>> groups = range(0, numbers.size())
            .boxed()
            .collect(groupingBy(index -> index / 4))
            .values()
            .stream()
            .map(indices -> indices
                    .stream()
                    .map(numbers::get)
                    .collect(toList()))
            .collect(toList());
    System.out.println(groups);

Output:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

[[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11]]

There was another question that was closed as being a duplicate of this one, but if you read it closely, it's subtly different. So in case someone (like me) actually wants to split a list into a given number of almost equally sized sublists, then read on.

I simply ported the algorithm described here to Java.

@Test
public void shouldPartitionListIntoAlmostEquallySizedSublists() {

    List<String> list = Arrays.asList("a", "b", "c", "d", "e", "f", "g");
    int numberOfPartitions = 3;

    List<List<String>> split = IntStream.range(0, numberOfPartitions).boxed()
            .map(i -> list.subList(
                    partitionOffset(list.size(), numberOfPartitions, i),
                    partitionOffset(list.size(), numberOfPartitions, i + 1)))
            .collect(toList());

    assertThat(split, hasSize(numberOfPartitions));
    assertEquals(list.size(), split.stream().flatMap(Collection::stream).count());
    assertThat(split, hasItems(Arrays.asList("a", "b", "c"), Arrays.asList("d", "e"), Arrays.asList("f", "g")));
}

private static int partitionOffset(int length, int numberOfPartitions, int partitionIndex) {
    return partitionIndex * (length / numberOfPartitions) + Math.min(partitionIndex, length % numberOfPartitions);
}

A one-liner in Java 8 would be:

import static java.util.function.Function.identity;
import static java.util.stream.Collectors.*;

private static <T> Collection<List<T>> partition(List<T> xs, int size) {
    return IntStream.range(0, xs.size())
            .boxed()
            .collect(collectingAndThen(toMap(identity(), xs::get), Map::entrySet))
            .stream()
            .collect(groupingBy(x -> x.getKey() / size, mapping(Map.Entry::getValue, toList())))
            .values();

}

Similar to OP without streams and libs, but conciser:

public <T> List<List<T>> getBatches(List<T> collection, int batchSize) {
    List<List<T>> batches = new ArrayList<>();
    for (int i = 0; i < collection.size(); i += batchSize) {
        batches.add(collection.subList(i, Math.min(i + batchSize, collection.size())));
    }
    return batches;
}