public class DistinctDataBag<E> extends SortedDataBag<E>
This data bag will gather distinct items in memory until a size threshold is passed, at which point it will write out all of the items to disk using the supplied serializer.
After adding is finished, call
iterator() to set up the data bag for reading back items and iterating over them.
The iterator will retrieve only distinct items.
IMPORTANT: You may not add any more items after this call. You may subsequently call
times which will give you a new iterator for each invocation. If you do not consume the entire iterator, you should
Iter.close(Iterator) to close any FileInputStreams associated with the iterator.
Additionally, make sure to call
SortedDataBag.close() when you are finished to free any system resources (preferably in a finally block).
Implementation Notes: Data is stored without duplicates as it comes in in a HashSet. When it is time to spill, that data is sorted and written to disk. An iterator that eliminates adjacent duplicates is used in conjunction with the SortedDataBag's iterator.
|Constructor and Description|
|Modifier and Type||Method and Description|
Find out if the bag is distinct.
Find out if the bag is sorted.
Returns an iterator over a set of elements of type E.
add, cancel, close, flush, isCancelled, isClosed
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
public boolean isSorted()
public boolean isDistinct()
Iter.close(Iterator)to be sure any open file handles are closed.
Licenced under the Apache License, Version 2.0