[cover] Improvements for small or homogeneous data
* The algorithm would bail as soon as it found one epoch that contained no new segments. Change it so it now has to fail >= 10 times in a row (10 for fastcover, 10-100 for cover). * The algorithm uses the `maxDict` size to decide the epoch size. When this size is absurdly large, it causes tiny epochs. Lower bound the epoch size at 10x the segment size, and warn the user that their training set is too small. Fixes #1554
This commit is contained in:
@@ -38,6 +38,35 @@ typedef struct {
|
||||
U32 score;
|
||||
} COVER_segment_t;
|
||||
|
||||
/**
|
||||
*Number of epochs and size of each epoch.
|
||||
*/
|
||||
typedef struct {
|
||||
U32 num;
|
||||
U32 size;
|
||||
} COVER_epoch_info_t;
|
||||
|
||||
/**
|
||||
* Computes the number of epochs and the size of each epoch.
|
||||
* We will make sure that each epoch gets at least 10 * k bytes.
|
||||
*
|
||||
* The COVER algorithms divide the data up into epochs of equal size and
|
||||
* select one segemnt from each epoch.
|
||||
*
|
||||
* @param maxDictSize The maximum allowed dictioary size.
|
||||
* @param nbDmers The number of dmers we are training on.
|
||||
* @param k The parameter k (segment size).
|
||||
* @param passes The target number of passes over the dmer corpus.
|
||||
* More passes means a better dictionary.
|
||||
*/
|
||||
COVER_epoch_info_t COVER_computeEpochs(U32 maxDictSize, U32 nbDmers,
|
||||
U32 k, U32 passes);
|
||||
|
||||
/**
|
||||
* Warns the user when their corpus is too small.
|
||||
*/
|
||||
void COVER_warnOnSmallCorpus(size_t maxDictSize, size_t nbDmers);
|
||||
|
||||
/**
|
||||
* Checks total compressed size of a dictionary
|
||||
*/
|
||||
|
||||
Reference in New Issue
Block a user