§ How to Improve Evalauation Metrics

• Create a histogram to burn down on.
• When you start, you will probably make it via just printing errors, sorted in say lex order.
• This will be too large. Therefore, you will now collapse these into larger equivalence classes, giving you a rougher partition of the error space, with a smaller number of buckets.
• The difficulty is to find the sweet spot between too coarse and too fine in the number of buckets.
• Great, now that you have the histogram, you can now start working on the largest buckets first.