Noob question! Of these three categorical distributions, do you agree that the leftmost is clearly bimodal, and the middle and rightmost are clearly unimodal? What quantity measures this? Entropy and Gini fail me here. @neu_rips help with this information theory stuff :)
@roydanroy @neu_rips let's say I have lots of data and know which ones have multiple possible answers vs just one, but don't know the answers or how many. I also have many predictions, and want to quickly identify which ones are making reasonably multi-modal predictions where necessary.
@giffmana @roydanroy @neu_rips > i know which have mult poss answers vs just one you could learn a classifier (e.g. linear probe) that takes in an output representation from your model and predicts if there were one or multiple answers. make "reasonableness of multimodal predictions" into a learnable task
@DigThatData @roydanroy @neu_rips Yup, that's a good approach, but linear wouldn't work, it's a set-based input and definitely a non-linear function that I want. Overall, making this almost a whole project on its own - can't afford that time for what I'm doing with it, and will prefer a closed-form heuristic.
@giffmana @roydanroy @neu_rips makes sense. really stupid simple heuristic you could use would be to sort the likelihoods in descending order and use some function of the gap between the dominant and second most dominant. maybe ratio of that delta over delta between most and least dominant