Skip to content
TheChatArchiveThe Chat Archive
NVIDIAUnspecified NVIDIA model (HelpSteer2)· NVIDIANVIDIA NVIDIA HelpSteer2

For context: I'm training a classification neural network and considering what options…

  1. User

    For context: I'm training a classification neural network and considering what options to try to deal with the class imbalance in the dataset. What do you think about the following ideas and suggestions: Ideas

      1. More data
    • With the current classification classes in requirements (H, Fw, A, Unknown) we need airliners the most.
    • Could collect more airliner data e.g. near an airport with @tt s camera-on-stick idea (one (usual annotations) or two (experiment with getting gt distance from stereo) cameras). (Is there a task for it?)
    • Could probably even get data with desired pixel-size by choosing distance to airport accordingly.
    • My intuition is that background and weather would still be similarly diverse as in current data and if we take videos from different positions also the orientation should be diverse.
    • Can be combined with other ideas
      1. Upsampling, is also the idea in T9658
    • Pobably most common method
    • Also done by @mathias [[ https://git-ng.daedalean.ai/daedalean/exp-spotter/src/commit/534eea23086233f41d89add2b19b6510ca147b28/distsclasses/models/train_cnn.py#L413 | with pytorch's WeightedRandomSampler ]] (Any results on this?)
    • Should improve/change samplers for this as suggested in T9658.
    • But as mentioned by @ccc it can also already be done in TP, manually computing (based on datastats) and setting the weight of each class there. (How to do it exactly, we select the same datagroup multiple times, each time setting a different filter and different weight? Do we already have tried this in the past and have an example of it?)
      1. Give more weight in the loss to samples from rare classes
    • In theory almost equivalent to 2.
    • Could be adapted without changing TP (I consider it separate from sampling), e.g. when we want to give more weight to Unknown because it's less bad when the model wrongly predicts Unknown
    • As I understand training duration would not increase compared to 2 (assuming we keep number of epochs for majority class the same).
      1. Sim or ROA
    • Did not manage to improve distance performance much so far, so probably for classification also would not be easy to make work
    • For the idea of "Fixing class imbalance with sim/roa data for the minority class" I can not find an example in literature where this has been done successfully, so I would guess this generally does not work well
      1. Synthetic/augmented datapoints from real data for rare classes
    • e.g. existing augmentations or combining/mixing multiple crops into one
    • Basically just combines upsampling with augmentations, should handle separately I think
      1. Other
    • I think less common for a good reason: more complicated and less successful. But maybe worth a try at some point. I suggest: Definitely do 1. Additionally compare 2. with 3., they should give similar results but it would still be nice to do the comparison and they can both be done without too much effort, probably manually choosing weights for classes for now.
  2. NVIDIA Assistant

    Your ideas for addressing class imbalance in your dataset are well thought out and cover many common approaches. Here's my take on each idea:

    1. Collecting more data: This is generally a good idea, as it can help improve the performance of your model and provide more diverse examples for the minority class. Existing annotations can be reused, and you can experiment with different camera setups to ensure the data is diverse and of high quality.

    2. Upsampling: This is a common and effective method for handling class imbalance. You can apply it during training by oversampling the minority class or using a weighted random sampler. The suggested improvements for the sampler in T9658 are worth considering, as they can lead to more efficient training and better performance.

    3. Giving more weight in the loss to samples from rare classes: This approach is conceptually similar to upsampling but deals with the imbalance at the loss level rather than the sampling level. The primary benefit of this method is that you can fine-tune the contribution of each class to the total loss, providing more flexibility in handling class imbalance.

    4. Sim or ROA: While you haven't seen many successful examples in literature, combining simulation/ROA data with real data could be an option if you can generate realistic and diverse synthetic samples. However, since you haven't had much success improving distance performance so far, it may not be the most promising approach.

    5. Synthetic/augmented datapoints from real data: This combines upsampling and augmentations, which can be helpful in some cases. Existing augmentations can be used, or you can develop novel augmentation techniques specific to your problem. Keep in mind that overly aggressive augmentations might lead to overfitting and negatively impact model performance.

    6. Other methods: While it's crucial to consider unconventional approaches, simpler solutions like upsampling and giving more weight to rare classes are often effective and easier to implement. However, it's worth exploring other methods if you have the resources and time to do so.

    Based on your suggestions, I would recommend starting with ideas 1, 2, and 3. While they are similar, comparing the results of upsampling and weighting the loss could give you valuable insights when working with class imbalance. Implementing these changes without too much effort is feasible, and manually choosing weights for classes is a reasonable starting point.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.