Applying a modality-specific loss weight allows you to explicitly control the importance of each modality during training. By increasing the weight of the image loss, you encourage the model to pay more attention to the image information and generate captions that are more representative of the visual content. Increasing BERT's learning rate could worsen the imbalance. PCA is a data reduction technique not a balancing technique, freezing the weights on resnet 50, will not allow the network to learn about the relationship, batch size is to do with training speed not modality balance.