You may not like it but your tree has a good idea - it is regulating what it can support. If the tree is young good crops are not often produced within the first 5 to 7 years. But even if you have an older tree fruit drop at this time is normal - in fact it is commonly called the June fruit drop. All citrus can experience three fruit drops or thinnings within a year.
Many small immature fruit drop during or shortly after flowering as they are being set on the limbs. The second drop occurs at this time of the year. The third drop is around September as the tree begins to mature the fruits.
Sometimes the fruit loss is excessive but the trees are going to thin out what cannot be supported from stored foods or due to growing conditions. Your job is to give the tree normal care and it will support what it can or maybe have a bumper crop next year.
Other fruit trees including mangos and avocado have years of fruit drop too. We don?t like it but the trees are better for it.