If haki has a cancelling or restraining effect on devil fruits, then it could also have a similar effect on your own devil fruit, which would mean that one reason some powerful people don't eat devil fruits could likely be because that would cause them to be weak in both. Or else why who Linlin, Whitebeard, and Kaido seem weaker physically than when they were younger. Maybe?