Maybe devil fruits have unlimited potential power, but the output is limited by the physique, so you have a lot of people who only rely on their devil fruits, and are kind of trashy, then you have people who lift, who are strong, and then there's innate freaks born with ungodly power and they still lift, so they reach the pinnacle of the world, but they could probably go even further if they had time to lift more