Most (halfway) decent DAPs have features in them no phone has - amplifiers that drive headphones a phone couldn't even touch, line output, dual card slots, optical outputs, balanced headphone jacks, etc.
But if you are talking one of those tiny little $20 ChiFi DAPs w/the 70hr battery life, then yeah I would just use your phone.
Also keep in mind that even though some of the better phones have decent dac chips in them, they are not purpose built for audio. Besides the radios and other electronics crammed in phones that could potentially cause EMI and other noise in the audio, decent DAPs always have better specs than any phone (noise, distortion, SNR, etc).
But it boils down to convenience and what works for you in any given situation. I went skating today, and wasn't going to lug my DAP + amp with me. I just streamed music from my iPhone to my KZ ZST via the KZ bluetooth cable. But when I'm at home or work I much prefer to use my DAP/amp with wired headphones/IEMs for best sound.
So to answer your question, I use both setups for various purposes (and definitely appreciate having both setups available).
Out of curiosity, what cheap DAPs have you compared to what smart phones to come to this conclusion?