I don't think the size slows your phone, the only thing can cause problems is how many files you have on it.
If you fill it up with 20000000 little pictures and themes, I think it can slow down the phone, because it has too many files to read, but else - no problem what so ever!