You are right that there are no direct trains to Perugia. But you can get an ES train, followed by a regional train with only a change in Florence. There are ES trains leaving Milano Centrale at 9am and 11am on Sundays with changes in Florence of about 20 minutes each. See
Trenitalia for full details.
Florence is a very easy station to change in, given that there are no stairs between platforms. This is the method I would recommend.
There is a direct bus from Milan to Perugia which runs over a Saturday night, arriving in Perugia at about midnight.
Here are the details, in case you hadn't already found it.
I am not sure about train and bus connections to the south. However I know there is a direct bus to Calabria, because my cousin's girlfriend takes that route regularly. It is very VERY long though. You would have to enquire at the bus depot in Perugia to find out the timetables.
In general, bus services in Italy are very localised and it is very hard to find out timetables and routes unless you enquire at the local bus depot. Even then, bus timetables are developed around work and school times for locals, rather than timetables which are suitable for tourists. For longer trips and inter-regional travel, trains are generally the best option.
Anyway, hope this helps. Enjoy Perugia - the travel guide in my signature should help!
Buon viaggio, Claire.