Interesting question!
It's not that milk is bad for your voice, it's the fact that many people who are lactose intolerant, tend to produce too much mucus when ingesting dairy products. Thus making you cough or have too much phlegm.
Easy solution - avoid milk a few hours prior to speaking in public. :)
Hope this helps!