Currently there is no other way to know whether there are new chat messages within a course than opening the chatbox and checking for yourself. (Auditive new message alerts arrive with a 10-minute delay :-/ )
My suggestion would be to display an unread message count in the course navigation and/or in the title of the inline chat panel (when collapsed). I've indicated these positions with an * in the screenshot below.
This would make the chat function a lot more useable!