Here's my solution using TimeUnit.
UPDATE: I should point out that this is written in groovy, but Java is almost identical.
def remainingStr = ""
/* Days */
int days = MILLISECONDS.toDays(remainingTime) as int
remainingStr += (days == 1) ? '1 Day : ' : "${days} Days : "
remainingTime -= DAYS.toMillis(days)
/* Hours */
int hours = MILLISECONDS.toHours(remainingTime) as int
remainingStr += (hours == 1) ? '1 Hour : ' : "${hours} Hours : "
remainingTime -= HOURS.toMillis(hours)
/* Minutes */
int minutes = MILLISECONDS.toMinutes(remainingTime) as int
remainingStr += (minutes == 1) ? '1 Minute : ' : "${minutes} Minutes : "
remainingTime -= MINUTES.toMillis(minutes)
/* Seconds */
int seconds = MILLISECONDS.toSeconds(remainingTime) as int
remainingStr += (seconds == 1) ? '1 Second' : "${seconds} Seconds"