When a judge cancels a trial, she declares a mistrial. In other words, she decides that some mistake has been made and the trial must begin again from the start, with a new jury.
There are several reasons that a judge might declare a mistrial, including a hung jury, which is when the jury can't come to a unanimous decision. There might also be misconduct by an attorney, or improperly introduced evidence. A mistrial usually means having to start all over again. The noun mistrial begins with the prefix mis-, which comes from Old English and means "bad or wrong."