In this paper we employ backward Viterbi search for speech recognition. Contrary to forward Viterbi search that is performed from the beginning to the end, and where a word depends on the preceding words, backward Viterbi search is performed from the end to the beginning and the current word depends from the following words. As the errors of the forward and the backward searches are not the same, improvement can be achieved by combining the forward and the backward Viterbi search. The fusion is attained by an expert system based on rover algorithm, and using confidence measure for the words and optimal confidence value for null arcs depending on its place in word transition network (WTN). The experimental result of the combined system showed significant improvement over both forward and backward Viterbi decoding system on the Number95 database.