Baima means "white horse" in Chinese. According to local legend, many years ago, a fierce and strong white horse lived in the Muyun section, terrorizing the local villagers. One day, Yang Yanzo, a famous general of the Song Dynasty, saw the horse playing in the river. He jumped onto the horse's back and succeeded in taming it after a long struggle. From then on, General Yang drove on the horse to fight every battle. At that time, the kingdom of Liao frequently invaded the Song's northern frontier. In order to stop the Liao's intrusion, General Yang left the horse on the Great Wall. When invading Liao soldiers saw it, they beat a hasty retreat because they thought General Yang and his troops were stationed there. Hence the name of the fort is.

Beside the gate, there are two sections of walls that once surrounded the fort still standing, Yidaobian and Erdaobian. Erdaobian is the outer wall, including eastern and western sections. The east part is only about 50 meters long, stopped near a cliff; while the 200-meter western part extends along a 40 to 50 degree slope, near which the remains of two guard towers can be made out. A little way south of Erdaobian is Yidaobian, the main body of the Baimaguan Great Wall.