Woo was born Wu Yu-Seng (Ng Yu-Sum in Cantonese) in Guangzhou, China, amidst the chaos of the Chinese Civil War at the end of October, 1946. Because of school age restrictions, his mother changed his birth date to 22 September 1948, which is what is what remains on his passport. The Christian Woo family, faced with persecution during Mao Zedong's early anti-bourgeois purges after the communist revolution in China, fled to Hong Kong when he was five.

Impoverished, the Woo family lived in the slums at Shek Kip Mei. His father was a teacher, though rendered unable to work by tuberculosis, and his mother was a manual laborer on construction sites.[6] The family was rendered homeless by the big Shek Kip Mei fire of 1953.[5] Charitable donations from disaster relief efforts enabled the family to relocate; however, violent crime had by then become commonplace in Hong Kong housing projects.

At age three he was diagnosed with a serious medical condition. Following surgery on his spine, he was unable to walk correctly until eight years old, and as a result his right leg is shorter than his left leg.[7] Woo went to Concordia Lutheran School and received a Christian education[citation needed] (his Christian background shows influences in his films[8]). As a young boy, Woo had wanted to be a Christian minister. He later found a passion for movies influenced by the French New Wave especially Jean-Pierre Melville.[2] Woo has said he was shy and had difficulty speaking, but found making movies a way to explore his feelings and thinking and would "use movies as a language".[2]