Over 60% of Africa's population is under 30.
"We have a young bulge that is going to need lots of work, lots of jobs and governments should focus on that," Annan told reporters at the World Economic Forum (WEF) in the Ethiopian capital.
"We consider trade schools, preparing them or giving them skills that could employ them when they are out of school or allow them to get their own companies and work."
Annan noted that Africa has shown "remarkable" progress in recent years, in part by attracting increased foreign investment.
To continue advancing, he said, governments must concentrate on "the welfare of their people and come up with social and economic policies that try to effect... the lives of the individuals."
Seven of the world's fastest growing economies are in Africa, and investment in agriculture, infrastructure and mineral resources has blossomed in the past decade.
China is the continent's largest trading partner and has major investments in infrastructure and minerals across Africa.
Gao Xinqing, president of the China Investment Corporation, said Africa was a strategic partner for the Asian powerhouse.
"From China, we look at Africa really not only as a target of investment but as a potential coveted partner in many ways: culturally, historically, economically [and] politically," said Gao, who is co-chairing this year's WEF conference.
Eight African leaders and former British prime minister Gordon Brown are among the more than 700 participants expected at the three-day Addis Ababa WEF meeting that wraps up on Friday.
The conference is focusing on boosting public-private investment and fostering economic diversity to kick-start development across Africa.