Welsh international midfielder Koumas fired Albion ahead after just five minutes from 25 yards - before limping off four minutes later with a leg injury.
Scottish international striker Scott Dobie doubled their advantage in the 24th minute when he pounced on a mistake by Millwall's central defender Stuart Nethercott.
While Nethercott went some way to making amends for his crucial mistake with Millwall's goal on the stroke of half time, the damage had already been done.
Albion manager Gary Megson was delighted with the success and he said: "It was a marvellous goal from Jason. He really is a special talent.
"There were a lot of good performances from my players - especially in the first half and I felt our win should have been more conclusive. Millwall's goal was a sucker punch."
Millwall boss Mark McGhee was disappointed with the start his side made and he said: "Any success we have had in the last couple of years has been built on having the kind of start Albion had.
"We did not cope with Albion's high tempo play and, although we huffed and puffed in the second half, we had left ourselves with too much to do."
Koumas opened the scoring with a stunning strike after a mazy run that took him deep into the Mil;wall half.
He was forced across the face of the penalty area by the Millwall defence only to whip in a left-foot drive from 25 yards that left goalkeeper Tony Warner stranded.
Albion made light of losing Koumas with an ankle injury and would have added to their advantage before Dobie's strike but for the heroics of Warner.
He pulled off smart saves to deny Thomas Gaardsoe and Andy Johnson before Dobie cashed in on Nethercott's mistake.
Nethercott tried to head clear a cross from James O'Connor only to deflect the ball into the path of Dobie who headed home his second goal in successive games.
Millwall were on the rack at this stage, but Albion failed to increase their lead despite their domination and allowed the Londoners a lifeline in first-half injury time.
After Albion keeper Russell Hoult had turned a shot from Neil Harris away for a corner, Dennis Wise whipped in a cross that Nethercott drove home from 14 yards through a crowded goalmouth.
McGhee went for broke in the second half by sending on an extra striker in Noel Whelan.
But Millwall toiled to create another opening and it was left to Warner to further frustrate Albion as they sought a decisive third goal.
He pulled off further smart saves to deny Bernt Haas and Johnson as battling Millwall slipped to only their third defeat of the season.
West Brom: Hoult, Gregan, Gaardsoe, Gilchrist, Haas, Johnson, O'Connor, Koumas, Clement, Hulse, Dobie. Subs: Murphy, Dichio, Hughes, Sakiri, Berthe.
Millwall: Warner, Lawrence, Nethercott, Ward, Craig, Ifill, Cahill, Wise, Livermore, Harris, Peeters. Subs: Gueret, Whelan, Robinson, Roberts, Dunne.
Referee: C Webster (Tyne & Wear).