Albion, who sacked manager Bryan Robson on Monday, were staring down the barrel of a major upset when Kayode Odejayi gave Cheltenham a half-time lead with his fourth goal of the season.
But a three-goal blast in the space of 19 minutes at the start of the second half ultimately earned Albion a comfortable passage into the next round.
Ronnie Wallwork headed them level and then Nathan Ellington and Nicholson both scored from the penalty spot.
It was Nicholson's first start at the Hawthorns but he has already shown a liking for the Carling Cup as he was on the mark in the previous round at Leyton Orient.
The final score was somewhat harsh on Cheltenham who were well worth their half-time advantage, but once Wallwork drew Albion level the outcome looked to be inevitable.
While Albion threatened in the early stages when Shane Higgs saved from Nicholson and Darren Carter's header was cleared off the line by Grant McCann, Cheltenham were dangerous on the break.
McCann gave a sign of what was to come in the 23rd minute when his snap-shot from the edge of the area crashed off the post.
Brian Wilson then lashed a shot over the bar before the powerful running of Odejayi was rewarded on the stroke of half-time.
He raced on to a pass from Mickey Bell and powered past the Albion central defenders Paul McShane and Chris Perry before sliding his low drive beyond Pascal Zuberbuhler.
Albion were stunned by the setback and could easily have slipped further behind seconds later when Odejayi fed Steve Guinan whose shot needed produced a smart save from Zuberbuhler.
It proved to be a costly miss by Guinan as Albion were back on level terms three minutes after the re-start.
McShane made amends for his earlier error by firing in a right-wing cross that Wallwork stooped low to head beyond Higgs.
Wallwork's equaliser changed the context of the match completely and Albion effectively booked their place in the third round 10 minutes later.
Ellington bustled into the area and forced Wilson into handling the ball.
Referee Tony Bates pointed to the spot on the advice of his assistant and Ellington calmly slotted home the spot-kick sending Higgs the wrong way.
To their credit, Cheltenham refused to buckle and it needed a wonder save by Zuberbuhler in the 66th minute to keep out a header from skipper John Finnigan.
It was a vital stop as Albion surged straight up the field and Jason Koumas was hauled down by Bell to give them their second spot-kick in eight minutes.
This time Nicholson strode forward and, although Higgs got a hand to his shot, he could not keep it out.