WASHINGTON -- The scandal surrounding Sen. David Vitter has sparked a partisan fight.

"Rather than proving to be a leader, he's proving to be an embarrassment," Chris Whittington, the Louisiana state Democratic chairman, said on Thursday night.

Phone records show that so-called D.C. Madam Deborah Jeane Palfrey called Vitter back when he was a House member and during hours that Congress was in session.

The Louisiana Republican acknowledged this week that his number was on Palfrey's call list. The father of four has dropped out of sight since the revelation, missing out on Senate votes.

Julie Vezinot, Whittington's spokeswoman, said the state Democratic Party would start a petition calling for Vitter's resignation on Friday.

But U.S. Rep. Rodney Alexander, R-La., said Vitter was "doing the right thing, taking some time to be with his family" and added that "we can't forget about all the positive things David has done for the state of Louisiana."

Sen. Jim DeMint, R-S.C., said he has traded e-mails with Vitter, adding that he expects Vitter to return to the Capitol by Tuesday for roll call votes.

DeMint, who generally shares Vitter's conservative philosophies, predicted that senators will welcome their colleague back despite his admission.

"It's a huge moral failure that reflects on the whole body," DeMint said. "And for that he's very sorry."

"Obviously he has a lot of remorse," DeMint said. "He seems to want to address it head on and not try to hide it."

One of the party's most high-profile members, Rep. Bobby Jindal, R-La., has keopt mum, however. Jindal is running for governor.

For his part, Vitter remained out of sight since he acknowledged the link to Palfrey on Monday.

Palfrey said Thursday she would return calls to her company to verify the caller's identity. She said she called Vitter's office five times. Four of those times, the House was in session and Vitter showed up for roll call votes.

Federal prosecutors accused Palfrey of running a prostitution ring. She said her business was a legitimate escort service.

The records do not indicate whether Vitter himself took the calls.

Hustler magazine publisher Larry Flynt claimed credit for outing Vitter's connection to Palfrey.

He said Wednesday that Vitter was fair game in what he called a sex scandal investigation by his magazine because Vitter advocated conservative positions while living a private life at odds with his public life.

"Sometimes I have to go bottom feeding" in an effort to remove "phonies" from lawmaking positions, Flynt said.

He said his staff called Vitter's office Monday, saying they knew his telephone number appeared on a list from Palfrey's escort service. Vitter then went to The Associated Press in an attempt to get ahead of the story, Flynt said.

In 1999, Hustler threatened to publish what it called evidence of martial improprieties into former U.S. Rep. Bob Livingston, which prompted the Louisiana congressman's resignation. Vitter, in turn, was elected to replace Livingston.

Flynt said his magazine has collected information on indiscretions among both Republicans and Democrats, in both the U.S. House and Senate, but he declined to provide other details.