The investigation also involves at least five and possibly 10 U.S. military personnel who were working with the Secret Service ahead of President Barack Obama's trip to the Summit of the Americas. They are accused of bringing prostitutes to their Cartagena hotel Wednesday night, and the chairman of the Joint Chiefs of Staff said Monday his personnel "let the boss down" with their conduct.
The Secret Service agents and officers involved range in experience from relative newcomers to nearly 20-year veterans, and all have been interviewed at least once, two government officials with knowledge of the probe told CNN. Their security clearances have been pulled while an investigation is under way and could be reinstated if they are cleared, the officials said.
Most if not all arrived in Cartagena on Wednesday, two days before Obama, and went out for drinks in several groups after arrival, the officials said.
The agency, which is responsible for presidential security, has briefed key congressional leaders on the investigation so far. Rep. Peter King, the chairman of the House Homeland Security Committee, told CNN that he believed 11 prostitutes were brought back to the hotel.
Over the weekend, King told CNN that one of the women refused to leave a hotel room Thursday morning. A hotel manager tried to get in the room, and eventually the woman emerged and said "they owed her money," according to King.
U.S. government sources also said there was a dispute between at least one Secret Service member and a woman demanding payment. At least one of the women brought to the hotel talked with police, and complaints were filed with the U.S. Embassy, the sources said.
The alleged misconduct occurred before Obama arrived in Cartagena, but the news broke while he was there. Gen. Martin Dempsey, chairman of the Joint Chiefs of Staff, told reporters at the Pentagon on Monday that the incident distracted attention "from what was a very important regional engagement for our president."
"So we let the boss down, because nobody's talking about what went on in Colombia other than this incident," Dempsey said.
Rep. Darrell Issa, the chairman of the House Oversight and Investigations Committee, said he met with Secret Service Director Mark Sullivan on Monday and said Sullivan was "shocked" by the accusations.
"This is an organization with a great history, a history of being disciplined," Issa, R-California, told CNN's "John King USA." But he said earlier that "to assume that 11 people did something on a one-time basis is a little bit questionable."
For 11 people to violate a basic security premise indicates a larger problem, Issa said.
The Americans don't seem to have broken the law in Colombia, where soliciting prostitutes is legal in some parts of the country, he said. But the congressman -- a leading Obama administration critic -- said the conduct could have posed a security risk to the president by exposing Secret Service agents to blackmail.
"What's legal in one country still could lead to a married man being blackmailed," he said.
Speaking at the end of the summit on Sunday, Obama called for a "thorough" and "rigorous" investigation into the allegations.
"We're representing the people of the United States, and when we travel to another country, I expect us to observe the highest standards," he said. "Obviously, what's been reported doesn't match up with those standards." But he added, "I'll wait until the full investigation is completed until I pass final judgment."
The Pentagon said Monday that the misconduct involved more service members than the five initially believed. Pentagon spokesman George Little could not say how many more might be involved, but he indicated the personnel might come from more than one branch of the military. Pentagon officials had originally thought only Army personnel were involved.
"We believe there may be more than five," Little said Monday. All were expected to return to the United States on Monday.
Little said the personnel were not directly involved in presidential security and did not have any contact with Obama. He defined their role as in "support" of the Secret Service.
Because presidential security was the overall mission, however, it is not clear whether the Pentagon will ever publicly describe what the military personnel were doing as part of their work in Colombia or what branches of the military they belonged to.
An officer who was already in Colombia gathered initial facts, according to Little. A more senior officer was leaving for Colombia on Monday to continue the investigation.
Secret Service spokesman Edwin Donovan said the agents were relieved of duty Thursday, before the president's arrival in Colombia. The agency's assistant director, Paul Morrissey, noted his agency's "zero tolerance policy on personal misconduct."
"This incident is not reflective of the behavior of our personnel as they travel every day throughout the country and the world performing their duties in a dedicated, professional manner," Morrissey said Saturday. "We regret any distraction from the Summit of the Americas this situation has caused."