Many perceived it as condescending to both Trump supporters and the popular restaurant chain.
"Look at them, they’re high-fiving each other for this deplorable display of completely unpatriotic, completely against law-and-order, completely unconstitutional behavior. It’s stunning," Cooper said on his show "AC360."
"And they’re going to go back, you know, to the Olive Garden and to the Holiday Inn they’re staying at, and the Garden Marriott and they’re going to have some drinks and they're going to talk about the great day they had in Washington."
The host Friday claimed his words were misinterpreted.
"I should have been more clear with something I said in the immediate moments after the attack as people who had broken into the Capitol were simply being allowed to leave, it seemed — just walk away," he clarified at the end of his show, admitting his words may have been unartful.
"Celebrating the criminal act they had taken part in. I was trying to remark about the seeming casualness of the behavior that we were all witnessing at that moment after the attack. The high-fiving, the laughing, the celebratory atmosphere, as if they had actually accomplished something."
Cooper, a wealthy son of late heiress and fashion designer Gloria Vanderbilt, went on to explain he actually likes Olive Garden.
He said that he had named the two hotels he could think of in the Capitol Hill area and name-checked the Olive Garden as a "normal" place anyone might eat. He further explained he was actually thinking of a specific restaurant in that area with the name Olive in it but his "brain froze," adding he realized later the closest Olive Garden was in Maryland.
Cooper said it "genuinely" makes him sad to think people would believe he’s too "fancy" to eat at a place like Olive Garden because of who he is.
"I’ve been kind of sad about it all day," he admitted.
"Just for the record, I like the Olive Garden. I like the artichoke dip even though it has spinach it in it, which I hate. Yes, I know, it’s an appetizer with breadsticks. It’s like a whole meal."
He added that his team broadcast from New York's Times Square Olive Garden during CNN’s New Year’s Eve broadcast a few years ago and he repeatedly praised the restaurant at the time.