A federal judge says a defamation lawsuit filed by two brothers who said they took part in a fake hate crime against Jussie Smollett can go forward. U.S. District Judge Mary Rowland ruled Friday that comments made by one of Smollett’s lawyers could be considered defamatory.

The attorney had suggested the brothers—who are Black—might have been in “whiteface” to disguise their identities during a real attack. Smollett was convicted of lying to police about what authorities say was a staged crime. Last week, he was sentenced to 150 days in jail.

On Wednesday, an appeals court ordered his release, pending appeal.