Its akin to the weekly call I get from "Revenue Canada " saying if I don't pay up now the police will arrest me. These scams are not new, call their bluff. Monday I got a scam call, so I kept asking him how he was dressed, and if he was craving my cock, till he hung up. A decade later and I am still waiting for these cops to show up.
Don't sweat it OP, just keep on doing your own thing, and don't give the scammers another thought, past blocking their number, which only takes 5 seconds to do.
Cheers