Gate Keepers 21

This anime doesn't involve gore but it involves some mystery and some possible government involvement. They are also similar in the fact that each of them has a super ditsy female as one of the main characters. 

Gate Keepers 21 was among the first attempts to deconstruct the mahou shoujo genre, while Madoka Magica is the ultimate «genre evisceration». In both series, doing magic appears not so cool and safe as it may seem, the enemies being far from what you'd like them to be. Also, some of the characters are similar. 

Both OVAs are extreme mixtures of comedy/parody, action and drama. Also, both share the "film noir" stylistics. 

Definitely, Utena has been a source of inspiration for the GK21 production team. The main character, Ayane, is similar to Himemiya in appearance and in some character traits. Both series feature shoujo-ai lines, in both works dark and funny scenes are mixed and melted together... And yes, those rocks flying apart all around the main characters in the opening of Utena are back in GK21. 

Both OVAs feature lots of humour, grotesque fighting (Yomiko's pavement-breaking papers vs Ayane's pavement-breaking cellphones), somewhat crazy characters, dramatic endings, some blood, a bit of shoujo-ai... The difference is that Gate Keepers 21 is dark and serious at its core, while R.O.D. is just a good piece of entertainment.