... And perhaps they should have. With the fourth episode, we're treated to the typical cliches and gags, with Louise wanting Saito on her own time but being her nasty, violent excuse for "tsun" the rest of the time. Admittedly, in the first episode, there was an argument that was fairly petty, but it was resolved surprisingly well for this series.
Furthermore, the fact that Siesta, who was a fairly enjoyable character in the first season, and now even Tabitha, have been reduced to, essentially, characters that just cling to Saito just to cause an issue for Louise, not to mention the fact that Louise constantly takes THEIR affections out on Saito, almost as often as he's pledged his devotion to her (he does, admittedly, ask for it sometimes) just makes things all the more ridiculous.
What's more, there's a plot device used to keep Siesta around, in that the queen asked her to be Saito's maid, but thinking about it - Wouldn't the queen know better than to do this after observing them for, oh, five minutes? Especially given that Siesta's self-proclaimed "love rival" is the queen's best friend, and as of season three, sister?
Ultimately, the flaw with what follows the great first three episodes (or perhaps just great THIRD episode - the first had its share of these problems as well) is that this series is trying to be a harem when it's got an established romance, and isn't letting its characters develop like they should in order to keep the harem aspect going when it's been clear (and even canon) from early on that there is no point to having a harem here.
Our protagonists have been saying from as early as the second season they love each other, we should not be arriving on the fourth season with the relationship still acting in a standstill, and with Louise acting in the same, petty way she has been all along, guided by shallow plot devices.
Aside from the romance/harem aspect of the series, however, the plot seems to be shaping up to be as good as always, and development outside of Louise has been decent. We've seen how Saito has grown since he's come to this world, and even how side characters like Guiche have taken to being warriors in their own way.
Overall, I'll be watching to the end, but unless things improve drastically in regards to Louise and her relationship with Saito this time around, I think I won't be too sad to see this one end here.
