Going into more detail, the reason it got so much backlash by certain people and drew in so much interest is what happens early on with the protagonist. He gets accused of something he didn't do by someone in power, he gets abandoned by everyone, robbed, he's hated by everyone because of lies and he's left with a defensive power that is nearly useless since he's alone and unable to even wield weapons.
The strength was not only the under dog trying to survive and rise up through adversity, it's that he wasn't brought down by someone who was just physically stronger, there's no monster he can just beat. He was brought down by his fellow man, and the writing of how he was treated was actually very realistic. You might wonder why they were so willing to throw away one of the heroes, but this is also explained later on. He was cast into a hated role against his will and was betrayed by those in power and demonized by nearly everyone in the kingdom. All while having innocent lives need the protection of the heroes, which leads to a conflict of wanting to help the people and hating them for betraying him. That was the interesting part.
Minor vague spoilers at this point.
As it goes on he starts getting companions. The first acquisition is pretty messed up, although you could argue it's better he took her in than leave her, morally grey at best. Then we find out her species just immediately matures and she just becomes a competent fighter, a fully grown adult who chooses to follow him, negating all the difficulties and moral conflicts nearly instantly. Then he buys an egg that by chance becomes an ultra rare sentient companion, which turns into a girl and ages up enough to be useful in battle. After this point he and his party become very powerful.
Not only are the accusations quickly cleared, but the accusers are stripped of power. On top of that, there's a whole show of it and they basically have magic televisions all over the kingdom broadcasting it and after the trial everyone believes him and that's the end of that. The writing becomes very unrealistic for the convenience of the plot. I mean unrealistic as far as human behavior, not because there's magic.
At that point it just becomes a more generic isekai of a guy and his all female group wandering around becoming more powerful and recruiting more female characters who will, yet again, end up being being overpowered because his group is the best. Everything that was good about the early parts is basically gone. I even read ahead a bit in the manga but I just got bored and stopped at one point, and I never went back. It just becomes very average after a while.
