(If you can't tell by my username) DBZ is very close to my heart. It's the anime that got me into anime, I've been lucky enough to meet 5 of the voice actors for it, I cosplay as DBZ character frequently, and DBZ merchandise is my weakness.
What went wrong with GT? Well I've heard that Akira Toriyama didn't write the story, but after some research I've learned that he outlined the story, designed SS4, drew out designs for all the villains and was in fact very involved. There are even quotes from him saying that he enjoys GT. Check out: http://dragonball.wikia.com/wiki/Dragon_Ball_GT for more on that.
The ideas in GT did have potential (SS4, relationship bonding between Goku and Pan, etc.) However, it just wasn't executed well.
So here is what went wrong in my opinion:
1. Goku becoming a kid: I think Goku never should have been reduced down to a child. All throughout DB and DBZ we watch Goku grow. He gets older, stronger, slightly smarter ;P , and he is always moving forward as an individual. He needs keep climbing higher and faster. That's just who he is as a person. So when he gets turned into a kid, that's a huge step back, and it didn't make any one happy.
2. Losing the competitive relationship with Vegeta: The friendly/not-friendly relationship between these two Sayain warriors is one of the most iconic relationships in all of anime. Their competition is what drives them to become stronger, faster. The DBZ community loves Goku and Vegeta whether they are fighting together or with one another. So it was a huge mistake to reduce Vegeta down to the level of a supportive character rather than a lead.
3. Losing the DBZ family. In addition to hardly any Vegeta, there was very little of the other beloved DBZ characters. Some of them had a few moments, but nothing really worth watching. I know the huge reason I love this anime is for the Z fighter interactions and relationships. I want to be a part of their family. Without them all being together, there is no family for me to want to be a part of.
4. Lame fighting. We all know there is nothing like a DBZ fight sequence! They are awesome! GT.....not so awesome. I guess the coolest thing that happened was the SS4 fusion, but it was out of character for Vegeta to transform not by his own power and the fight itself drug on for so long that it almost became boring. Every other fight in the show was predicable and repetitive.
I could go on, but these 4 points are what I believed killed the show. If you are watching Dragon Ball Super now you can see that all of these have been corrected which would be the reason for it's success as well as the success of Battle of the God and Resurrection F.
