Then what was the point of offering it to Yaguruma if they knew it wouldn't take him?
They were asking for Yaguruma to return. If he chose to return to Zect, it would accept him, but he rejected it because he wasn't going to do it.
How do you even know that loyalty is necessary to use Zabee?
It's the only Zecter that always left users who went against Zect?
Then what was the point of offering it to Yaguruma if they knew it wouldn't take him? How do you even know that loyalty is necessary to use Zabee? It left Yagu the first time even when he was still loyal to ZECT.
When it left him, he hadn't betrayed Zect, but he had received orders to kill worms, and went directly against them, ignoring the worms to fight against Tendo.
And why would Kagami have to give it up to not fight Tendou? The Zecter wasn't forcing him to fight.
That's exactly my point. The Zecter wasn't forcing Kagami to fight, yet, because he didn't want to fight against Tendo, he had to give up the Zecter. He had no such problem as Gattack.
First Tendou takes Daisuke and Tsurugi's Zecters away, and then later they're still using them. Tendou even bought the Sasword Zecter from Tsurugi, and then in a later episode he gets mad at Tendou for taking it away mid-battle.
That WAS addressed in the show. He wanted to take every Zecter so no Rider would attack Hiyori, but after the Red Shoes incident he realized that he can't do everything by himself and gives up on that idea, letting people take their Zecters back. The Perfect Zecter only appears after he had already returned the Zecters to their owners.
Why do you seem to write this off so nonchalantly?
Because that's part of the basic premise of Kabuto's setting. For whatever reason, the Zecters needed real AIs to work and Zect didn't have much control over those AIs. It'd be a very different show if that element didn't exist. I mean, it's like complaining about Ryuuki because Kanzaki had to make a Rider battle rather than just kill a bunch of people to get their life force.