My current theory: Wiseman isn't a Phantom at all, but a Philosopher's Stone - the Stone was often referred to in alchemical lore as 'the white stone' or 'the Stone of the Wise' and one of the abilities attributed to it was the production of precious gems, not to mention the fact that what few consistent design features we have for the Phantoms (a gold-ringed coloured gem, fairly uniform in size, located on the abdomen; Greater Phantoms having bird-themed shoulders) don't seem to apply to him - and the White Wizard is his creator.
He may or may not be somehow flawed, and either Koyomi or Haruto is a second attempt (or both - 'this one is complete' after all). It's notable that in a lot of alchemical texts, the Philosopher's Stone was described as having two forms - a white form where it was 'immature' and a red form where it was completed.
I wouldn't be surprised if this is wrong - after all, another name for the Stone in lore was 'Phoenix', so you could just as easily make an argument that Phoenix was a Philosopher's Stone, even though he almost certainly isn't - but eh, it's a theory.