No, it's definitely not mainly genetics.
I'm not going to pretend that size plays no role at all (the average Japanese male is 170cm tall / the average Canadian 178) - because obviously there is a difference there. But Japan produces good players in other disciplines - including soccer, baseball, (granted, not as physical sports, but what about Rugby union?) - so I don't think that's the main hinderance.
Rather, it mostly comes down to a lack of opportunity / infrastructure. There's very few clubs and the major league only had 6 teams for decades before folding entirely. It's only fairly recently been replaced by an 8 team pan-asian league. It's pretty clear that most of the registered players are basically hobby-level players. In that respect it's a bit like Canada in soccer. Tons of people play the game - but (at least until recently) there were few avenues to progress. A solid domestic league, good professional teams, and a program focused on player development are needed to translate some of that participation into success.