I think this narrative is deceiving.
I'll start with the Mens Worlds. USA never sends their best to the Mens Worlds. Every country other than Canada (who let's face it has more good players to pick from) has regular National Team players that play in Europe, and are regularly playing in these International tournaments throughout the year that helps team cohesion. American players are typically also less loyal to their country in trying to win a medal. Players from the European countries view it as important to help their country win these tournaments. American players don't care, unless it's the Olympics. Another issue that's arisen with American players is that a bunch of them end up finishing their degrees (because few of the good ones end up playing 4 years in college) during the summer, which makes it hard to recruit players to play at the tournament, and this is a challenge only the USA faces in big numbers. Put it all together, and you get a country that doesn't take the tournament seriously.
I'm not sure either why we should be putting it on the same level as Best on Best tournaments like the Olympics and WCH, although it's true we haven't seen many of those recently. I realize it's importance in Europe, but when the tournament has at most half of the best players (and truthfully it almost never has that many) it's hard to be using it as an important metric of who the best international hockey countries are. In the last 5 editions, we've had Latvia, Germany, and Switzerland medal. The last five best on best Olympics, we've gotten no one other than the big six medaling. When you get a tournament that allows for the smaller hockey countries to regularly medal, it tells you how watered down it is.
U18's isn't the only junior international tournament USA does well at. We have the most U20 Golds since 2010 other than Canada with four. Finland has three. Russia (they've missed a few tournaments) only has one, as does Sweden.
This is what the medal tallies look like at the U20 Worlds since 2010:
USA: 9 in 14
Canada: 10 in 14
Finland: 5 in 14
Sweden: 7 in 14
Russia: 9 in 12
USA is quite a bit ahead of Finland and Sweden in medaling. One behind Canada, and then the same as Russia (although in more tournaments), yet Russia only has 1 Gold compared to 4 for USA. So when you put it all together, USA essentially has the best record of the non-Canada countries in this tournament. U18 Worlds, I don't even need to elaborate much on that. The playing field isn't level, and I agree, but it's still the official U18 World Championship. No one other than the big 6 has medaled there since 2003. I think that's a better judge of hockey prowess than tournaments where Germany, Latvia, and Switzerland types of countries are medaling with regularity.
As for your soccer comparison, I think the England comparison is unfair. International soccer is so different from international hockey. International soccer sees a best on best tournament (with no real interruptions) every four years, and you could argue every 2 years at times (with Euros, Copa America, etc.). How many chances has USA even had to win a best on best tournament since 2000? Maybe like 5 or 6. And none in the last 6+ years. American hockey has only gotten better in that stretch. No one would've dared suggest we were close to as good as Canada 6 years ago. Now the comparison is taking place. So I'd say USA is more of a newer giant than an underachieving giant. The soccer comparison I'd use is France or Spain.