I personally consider the GameCube's WaveBird to be the most comfortable controller ever made. It just perfectly mushes with your hands.
As we all know, the Atari 5200's controllers were atrocious due to the lack of a self-centering mechanism. But the Magnavox Odyssey controllers were also very difficult to use.
The Wii controls hurt my arms. I really hate having to constantly swing around like a lunatic.
Some older systems used these little "knobs" as directional pads - Channel F, ColecoVision, and so on. The Channel F is especially unpleasant, since the knob is on top. Some other systems had the "controllers" built-in for especially painful gaming.
I like how the XaviX has a different motion sensor controller for every game. The GoGo TV Game System tried this out, but it was very unresponsive.
The Sega Pico's bizarre pen-and-book controls are really awkward and make for difficult playing.
But the award for weirdest controls would be a tie between the Compact Vision TV Boy and the InteractiveVision:




