That depends heavily on the game. You can see runs of games that aren't super popular to run with so many large mistakes in them that you can very realistically say: "Yeah, I can do that better.". Spend one or two hours every evening on running and you can break world records in weeks.
Most runners that aren't mental are either still in school and have fuckloads of time or run not so competitive games.
But somebody that's grown up and thinks he can go and break records in something like Golden Eye or SM64? They will either get a rude awakening or are somewhat crazy and have much too much time on their hands.
Repeating something loads of time isn't in itself crazy, however. Tetris or Pinball are basically always the same with minor randomness to them. Is it crazy to try and get better for fun at a repetitive task?