Custom Post Templates

Custom Post Templates

The Issue: I am looking for custom single post templates, to add or remove individual elements as a function of a normal single post.
There are a lot of ways to create custom post templates for single posts within WordPress. Especially post formats are a great chance to use default templates for default cases; however I need real custom templates for every post.
The Idea: My first approach was to add an if/else statement according to the post ID:
// check if custom post
if ( is_single('999') )
    // check if there is a custom post template file
    if ( file_exists(TEMPLATEPATH . '/single-99.php' )
        // use custom post template
        return TEMPLATEPATH . '/single-99.php';
// use normal post template for everything else
include(TEMPLATEPATH . '/single.php');
Well, this isn't wrong, but it would totally mess up my template code if there will be more and more special cases. So maybe I can use a filter to always use a custom template if there is one that corresponds with the post ID:
add_filter( 'single_template', function( $template ) {
    // check if there is a custom post template file
    if ( file_exists(TEMPLATEPATH . '/single-' . $GLOBALS['post']->ID . '.php') )
        // use custom post template
        return TEMPLATEPATH . '/single-' . $GLOBALS['post']->ID . '.php';
    // use normal post template for everything else
    return $template;
});
But now I will end up with a lot of custom template files, that are all the same except some minor changes.
More Thoughts: I think I would rather like to add dynamic sections within the single.php template, to hook in with custom includes/filters?
The Question: Is there a different (more exclusive) approach to get those kind of custom post templates?