Thoughts on "What 50,000 Runs of a 5-Line Eval Taught Us"

A couple of thoughts on "What 50,000 Runs of a 5-Line Eval Taught Us".

Many simple paths

TLDR: The say_hello task has one correct answer, i.e. a file called "hello.txt" containing the text "HELLO". I don't think it's useful to focus on the method, but instead I'd focus on the token count i.e. the elegance or the "over thinking" that a model performs to get to the desired result.

There's an assessment made in "How models spend their overhead" that running the terminal command echo HELLO > HELLO.txt is more complex than the "simpler file-creation API" create_file.

Initially I thought I could see what they meant in terms of meaning making. If you asked a human to perform this task, and told them about the file-creation API and they then turned to previous ways of performing this task instead of using this new API then you might say "there was a simpler way". But actually I wouldn't say that, they might use echo HELLO > HELLO.txt, or they might use the IDE's file-creation functionality, or some other method instead. All are valid ways to achieve the same result.

Reflecting further, the advantage of using echo HELLO > HELLO.txt or the IDE's functionality is that you can avoid specifying the full path in the file-creation API which is more complex and error-prone i.e. you don't need to specify the /path/to/workspace/HELLO.txt below:

tool : create_file
args : {
    "filePath": "/path/to/workspace/HELLO.txt",
    "content": "HELLO"
}

In fact Model-W which along with Model-Z, and Model-V are highlighted as "Representative models" of this "more complex" approach of using the terminal command echo, has a lower token count (95) than Model-A which chooses the file-creation API each time and uses on average 102 tokens. Perhaps though Model-W is not always creating the file successfully? This data is not shown as the chart "Can the model "just create the file"?" seems to only assess if the model chose to run the single call to the file-creation API create_file.

Inverse Model size _does_ predict overhead.

Under the section "Model size does not predict overhead" it shows that the larger the model the smaller the token usage to achieve a similar result. This makes intuitive sense as the more you learn, the easier it is to make the correct action given some circumstances and the more elegant it is - the more you can achieve with less effort (in this case fewer tokens).

So model size does (inversely) predict overhead for this simple task.