Neural networks as DSL primitives

Consider solving puzzle 3e980e27 (in ARC-1) while only being exposed to the first example in test-time. Our brains seem to implicitly synthesize something similar to the following program:

def p(g: Grid) -> Grid:
    x = copy(grid=g, obj=obj, loc=red_square_in_obj)
    x = hflip(obj=x)
    return paste(grid=g, obj=x, loc=red_square_not_in_obj)

We copy the grid at the red square inside the object, flip it, and paste it at the red square outside the object. What's interesting is that we don't have pre-coded definitions of red_square_in_obj or red_square_not_in_obj, but are still able to manipulate them in accordance to the above program.

Consider the next example. Our brain immediately makes a generalization: the red squares in Example 1 were an instantiation of the more abstract square entity; in Example 2 we see another instantiation, which happens to be green. Also, there are n squares, and n happened to be 2 in Example 1. So instead of red_square_in_obj and red_square_not_in_obj, our brain generalizes to an n-dimensional squares structure.

Also, for some reason, we don't flip the object before pasting. I'm not sure why, so I'll leave that as condition():

def p(g: Grid) -> Grid:
    x = copy(g=grid, obj=obj, loc=squares["green"][0])
    if condition(): x = hflip(obj=x)
    for sq in squares["green"][1:]:
        g = paste(grid=g, obj=x, loc=sq)

def condition() -> bool:
    return FF()

We see that condition() is a forward pass on small neural network. Because we don't know the rule, we can leave it a "black-box"; i.e., "for reason X, we don't call flip()—I have some vague intuition when we would call flip() from seeing examples, but I don't know what reason X is."

• More Task n examples will:
  • Serve as training examples. After enough examples, humans are able to attend on the right object-pattern features, training the FF network for condition() without knowing the actual rule
  • Serve as constraints in program search, pruning the search space and enabling finding a rule for condition() that is satisfiable for all Task n-1 examples we saw thus far
• Once the rule is found
  • I don't think we discard the FF() implementation of condition, we still use our intuition
  • The intuition is sharply adjusted to match the rule
  • Maybe sometimes we use one and sometimes we use the other

Let's continue to the remaining 2 examples. The program is pretty much the same; I've replaced condition() with [ ? ] to make it more concise.

def p(g: Grid) -> Grid:
    for color in {"green", "red"}:
        x = copy(obj=obj, loc=squares[color][0])
        if [ ? ]: x = hflip(obj=x)
        for sq in squares[color][1:]:
            g = paste(grid=g, obj=x, loc=sq)

**Note that if we had enough data, we could certainly start to solve these puzzles by training the condition() model without knowing what the actual condition is!

Here's the final example: It was here where it finally clicked that condition() is actually color == "red"; in other words, only when the square is red we flip. We can "step into" the condition() model, replacing it with the symbolic rule that is maintained across all 4 examples.

def p(g: Grid) -> Grid:
    for color in {"green", "red"}:
        x = copy(obj=obj, loc=squares[color][0])
        if color == "red": x = hflip(obj=x)
        for sq in squares[color][1:]:
            g = paste(grid=g, obj=x, loc=sq)