I asked Claude, one of the AIs supposedly "good for programming," to help me out with two tasks today. The first was "I have a page with a collection of cards on it. The cards are web components. Each component has a button inside its shadowDOM, and the label on each button is different. Using WebdriverIO, I want to find the card that has a button with a specific label."

It failed utterly. Then it failed again.

For this problem, it gave me some anodyne "push this to the browser and traverse it manually," that was about as inefficient as one could manage, and it was several versions of WebdriverIO out of date. Now, WebdriverIO 9, the latest version, hasn't solved this, but still; you'd think it would have done better.

The real solution isn't pretty, but at least you can run it from the server. The isDisplayed clause is necessary because this is Lit, and Lit has a lot of little tricks about keeping things in Document Fragments off-page until they're needed.

const comparator =
    typeof label === "string" ? (sample) => sample === label : (sample) => label.test(sample);

const card = await (async () => {
    for await (const card of $("ak-radio-card").$$(
        '[data-ouid-component-type="ak-radio-card-card"]',
    )) {
        if (await card.isDisplayed() && comparator(await card.$(".card__title").getText())) {
            return card;
        }
    }
})();

Yes, I use IIFEs for namespace segmentation. Deal.

The second problem was "Help me write a parser for a JavaScript Template Literal. The acceptance criteria requires that the IDE and the programming language formatter must both recognize the contents of the template as a different programming language from JavaScript."

Claude cheerfully spat out the internals of a Template Literal engine that, "for example, parsed HTML." I recognized it immediately. It's the insides of Lit-HTML. Like, I have already read that source code myself.

And it didn't work! The IDE didn't recognize the insides as HTML, and neither did Prettier. I don't know if there's special code to recognize a JS Template Literal parser named html and go from there inside Prettier or not, but just tagging the output of the function as HTML after the fact didn't make the IDE or the formatter work backward and figure out the insides of any template supplied to it should have been formatted and syntactically checked as "valid HTML with template variables."

I have found that working with LLMs is great if I have to backfill my knowledge and learn the priors of something that's fairly well known. If I want to learn, for example, Rust, Claude is pretty good at explaining something to me that's been widely explained on the Internet already. But LLMs cannot create, cannot reason, and cannot assemble a new idea in the gap between of two existing ideas. At best, they can tell you more about the priors you might need to fill in that gap, and maybe explain why the gap exists at all, but even then, no LLM can fill in that gap.

Inventing new things is still the province of human beings. Deriving meaning from the random shuffling about in a box full of image fragments is still the province of human beings. LLMs are a long, long way from being General Artificial Intelligence.

At best, LLMs are an alternative form of search engine; one that can give you much better answers to well-known problems than Google's older form, because LLMs boil down all the answers into a readable solution, tuned to the vocabulary and attention of the user. But both Lit and WebdriverIO 9's support for the ShadowDOM are new, and solving problems about how they interact is very new, and no LLM can help there. Only experience.