Two Years with a GPT
My first experience with a “GPT” was copilot, integrated into Visual Studio Code. I was working inside Microsoft, and the dates seem fuzzy now, but it’s been roughly two years. What is a “GPT” anyway? Generative Pre-Trained Transformer. It says what it is right on the box. Well, at least in nerd speak. Let’s break it down though, because it’s instructive as to the state of “AI” at the moment.
Pre-Trained – A bunch of data is used create a ‘model’, which is then used to do parlor tricks that make you think something is showing signs of intelligence. Just imagine having a 10 year old who has read every book and web page on the planet, and then ask them questions. This 10 year old’s trick is being able to read and remember everything they’ve read, and make some predictions when answering questions, but they don’t have the years of experience and insight necessary for true ‘wisdom’ or ‘intelligence’.
Generative, Transformer – “Generative” is the part where you can ask a seemingly ‘new’ question, and get an answer. Again, it’s as if my ten year old has read everything, and I ask, “How would Mark Twain answer this question?”. Well, having read all of Mark Twain, they can make an educated guess as to what response might be generated from a particular query. Basically, doing some statistical analysis of past responses, and putting together a “most likely” response. It’s not thinking about Samuel Clemons’ actual thought processes, it’s just stitching together the words that are most likely to appear together based on all the reading they’ve done of the author. You could ask the same question twice and get two very different answers. There’s no ‘intelligence’ there.
That’s a gross simplification, and those versed in the trade will smack their foreheads in disbelief, but the rest of us can carry on.
What this has meant to me as a fairly seasoned programmer is this. Copilot is a great copy/paste tool within the Visual studio editor. From the beginning, I’ve marveled at how I can type a comment in my code, and it will fill in the blanks with what appears to be functional code. This works great for code areas where the answer is well known, and having read all the publicly available code on GitHub, it can fill in a reasonable approximation of what I want. When I already know what I want, CoPilot is great at doing the boilerplate work for me, saving me let’s say 30% of my time.
It is also great providing some guidance on things I’m not familiar with. For example, I’ve never written an app for Android. With CoPilot, I can say “generate a simple app that does the following, on Android”, and it will do a reasonable first approximation, not because it’s intelligent, but because it’s read a bunch of Android code, and can stitch together a reasonable approximation of what I’m looking for.
Where it does not do well is when you’re asking for something it hasn’t seen before. When you’re looking for precision, novelty, efficiency, beyond what you’ve typically seen in online available training sets of code. I tried this experiment, I needed to be able to parse a time value for SMIL animations. I fed an eBNF into ChatGPT, and had a multi-hour interactive session to try and come up with a solution. The actual chat of the conversation can be found here: Chat about time parsing. It was an interesting conversation. We went off into the weeds a few times, and I had to do a hard course correction to get it back on track to helping me. In the end, I needed to come up with the novel implementation, and used ChatGPT to help me clean it up. It was great for coming up with test cases, not so great at generating efficient code to cover all the corner cases.
General conclusion on the ‘AI generates code’ front is; Yes, the various “AI” tools out there can generate code. Those tools are best wielded by sharp eyed experienced devs who know when the thing is making a mistake, or coding itself into a corner. I know this runs counter to the “AI Agents to rule them all” narrative that’s currently circulating, but it’s been my lived experience. At the same time, I an also say, over the two years that I’ve been using CoPilot and ChatGPT, these things have improved a lot, and will continue to do so. Also, OpenAI and MSFT tools are not the only ones in this area. There are plenty others, and more coming all the time. As we programmers use them more, providing guidance and more code for them to look at, they’re getting better. Less hallucinations, and more actual working code.
Besides code, there’s been a ton of work going on in the ‘generative image’ part of the AI game.
“create an image of a teddy bear washing dishes”