How I used ChatGPT Plus to automate email support for a small business – with Claude’s help

ZDNET’s key takeaways

• ChatGPT Plus can do this job, but it could not build the system on its own.
• It is important to have one AI check the work of another AI.
• Answering customer emails does not have to be exhausting.

This is the story of how I successfully automated customer support emails for a small online store. It is also the story of how the process slowly drove me to lose patience and seriously consider strangling ChatGPT.

It all started because my wife’s online store was getting far too many customer support emails. I also write a column about AI, so I decided to see whether I could use it to help.

My wife’s online store sells digital products for a popular hobby. Most real customer support emails ask for help downloading products, fixing password or login problems, or using the products themselves.

About half of these requests can be answered with standard replies. My wife had already been handling most of them this way. She had created several TextExpander templates, which were enough to answer many common, genuine requests.

Only about 1–2% of the emails she receives are actual customer questions. The rest are different kinds of spam, PayPal promotional emails, and many newsletters that she has hardly ever signed up to receive.

All these messages arrive at the company email address, which is separate from her personal email. That was helpful because I could work on automating the company’s email without putting her personal messages at risk.

The AI needed to handle two main jobs: first, identify and sort incoming emails; second, process each message in the right way after identifying it.

Unfortunately, Gmail’s filters were not good enough because the emails varied so much. For example, the filters could not reliably tell the difference between a PayPal marketing email and a customer complaint that needed an immediate response. AI systems are good at spotting these kinds of differences.

I have been writing step-by-step, or rule-based, programs for decades. In this kind of programming, I define each step and the conditions that determine what happens next. This was my first time building a program in which the workflow itself decides what to do next.

I started by looking for a way to process incoming customer support emails. I was determined to find out whether my wife’s $20-per-month ChatGPT Plus plan could handle the whole job. It is now doing that.

The problem was getting the AI to work as a customer support assistant. The process involved so much frustration that I began questioning some of my life choices. More than once, I wished I had been raised by wolves and had never encountered computers, the internet, email, keyboards, or AI.

Attempt #1: The one with the spreadsheet

My biggest and most time-consuming mistake was letting ChatGPT design the whole system and then trusting it for almost four weeks to know what it was doing.

Its design looked solid and believable. When this stage ended, I felt like a homeowner who had hired a contractor with impressive plans and paid half the bill upfront, only to discover that the contractor had no idea how to turn those plans into real work.

ChatGPT designed a system with several layers, all organized around a shared Google spreadsheet. The layers covered:

• Controlling the program and collecting incoming messages
• Classifying emails
• Checking orders and payments
• Sorting customer support requests by priority
• Checking for spam and safety risks
• Taking actions in Gmail
• Mapping messages to the right actions, choosing replies, and checking the results
• Scheduling tasks and setting checkpoints

The idea was that every email, identified by its unique message ID, would pass through one or more stages until it had been answered and filed or simply filed. My wife already had a carefully organized system of email labels. The AI was taught to use those labels. Any message it could not safely handle would go into a folder called “Human Attention Required,” where my wife could deal with it herself.

At first, the AI collected 240 messages from the backlog in the inbox. These messages gave us a good sample of the kinds of emails the support account received. We used them to build a system that covered most types of mail: customer messages, orders, payments, vendors, security notices, newsletters, sales pitches, and spam.

Then we entered what seemed like an endless cycle. The AI tried to find features that would help it classify each email. It created a spreadsheet where I rated how well it had classified every message. It then used my feedback to adjust its understanding.

Once the AI understood what each email was about, it had to decide what to do with it. The possible actions included replying and filing the message, filing it without replying, moving it to the “Human Attention Required” folder, or marking it as spam. At least during the first few weeks, it sometimes came up with completely bizarre AI-generated categories that made no sense at all.

We kept reviewing unfamiliar messages and unusual cases until the AI finally learned to understand the customer support emails that arrived each week.

The AI created separate ChatGPT conversations for each stage of the process. It kept asking me to copy and paste instructions from one conversation to another. One conversation would classify the emails, another would check for orders and payment issues, another would sort support requests, and another would write replies, with more conversations handling other tasks.

But ChatGPT could not use those separate conversations as working services that other parts of the system could call on. That was exactly what we were trying to build. The AI gave each conversation a name that sounded like a software component and kept insisting it could use the conversations as real software components.

This was my first attempt at building this kind of system with AI, so I kept going along with the plan.

Eventually, I ran out of patience. That led to my second failed attempt.

Attempt #2: The first one with a skill

I finally lost my patience and challenged ChatGPT. It kept insisting that the process we had been working on would succeed. But I have worked with contract programmers before, and I know the feeling when someone is making promises without having the skills to deliver.

ChatGPT was no different, although it was more eager to please and seemed more willing to help than many contractors I have argued with.

Eventually, I convinced it that using several separate conversations was not working and that we needed a better approach. I asked, “What about turning this all into a skill?” ChatGPT thought that was a great idea.

It then examined the spreadsheet, which had grown into a monster with 73 tabs. Using that information, it created a skill with a 12-stage processing pipeline, formal rules, checkpoints, checks to confirm results, and safeguards meant to prevent messages from being processed twice or lost. The earlier system of separate conversations became a 12-stage process inside the ChatGPT skill.

The test runs appeared to work. The AI ran 59 unit tests and 139 regression tests to check whether the process was reliable. It reported no mismatches. But there were problems. Oh, were there problems.

First, despite all those tests, the system did not actually work. The tests checked rules inside the AI model and simulated what might happen. They did not test whether the system could process real emails on its own without someone watching it.

The scheduled automations also failed. They could not coordinate the different pipeline stages, made no actual changes in Gmail, and then stopped. There was no way for them to recover and continue working.

My second attempt was supposed to turn the overly complicated email system into a reliable service that could run by itself. Instead, it made the system even more complicated without fixing the main problem: it could not reliably carry out the full process.

Attempt #3: The one where I brought Claude in to consult

I have found that Claude works very well as a coordinator that helps other AI systems. I have used it to build several AI systems on my local Mac Studio home lab computer. I may write more about that in a future article. Since Claude had already helped me review and guide other AIs successfully, I decided to ask it to help solve my ChatGPT problems.

First, I asked ChatGPT to prepare a complete report describing what it had done so I could give it to Claude. Claude was not impressed. It called ChatGPT’s work, “A failed bank-grade architecture for a hobby-download site which contradicted David’s requirements and resulted in excess complexity without value.”

Have you ever seen one contractor praise another contractor’s work? Whether the job is computer programming or plumbing, people often think someone else’s work is terrible. Still, Claude had a point. ChatGPT’s plan did not work and was far more complicated than it needed to be.

I still wanted to run the system through my wife’s $20-per-month ChatGPT Plus account. But I decided to use my $100-per-month Claude Max account to develop the skill and operating process, which would then be installed in my wife’s e-commerce service accounts. This approach worked very well.

First, Claude threw away most of the information we had collected in the spreadsheet. Instead, it took a different approach. It asked me to visit the support account’s Google Takeout and export the account’s entire email history. Claude then read that export. By studying the messages, it worked out what kinds of emails the account received and how the replies had been handled.

The export was a 5.24GB file containing 88,777 messages, including a record of 40,580 sent messages.

This was a customer support account, so I was not too worried about sharing its full history with Claude. I would not have felt comfortable doing the same with a personal account or one containing important confidential business information. Even so, Claude learned the full history of my wife’s e-commerce support operation in less than an hour. I had time to eat lunch while it worked, instead of spending weeks testing and correcting the system.

Claude then worked out how the emails had been filed. This was an important part of the process and had taken a lot of time before. It identified most of the main customer support questions and how they had been answered, allowing us to use a set of reply templates. It also recognized sales pitches, spam, and other messages that could be filed away or discarded.

All this information went into a Google Doc called the “Mail Bot Rulebook.” We will return to that shortly.

Claude built the workflow

Next, Claude began building the workflow.

First, it tested what had been going wrong with ChatGPT’s system. The main problem was that ChatGPT could not reliably send messages through Gmail. A common recommendation is to leave AI-written messages in the Drafts folder so a person can check them before sending.

But a major goal of this project was to let the system reply to customers automatically. That meant the AI had to be able to send emails, not just prepare drafts.

Claude suggested splitting the work into two clear parts: a scheduled ChatGPT event and a scheduled Google Apps Script. The ChatGPT event uses a process that can make different decisions, while the script follows a fixed set of steps.

Once a day, a scheduled ChatGPT event reads the rulebook and processes the new messages that arrived that day. It files messages that it can handle. For messages that need a reply, the rulebook tells the AI whether to move them to the “Human Attention Required” folder or prepare a reply using one of ten ready-made templates.

When a request can be answered with a template, ChatGPT creates a draft reply and puts it in Gmail’s Drafts folder. The original customer email is left unchanged in the inbox.

Then, every hour, a Google Apps Script runs separately from ChatGPT. It checks the messages in the Drafts folder and decides which ones it can send, using a clear set of rules:

• It sends messages signed with the name assigned to the AI support representative. It also sends messages containing a specific phrase that marks them as urgent and critical. Messages of this second kind go straight to my wife. Serious system problems and issues with payment services count as critical.
• It sends only drafts that are at least 10 minutes old, and it sends no more than 10 drafts each time it runs.
• After sending a draft, it checks that the message appears in Sent Items. It then gives the original incoming email an Inquiries label and removes it from the Inbox. If it cannot confirm that the reply was sent, the original message stays in the Inbox. So far, the only messages that remain stuck there are Google’s own security alerts, which neither Apps Script nor ChatGPT is allowed to change.

During the first two weeks, we updated the rulebook 15 times to make small improvements. The system has now been running reliably for some time.

Claude still has knowledge of the work it did, but it is no longer actively involved in the project. The system now runs on its own using ChatGPT and Google Apps Script, without someone having to manage it all the time.

The numbers justify the system

Before this system was in place, my wife faced a wall of junk every time she opened the customer support inbox. We tested one group of old messages to see how much of the email was genuine. Out of 1,689 messages in that backlog, only 25 were from real customers. That was about 1.5% of the email in that batch.

About half of those 25 messages could be answered automatically. Only the other half really needed a person’s attention. We could have added more automatic replies, but my wife likes seeing messages such as “look what I did” and “I love your product,” so we set up the AI to leave those messages unanswered by automation.

That leaves her with 11 messages out of the 1,689 in that batch that actually need her to respond. This is not only a big reduction in work. It also greatly reduces the mental effort and stress involved.

Previously, every time she checked the new email, she had to brace herself for what she might find. It took a lot of willpower. Now, checking customer support email is something she looks forward to.

Here is a comparison that explains the result. Imagine the mailroom of a large company. At first, the company’s leading product expert had to look through every incoming email to find the few that needed her specialist knowledge. Now, only those important messages are delivered directly to her desk. She can spend more time creating value for the business and doing all the other wonderful things my wife does each day.

If you decide to try a project like this, I recommend keeping six things in mind:

• Keep the project as small and focused as possible.
• Try not to use it for anything essential to the business.
• Expect it to take longer and cause more trouble than you can imagine.
• Do not hesitate to ask one AI to review another AI’s plans.
• Remember that AI systems can struggle with step-by-step workflows.
• Limit the AI’s access to email and company systems. Never give it access to your personal email or confidential business services. You cannot know in advance what it might try to send.

Building this system was a long, difficult job, and I did not enjoy it. But the value we get from using my wife’s $20-per-month ChatGPT Plus plan for this task is almost impossible to measure.

This is where AI can really help. We do not have a team of customer support workers whose jobs are being taken over by AI. My wife runs the business by herself. The AI did not put anyone out of work. Instead, it gave her more time and improved the service customers receive. We have already received several emails from customers saying things like “Thank you, that helped” after getting quick, useful replies from the AI.

And yes, I know that messages saying “Thank you, that helped” are still incoming emails. But those are exactly the kinds of messages you want to see every morning.

Good luck! If you have tried or are trying something like this, tell us how it went in the comments below.