I’ve said this to owners in almost these words: you tell your people, go use AI, go use ChatGPT, go use Claude. They go and do it, but they have no clue what they’re doing. They might be leaking private data without knowing.
The Australian Bureau of Statistics counted 12% of Australian businesses using AI in 2024 to 2025, up from 1% three years earlier. Surveys that ask staff come back far higher. Some of that gap is method, because the ABS counts businesses and those surveys count people. Not all of it.
Which account they’re signed into decides most of what happens next, and almost nobody has checked. This article ends with how to check yours.
What happens to what you type in?
Three questions hide inside that one: is it used to train the model, how long is it kept and who can reach it?
The Australian Signals Directorate, writing for small business with its New Zealand counterpart and the Council of Small Business Organisations Australia, answers the first one directly. Some providers may use customer-submitted data to train or refine their models, it says, and this “can depend on the configuration settings or the type of subscription”. Information entered “could potentially be reused or disclosed in unexpected contexts later”.
The specifics, as they stand today. On ChatGPT’s consumer plans, free, Plus and Pro, OpenAI’s help centre says the product improves by further training on the conversations people have with it, unless you opt out. On Claude’s consumer plans, free, Pro and Max, it’s a choice rather than a default: Anthropic asked users to decide. Allow training and conversations are retained for five years. Decline, and deleted conversations are purged within 30 days instead.
That last point is widely misread. Declining doesn’t make your chat history disappear after a month. It sits in the account until someone deletes it. The 30 days is how long deletion takes to reach the back end.
Does the plan you’re on change the answer?
Yes. More than anything else you could do this month.
OpenAI’s enterprise privacy page states that on its business, enterprise, education and API products it doesn’t train on your data by default. Anthropic’s commercial tiers, Claude Team, Enterprise and the API, were explicitly excluded from the consumer terms change above.
So the practical question is which account your bookkeeper was signed into on Tuesday.
This is where it comes apart in smaller businesses. Someone starts on a personal subscription because they wanted to try it, it works and it quietly becomes part of how they do their job. There’s never a moment where a decision gets made. The firm now has client information going through an account it doesn’t own, can’t see and can’t switch off when that person leaves.
Here’s a qualifier that cuts against my own advice. A business plan changes the vendor’s default, and the vendor still has to obey the law where it operates. In May 2025 a US court ordered OpenAI to preserve output logs it would otherwise have deleted, including for free, Plus, Pro and Team subscribers. Enterprise, education and zero-retention API customers were outside it. That order ended in September 2025, but data already preserved stayed preserved. Published terms are a policy, and a policy isn’t a guarantee.
Has this gone wrong in Australia?
Twice, publicly, and neither was caught by a security alert.
The older case is the more uncomfortable. In September 2024 the Victorian information commissioner published an investigation into a child protection worker’s use of ChatGPT to help draft a Protection Application Report for the Children’s Court, arising from an incident reported in December 2023. The worker said they’d removed identifying details. The regulator concluded, on the balance of probabilities, that client names had been entered, and found the department in breach of two Information Privacy Principles.
What gave it away was the writing. The draft mischaracterised evidence in the case badly enough that a departmental lawyer noticed the language about a week later, in a report going to a court about a child’s safety. The privacy failure and the accuracy failure arrived together, which is the part most accounts miss.
The scale is the detail I’d take from it. The department found that almost 900 staff had accessed the ChatGPT website over six months, close to 13% of its workforce. Nobody had decided that. It had simply happened.
Then, in March 2025, a former temporary staff member at the NSW Reconstruction Authority uploaded a spreadsheet to ChatGPT, which wasn’t an approved tool there. It held more than 12,000 rows of applicant data from a flood recovery program. Investigation later established that 2,031 people had some of their data uploaded: names, contact details, dates of birth and sensitive personal and health information. The NSW Privacy Commissioner was notified, and investigators found no evidence the data had been accessed by anyone else or appeared publicly. ASD’s small business guidance now carries an anonymised example that reads unmistakably like this one.
Is this a privacy problem or a security problem?
Both. The privacy half is the one with a regulator attached.
The OAIC’s guidance on commercially available AI products says that, as a matter of best practice, organisations should not enter personal information, and particularly sensitive information, into publicly available generative AI tools. That’s been the regulator’s position since late 2024.
Whether the Privacy Act applies to you turns on your turnover and on what you do. The small business exemption still sits at $3 million, but the carve-outs catch more firms than owners expect, and since 1 July 2026 the anti-money-laundering reforms have brought accountants, lawyers, conveyancers and real estate businesses into scope for the personal information they handle under those obligations. The thresholds, and what the December 2026 obligation on automated decisions reaches, are in whether Australian businesses still need AI governance. The commercial exposure doesn’t track the legal one, though. A client who finds their information went into a tool they were never told about won’t check whether you were technically covered.
There’s a fair objection to all this. Notifiable data breaches reported to the OAIC hit an all-time high in 2025, 1,205 notifications, up 8% on the year before. AI isn’t among the causes. It’s never been a category in that reporting at all.
Two reasons not to take much comfort from it. The federal scheme has no AI cause code, so a breach that began with an AI tool gets recorded as human error and disappears into a number that rose sharply last year anyway. And both Australian cases above sat with state regulators, Victorian and NSW, so neither was ever in the OAIC’s figures to begin with. The Victorian investigation added the sharpest point: the regulator recorded that it was impossible to verify what had actually been typed in, because organisations have no visibility of what staff put into these tools. That’s a gap in the measurement, and it says nothing either way about the risk.
What about the tools nobody approved?
You can’t manage what you’re pretending isn’t happening.
A survey of 500 Australian technology decision-makers by Josys, a vendor that sells software for exactly this problem, found 36% reporting staff uploading sensitive company information to AI tools and 24% reporting customer personal information going the same way. Read it with that interest in mind. The figure I find more telling is a quieter one: only 30% of firms under 250 staff felt equipped to assess AI risk, against 42% of larger ones. The survey’s from September 2025, so treat it as a direction of travel.
The instinct is to ban it. That fails in a specific way: the tool moves to a personal phone, the work carries on and you lose the only advantage you had, which was being able to see it.
Does the vendor’s security handle this for you?
No, and this is the assumption I most often have to take apart.
Vendor certifications tell you the vendor runs a tight operation. They say nothing about which of your client folders somebody connected to it last month. The vendor secures their side. What reaches the tool from your side is yours to decide, and no product makes that decision for you.
The vendor’s own controls don’t always hold either. In early 2026 a defect in Microsoft 365 Copilot’s work chat summarised confidential-labelled emails in users’ own Sent Items and Drafts, despite data loss prevention policies set up to keep that mail out. Microsoft detected it on 21 January, began rolling out a fix in early February and reported the fix in place across all affected environments in late February. It hasn’t said publicly how many tenants were affected. The controls you set are a claim about how a product should behave, and products have bugs.
That gap widens as soon as you connect anything. Once a tool can see your files or your mail, anything it can reach is in play, and permissions are always easier to widen than to pull back. I’ve written about scoping that in deciding what to let an AI system touch.
How do you check your own setup?
Half an hour, and you can do it yourself today.
Find out what people are signed into. Ask, and make it safe to answer honestly. Then check the expense claims and the company card statements for the last three months. Personal AI subscriptions show up there as small recurring charges, and they’re the clearest evidence of work happening on accounts you don’t control.
Open the settings on each account. In ChatGPT it’s Settings, then Data controls, where the switch called “Improve the model for everyone” decides whether your conversations are used to train the model. In Claude it’s Settings, then Privacy, and the switch is “Help improve our AI models”. Both take under a minute and both tell you what’s been true for however long that account has existed.
Check whether you’re already paying for a better option. If you run Microsoft 365 or Google Workspace, the assistant bundled with your plan has different data handling from the consumer app of the same name, and it may be sitting there switched off. Ask your IT provider which tier you hold and what its terms say. The brand name won’t tell you. For Copilot, the label under each person’s name will.
Then do the four things that stick. Move everyone doing client work onto a business plan. Write down what may not go in, in a short paragraph: names, health information, financial detail, anything under a confidentiality clause. Ask what each tool is currently connected to and remove anything nobody needed. Name someone who owns it, including the bad weeks.
ASD’s own list for small business runs along the same lines: know what data you hold, read the vendor’s handling terms, write down what may not be uploaded, train people and strip identifying detail before it goes anywhere.
All of that is about the setup. What it doesn’t settle is what the output then decides, and that’s where exposure changes if AI is anywhere near decisions about people. The AI risk assessment takes one use case through both and ends on a decision: proceed, proceed with conditions or not yet.
Once the setup’s right, getting everyone using AI the same way is the next job. If you’d like help working through your own setup, that’s what AI training is for. Or tell us how your team uses AI on a discovery call, and we’ll come back to you with what to change first.
FAQ
Is it safe to put client information into ChatGPT?
On a personal plan, treat it as unsafe. Consumer ChatGPT may use conversations to improve the model unless you’ve opted out, and the account isn’t yours to control. On the business and enterprise tiers OpenAI states it doesn’t train on your data by default, which is a materially different position. Separately, the Australian privacy regulator recommends as best practice not entering personal information, and particularly sensitive information, into publicly available generative AI tools at all.
Does ChatGPT train on what I type?
On the consumer plans, yes, unless you turn it off in the data controls section of Settings. On the business, enterprise, education and API products, training on your data is off by default. Retention is a separate question from training. A court order in 2025 required OpenAI to keep logs it would otherwise have deleted, on consumer and team plans alike, which is worth knowing before you treat any published retention period as a promise.
Does Claude train on my conversations?
On the consumer plans it’s a setting each person chooses. Since Anthropic changed its consumer terms in August 2025, new users pick at sign-up, existing users had to choose by 8 October 2025 and anyone can change it later in the privacy settings. Allowing it means new and resumed conversations are kept for five years. Declining means deleted conversations are purged within 30 days, which isn’t the same as your history clearing itself. The commercial tiers, team and enterprise and API, were excluded from that change.
Should we ban AI tools at work?
I wouldn’t. A ban moves the same behaviour onto personal devices and personal accounts, where you have no visibility and nothing to switch off. A permitted tool on a business plan, with a short written rule about what may go into it, gets you further than a prohibition everyone quietly works around.
Someone has already put client data into a personal account. What now?
Don’t delete anything yet, and don’t start with blame or you won’t hear about the next one. First preserve the record: export or screenshot the conversation, because it’s the only evidence of what actually went in, and Australian regulators have found that organisations usually can’t reconstruct this afterwards. Then assess whether anyone’s personal information was involved, which is the point to take advice, and note that the notifiable data breach scheme gives you 30 days to complete that assessment once you suspect a breach. Clean-up, settings and moving the person onto a business plan come after that.
General information only, not legal advice. Current as at October 2026: vendor terms and retention periods in this area change frequently, and the position may have moved by the time you read it.
