This is a REALLY random question. I have never (knowingly) used AI, and I’m wondering if AI can determine the probability a document was largely written by AI (or using templates available on-line).
And, if AI can do so, does anyone have a sense of how accurate this is? The document in question are legal documents (LLC Operating Agreement and an Occupancy Agreement).
And, finally, does anyone have strong preferences about which AI agent might be the most useful for such a task?
+1 for Claude. Recent models on high reasoning detect it very well. Here’s the sample I ran on one of the posts here recently:
Yes, likely AI. Tells:
Uniform hedging cadence — every paragraph opens “I’d definitely / I’d ask / I’d also ask / Your best move is probably.” Human advice clusters emphasis; this spreads it evenly.
Balanced-closer reflex — ends on “sounds like a pretty reasonable compromise.” AI loves the reassuring both-sides wrap-up.
List-of-three padding — “Grand Highlander Hybrid, CX-90 PHEV, and other 3-row options.” Named examples plus a vague catch-all = classic filler.
No stumbles — no typos, no fragments, no personal aside, perfect parallel structure across five paragraphs. Real forum/text advice is messier.
Safe-hedge on every claim — “may be,” “it may have,” “I wouldn’t assume.” Nothing committed, nothing risked.
Counter-signal: content is actually competent and specific. So either AI with good domain prompt, or a knowledgeable human who writes clean.
Verdict: ~75% AI-assisted or AI-written. Substance good, voice synthetic.
they can but public models vary widely. it also depends on the type of content you’re having it check. a resume, for example, will often show written by AI due to their buzzwordy and listy nature even if written entirely by a human. contracts might be similar
Part of the reason I asked the question(s) is b/c an LLC of which I’m a part has had a new attorney re-draft our governing docs. The new docs are easily double to triple the length of the old ones (unnecessarily so), written in a very confusing and “dense” manner, contain sections are not “do-able” from an accounting or enforcement perspective, contain numerous (new) provisions that could be used to intimidate and bully partners (we were in something akin to a joint venture), and often contain things that are largely irrelevant. All of this made me wonder if the docs were largely drafted using AI.
But I guess the significant thing is that, regardless of how the docs might have been drafted, I think the docs are fundamentally flawed for both macro and specific reasons.
The real problem is that I think I work w/ a group of people who are largely very apathetic and likely don’t understand that governing docs are a legally-binding contract and not simply suggestions (so they are prone to signing things w/o reviewing them fully)…
Could be AI but could also be someone took a form or precedent off the shelf and filled in the name, etc. without giving any thought to whether it works or reflects the commercial understanding. Happens all the time even with an attorney involved.
This. From my experience, unless directed, AI does not generally take overly aggressive positions (especially in governing docs). My guess is also that this is a precedent document that the attorney just churns for a variety of Clients (and since most Clients are unsophisticated and do not even glance at these documents, no one has ever raised any of these issues to him/her).
Also, even if it is AI generated, it sounds like the attorney did not take the time to review and revise as needed to make it streamlined and applicable to the commercial reality…which I’d argue is worse
This is helpful to know (but disturbing to read b/c much of the docs are pretty aggressive, IMHO, but IANAL).
No, it’s even worse than that, I think. There have apparently been a series of meetings btw managing group and attorney over the past 8 mos and already a lot of back and forth. And the attorney has charged the LLC (are you sitting down?), $8K to come up w/ this pile of garbage.
When I asked one insider (I am on the outs w/ much of the managing committee) why there are so many sections that are irrelevant or could be used to bully/intimidate, the responses I got were (paraphrasing), “The docs are so long that I’m not sure if any of us read through all of them” and “The committee wasn’t worried about sections being enforced b/c that particular situation hasn’t occurred in the past.”
The attorney has also inserted himself into the agreements, giving himself final approval power in certain situation that will most definitely occur (unless none of us ever sell or transfer a share).
And, yes, I have already alerted all of the partners that I am concerned about this attorney (and there is only one other partner that seems equally concerned, and that partner is not one of the managing group).
My vague memory is that we did actually try Discourse’s built-in AI detector for a bit, but I think it was overly-aggressive in flagging things, so we turned it off? Can’t remember distinctly…
AI can estimate this probability, but it is highly unreliable for legal documents because standard boilerplate text naturally mimics AI patterns. Dedicated tools like GPTZero or Copyleaks can analyze the text, while advanced models like Claude are better at spotting online templates. However, you should treat any “AI score” as a measure of how generic the document is, rather than definitive proof of its origin.
