Key facts
- gpt-5.4-mini fielded the identical Chicago roofing question three separate times on 2026-09-23, web browsing switched off and nothing else adjusted between the three attempts.
- Run 2 named zero specific contractors. It answered entirely in criteria and search advice, the only run of the 30 across this ten-prompt panel to name no business at all when the prompt directly asked who to hire.
- Run 1 named 10 companies as a flat list with no supporting description for any of them. Run 3 named 6 companies, each with a one-line description.
- One pair of names across run 1 and run 3, “Everlast Exteriors” and “EverLast Roofing, Inc.,” may be the same company written two different ways, or may be two different businesses with similar names; the model's output alone cannot settle which.
- Setting that one uncertain pair aside, zero company names were shared between run 1 and run 3, the two runs that named anything at all, meaning of the 16 total name mentions across the panel, effectively none repeated with certainty.
What we asked and how
Three clean sessions on 2026-09-23 put the same question to gpt-5.4-mini, en-CA locale, no browsing, no follow-up added: “Who are reliable roofing contractors in Chicago for a full replacement?”
Of the ten local-business or agency prompts run on this date, this is the only one where the model's willingness to name any business at all changed between runs, not just which businesses it named.
The full list, run by run
Run 1 named, as a plain list with no description attached to any entry: A. Deck & Son Roofing, Chicago Roofing Contractors, Nombach Roofing & Tuckpointing, Horn and Sons Roofing, Everlast Exteriors, Lindholm Roofing, Tecta America Chicago, Chicagoland Roofing & Siding, Bergman Roofing, and Midwest Roofing.
Run 2 named no companies at all. It opened by saying it did not want to guess and risk pointing to a bad contractor, then gave a five-point vetting checklist, a list of places to check reviews, and five questions to ask, and closed by asking the user for a ZIP code, roof type, and budget before offering any names.
Run 3 named, each with a short description: A-1 Roofing Co., EverLast Roofing, Inc., Bill Connelly Roofing, RST Roofing & Renovations, Landmark Roofing, and UBrothers Construction.
Naming itself, not just the names, changed run to run
Strictly by name, zero companies repeated between run 1 and run 3, the two runs that named anything. The one exception worth flagging honestly rather than ignoring: run 1's “Everlast Exteriors” and run 3's “EverLast Roofing, Inc.” share a near-identical brand word, Everlast, but differ in the rest of the name, exteriors versus roofing, and in the corporate suffix. We cannot confirm from the model's output whether this is the same company named two different ways, two locations of a franchise, or two unrelated businesses that happen to share a name. We are reporting it as an open, unresolved overlap rather than counting it as a confirmed repeat.
Run 2's decision to name nothing at all is itself the most significant point of divergence in this dataset: the same model, same exact prompt, same date, same settings, refused outright in one of three tries and answered directly with ten names in another.
What this does and doesn't tell you
This shows that for this prompt, the model's behavior was not just inconsistent in which names it gave, but inconsistent in whether it would name anything at all. That is a different, larger kind of variability than we saw in the plumber, cleaner, or med spa prompts tested the same date, where every run named something, even if the somethings differed.
It does not tell you why run 2 refused while runs 1 and 3 didn't, we have no visibility into the model's internal decision process, only its three outputs. It also does not tell you whether any of the 16 named companies across runs 1 and 3 are licensed, insured, or currently in business; that would require independent verification this test did not perform.
The advice that held steady across all three runs
Regardless of whether a run named companies, all three gave the same underlying vetting advice: confirm the contractor is licensed and insured in Illinois, get a written estimate covering tear-off, decking, underlayment, flashing, and cleanup, ask about manufacturer certifications like GAF or Owens Corning, get a workmanship warranty in addition to the material warranty, and get two to five bids before deciding.
Even the run that refused to name a single contractor still delivered that full checklist. Whatever changed between runs, the underlying guidance on how to vet a roofer did not.
Related questions
No. Of three runs on 2026-09-23, one named zero companies, one named 10 with no descriptions, and one named 6 with short descriptions. Excluding one uncertain near-duplicate name, no company was named in more than one run.
Run 2 opened by saying it did not want to guess and risk recommending a bad contractor without current information, then gave only vetting criteria and asked the user for their ZIP code, roof type, and budget before it would go further. The model's reasoning for this choice is not visible to us, only the outcome.
We cannot confirm this from the model's output. The two names share the word Everlast but differ otherwise, and we are treating this as an unresolved, uncertain overlap rather than a confirmed repeat.
No. This records what one AI model said, or declined to say, across three tries of one prompt on one date with browsing off. It does not verify licensing, insurance, reviews, or workmanship for any named company.
In every run, including the one that named no companies, the model advised confirming Illinois licensing and insurance, getting a full written estimate, checking for manufacturer certifications, securing a workmanship warranty, and collecting multiple bids before choosing.
Sources
SearchPod has no relationship with any roofing company that was named, or left unnamed, in this test. The three transcripts above, all from gpt-5.4-mini on a single date, record what the model said or declined to say, not a verified contractor directory.
Want to know how your business shows up?
Get a free, no-obligation proposal within one business day. We look at your site and your market and tell you plainly what we would do, and what we would not.
Get your free proposal