Three investment staff. 450 companies on their watchlist. Before every quarterly committee meeting, someone spent two to three days reading PDFs to prepare a qualitative brief. This is what we built to fix that and what it took to get it right.
Emerald Wealth Partners is a private family office. Its actual portfolio is small on purpose, roughly 20 to 25 positions. But the watchlist behind it is deliberately wide: 450 public companies tracked as future candidates, competitive benchmarks, and sector context. Three investment staff (a principal and two associates) do all of the research.
FactSet covers the financial data. Everything else, what management said about pricing power, capital discipline, competitive position, or AI investment, still came from reading the filings by hand.
The problem was time. Before every quarterly investment committee meeting, someone spent two to three days reading PDFs to prepare a qualitative summary. For a new position, deep research meant an associate working through a company’s last four annual reports plus recent earnings call transcripts (roughly 1,200 pages) over several days.
We built a private AI search tool over their entire research corpus: 2,200 annual reports and earnings call transcripts across 400 companies. The 50 remaining watchlist names were excluded because they lacked four years of SEC filing history at the time of corpus assembly. The corpus covers the four most recent annual filings per company (10-K and 20-F), plus transcripts sourced from 8-K exhibits where filed with the SEC.
Ask a plain-English question. Get a short, cited answer. See the exact passage it came from. Click through to the original SEC filing or transcript. Everything runs on their infrastructure. Their queries never leave their environment.
10 min
Investment committee research prep
Down from 2–3 days of manual reading per meeting cycle.
3×
New positions evaluated per quarter
Team now evaluates 12–15 new names per quarter, up from 4–5.
0%
Citation errors since go-live
Every answer links to the exact passage in the original filing or transcript.
50+
Queries per week at 60 days
Cross-company comparisons and historical tone analysis are the most common types.
The single number that mattered most to the team, though, was the one on research prep time itself.
Before / After
450
companies on the watchlist
2,200
filings & transcripts indexed
0
citation errors since go-live
Quarterly investment committee meetings at Emerald are structured around a qualitative brief: how has management tone shifted across the portfolio, what’s changed in the risk factor language, are the things companies said they’d do in the last annual report showing up in recent earnings calls. Preparing that brief took two to three days every quarter.
Before one particular meeting, the principal wanted a simple answer. Had 15 retail and consumer companies stayed consistent in how they talked about tariffs and rising costs over the last two years, or had the language quietly shifted? Answering that meant reading roughly 60 quarterly filings across those 15 companies, then cross-checking it all against what was said on each earnings call.
The associate spent three days on it. The committee meeting happened on day two. The principal walked in with partial notes.
Afterwards, she said the thing she’d been saying for a year: “There has to be a better way to do this.” This time, she asked us to scope it.
One of the associates had built a prototype herself: OpenAI’s API, a few PDFs uploaded per session and a simple chat interface. It worked for single-company, single-filing questions. It couldn’t hold more than one or two documents in context, which ruled out any cross-company or cross-year question. And occasionally the answers referenced passages that, when checked against the source filing, didn’t say quite what the model claimed.
She called it plausible hallucination. The wrong answer sounds exactly right. It’s worse than no answer because you don’t know to verify.
Two investment memos went out summarizing a company’s management language on pricing power using citations the model had partially fabricated. The passages cited existed. The specific sentences quoted didn’t. It wasn’t a big deal because both memos were for internal discussion.
When they came to us, the requirement was simple: either the answer comes with a source you can click through and verify, or there is no answer.
Their FactSet subscription handles quantitative data precisely. When they need revenue, margins, debt ratios, or consensus estimates, FactSet is the right tool. What FactSet’s standard research workstation doesn’t do is answer “how has this management team described its approach to capital allocation across the last five annual reports, and has that language changed?” That question requires reading. FactSet has added AI-powered features in recent years, but those work across FactSet’s own data universe, not a curated private corpus scoped to the 400 companies they actually track.
General AI search tools (Perplexity, built-in AI features in various research platforms) cite news articles and analyst commentary. For fundamental research built on primary sources, a paraphrase in a news article is not a citation. You want the 10-K sentence, not someone’s summary of it.
The associate’s prototype had the right instincts: private, local, working straight from primary sources. But it only worked one session at a time. It couldn’t hold a permanent library of 2,200 documents. It couldn’t answer cross-company questions because it couldn’t hold more than a document or two in view at once. And it made things up just often enough to be unusable.
What they needed was something that kept the whole corpus in memory permanently, searched it by meaning and by exact words simultaneously, and refused to answer when the evidence wasn’t strong enough to support a confident response. That last behavior (saying “no data” instead of guessing) turned out to be the hardest thing to build correctly.
There was also a data question they raised early and we took seriously. What Emerald is researching is the most sensitive possible signal for an investment office. Having any of that processed through a third-party vendor’s systems creates risk they weren’t willing to accept. The tool runs on their infrastructure with their API keys. A query about what Nvidia said about capacity in its last three earnings calls never touches a server they don’t own.
Seven weeks from kickoff to a tool the team could put in front of a real committee memo. Each week bought something specific, and skipping any of them would have left a tool nobody could actually trust.
Discovery + corpus scoping
Defined which 400 companies actually mattered to the watchlist, so the tool would answer questions about the right universe from day one
Ingestion pipeline: filings + transcripts
2,200 annual reports and earnings call transcripts pulled into one searchable corpus, updating itself weekly with no engineering involvement required
Smarter search + company matching
The engine that makes sure a question about one company doesn't return answers about a different one
Evaluation suite (13 → 83 grounded cases)
A standing test of whether the tool gives correct answers, not just plausible-sounding ones, before anyone on the investment team saw it
Retrieval tuning + three versions of the same bug
Closed the gap between "returns an answer" and "returns the right answer every time," including bugs that would have misattributed quotes to the wrong company
Faithfulness layer + source viewer
Every answer links to the exact sentence it came from, so nothing goes into a committee memo that can't be verified in one click
CI gates + delivery
Locked in the accuracy bar permanently, trained the team, and handed over a tool that keeps itself current
Built with: OpenAI for the search engine, Supabase to store and search the corpus, SEC EDGAR’s public data feed to pull filings, and a custom pipeline to keep everything organized and searchable. The query interface is a lightweight internal web app. Infrastructure and API costs run roughly $150–250 per month at the team’s current query volume.
The first two weeks were pure setup. Nothing else could get built until all 2,200 documents were pulled in, broken into searchable pieces, and stored.
The pipeline pulls each filing straight from SEC EDGAR, strips out the formatting, and breaks the document into two layers. Small passages, about 220 words each, are what the search actually scans to find the right answer. Larger windows, about 650 words each, are what the tool reads back to write the answer. A 220-word snippet rarely has enough surrounding context to make sense on its own. The tool needs the fuller passage to explain what it found.
Earnings call transcripts needed a different approach. Many companies file their transcripts with the SEC shortly after the call, as an attachment to a routine report. The pipeline pulls those in the same way it pulls other filings, but it treats transcripts differently once it has them: it keeps each analyst’s question paired with the answer, instead of splitting the conversation apart.
The corpus also includes foreign companies on the watchlist, like ASML, TSMC, and SAP, which file different report types than U.S. companies do. The pipeline handles both. Some of those foreign filings turn out to be blank cover pages with no real content. Those get logged and skipped automatically, without stopping everything else.
End of week 2: 400 companies, 2,200 annual reports and transcripts, roughly 160,000 searchable passages.
The pipeline also runs itself going forward, on a weekly schedule. As new filings land on the SEC’s site, it pulls them in and adds them to what’s already there, without duplicating anything or needing to rebuild from scratch. Each weekly run produces a report showing what got updated and what, if anything, got skipped. None of it requires an engineer to babysit it.
The first version of the search was barebones. A question came in, the tool found the 12 passages that sounded closest to it, and that was the answer. No check for which company the question was actually about. No keyword matching alongside the meaning-based search. No way to say “I don’t know” when the corpus had nothing relevant.
It worked on straightforward questions. “How did Nvidia discuss AI supply chain risk?” returned Nvidia passages. Everything else failed.
A question about input cost pressure could return three passages from the same filing on a completely unrelated topic, just because the wording happened to sound similar. A broad question like “which companies mentioned pricing discipline on recent earnings calls?” returned the same company six times, because nothing was pushing the tool to pull from multiple companies. A question about a company that wasn’t even in the corpus still got a confident-sounding answer.
We measured it. Roughly two-thirds of the time, the correct passage wasn’t even the top result. Every fix after this was aimed at one of these specific failures.
The most instructive failure showed up three separate times in different disguises.
The first version matched company names by checking whether any piece of the name showed up anywhere in the question. Single-letter stock tickers broke this immediately. AT&T’s ticker is “T.” The word “discussed” contains the letter T. Every question about AI strategy was getting routed to AT&T, and to Disney too, since Disney’s ticker DIS sits right inside the word “discussed.” The fix: only match whole words, never letters buried inside other words.
Then the same problem came back in a new shape, this time with ordinary words baked into legal company names. “Companies” is part of “Lowe’s Companies, Inc.” A question like “which companies mentioned pricing discipline?” was filtering results down to Lowe’s. “Electric” is part of General Electric’s name. “Technology” appears in the full legal name of several companies in the corpus. The fix: any word that shows up in more than one company’s name gets dropped from the matching list, leaving only the words that actually identify a single company.
Third, abbreviations. Someone typing “AI” in a question didn’t match filings that spelled out “artificial intelligence” in full. An analyst searching for AI investment was missing half the results without knowing it. The fix: the tool now expands common abbreviations before searching.
None of these problems showed up just by reading the code. All three were caught because the test suite kept asking a simple question after every search: which companies actually came back, and is that the right list?
“What did management say about AI strategy?”
Returned AT&T (ticker "T") and Disney (ticker "DIS", found inside "discussed")
Only whole-word ticker matches count. AT&T and Disney no longer appear unless named.
“Which companies mentioned pricing discipline?”
Filtered down to Lowe’s, because "Companies" is part of "Lowe’s Companies, Inc."
Words shared by more than one company name are dropped from matching.
“Which companies discussed AI investment?”
Missed filings that spelled out "artificial intelligence" instead of "AI"
Common abbreviations now expand automatically before searching.
For about a week, searches took 40 seconds. We’d built the search index, confirmed it was working, and just watched every question crawl.
The cause was a single setting buried in the database’s search configuration. It was silently capping every search at 40 results, no matter how many were actually requested. Broad questions that needed 80 results to pull in answers from multiple companies were only getting 40 back, with no warning that anything had been cut off. Results just looked thin, and the ranking looked wrong. On top of that, the search was doing extra, unnecessary work behind the scenes for every large question, which is what was slowing it down.
Changing one setting fixed it. But finding that setting only happened because the test suite was already asking the right question: why does a broad search keep returning results from the same company?
Before touching any of the search settings, we built a labeled test suite. Thirteen hand-written cases to start: questions that must return specific companies, questions that must surface specific topics, and questions that must return nothing: companies not in the corpus, themes absent from any filing.
From there, the suite grew by pulling real topics out of the actual corpus and generating test questions around six categories: single-company, side-by-side comparisons, historical trends, alias-matching, filtered searches, and deliberately unanswerable questions meant to check that the tool says “no data” instead of guessing. Every generated test question gets run against the live tool before it’s added to the suite. If the tool genuinely can’t answer it with the current corpus, the question doesn’t make the cut. Nothing in the test file expects an answer the corpus can’t actually give.
The suite grew to 83 grounded cases. Building it found more problems than the code review did.
One recurring lesson: the test questions need to grow along with the corpus. “Which companies mentioned AI investment?” was written back when only a handful of companies were indexed. Against the full 400, a good answer surfaces 20 or more distinct companies. The original test, written to expect a short fixed list, was wrong, not the tool. Any test like that now checks for a minimum number of companies instead of an exact list, so it keeps making sense as the corpus grows.
Once the test suite was large enough to trust, we tried tuning the search: how much weight to give meaning-based matching versus keyword matching, how many results to pull in for a broad question, and how strict to be before the tool trusts a match enough to answer at all. 29 different combinations, tested one change at a time against the same set of cases.
The starting settings were already close to right. The one real improvement: pulling in fewer results for broad questions (40 instead of 80) made the correct passage rank higher more consistently, with no drop in accuracy. A smaller pool meant the passages that actually answered the question weren’t getting buried under weaker matches.
The counterintuitive part: combining the best setting from each category separately made things worse than just using the starting settings. The best keyword-versus-meaning balance broke one comparison question on its own. The best pool size didn’t touch that same question on its own. Put them together, and both did damage. Every change we kept got tested against the full suite again before it went live.
Checking that every citation actually points to a real, retrieved passage was the first safeguard. Necessary, but not enough on its own. An answer can cite real passages and still claim something those passages don’t actually say.
So a second AI model checks the first one’s work. After every answer, it re-reads the answer against its cited passages and flags any sentence that goes beyond what the evidence actually states. The first run of this check caught two problems the citation check had missed.
One answer about Apple, Cisco, and IBM included a sentence that sounded reasonable but wasn’t in any cited passage. It was a small inference the model drew, not something any filing said. One comparison answer retrieved passages from both Tesla and Rivian but generated an answer discussing only Rivian, silently dropping half the comparison.
Two fixes: the tool now tells the model exactly which citations are valid to use, so it stops inventing citation numbers that point nowhere real. And when nothing the model writes passes the accuracy check, the fallback answer now shows 5 raw passages instead of 3, so both sides of a comparison still show up even in the fallback.
Those two fixes took the accuracy check’s pass rate from 8 out of 10 to a clean 10 out of 10.
Three weeks in, we ran the suite and got a pass rate that looked healthy. Then we looked at what “pass” actually meant for one case category.
Several questions had fixed company-list expectations written when the corpus had a handful of companies. Against 400 companies, a good answer to “which management teams mentioned supply chain pressure on recent earnings calls?” surfaces 30 or 40 distinct companies. The three-company test we’d written was wrong, not the tool. We were marking correct answers as failures and getting a pass rate that looked healthy for the wrong reason.
Fixing those tests took half a day. The same thing happened again two weeks later, once more foreign companies got added. Test questions need ongoing upkeep. Writing them once and forgetting them doesn’t work.
The day after we added 28 new companies to the corpus, the tool started missing some of them in its results. Not often. But often enough to matter.
Two of the new companies had name words that should have been excluded from matching, the same “shows up in more than one company’s name” rule from the ticker bug above. But that exclusion list had been built by hand at the start and never checked again as new companies got added. It had gone stale.
The fix: every time a new company is added now, the system automatically checks its name against the existing matching list, flags any conflicts before the company goes live, and writes out a plain summary of what it will match on. Someone can check that summary without reading any code. This check now runs on every update, not just the first time.
The tool has a built-in rule for questions it can’t actually answer. If the evidence it finds is too thin to support a real answer, it says “no data found” instead of guessing anyway. This was a core requirement from day one. The team had already been burned once by answers that sounded confident and were wrong.
But while the corpus was still being built out, some companies only had two or three filings indexed so far. Questions about those companies sometimes cleared the bar for “enough evidence” on a technicality, and produced answers that were technically grounded but too thin to be useful. An associate couldn’t tell whether a weak answer meant the tool had a real problem, or just meant that company’s filings weren’t fully loaded yet.
We added a coverage flag: when the queried company has fewer than five indexed documents, the answer includes a note that coverage is limited. “The system can’t answer this” and “the corpus doesn’t have enough yet” are different situations with different responses: the second one has a resolution path.
Emerald’s team started using the tool from the first week after go-live. The qualitative feedback has been consistent: the tool removed the bottleneck, not the thinking. Finding what Nvidia said about capacity investment across its last four earnings calls takes 10 minutes and returns the exact passages with source links. What that means for the investment thesis is still the associate’s job and still takes as long as it needs to.
10 min
Investment committee prep
A research brief that used to take 2–3 days is now prepared the morning of the meeting.
12–15
New positions evaluated per quarter
Up from 4–5. Same three-person team.
50+
Queries per week
Mostly cross-company comparisons and historical tone analysis: the questions manual reading could not answer at all.
0
Citation errors since go-live
Every [1] has been verified against the original source. The faithfulness gate runs on every answer.
The bigger shift wasn’t speed on individual queries. It was which questions the team bothers asking at all. Something like “how has this management team described its capital allocation philosophy over the last five years, and has the language become more or less specific?” used to be a week of work, so nobody ran it. Now it takes 15 minutes, so they do.
Reading filings by hand because your data provider stops at the numbers?
Tell us the corpus size and the questions your team keeps asking. We'll tell you in one call whether a custom search tool is worth building, or whether it isn't yet.
For teams with small document sets and occasional research needs, a general-purpose AI tool (or even a well-maintained NotebookLM workspace) is likely sufficient. This project made sense for Emerald for three specific reasons.
First, the corpus is too large and changes too often for any tool that only works one session at a time. 2,200 documents at delivery, and growing every week as new annual reports and transcripts get filed. Uploading documents fresh each session can’t keep up with that.
Second, citation integrity is non-negotiable. Investment decisions (even internal ones) depend on being able to verify the source of every claim. The faithfulness gate and the deep links to original filings aren’t nice-to-haves. They’re the product.
Third, data sensitivity. An investment office’s research activity (what they’re studying, which companies they’re probing, what questions they’re asking about management credibility or capital discipline) is proprietary signal. Having that processed through a third-party AI platform creates exposure they weren’t willing to accept. Every query stays within their environment.
Small watchlist (< 150 companies), research is occasional and single-filing
Your data provider + manual EDGAR is sufficient. No custom build needed.
Medium watchlist, cross-company research quarterly, some tolerance for citation limitations
Evaluate a managed AI search tool or NotebookLM. Custom may be overkill at this stage.
Large persistent corpus, cross-time qualitative research, citation integrity required, data must stay internal
Custom build. What this requires is a different foundation, not a bigger version of the same tool.
We’d start the test suite in week 1, not week 4. The ticker bug would have been caught in the first week instead of three weeks in. The 40-second slowdown would have been obvious from day one, since “why does a broad question keep returning results from the same company?” is exactly the kind of question a test suite is built to ask, not something you’re supposed to stumble into later by accident. We built the test suite as a development tool and only discovered its real value from what it caught along the way. We should have known what it was for before we needed it.
We’d also be more aggressive about corpus composition in week 1. Emerald had a clear sense of which companies mattered but no written definition of what the tool’s coverage universe actually was. Some companies were on the watchlist for clear reasons. Some were included for historical reasons nobody could reconstruct. Re-running a company through the pipeline just updates it instead of creating a duplicate, so the earlier the company list gets locked in, the earlier the name-matching stays stable and the test questions stay valid.
Third, we’d put a read-only version in front of one associate in week 4 instead of week 7. Our own tests catch logic bugs. Real research questions catch everything else: company names we hadn’t accounted for, earnings call transcripts that got split awkwardly, questions phrased the way associates actually talk instead of the way we imagined they would. One person running a week of real questions taught us more than another 20 questions we wrote ourselves.
Start the evaluation suite on day one, finalize the corpus before you tune anything, and put a real user in front of a working version as early as you can. The build gets easier once you know what you’re actually measuring. Everything before that is educated guessing.
450 companies on a watchlist and three days to read before every meeting?
That's the exact problem we solved. Fixed scope, fixed price, live in 6 to 8 weeks, and we'll tell you honestly if your corpus is too small to justify it yet.
Himanshu runs NocodeAssistant, a development agency that builds internal tools and SaaS products for growing companies. He's worked directly with every client since 2019. Same person from kickoff to post-launch.
Connect on LinkedInLet's talk
Book a relaxed 30-minute call. Bring whatever you're wondering about and we'll help you think it through, whether or not you ever work with us.
Continue reading
18k orders/month, 4,200 tickets, 3 support agents drowning. We built a custom AI chatbot in 5 weeks. Median time to a full answer went from 4.2 hours to 8 seconds. Full case study.
What it takes to migrate a Bubble app to custom code: 3–5 months, $15k–$25k, and the mistakes that lose data. A practical guide built around AI agents.
Learn why internal tools fail as teams grow past 50 people, and the architecture and process fixes that prevent recurring breakdowns.
