In the previous article I described the architecture of a system that enables a user to ask questions of a long, complex document. As I generate complex legal documents for clients, I’m interested in the idea of allowing my clients to understand those documents and get answers about the implications of those documents without necessarily booking time with me. To do this, I’m taking advantage of some of the large language AI models (LLMs) out there. In this example, I’m using one of OpenAI’s GPT-3 models. On top of that model, I’m layering some programming logic in Python. I’m also converting both the user’s query and the document content to vector embeddings that power a more precise content search within the document.
Now I’d like to take that system for a test drive. For this demonstration, I’m using the 128-page document “Medicare & You 2023” as the source document. The question I’ll run through the system: “Is long-term care covered?” I want to compare the answer I arrive at as a human to the answer(s) provided by this system. You can see details of all the results in a spreadsheet here.
Of course while the question seems straight-forward, when the user asks this question, there may be any number of implicit questions that they actually want answers to which include:
- Do I have long-term coverage?
- How do I get long-term coverage if I have Medicare?
- Can I afford to live in a nursing home?
The user may not know that “long-term care” is actually a term of art and may not be using it correctly in their question. A more sophisticated system would account for these implicit questions in generating the answer. For the sake of this exercise however, I’m going to take the question at face value.
Human vs. AI Answers
After reviewing the Medicare document, after about 15-minutes, this is the answer I came up with:
Is long-term care covered?
“It depends on what you mean by “long-term care.” As a general rule, Medicare does not pay for non-medical long-term care. However, if you have a medical event that requires a 3-day minimum inpatient hospital stay (with some exceptions), then Medicare may cover skilled nursing care. Generally, this is limited to 100 days of care within each benefit period.”
Julian’s Answer
And this is the answer generated by my system in about 10 seconds:
Is long-term care covered?
“Original Medicare is a type of Medicare health coverage. With Original Medicare, you generally have to pay a portion of the cost for each service that is covered. There is no limit to what you can pay out of pocket in a year, unless you have other coverage, like Medigap, Medicaid, or employee or union coverage. Original Medicare does not cover most drugs, with a few exceptions. You can add Medicare drug coverage by joining a separate Medicare drug plan. In most cases, you can go to any Medicare-enrolled doctor or other health care provider that accepts Medicare patients anywhere in the United States. Long-term care is not covered by Medicare. This includes care in a skilled nursing facility, which is only covered if the patient has been hospitalized for at least three days. If the patient is in a Medicare Advantage Plan, they may not need to be hospitalized for three days.”
Python+GPT Answer
I find my answer a bit more concise and on-point than the AI answer. That said, the system does eventually answer the question. However, the final answer isn’t as interesting as the thought processes and source texts used to get to the answer.
To formulate my answer, I pulled information from the following parts of the document. All come from “Section 2: Finding out if Medicare covers your test, item, or service” and are listed in order of importance to answering the question:
- “What’s Not Covered by Part A and Part B” (Page 55): States that long-term care is not covered.
- “Paying for Long-term Care” (Page 56): Definition of “long-term care” and scope of Medicare coverage.
- “What Service Does Medicare Cover?” (Page 25): States thatin-patient care in skilled nursing facility is covered, but long-term care is not.
- “Skilled Nursing Facility Care” (Page 28): Describes the 3-day minimum stay requirement and cost structure.
While I probably should have read the table of contents carefully, identified potentially relevant sections, and read each section in context; I took the quick-and-dirty approach of searching the document for “long-term” (Ctrl-F) and jumped right to the sections that appeared in the results. 😅 I took what I thought was the most human approach to the task.
AI Answers and Sources
In contrast, let’s look at the material the system used to answer the question. Here are the sections of information (chunks as described in the previous article) used as a basis for the AI response. This is, in a sense, the system’s thought process. As outlined in my earlier article, I use a vector-based search to get the top 3 chunks of relevant text and then ask GPT-3 to generate a distinct answer based just on that chunk.
| Question: “Is long-term care covered?” |
| Document Chunk: “What’s Not Covered by Part A & Part B” / “Paying for Long Term Care” (Pages 55-56) |
| Sub-Answer #1: “Original Medicare is one of your Medicare health coverage choices. Youll have Original Medicare unless you choose a Medicare Advantage Plan or other type of Medicare health plan.You generally have to pay a portion of the cost for each service Original Medicare covers. Theres no limit to what youll pay out of pocket in a year unless you have other coverage (like Medigap, Medicaid, or employee or union coverage) or join a Medicare Advantage Plan. Original Medicare doesn’t cover most drugs, with a few exceptions (see pages 2627, 47, and 51). You can add Medicare drug coverage (Part D) by joining a separate Medicare drug plan. In most cases, you can go to any Medicare-enrolled doctor, other health care provider, hospital, or other facility that accepts Medicare patients anywhere in the U.S. Visit Medicare.gov/care-compare to find and compare providers, hospitals, and facilities in your area. You generally pay a set amount for your health care (deductible) before Medicare begins to pay its share. Once Medicare pays its share, you pay a coinsurance or copayment for covered services and supplies. Theres no yearly limit for what you pay out of pocket unless you have other insurance (like Medigap, Medicaid, or employee, retiree, or union coverage).” |
| Question: “Is long-term care covered?” |
| Document Chunk: “Hospice Care” and “In-Patient Medical Care” (Pages 26-27) |
| Sub-Answer #2: “According to the passage, long-term care is covered under Medicare. This includes room and board in a facility, as well as short-term inpatient care if it is deemed necessary by the hospice medical team. In addition, Medicare also covers inpatient respite care, which is care given in a Medicare-approved facility so that the patient’s usual caregiver can rest. There is no charge for hospice care, but there may be a copayment for outpatient drugs and a five percent coinsurance for inpatient respite care.” |
| Question: “Is long-term care covered?” |
| Document Chunk: “Skilled Nursing Facility Care” (Page 28): Describes 3-day minimum stay requirement and cost structure. |
| Sub-Answer #3: “According to the passage, long-term care is not covered by Medicare. This includes care in a skilled nursing facility, which is only covered if the patient has been hospitalized for at least three days. If the patient is in a Medicare Advantage Plan, they may not need to be hospitalized for three days.” |
Note that sub-answer #1 doesn’t really answer the question clearly at all. Also note that sub-answer #2 states that long-term care IS covered by Medicare. I believe this is because “long-term” is not defined clearly and GPT-3 interpreted hospital stays of longer than 3 days as long-term (and it probably feels long if you’ve been in the hospital more than 3 days).
Conclusion & Next Steps
Even though different sub-answers come to different conclusions, when GPT-3 is asked to take all three sub-answers and generate a detailed summary of them, it does eventually output the correct answer. This is the GPT+Python answer quoted above. Because sub-answer #1 discusses Medicare in general, the final summary answer also begins with a long discussion of what Medicare is before finally discussing long-term care. Unfortunately, the prompt for the final answer (“Write a detailed summary of the following.”) did not include any reference to the original question. This is probably a design mistake on my part. GPT-3 did not know the question it was supposed to answer and therefore spent a time rambling about things that were not necessarily on-point to begin it’s summary.
You can see a spreadsheet of all the search information results here. I hope this provided a useful example of how LLMs like GPT-3 can be layered with programming logic to create some interesting custom information systems.  I’ve already made some improvements to this system in terms of optimizing code and prompts such that you can just drag and drop any .pdf file and then ask natural language questions against it via a web interface. I’ll talk about this in the next article.
