The LLM Data Company
Teaching models to learn and play at scale.
Tasks & Rubrics
Tasks & Rubrics
TLDC’s domain experts create tasks and rubrics for non-verifiable domains. A task defines the initial context, and a rubric specifies how outcomes are evaluated where there is no ground truth. Domain experts translate professional reasoning standards into objective, verifiable criteria. Each criterion is a weighted binary proposition that undergoes multiple rounds of iterative peer and model refinement to sharpen reward discrimination, mitigate reward hacking, and maximize throughput.
Clinical reasoning and bedside support spanning diagnosis, treatment planning, and longitudinal care. Example tasks include identifying etiologies, managing treatment-resistant conditions, and adjusting medication and dosing.
Task
A 70-year-old woman on warfarin 5mg daily for atrial fibrillation is referred to you. She is presenting with 4 days of bleeding gums and nosebleeds. Her INR is 5.8, up from 2.3 one week prior, and she started trimethoprim-sulfamethoxazole DS twice daily 3 days ago for dysuria and frequency. She reports no falls, no new medications, and no missed doses.
The patient’s vital signs are stable: temperature 37.1°C, heart rate 82, blood pressure 145/88, and oxygen levels are good. Blood work shows her hemoglobin is 11.2 (baseline 12.8), platelets are normal at 220, and kidney function is normal with creatinine at 0.9. She was given 2.5 mg of vitamin K intravenously, but her INR is still high at 4.3 after 24 hours. Her albumin is a bit low at 3.2 (baseline 3.8), though her liver enzymes are normal. She had a nearly identical episode 8 months ago while on amoxicillin-clavulanate, when her INR rose to 6.1 and took 5 days to come back down.
Given that vitamin K was administered and no drugs were changed apart from antibiotics, what delayed metabolic interaction explains the sustained anticoagulant effect?
Rubric
+/-
Weight
Criterion
+
10
Indicates antibiotic-induced suppression of intestinal microbiota reduces endogenous
vitamin K production
+
10
States warfarin inhibits vitamin K epoxide reductase
+
5
Indicates inhibition limits vitamin K recycling
+
10
Indicates impaired synthesis of vitamin K-dependent clotting factors, at least 2 of II, VII, IX or X
+
5
Indicates exogenous vitamin K temporarily replenishes active stores
+
5
Lists at least 2 of trimethoprim-sulfamethoxazole, sulfonamides or beta-lactams as antibiotics implicated in the interaction
+
5
States broad-spectrum agents reduce endogenous vitamin K production
+
5
Indicates lower albumin increases free warfarin exposure
+
3
States warfarin is highly protein-bound
+
8
Lists at least 1 of INR remained elevated after vitamin K administration or delayed
INR normalization after similar antibiotic courses as clinical evidence supporting
the interaction
+
8
Indicates the interaction persists beyond the antibiotic course
+
3
Indicates restoration of gut flora takes days to weeks
+
5
Indicates vulnerability to bleeding despite standard reversal attempts
+
8
Lists at least 3 of frequent INR checks, temporary warfarin dose reduction, warfarin
interruption, or ongoing low-dose vitamin L supplementation as indicated
management actions
+
5
Indicates low-dose vitamin K supplementation continues until intestinal flora
normalize
+
5
Recommends prioritizing narrow-spectrum antibiotics in patients at high bleeding
risk
+
5
States normal liver enzymes rules out impaired clotting factor synthesis from liver
disease
–
10
States dietary vitamin K restriction is the primary explanation for sustained
anticoagulation in the setting of recent antibiotics
–
10
Indicates INR follow-up can be deferred until symptoms occur
Clinical reasoning and bedside support spanning diagnosis, treatment planning, and longitudinal care. Example tasks include identifying etiologies, managing treatment-resistant conditions, and adjusting medication and dosing.
Task
A 70-year-old woman on warfarin 5mg daily for atrial fibrillation is referred to you. She is presenting with 4 days of bleeding gums and nosebleeds. Her INR is 5.8, up from 2.3 one week prior, and she started trimethoprim-sulfamethoxazole DS twice daily 3 days ago for dysuria and frequency. She reports no falls, no new medications, and no missed doses.
The patient’s vital signs are stable: temperature 37.1°C, heart rate 82, blood pressure 145/88, and oxygen levels are good. Blood work shows her hemoglobin is 11.2 (baseline 12.8), platelets are normal at 220, and kidney function is normal with creatinine at 0.9. She was given 2.5 mg of vitamin K intravenously, but her INR is still high at 4.3 after 24 hours. Her albumin is a bit low at 3.2 (baseline 3.8), though her liver enzymes are normal. She had a nearly identical episode 8 months ago while on amoxicillin-clavulanate, when her INR rose to 6.1 and took 5 days to come back down.
Given that vitamin K was administered and no drugs were changed apart from antibiotics, what delayed metabolic interaction explains the sustained anticoagulant effect?
Rubric
+/-
Weight
Criterion
+
10
Indicates antibiotic-induced suppression of intestinal microbiota reduces endogenous
vitamin K production
+
10
States warfarin inhibits vitamin K epoxide reductase
+
5
Indicates inhibition limits vitamin K recycling
+
10
Indicates impaired synthesis of vitamin K-dependent clotting factors, at least 2 of II, VII, IX or X
+
5
Indicates exogenous vitamin K temporarily replenishes active stores
+
5
Lists at least 2 of trimethoprim-sulfamethoxazole, sulfonamides or beta-lactams as antibiotics implicated in the interaction
+
5
States broad-spectrum agents reduce endogenous vitamin K production
+
5
Indicates lower albumin increases free warfarin exposure
+
3
States warfarin is highly protein-bound
+
8
Lists at least 1 of INR remained elevated after vitamin K administration or delayed
INR normalization after similar antibiotic courses as clinical evidence supporting
the interaction
+
8
Indicates the interaction persists beyond the antibiotic course
+
3
Indicates restoration of gut flora takes days to weeks
+
5
Indicates vulnerability to bleeding despite standard reversal attempts
+
8
Lists at least 3 of frequent INR checks, temporary warfarin dose reduction, warfarin
interruption, or ongoing low-dose vitamin L supplementation as indicated
management actions
+
5
Indicates low-dose vitamin K supplementation continues until intestinal flora
normalize
+
5
Recommends prioritizing narrow-spectrum antibiotics in patients at high bleeding
risk
+
5
States normal liver enzymes rules out impaired clotting factor synthesis from liver
disease
–
10
States dietary vitamin K restriction is the primary explanation for sustained
anticoagulation in the setting of recent antibiotics
–
10
Indicates INR follow-up can be deferred until symptoms occur
Clinical reasoning and bedside support spanning diagnosis, treatment planning, and longitudinal care. Example tasks include identifying etiologies, managing treatment-resistant conditions, and adjusting medication and dosing.
Task
A 70-year-old woman on warfarin 5mg daily for atrial fibrillation is referred to you. She is presenting with 4 days of bleeding gums and nosebleeds. Her INR is 5.8, up from 2.3 one week prior, and she started trimethoprim-sulfamethoxazole DS twice daily 3 days ago for dysuria and frequency. She reports no falls, no new medications, and no missed doses.
The patient’s vital signs are stable: temperature 37.1°C, heart rate 82, blood pressure 145/88, and oxygen levels are good. Blood work shows her hemoglobin is 11.2 (baseline 12.8), platelets are normal at 220, and kidney function is normal with creatinine at 0.9. She was given 2.5 mg of vitamin K intravenously, but her INR is still high at 4.3 after 24 hours. Her albumin is a bit low at 3.2 (baseline 3.8), though her liver enzymes are normal. She had a nearly identical episode 8 months ago while on amoxicillin-clavulanate, when her INR rose to 6.1 and took 5 days to come back down.
Given that vitamin K was administered and no drugs were changed apart from antibiotics, what delayed metabolic interaction explains the sustained anticoagulant effect?
Rubric
+/-
Weight
Criterion
+
10
Indicates antibiotic-induced suppression of intestinal microbiota reduces endogenous vitamin K production
+
10
States warfarin inhibits vitamin K epoxide reductase
+
5
Indicates inhibition limits vitamin K recycling
+
10
Indicates impaired synthesis of vitamin K-dependent clotting factors, at least 2 of II, VII, IX or X
+
5
Indicates exogenous vitamin K temporarily replenishes active stores
+
5
Lists at least 2 of trimethoprim-sulfamethoxazole, sulfonamides or beta-lactams as antibiotics implicated in the interaction
+
5
States broad-spectrum agents reduce endogenous vitamin K production
+
5
Indicates lower albumin increases free warfarin exposure
+
3
States warfarin is highly protein-bound
+
8
Lists at least 1 of INR remained elevated after vitamin K administration or delayed INR normalization after similar antibiotic courses as clinical evidence supporting the interaction
+
8
Indicates the interaction persists beyond the antibiotic course
+
3
Indicates restoration of gut flora takes days to weeks
+
5
Indicates vulnerability to bleeding despite standard reversal attempts
+
8
Lists at least 3 of frequent INR checks, temporary warfarin dose reduction, warfarin interruption, or ongoing low-dose vitamin L supplementation as indicated management actions
+
5
Indicates low-dose vitamin K supplementation continues until intestinal flora normalize
+
5
Recommends prioritizing narrow-spectrum antibiotics in patients at high bleeding risk
+
5
States normal liver enzymes rules out impaired clotting factor synthesis from liver disease
–
10
States dietary vitamin K restriction is the primary explanation for sustained anticoagulation in the setting of recent antibiotics
–
10
Indicates INR follow-up can be deferred until symptoms occur
Clinical reasoning and bedside support spanning diagnosis, treatment planning, and longitudinal care. Example tasks include identifying etiologies, managing treatment-resistant conditions, and adjusting medication and dosing.
Task
A 70-year-old woman on warfarin 5mg daily for atrial fibrillation is referred to you. She is presenting with 4 days of bleeding gums and nosebleeds. Her INR is 5.8, up from 2.3 one week prior, and she started trimethoprim-sulfamethoxazole DS twice daily 3 days ago for dysuria and frequency. She reports no falls, no new medications, and no missed doses.
The patient’s vital signs are stable: temperature 37.1°C, heart rate 82, blood pressure 145/88, and oxygen levels are good. Blood work shows her hemoglobin is 11.2 (baseline 12.8), platelets are normal at 220, and kidney function is normal with creatinine at 0.9. She was given 2.5 mg of vitamin K intravenously, but her INR is still high at 4.3 after 24 hours. Her albumin is a bit low at 3.2 (baseline 3.8), though her liver enzymes are normal. She had a nearly identical episode 8 months ago while on amoxicillin-clavulanate, when her INR rose to 6.1 and took 5 days to come back down.
Given that vitamin K was administered and no drugs were changed apart from antibiotics, what delayed metabolic interaction explains the sustained anticoagulant effect?
Rubric
+/-
Weight
Criterion
+
10
Indicates antibiotic-induced suppression of intestinal microbiota reduces endogenous vitamin K production
+
10
States warfarin inhibits vitamin K epoxide reductase
+
5
Indicates inhibition limits vitamin K recycling
+
10
Indicates impaired synthesis of vitamin K-dependent clotting factors, at least 2 of II, VII, IX or X
+
5
Indicates exogenous vitamin K temporarily replenishes active stores
+
5
Lists at least 2 of trimethoprim-sulfamethoxazole, sulfonamides or beta-lactams as antibiotics implicated in the interaction
+
5
States broad-spectrum agents reduce endogenous vitamin K production
+
5
Indicates lower albumin increases free warfarin exposure
+
3
States warfarin is highly protein-bound
+
8
Lists at least 1 of INR remained elevated after vitamin K administration or delayed INR normalization after similar antibiotic courses as clinical evidence supporting the interaction
+
8
Indicates the interaction persists beyond the antibiotic course
+
3
Indicates restoration of gut flora takes days to weeks
+
5
Indicates vulnerability to bleeding despite standard reversal attempts
+
8
Lists at least 3 of frequent INR checks, temporary warfarin dose reduction, warfarin interruption, or ongoing low-dose vitamin L supplementation as indicated management actions
+
5
Indicates low-dose vitamin K supplementation continues until intestinal flora normalize
+
5
Recommends prioritizing narrow-spectrum antibiotics in patients at high bleeding risk
+
5
States normal liver enzymes rules out impaired clotting factor synthesis from liver disease
–
10
States dietary vitamin K restriction is the primary explanation for sustained anticoagulation in the setting of recent antibiotics
–
10
Indicates INR follow-up can be deferred until symptoms occur
The LLM Data Company
works with frontier AI teams to create bespoke tasks, graders, and environments for models to play and learn at scale.
Copyright © 2025 The LLM Data Company, Inc. All rights reserved.
Privacy Policy
Legal
Disclaimers
The LLM Data Company
works with frontier AI teams to create bespoke tasks, graders, and environments for models to play and learn at scale.
Copyright © 2025 The LLM Data Company, Inc. All rights reserved.
Privacy Policy
Legal
Disclaimers
The LLM Data Company
works with frontier AI teams to create bespoke tasks, graders, and environments for models to play and learn at scale.
Copyright © 2025 The LLM Data Company, Inc. All rights reserved.
Privacy Policy
Legal
Disclaimers