Inclusion
Documented informed consent of the participant and/or legally authorized representative
Assent, when appropriate, will be obtained per institutional guidelines
Be willing to provide tissue (either from a fresh core or excisional biopsy performed as standard of care, or from archival tissue) of a biopsy that was performed after frontline systemic therapy, and prior to starting protocol therapy
If unavailable, exceptions may be granted with study principal investigator (PI) approval
Eastern Cooperative Oncology Group (ECOG) =\< 2
Histologically confirmed diagnosis of classical Hodgkin lymphoma (excluding nodular lymphocyte predominant Hodgkin lymphoma) according to the World Health Organization (WHO) classification, with hematopathology review at the participating institution
Relapsed or refractory disease after no more than 1 line of prior therapy (not counting radiotherapy). However, a maximum of 5 patients with primary refractory disease may be enrolled in this study.
Not a candidate for ASCT, based on age, co-morbidities, or patient preference. The reason for ASCT non-candidacy must be documented in the Case Report Form and verified by the site PI
Measurable disease (at least one non-bony fludeoxyglucose F-18 \[FDG\]-avid lesion \>= 1.5 cm in long axis)
Absolute neutrophil count (ANC) \>= 1,000/mm\^3
NOTE: Growth factor is not permitted within 7 days of ANC assessment unless cytopenia is secondary to disease involvement
Platelets \>= 50,000/mm\^3
NOTE: Platelet transfusions are not permitted within 7 days of platelet assessment unless cytopenia is secondary to disease involvement
Hemoglobin \>= 8 g/dL (no transfusion allowed within 3 days prior to screening)
Total bilirubin =\< 1.5 x upper limit of normal (ULN) or direct bilirubin =\< 1.5 x ULN for patients with Gilbert's disease
Aspartate aminotransferase (AST) =\< 2.5 x ULN
Alanine aminotransferase (ALT) =\< 2.5 x ULN
Creatinine clearance of \>= 40 mL/min per 24 hour urine test or the Cockcroft-Gault formula
Women of childbearing potential (WOCBP): negative urine or serum pregnancy test. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required
Agreement by women and men of childbearing potential\* to use an effective method of birth control or abstain from heterosexual activity for the course of the study through at least 5 months (women) or 7 months (men) after the last dose of protocol therapy
Childbearing potential defined as not being surgically sterilized (men and women) or have not been free from menses for \> 1 year (women only)
Exclusion
Concomitant investigational therapy
Live vaccine within 30 days prior to day 1 of protocol therapy (e.g. measles, mumps, rubella, varicella, yellow fever, rabies, bacillus Calmette-Guerin \[BCG\], oral polio vaccine, and oral typhoid)
Grade \>= 2 peripheral neuropathy
History of prior \>= grade 3 hypersensitivity to either brentuximab vedotin or nivolumab
Known active central nervous system (CNS) involvement by lymphoma, including parenchymal and/or lymphomatous meningitis
History of another primary malignancy that has not been in remission for at least 3 years, with the following exceptions:
Non-melanoma skin cancer treated with curative intent
In situ cervical cancer
If the malignancy is expected to not require any treatment for at least 2 years (this exception should be discussed with the study PI)
Condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone or equivalent) or other immunosuppressive medications within 14 days of study drug administration. Exceptions are:
Inhaled or topical steroids and
Adrenal replacement doses \> 10 mg daily prednisone equivalents in the absence of active autoimmune disease
History of progressive multifocal leukoencephalopathy (PML)
Prior diagnosis of inherited or acquired immunodeficiency
Active pneumonitis or interstitial lung disease
Active, known or suspected autoimmune disease. The following are exceptions:
Vitiligo
Psoriasis not requiring systemic treatment
Hemolytic anemia associated with the lymphoma
Type I diabetes mellitus, if adequately controlled with therapy
Thyroid disease, if adequately controlled with therapy
Conditions not expected to recur in the absence of an external trigger (such exceptions should be discussed with the study PI)
Active history of:
Hepatitis B (hepatitis B virus \[HBV\]) or C (hepatitis C virus \[HCV\]) infection. Patients with past HBV infection (defined as negative hepatitis B surface antigen \[HBsAg\] and positive hepatitis B core antibody \[HBcAb\]) are eligible if HBV DNA is undetectable. Patients who are positive for HCV antibody are eligible if polymerase chain reaction (PCR) is negative for HCV ribonucleic acid (RNA)
Human immunodeficiency virus (HIV) infection. Subjects who have an undetectable or unquantifiable human immunodeficiency virus (HIV) viral load with CD4 \>= 200 and are on highly active antiretroviral therapy (HAART) medication are allowed. Testing to be done only in patients suspected of having infections or exposures
History of a cerebral vascular event (stroke or transient ischemic attack), unstable angina, myocardial infarction, or cardiac symptoms consistent with New York Heart Association class III-IV within 6 months prior to day 1 of protocol therapy
Pregnant or breastfeeding
Any other condition that would, in the Investigator's judgment, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures
Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)