Use of Special Purpose Entities (SPE) and Variable Interest Entities (VIE) - Assets and Liabilities of Consolidated VIEs (Details) - USD ($) $ in Thousands		Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents		$ 293,183	$ 118,763	$ 103,724
Investment securities available for sale, at fair value		311,284	1,760,812
Residential loans, carrying value		3,049,166	2,961,396
Loans held in securitization trusts, at fair value		0	17,816,746
Other assets		165,824	169,214
Total assets	[1]	4,655,587	23,483,369
Other liabilities		138,498	177,260
Total liabilities	[1]	2,348,014	21,278,340
Non-controlling interest in consolidated variable interest entities		6,371	(704)
Residential Collateralized Debt Obligations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations, at fair value		569,323	40,429
Residential collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations, at fair value		1,054,335	17,777,280
VIE, Primary Beneficiary
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Total assets		2,150,984	19,270,384
Total liabilities		1,667,306	17,878,314
VIE, Primary Beneficiary | Residential Loan Securitizations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents		0	0
Investment securities available for sale, at fair value		0
Operating real estate		0
Residential loans, carrying value			44,030
Other assets		24,959	1,328
Total assets		716,410	45,358
Collateralized debt obligations, at fair value		554,067
Mortgages payable in consolidated variable interest entities		0
Other liabilities		2,610	14
Total liabilities		556,677	40,443
Non-controlling interest in consolidated variable interest entities		0	0
Variable interest entity, primary beneficiary, maximum loss exposure, amount		159,733	4,915
VIE, Primary Beneficiary | Non-Agency RMBS Re-Securitization
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents		0
Investment securities available for sale, at fair value		109,140
Operating real estate		0
Other assets		535
Total assets		109,675
Collateralized debt obligations, at fair value		15,256
Mortgages payable in consolidated variable interest entities		0
Other liabilities		70
Total liabilities		15,326
Non-controlling interest in consolidated variable interest entities		0
Variable interest entity, primary beneficiary, maximum loss exposure, amount		94,349
VIE, Primary Beneficiary | Consolidated SLST
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents		0	0
Investment securities available for sale, at fair value		0
Operating real estate		0
Residential loans, carrying value			1,328,886
Other assets		4,075	5,244
Total assets		1,270,860	1,334,130
Collateralized debt obligations, at fair value		1,054,335	1,052,829
Mortgages payable in consolidated variable interest entities		0
Other liabilities		2,781	2,643
Total liabilities		1,057,116	1,055,472
Non-controlling interest in consolidated variable interest entities		0	0
Variable interest entity, primary beneficiary, maximum loss exposure, amount		213,744	278,658
VIE, Primary Beneficiary | Other
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents		462
Investment securities available for sale, at fair value		0
Operating real estate		50,532
Other assets		3,045
Total assets		54,039
Collateralized debt obligations, at fair value		0
Mortgages payable in consolidated variable interest entities		36,752
Other liabilities		1,435
Total liabilities		38,187
Non-controlling interest in consolidated variable interest entities		6,371
Variable interest entity, primary beneficiary, maximum loss exposure, amount		9,481
VIE, Primary Beneficiary | Consolidated K-Series
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents			0
Residential loans, carrying value			0
Other assets			59,417
Total assets			17,876,163
Collateralized debt obligations, at fair value			16,724,451
Other liabilities			57,873
Total liabilities			16,782,324
Non-controlling interest in consolidated variable interest entities			0
Variable interest entity, primary beneficiary, maximum loss exposure, amount			1,093,839
VIE, Primary Beneficiary | KRVI
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents			107
Residential loans, carrying value			0
Other assets			14,626
Total assets			14,733
Collateralized debt obligations, at fair value			0
Other liabilities			75
Total liabilities			75
Non-controlling interest in consolidated variable interest entities			(704)
Variable interest entity, primary beneficiary, maximum loss exposure, amount			15,362
VIE, Primary Beneficiary | Financing And Other VIEs
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents		462	107
Investment securities available for sale, at fair value		109,140
Operating real estate		50,532
Residential loans, carrying value			1,372,916
Other assets		32,614	80,615
Total assets		2,150,984	19,270,384
Collateralized debt obligations, at fair value		1,623,658
Mortgages payable in consolidated variable interest entities		36,752
Other liabilities		6,896	60,605
Total liabilities		1,667,306	17,878,314
Non-controlling interest in consolidated variable interest entities		6,371	(704)
Variable interest entity, primary beneficiary, maximum loss exposure, amount		477,307	1,392,774
VIE, Primary Beneficiary | Residential Collateralized Debt Obligations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations, at fair value		569,323	40,429
VIE, Primary Beneficiary | Residential Collateralized Debt Obligations | Residential Loan Securitizations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations, at fair value			40,429
VIE, Primary Beneficiary | Residential Collateralized Debt Obligations | Financing And Other VIEs
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations, at fair value			17,817,709
VIE, Primary Beneficiary | Residential collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations, at fair value		1,054,335	17,777,280
VIE, Primary Beneficiary | Residential Loans
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential loans, carrying value			44,030
VIE, Primary Beneficiary | Residential loans, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential loans, at fair value			1,328,886
VIE, Primary Beneficiary | Residential loans, at fair value | Residential Loan Securitizations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential loans, at fair value		691,451
VIE, Primary Beneficiary | Residential loans, at fair value | Non-Agency RMBS Re-Securitization
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential loans, at fair value		0
VIE, Primary Beneficiary | Residential loans, at fair value | Consolidated SLST
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential loans, at fair value		1,266,785
VIE, Primary Beneficiary | Residential loans, at fair value | Other
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential loans, at fair value		0
VIE, Primary Beneficiary | Residential loans, at fair value | Financing And Other VIEs
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential loans, at fair value		$ 1,958,236
VIE, Primary Beneficiary | Multi-family Loans Held In Securitization Trusts, At Fair Value | Residential Loan Securitizations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Loans held in securitization trusts, at fair value			0
VIE, Primary Beneficiary | Multi-family Loans Held In Securitization Trusts, At Fair Value | Consolidated SLST
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Loans held in securitization trusts, at fair value			0
VIE, Primary Beneficiary | Multi-family Loans Held In Securitization Trusts, At Fair Value | Consolidated K-Series
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Loans held in securitization trusts, at fair value			17,816,746
VIE, Primary Beneficiary | Multi-family Loans Held In Securitization Trusts, At Fair Value | KRVI
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Loans held in securitization trusts, at fair value			0
VIE, Primary Beneficiary | Multi-family Loans Held In Securitization Trusts, At Fair Value | Financing And Other VIEs
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Loans held in securitization trusts, at fair value			$ 17,816,746

[1] Our consolidated balance sheets include assets and liabilities of consolidated variable interest entities (“VIEs”) as the Company is the primary beneficiary of these VIEs. As of December 31, 2020 and December 31, 2019, assets of consolidated VIEs totaled $2,150,984 and $19,270,384, respectively, and the liabilities of consolidated VIEs totaled $1,667,306 and $17,878,314, respectively. See Note 7 for further discussion.