11-K

UNITED STATES

SECURITIES AND EXCHANGE COMMISSION

WASHINGTON, D.C. 20549

FORM 11-K

(Mark one)

x ANNUAL REPORT PURSUANT TO SECTION 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934

For the fiscal year ended December 31, 2010

or

¨ TRANSITION REPORT PURSUANT TO SECTION 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934

For the transition period from                      to                     

Commission File Number 01-34219

CASUAL MALE RETAIL GROUP, INC.

401(K) HOURLY SAVINGS PLAN

(Full title of the plan)

CASUAL MALE RETAIL GROUP, INC.

555 Turnpike Street

Canton, Massachusetts 02021

(Name of issuer of the securities held pursuant to the plan and the address

of its principal executive office)

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Financial Statements and Supplemental Schedule

Year Ended December 31, 2010

Table of Contents

Report of Independent Registered Public Accounting Firm	1
Statements of Net Assets Available for Benefits as of December 31, 2010 and 2009	2
Statement of Changes in Net Assets Available for Benefits For the Year Ended December 31, 2010	3
Notes to Financial Statements	4-14
Supplemental Schedule	15
Schedule H, Line 4i – Schedule of Assets Held for Investment Purposes at End of Year (December 31, 2010)	16

Report of Independent Registered Public Accounting Firm

Plan Administrator

Casual Male Retail Group, Inc.

401(k) Hourly Savings Plan

We have audited the accompanying statements of net assets available for benefits of the Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan (the “Plan”) as of December 31, 2010 and 2009, and the related statement of changes in net assets available for benefits for the year ended December 31, 2010. These financial statements are the responsibility of the Plan’s management. Our responsibility is to express an opinion on these financial statements based on our audits.

We conducted our audits in accordance with standards of the Public Company Accounting Oversight Board (United States). Those standards require that we plan and perform the audit to obtain reasonable assurance about whether the financial statements are free of material misstatement. The Plan is not required to have, nor were we engaged to perform, an audit of its internal control over financial reporting. Our audits included consideration of internal control over financial reporting as a basis for designing audit procedures that are appropriate in the circumstances, but not for the purpose of expressing an opinion on the effectiveness of the Plan’s internal control over financial reporting. Accordingly, we express no such opinion. An audit includes examining, on a test basis, evidence supporting the amounts and disclosures in the financial statements. An audit also includes assessing the accounting principles used and significant estimates made by management, as well as evaluating the overall financial statement presentation. We believe that our audits provide a reasonable basis for our opinion.

In our opinion, the financial statements referred to above present fairly, in all material respects, the net assets available for benefits of the Plan as of December 31, 2010 and 2009, and the changes in net assets available for benefits for the year ended December 31, 2010, in conformity with U.S. generally accepted accounting principles.

Our audits were performed for the purpose of forming an opinion on the basic financial statements taken as a whole. The supplemental schedule, as listed in the accompanying index, is presented for the purpose of additional analysis and is not a required part of the basic financial statements, but is supplemental information required by the Department of Labor’s Rules and Regulations for Reporting and Disclosure under the Employee Retirement Income Security Act of 1974. This supplemental schedule is the responsibility of the Plan’s management. The supplemental schedule has been subjected to the auditing procedures applied in the audits of the basic financial statements and, in our opinion, is fairly stated in all material respects in relation to the basic financial statements taken as a whole.

/s/ Mayer Hoffman McCann CPAs

(The New York Practice of Mayer Hoffman McCann P.C.)

New York, New York

June 29, 2011

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Statements of Net Assets Available for Benefits

December 31, 2010 and 2009

	December 31,
	2010			2009
Assets
Investments, at fair value (Notes 3, 4 and 5)	$	11,240,822		$	10,772,580
Receivables:
Employee contributions		—			18,646
Employer contributions		—			84,639
Notes receivable from participants		317,934			346,581
		317,934			449,866
Net assets reflecting all investments at fair value		11,558,756			11,222,446
Adjustment from fair value to contract value for fully benefit-responsive investment contracts		(75,309	)		5,724
Net assets available for benefits	$	11,483,447		$	11,228,170

See accompanying notes to financial statements.

2

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Statement of Changes in Net Assets Available for Benefits

For the Year Ended December 31, 2010

Additions to net assets attributed to:
Contributions:
Employee	$	478,562
Rollovers		6,149
		484,711
Investment income:
Net appreciation in fair value of investments (Note 3)		819,480
Interest and dividend income		199,561
		1,019,041
Interest income on participant loans		20,721
Transfers into the Plan (Note 1)		314,198
Total additions		1,838,671
Deductions from net assets attributed to:
Benefits paid directly to participants		1,554,031
Transfers out of the Plan (Note 1)		8,603
Administrative fees		20,760
Total deductions		1,583,394
Net increase		255,277
Net assets available for benefits at beginning of year		11,228,170
Net assets available for benefits at end of year	$	11,483,447

See accompanying notes to financial statements.

3

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 1- Description of the Plan

The following description of Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan (the “Plan”) provides only general information. Participants should refer to the Plan document for a more complete description of the Plan’s provisions.

General

The Plan is a defined contribution plan covering eligible employees of Casual Male Retail Group, Inc. (the “Company”). The Plan is subject to the provisions of the Employee Retirement Income Security Act of 1974 (ERISA).

Eligibility

All hourly employees of the Company who have completed one year of employment, worked 1,000 hours in one year, and are at least 21 years of age are eligible to participate in the Plan. After completing age and service requirements, the employee can enter the Plan on the first day of any subsequent month.

As participants change positions at the Company, they may change from an hourly to salary status or vice versa. Assets transferred to the Casual Male Retail Group, Inc. 401(k) Salaried Savings Plan during the year ended December 31, 2010 were $8,603. Assets transferred from the Casual Male Retail Group Inc. 401(k) Salaried Savings Plan during the year ended December 31, 2010 were $314,198.

Contributions

Each year, participants may contribute from 1% to 80% of pretax annual compensation as defined in the Plan, subject to the provisions of ERISA. Participants may also contribute amounts representing distributions from other qualified defined benefit or defined contribution plans.

Effective January 1, 2008, the Plan approved and adopted the Safe Harbor Matching provisions permitted under the Pension Protection Act of 2006. Effective May 31, 2009, the Company cancelled the Safe Harbor Matching contribution.

Effective January 1, 2011, the Company reinstituted the Safe Harbor Matching provisions permitted under the Pension Protection Act of 2006 for the period January 1, through December 31, 2011, and will contribute a match of 100% of the participant’s elective deferral for the first 1% of compensation plus 50% of the participant’s elective deferral for the next 5% of compensation.

4

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 1- Description of the Plan (continued)

Automatic Enrollment

Each employee who has satisfied the eligibility requirement shall be automatically enrolled and shall have an amount equal to 3% of their compensation automatically deducted unless the employee affirmatively elects a different elective deferral contribution percent (including a zero percent election). Unless an eligible employee affirmatively elects not to contribute or changes their deferral rate, the pre-tax deferral will increase annually by 1% not to exceed 10%.

Participants’ Accounts

Each participant’s account is credited with the participant’s contributions, the Company’s matching and discretionary contribution, plan earnings and an allocation of plan expenses. The benefit to which a participant is entitled is the benefit that can be provided from the participant’s account.

Forfeitures

Amounts forfeited under the Plan are used to reduce future employer contributions and administrative expenses. At December 31, 2010 and 2009, forfeited non-vested accounts totaled $2,092 and $1,039, respectively.

Vesting

Participants are immediately vested in their contributions plus actual earnings thereon. Vesting in the Company contribution portion of their accounts plus actual earnings thereon is based on years of continuous service. A participant is 100% vested after two years of credited service.

Investment Options

Upon enrollment in the Plan, a participant may direct their elective contribution and Company contributions into various investment options offered by the Plan.

Participants may change their investment options at any time.

5

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 1- Description of the Plan (continued)

Notes Receivable from Participants

Participants may borrow the lesser of $50,000 reduced by the individual’s highest outstanding loan balance during the preceding twelve months or 50% of the individual’s vested balance with a minimum of $1,000 per loan. Loan transactions are treated as a transfer to / (from) the investment fund from / (to) the participant loans fund. Loans are secured by the balance in the participant’s account. Loan terms range from one to five years unless the loan is used to purchase a primary residence, in which case the loan may be repaid over a ten-year period. Other restrictions, as specified in the Plan agreement, may apply to a participant’s loan transaction. Interest rates range from 4.75% to 9.25% at December 31, 2010. Principal and interest is paid ratably through payroll deductions.

Plan Expenses

In accordance with the Plan, all administrative expenses may be paid out of the Plan unless paid by the Company. During 2010, there were no expenses paid by the Company.

Payment of Benefits

Distribution can be made if requested due to disability, retirement, or termination of employment. On termination of service for any reason, a participant may receive a lump sum amount equal to the vested value of the participant’s vested interest in his or her account, if the vested interest is $5,000 or less. If the participant’s vested interest is over $5,000, the participant may elect to receive payment in a lump-sum amount or installments paid over a certain number of years selected by the participant.

Beneficiaries may request a distribution of vested Account balance in the event of death. The Account balance will continue to increase or decrease, as appropriate, based on the investment returns until it is distributed.

Distribution can be deferred to a later date, however, it cannot be postponed if the vested Account balance is $5,000 or less; in which case the Plan Administrator will direct the Trustee that any amount exceeding $1,000 be distributed to an Individual Retirement Account or Annuity (“IRA”). If the vested Account balance is $1,000 or less, the Plan Administrator will direct the Trustee to distribute it as a Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan lump sum distribution without the participant’s consent. Prior to such distribution, participants still have the right to request that the amount be distributed directly to them in the form of a lump sum payment or to request that it be rolled-over to a different IRA provider or another retirement plan eligible to receive rollover contributions.

6

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 1- Description of the Plan (continued)

If the participant fails to request a different treatment of an automatic distribution under the Plan’s Cash-Out Provision, the distribution will be paid over to an IRA provider chosen by the Plan Administrator and invested in a product designed to preserve the principal of that distribution while still providing a reasonable rate of return and preserving liquidity. The fees assessed against this newly established IRA by its provider will be paid by the participant.

Plan Termination

Although it has not expressed any intent to do so, the Company has the right under the Plan to discontinue its contributions at any time and to terminate the Plan subject to the provisions of ERISA. In the event of a plan termination, participants will become 100% vested in their accounts.

Note 2- Summary of Significant Accounting Policies

Basis of Accounting

The financial statements have been prepared on the accrual basis of accounting

The Fidelity Advisor Stable Value Fund invests in fully benefit-responsive investment contracts which are reported at fair value (see Note 5); however, contract value is the relevant measurement attribute for that portion of the net assets available for benefits attributable to fully benefit-responsive investment contracts because contract value is the amount participants would receive if they were to initiate permitted transactions under the terms of the Plan. The Statement of Net Assets Available for Benefits presents the fair value of the investment contracts as well as the adjustment of the fully benefit-responsive investment contracts from fair value to contract value. The Statement of Changes in Net Assets Available for Benefits is prepared on a contract value basis.

Investment Valuation and Income Recognition

Investments are stated at fair value. Fair value is the price that would be received to sell an asset or paid to transfer a liability in an orderly transaction between market participants at the measurement date (an exit price). See Note 5 for further discussion of fair value measurement.

7

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 2- Summary of Significant Accounting Policies (continued)

Purchases and sales of securities are recorded on a trade-date basis. Interest income is recorded on the accrual basis. Dividends are recorded on the ex-dividend date. Net appreciation includes the plan’s gains and losses on investments bought and sold as well as held during the year.

Notes Receivable from Participants

Notes receivable from participants are measured at their unpaid principal balance plus accrued but unpaid interest. Delinquent participant loans are reclassified as distributions based upon the terms of the plan document.

Use of Estimates

The preparation of financial statements in conformity with accounting principles generally accepted in the United States of America requires management to make estimates and assumptions that affect the reported amounts of assets and liabilities and changes therein and disclosure of contingent assets and liabilities. Actual results could differ from those estimates.

Payment of Benefits

Benefits are recorded when paid.

Recent Accounting Pronouncements

Fair Value Disclosures - In January 2010, the Financial Accounting Standards Board (“FASB”) issued guidance which expanded the required disclosures about fair value measurements. In particular, this guidance requires (i) separate disclosure of the amounts of significant transfers in and out of Level 1 and Level 2 fair value measurements along with the reasons for such transfers, (ii) information about purchases, sales, issuances and settlements to be presented separately in the reconciliation for Level 3 fair value measurements, (iii) fair value measurement disclosures for each class of assets and liabilities and (iv) disclosures about the valuation techniques and inputs used to measure fair value for both recurring and nonrecurring fair value measurements for fair value measurements that fall in either Level 2 or Level 3. This guidance is effective for annual reporting periods beginning after December 15, 2009 except for (ii) above which is effective for fiscal years beginning after December 15, 2010. Net assets available for benefits and changes in net assets available for benefits of Plan were not affected by the adoption of the new guidance.

8

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 2- Summary of Significant Accounting Policies (continued)

Notes Receivable from Participants - In September 2010, the FASB issued guidance clarifying the classification and measurement of participant loans by defined contribution pension plans. Participant loans are required to be classified as notes receivable from participants (rather than investments) and measured at their unpaid principal balance, plus any accrued but unpaid interest. The guidance, which must be applied retrospectively, is effective for fiscal years ending after December 15, 2010 with early adoption permitted. The Plan adopted this guidance in its December 31, 2010 financial statements and has reclassified participant loans at December 31, 2009 from investments to notes receivable from participants. Net assets available for benefits of the Plan were not affected by the adoption of the new guidance.

Note 3- Investments

During 2010, the Plan’s investments (including investments purchased, sold as well as held during the year) appreciated in fair value as follows:

	Net Appreciation in Fair Value of Investments
Mutual funds	$	705,084
Common Stock		114,396
	$	819,480

Investments that represent 5% or more of the Plan’s net assets available for benefits are as follows:

	December 31,
	2010				2009
Fidelity Advisor Stable Value Fund *	$	3,951,724	34.4	%	$	4,380,364	39.0	%
Fidelity Advisor Freedom 2005		1,007,336	8.8	%		957,083	8.5	%
JP Morgan Equity Index		992,458	8.6	%		955,472	8.5	%
Fidelity Advisor Freedom 2010		771,837	6.7	%		792,838	7.1	%
Fidelity Advisor New Insights		629,535	5.5	%		—	—
T. Rowe Price Growth Stock		—	—			615,158	5.5	%
	$	7,352,890	64.0	%	$	7,700,915	68.6	%

* The amount represents contract value for this investment.

9

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 4- Non Participant Directed Investments

Information about the net assets and the significant components of the changes in net assets relating to the non participant directed investments are as follows:

	2010		2009
Net Assets:
Fidelity Advisor Stable Value Fund	$	2,092	$	1,039

	Year Ended December 31, 2010
Changes in Net Assets:
Interest and dividend income	$	9
Current year forfeitures		2,644
Forfeitures used for employer match		(1,600	)
	$	1,053

Note 5- Fair Value Measurements

The Plan measures fair value in accordance with the FASB’s Accounting Standards Codification (“ASC”) 820, Fair Value Measurements and Disclosures, which provides a fair value hierarchy that prioritizes the inputs to valuation techniques used to measure fair value. The hierarchy gives the highest priority to unadjusted quoted prices in active markets for identical assets or liabilities (Level 1 measurements) and the lowest priority to unobservable inputs (Level 3 measurements). The three levels of the fair value hierarchy under FASB ASC 820 are described as follows:

Level 1	Inputs to the valuation methodology are unadjusted quoted prices for identical assets or liabilities in active markets that the Plan has the ability to access.
Level 2	Inputs to the valuation methodology include
	•    Quoted prices for similar assets or liabilities in active markets;
	•    Quoted prices for identical or similar assets or liabilities in inactive markets;
	•    Inputs other than quoted prices that are observable for the asset or liability;
	•    Inputs that are derived principally from or corroborated by observable market data by correlation or other means.

10

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 5- Fair Value Measurements (continued)

	If the asset or liability has a specified (contractual) term, the Level 2 input must be observable for substantially the full term of the asset or liability.
Level 3	Inputs to the valuation methodology are unobservable and significant to the fair value measurement.

The asset or liability’s fair value measurement level within the fair value hierarchy is based on the lowest level of any input that is significant to the fair value measurement. Valuation techniques used need to maximize the use of observable inputs and minimize the use of unobservable inputs.

Following is a description of the valuation methodologies used for assets measured at fair value. There have been no changes in the methodologies used at December 31, 2010 and 2009.

Common stock and Mutual funds: Valued at the closing price reported on the active market on which the individual securities are traded.

Common/collective trust fund: Valued at fair value by discounting the related cash flows based on current yields of similar instruments with comparable durations considering the credit-worthiness of the issuer of the specific instruments of the fund at year-end.

The preceding methods described may produce a fair value calculation that may not be indicative of net realizable value or reflective of future fair values. Furthermore, although the Plan believes its valuation methods are appropriate and consistent with other market participants, the use of different methodologies or assumptions to determine the fair value of certain financial instruments could result in a different fair value measurement at the reporting date.

11

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 5- Fair Value Measurements (continued)

The following table sets forth by level, within the fair value hierarchy, the Plan’s investments at fair value as of December 31, 2010 and 2009:

Investments at Fair Value as of December 31, 2010

	Level 1		Level 2		Level 3		Total
Mutual funds:
Large cap stock funds	$	1,960,779	$	—	$	—	$	1,960,779
Mid cap stock funds		502,722		—		—		502,722
Small cap stock funds		203,096		—		—		203,096
International stock funds		676,245		—		—		676,245
Target funds		2,876,422		—		—		2,876,422
Fixed income funds		844,784		—		—		844,784
Total mutual funds		7,064,048		—		—		7,064,048
Common stocks:
Retail		149,742		—		—		149,742
Common/collective trust funds:
Fixed Income funds		—		4,027,032		—		4,027,032
Total investments at fair value	$	7,213,790	$	4,027,032	$	—	$	11,240,822

Investments at Fair Value as of December 31, 2009

	Level 1		Level 2		Level 3		Total
Mutual funds:
Large cap stock funds	$	1,943,741	$	—	$	—	$	1,943,741
Mid cap stock funds		462,568		—		—		462,568
Small cap stock funds		117,450		—		—		117,450
International stock funds		569,120		—		—		569,120
Target funds		2,762,843		—		—		2,762,843
Fixed income funds		442,682		—		—		442,682
Total mutual funds		6,298,404		—		—		6,298,404
Common stocks:
Retail		99,536		—		—		99,536
Common/collective trust funds:
Fixed Income funds		—		4,374,640		—		4,374,640
Total investments at fair value	$	6,397,940	$	4,374,640	$	—	$	10,772,580

12

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 6- Related Party Transactions

The Plan owned 31,591 and 42,720 shares of Casual Male Retail Group, Inc. valued at $149,742 and $99,536 at December 31, 2010 and 2009, respectively. Additionally, certain Plan investments are shares of mutual funds and units in the Fidelity Advisor Stable Value Fund managed by Fidelity Management Trust Company who is the Trustee as defined by the Plan, and therefore, these transactions qualify as party-in-interest transactions.

Note 7- Risks and Uncertainties

The Plan invests in various investment securities. Investment securities are exposed to various risks such as interest rate, market and credit risks. Due to the level of risk associated with certain investment securities, it is at least reasonably possible that changes in the values of investment securities will occur in the near term and that such changes could materially affect participants’ account balances and the amounts reported in the Statements of Net Assets Available for Benefits.

Note 8- Income Tax Status

The underlying non-standardized prototype plan has received an opinion letter from the Internal Revenue Service (IRS) dated March 31, 2008 stating that the form of the Plan is qualified under Section 401 of the Internal Revenue Code (the “Code”) and, therefore, the related trust is tax-exempt. In accordance with Revenue Procedure 2002-6 and Announcement 2001-77, the Plan Sponsor has determined that it is eligible to and has chosen to rely on the current IRS prototype plan opinion letter. Once qualified, the Plan is required to operate in conformity with the Code to maintain its qualification. The Plan has been amended since receiving the opinion letter. However, the plan administrator believes the Plan is being operated in compliance with the applicable requirements of the Code and, therefore, believes that the Plan is qualified and the related trust is tax-exempt.

Accounting principles generally accepted in the United States of America require plan management to evaluate tax positions taken by the Plan and recognize a tax liability (or asset) if the Plan has taken an uncertain tax position that more likely than not would not be sustained upon examination by the taxing authorities. The plan administrator has analyzed the tax positions taken by the Plan, and has concluded that as of December 31, 2010, there are no uncertain positions taken or expected to be taken that would require recognition of a liability (or asset) or disclosure in the financial statements. The Plan is subject to routine audits by taxing jurisdictions; however, there are currently no audits for any tax periods in progress. The plan administrator believes it is no longer subject to income tax examinations for 2006 and prior.

13

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

Notes to Financial Statements

Note 9- Reconciliation of Financial Statement to Form 5500

The following is a reconciliation of net assets available for benefits according to the financial statements to Form 5500 December 31, 2010 and 2009:

	2010		2009
Net assets available for benefits per the financial statements	$	11,483,447	$	11,228,170
Adjustment from fair value to contract value for fully benefit-responsive investment contract		75,309		(5,724	)
Net assets available for benefits per Form 5500	$	11,558,756	$	11,222,446

The following is a reconciliation of net appreciation in fair value of investments, interest and dividend income, and interest income on participant loans for the year ended December 31, 2010 per the financial statements to Form 5500:

Net appreciation in fair value of investments, interest and dividend income, and interest income on participant loans per the financial statements	$	1,039,762
Add: Adjustment from fair value to contract value for fully benefit-responsive investment contracts in 2010		75,309
Add: Adjustment from fair value to contract value for fully benefit-responsive investment contracts in 2009		5,724
Net appreciation in fair value of investments, interest and dividend income and interest income on participant loans per Form 5500	$	1,120,795

14

Supplemental Schedule

15

Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan

EIN: 04-2623104/Plan Number: 005

Schedule H, Line 4i – Schedule of Assets Held For Investment Purposes at

End of Year

(December 31, 2010)

Identity of Issue, Borrower, Lessor, or Similar Party	Description of investment including maturity date, rate of interest, collateral, par, or maturity value	Shares/Units		Current Value
Commingled funds held through Fidelity Investments:
	* Fidelity Advisor Stable Value Fund (1)		3,951,724	$	4,027,032
Mutual funds held through Fidelity Investments:	Cash – interest bearing		—		1,262
	* Fidelity Advisor Leveraged Company Stock		3,121		107,386
	* Fidelity Advisor High Income Adv.		10,492		104,181
	* Fidelity Advisor Freedom 2010		66,884		771,837
	* Fidelity Advisor Freedom 2020		19,975		239,705
	* Fidelity Advisor Freedom 2030		5,306		64,682
	* Fidelity Advisor Freedom 2040		6,122		75,670
	* Fidelity Advisor Freedom Income		1		13
	* Fidelity Advisor New Insights		31,540		629,535
	* Fidelity Advisor Freedom 2005		90,425		1,007,336
	* Fidelity Advisor Freedom 2015		39,133		449,637
	* Fidelity Advisor Freedom 2025		8,413		97,595
	* Fidelity Advisor Freedom 2035		4,430		51,254
	* Fidelity Advisor Freedom 2045		4,769		45,830
	* Fidelity Advisor Freedom 2050		7,702		72,863
	Federated Bond Fund		55,197		504,498
	Columbia Acorn USA		4,229		116,473
	Columbia Acorn International		13,506		552,000
	Allianz NFJ International Value		5,996		124,245
	American Century Investment Mid Cap Value		3,561		44,795
	JP Morgan Equity Index		34,786		992,458
	Columbia Acorn		11,988		350,541
	Goldman Sachs Large Cap Value		28,735		338,786
	Perkins Small Cap Value		3,621		86,623
	Franklin Strategic Income		22,534		234,804
	Blackrock Inflation Protected Bond Fund		4		39
Common Stock held through Fidelity Investments:	*Casual Male Retail Group, Inc.		31,591		149,742
					11,240,822
*Participant loans	4.75% - 9.25%				317,934
				$	11,558,756

* Indicates party-in-interest to the Plan.

(1) Includes $2,092 of nonparticipant-directed, unallocated forfeitures for which historical cost approximates current value.

16

Index to Exhibits

23.1	Consent of Independent Registered Public Accounting Firm
32.1	Certification pursuant to 18 U.S.C. Section 1350, as adopted pursuant to Section 906 of the Sarbanes –Oxley Act of 2002

Signatures

The Plan. Pursuant to the requirements of the Securities Exchange Act of 1934, the trustees (or other persons who administer the employee benefit plan) have duly caused this annual report to be signed on its behalf by the undersigned hereunto duly authorized.

Casual Male Retail Group, Inc. 401(k) Hourly Savings plan
By:	/s/    DENNIS R. HERNREICH
	Dennis R. Hernreich, Executive Vice President, Chief Financial Officer and Chief Operating Officer of Casual Male Retail Group, Inc., the Plan Administrator

June 29, 2011

EX-23.1

Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the Registration Statement (No. 333-112218) on Form S-8 of Casual Male Retail Group, Inc. of our report dated June 29, 2011, relating to the financial statements and supplemental schedule of the Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan included in the December 31, 2010 Annual Report on Form 11-K for the Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan.

                /s/ Mayer Hoffman McCann CPAs                        

(The New York Practice of Mayer Hoffman McCann P.C.)

New York, New York

June 29, 2011

EX-32.1

Exhibit 32.1

Certification Pursuant to

18 U.S.C. Section 1350,

As Adopted Pursuant to Section 906

Of the Sarbanes-Oxley Act of 2002

In connection with the Annual Report on Form 11-K of the Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan (the “Plan”) for the period ending December 31, 2010 as filed with the Securities and Exchange Commission on the date hereof (the “Report”), I, Dennis R. Hernreich, Executive Vice President, Chief Financial Officer and Chief Operating Officer of Casual Male Retail Group, Inc., the Plan Administrator, certify pursuant to 18 U.S.C. Section 1350, as adopted pursuant to Section 906 of the Sarbanes-Oxley Act of 2002, that:

(1) The Report fully complies with the requirements of Section 15(d) of the Securities Exchange Act of 1934, as amended (the “Exchange Act”); and

(2) The information contained in the Report fairly presents, in all material respects, the net assets available for benefits, and changes in net assets available for benefits of the Plan.

A signed original of this written statement required by Section 906, or other document authenticating, acknowledging, or otherwise adopting the signature that appears in typed form within the electronic version of this written statement required by Section 906, has been provided to Casual Male Retail Group, Inc. and will be retained by Casual Male Retail Group, Inc. and furnished to the Securities and Exchange Commission or its staff upon request.

/s/ Dennis R. Hernreich

Dennis R. Hernreich

Executive Vice President, Chief Financial Officer,

Chief Operating Officer of Casual Male Retail

Group, Inc., the Plan Administrator

June 29, 2011