In his new book, Anderson Cooper tells the incredible story of his ancestors, including his late heiress turned fashion designer mother, Gloria Vanderbilt

Anderson Cooper is opening up about the special bond he had with his heiress turned fashion designer mother, Gloria Vanderbilt, who died just over two years ago.

“My mom was extraordinarily unique, and I miss that uniqueness in my life,” Cooper, 54, says in this week’s PEOPLE cover story.

“It was sometimes very frustrating and stressful, because she never had a plan for things,” he continues. “She was incredibly creative, so things were always changing. I was always having to adapt to her new ideas and encourage some of those, and try to discourage her from things that didn’t make any sense.”

In his new book (co-written with historian and novelist Katherine Howe), Vanderbilt: The Rise and Fall of an American Dynasty (available Sept. 21), Cooper dives deep into the past to tell the incredible story of how his ancestors were once among the richest people in the world before losing it all.

Ben Watts

He also sits down PEOPLE to share what he’s learned about love and loss from his famous family, and how he’s raising his son Wyatt, whom he co-parents with his former partner of 10 years, Benjamin Maisani, 48.

Gloria Vanderbilt was born in New York City on February 20, 1924 to one of the wealthiest families in America. Her father was railroad heir Reginald Claypoole Vanderbilt — the great-grandson of Cornelius Vanderbilt, who built the New York Central Railroad — and died when she was 18 months old. Her mother was his second wife, 19-year-old Gloria Morgan.

While her mother attended parties and traveled, her grandmother “Naney” and her beloved nanny “Dodo” quietly raised Vanderbilt. But by age 10, she was back in the news. Dubbed the “poor little rich girl” by press when her socialite mother and her aunt Gertrude became locked in a courtroom fight, Vanderbilt found herself at the center of a high-profile custody battle.

Years later, as a teen, Vanderbilt traveled to Beverly Hills to reconnect with her mother for a visit that was supposed to last only two weeks. But two weeks in Los Angeles turned into months – and the trip eventually led her to meeting her first husband.

“The summer after my junior year in high school, I went out on dates with movie stars every night: Errol Flynn, George Montgomery, Ray Milland,” she told PEOPLE in 2016. “My mother let me do anything that I wanted to do and I hardly ever saw her.”

Vanderbilt also had love affairs with Frank Sinatra and Howard Hughes. She would marry three more times, her final time to Cooper’s father, author and screenwriter Wyatt Emory Cooper. He later died while undergoing open-heart surgery in 1978.

Anderson Cooper and mother Gloria Vanderbilt. Dimitrios Kambouris/Getty

As Vanderbilt coped with the loss, she juggled single motherhood and her career. She found success as an actress and artist designing home furnishings, sheets and china, but it was her line of Gloria Vanderbilt designer jeans that made her millions — though she would eventually lose it all at the hands of a fraudulent lawyer.

When she turned 91, Vanderbilt and Cooper cowrote a memoir together, The Rainbow Comes and Goes. In 2019, she died from stomach cancer. She was 95.

“I miss the random phone calls with a crazy idea, or her asking me ludicrous relationship advice,” Cooper tells PEOPLE. “I’m certainly not anyone to give relationship advice, but I would give my mom feedback when she would go on dates, what I thought about the guys. Some of them I really didn’t think much of at all! We had a really unique relationship.”