Replication Studies and The Miami Project
Replication Studies and The Miami Project: A Firsthand Report From the Trenches by Dr. Vance Lemmon
Over the past ten years or so a consensus has emerged that, before a new therapy is tested in humans with spinal cord injuries, the original animal studies should be retested to ensure the method is robust and reliable. While there are many reasons for this, the famous “Facilities of Research Excellence-Spinal Cord Injury” (FORE-SCI) program launched in 2003 by the National Institute of Neurological Disorders and Stroke (NINDS), and performed at the University of California, Irvine, the Ohio State University and the University of Miami, clearly demonstrated that many promising studies could not be replicated (PMID: 22078756).
This is not unique to SCI in replication studies. Similar lack of reproducibility in other fields has led the National Institutes of Health (NIH) and international organizations to make specific recommendations about strategies to improve the rigor of biomedical research. Chief among these are improved reporting of methods, ensuring that investigators are blinded to treatments, and that there are enough samples that the results are statistically meaningful (PMID: 20613859, PMID: 23060188).
The SCI research community, including Miami Project faculty (Bixby, Bunge, Guest, Lee, Lemmon, Oudega, Park), has been at the forefront of these efforts – establishing reporting standards and data repositories (PMID: 24870067, 27055827, 28576567) and implementing best practices (PMID: 20507235, 23727091, 25902036, 28716559).
Because The Miami Project has been so successful at conducting a variety of U.S. FDA approved clinical trials, SCI researchers from around the world often reach out to us for advice about pre-clinical experiments and clinical trial design. Even me, a cell and molecular neuroscientist, gets emails and phone calls from people asking for advice. While it is true I am co-inventor on some patents about potential SCI therapeutics, our lab’s work is at the beginning of the drug discovery pipeline, with little involvement in the translational efforts involving clinical trials.
In late 2016, I received an email from a professor at UCLA, Dr. Yi E. Sun. I knew her when she was a Ph.D. student at Case Western Reserve University. From there she went to Harvard where she published high impact papers on development of the brain. At UCLA, she started studying stem cells as a way to understand developmental diseases like Autism. This resulted in collaborations with a Noble Prize winner, Thomas Sudhof, at Stanford and a group of SCI scientists in Beijing. The Beijing team, led by Professor Xiaoguang Li, developed a clever biomaterial that slowly releases a growth factor that stimulates endogenous stem cells in the spinal cord. The stem cells proliferate and form new neurons and glia that improve sensory and motor recovery after a large spinal cord transection in rats (PMID: 26460015, 26460053). Dr. Sun asked if I could travel to Beijing, along with other Miami Project scientists, to advise about a potential clinical trial. After reminding her that I had never been involved in a clinical trail myself, I agreed to come with colleagues to help evaluate the preclinical data and make suggestions about the feasibility of a trial.
In early February of 2017, I flew to Beijing for a kick-off meeting, along with Anil Lalwani, a biomedical engineer who had worked at the MP and been heavily involved in the Investigational New Drug (IND) and Investigational Device Exemption (IDE) submissions to the FDA for the Schwann cell clinical trial and medical device studies for pain. At this meeting we saw the rat data as well as unpublished data from studies in monkeys, showing that the biomaterial/growth factor combination had dramatic effects on walking. I and other scientists at the conference, mindful of the poor replication track record of previous “breakthroughs”, strongly recommended that an independent replication study be done prior to moving forward with clinical development of this novel combination approach. Professor Li and his colleagues agreed, in principle, to do this. We then went to Shanghai to visit Tongji Hospital, where they were considering doing the clinical trail. This giant hospital (over 4000 beds) had a large unit set up for conducting early stage clinical research and clinical trials that complied with the Chinese FDA regulations. It also had a very busy spine surgery department that received sufficient numbers of SCI patients to conduct a meaningful phase one safety trial in a relatively short amount of time, especially compared to U.S. hospitals.
After returning to the U.S., Professor Li asked me to help organize the replication study. Since I had urged him to do this it was difficult to say no. But figuring out how to do the replication study was extremely difficult. There were at least five non-trivial problems that needed to be resolved. 1) How to fund the replication project. 2) How to overcome institutional concerns about protecting intellectual property regarding the biomaterial. 3) Deciding exactly what experiments needed to be replicated. 4) How to get it done efficiently. 5) How to not go astray of U.S. laws and regulations regarding transfer of technology to China. To deal with this last issue we got excellent training from the U.M. Export Control officer on how to comply with U.S. Department of Commerce regulations.
In May I returned with a bigger team to try to resolve these and other issues. Our replication studies team members were Dr. Rachel Cowan, a Miami Project faculty member who has a spinal cord injury, Stephan Zuchner, a neurologist and chair of the UM genetics department, Yan Shi, a MP scientist who is an expert in imaging (and importantly, a native from Beijing), Monica Perez, a MP professor and expert on physical therapy and outcome measures, Anil Lalwani as a regulatory consultant and myself as a neuroscientist and the team leader. We were joined by Mark Tuszynski, a world famous SCI researcher from UCSD in La Jolla, CA. We visited the lab at Capital City Hospital where the animal work was done, looked at tissue and saw how they did various experiments. We had long discussions about the experiments to be done. One big road block was that the Chinese institutions would not allow the biomaterial to leave China. Consequently, the U.S. scientists would have to come to China to conduct the experiments. Since SCI experiments take months, we agreed to come for the initiation and completion of these studies, and ensure that investigators in China were blinded by using electronic tags to identify animals that were coded by UM investigators.
Prof. Tuszynski made great suggestions about how we could best do this. We then went to Shanghai to revisit Tongji Hospital and discussed details of clinical trial design. Aspects like dose and size of the implant, inclusion/exclusion criteria. Outcome measures for SCI clinical trials were discussed with the intention of planning a safe and well thought out investigation in humans – one that would satisfy SCI experts around the world as well as global regulatory authorities.
Traveling with Rachel and her wheelchair in China was an eye opening experience. From getting off an airplane, to narrow door openings in hotels, to a virtual absence of wheel chair accessible restrooms in any public buildings; being an SCI individual in China is incredibly difficult. Thank goodness for the “Americas with Disabilities Act”!
We had planned to return to Beijing in July to start the experiments but the university’s animal facility was closed for complete renovation. We used this time to solve the funding problem, receiving a grant from Tongji Hospital in Shanghai, to do the experiments in Beijing and pay for the travel of the U.S. scientists. We were unable to return until the beginning of September. On this trip Prof. Dan Liebl from the MP accompanied Yan Shi and myself to participate in all the surgeries. These were grueling days that lasted 10 hours in the surgery suite, bookended by commutes in Beijing traffic. Unfortunately, Hurricane Irma set its sights on South Florida so Dan had to rush home to take care of his family. Yan and I stayed to complete our observations of all phases of the surgeries. Only two members of the large Chinese team spoke even a little English. This was a major complicating factor, so having a Miami Project scientist, Yan Shi, on our team who was a native speaker of Chinese was essential.
In December of 2017, Martin Oudega, a member of the MP and the Principal Investigator (P.I.) on the grant form Tongji, flew to Beijing with Yan Shi and myself. Martin was the ideal P.I. He does research on biomaterials using rats and has first-hand experience with all the outcome measures the Beijing team used. We observed and videotaped the behavioral and electrophysiological tests and then observed the processing of histologic tissue and examined the specimens with confocal microscopes. These were 12 hour days in the lab for 10 straight days. Actually, the first day, we went from the airport directly to the laboratory to do behavioral testing and then went to the hotel to take a shower. By good luck, Rosario Isasi, an attorney and bioethicist and assistant professor from the U.M. genetics department was in Beijing and we took her to Beijing Changgeng Hospital, a second site being considered for a clinical trail, so we could get her advice about informed consent.
When we returned to Miami, we had two different teams score the animal behavior videos in a blinded fashion. The Chinese team also scored the videos. We even had a colleague at OSU examine some of the videos to give us additional confidence. In Miami, we examined the histological images, assembled figures and performed statistical analysis on the data. We prepared a manuscript, including a data table that reported all the results for each individual animal. When we circulated the manuscript to the co-authors, there was some back and forth about the meaning of various tests and words. But we came to consensus and Martin, as P.I., was the final arbitrator.
In the past, journals have been reluctant to publish replication studies. But the recent concerns about scientific rigor have changed everyone’s mind about this. Indeed, the U.S. V.A. has recently announced a program to expressly fund replication studies. We submitted our manuscript to Experimental Neurology, an important journal in the CNS injury field and the journal that published the papers from the FORE-SCI program. The reviewers were very positive about the project and had some useful suggestions for improving the manuscript (mostly to put even more data in the appendix), which we were happy to comply with.
After the paper was published (PMID: 30471251), the editor of the journal invited a German SCI researcher, Frank Bradke, to write a short commentary about it (PMID: 30605623). In the commentary, Dr. Bradke and his student, Barbara Schaffran, commented on the value of doing rigorous replications and hope our success inspires others to undertake similar replication studies to help the SCI field move forward more quickly.
In the middle of 2018, the Chinese team published their monkey results in the U.S.A. Proceedings of the National Academy of Science (PMID: 29844162). They now have a very strong package of positive results in two species along with our replication study to apply for funding for a clinical trial. If they are successful, there is a good chance they will be asking for more help from The Miami Project on next steps.