"Zidong Taichu" Full Modal Large Model Released, Precisely Positioned 3D Scene, Listening to "Moonlight Song" and Talking about Beethoven Images | Applications | Beethoven

Release time:Apr 13, 2024 22:00 PM

Not only can you hear Beethoven talk freely in "Moonlight", but you can also achieve precise positioning in three-dimensional scenes, and complete scene analysis through the combination of images and sound. On June 16, at the AI Framework Ecological Summit, the Institute of Automation of the Chinese Academy of Sciences officially released the full mode large model of "Zidong Taichu".

This model is the 2.0 version upgraded from the 1.0 version of the 100 billion parameter multimodal model "Zidong Taichu". On the basis of voice, image and text three modes, it adds video, sensor signal, 3D point cloud and other modal data, breaks through the key technologies such as multimodal correlation for cognitive enhancement, and has full modal understanding, generation and correlation capabilities.

At the meeting, Xu Bo, the director of the Institute of Automation, presented in real-time for the first time the new features of the "Zidong Taichu" full modal cognitive model in music understanding and generation, three-dimensional scene navigation, signal understanding, multimodal dialogue, and invited on-site audiences to interact with the model in real-time.

Continuous Exploration from Multimodal to Full Modal

When humans perceive the world, they often involve information such as speech, images, and text simultaneously. Machines need to achieve higher levels of intelligence, just like humans, by developing larger models that connect more modalities such as graphics, text, and sound. Since 2019, the Institute of Automation has adhered to the core of "image audio text" multimodal technology, established a multimodal large model layout, integrated the advantageous resources of research directions such as images, text, and speech within the institute, carried out group style research, and successfully created the "Zidong Taichu" 1.0 multimodal large model in September 2021. "Zidong Taichu" 1.0 has propelled artificial intelligence from "one specialization and one capability" to "multiple specialties and multiple capabilities", taking a solid first step towards the development of universal artificial intelligence.

Entering the era of digital economy, the scope of data is constantly expanding, including not only human generated voice, image, text and other data, but also a large amount of structured and unstructured data generated by machines. In response to new demands and trends, "Zidong Taichu" 2.0 has achieved full modal open access to structured and unstructured data from a technical architecture perspective; Breaking through the multimodal grouping cognitive encoding and decoding technology that can fully understand and flexibly generate information, the multimodal cognitive ability of large models has been greatly improved.

From 1.0 to 2.0, the "Zidong Taichu" big model has broken through the interactive barriers of perception, cognition, and even decision-making, enabling artificial intelligence to further perceive and recognize the world, thereby extending more powerful universal capabilities.


"Zidong Taichu" Full Modal Large Model Released, Precisely Positioned 3D Scene, Listening to "Moonlight Song" and Talking about Beethoven Images | Applications | Beethoven

[Broad prospects for industrial applications]

The "Zidong Taichu" 2.0 is based on the self-developed algorithm of the Institute of Automation, and is based on the Shengteng AI hardware and Shengsi MindSpore AI framework. With the support of the computing power of the Wuhan Artificial Intelligence Computing Center, it focuses on creating a full stack domestically produced universal artificial intelligence base.

At present, the "Zidong Taichu" model has shown broad industrial application prospects, and has begun a series of applications in fields such as neurosurgical surgical navigation, short video content review, legal consultation, medical multimodal differential diagnosis, and traffic violation image study.

In medical scenarios, the "Zidong Taichu" large model is deployed on the neurosurgical robot MicroNeuro, which can fuse multimodal information such as vision and touch in real-time during surgery, assisting doctors in real-time inference and judgment of surgical scenes. The research team is collaborating with Peking Union Medical College Hospital and utilizing the strong logical reasoning ability of "Zidong Taichu" to attempt breakthroughs in the diagnosis and treatment of rare human diseases.

Xu Bo stated that the Institute of Automation will continue to explore the integration of technology paths such as neuromorphic intelligence and game intelligence based on the "Zidong Taichu" model.

Shanghai International Sister City Youth "Play" Summer Camp Experience Traditional Culture, Make Pankou, Learn Paper Cuttings, Make Dumplings International | Youth | Make Dumplings
Shanghai International Sister City Youth "Play" Summer Camp Experience Traditional Culture, Make Pankou, Learn Paper Cuttings, Make Dumplings International | Youth | Make Dumplings

Making coils, learning Paper Cuttings, making dumplings, walking into the home of summer camp volunteers and feeling the life of Shanghai people... On the 20th, the 2023 Shanghai International Sister City Youth Summer Camp officially opened in Shanghai Shidong Experimental School. 73 campers from 13 cities in 12 countries gathered in Shanghai to open the annual international sister city youth exchange event with their peers in Shanghai. The young partners together carried out activities such as learning excellent traditional Chinese culture courses such as Chinese and traditional Chinese painting, intangible cultural heritage, Chinese clothing, disco, Paper Cuttings, seal cutting, calligraphy, pottery, tea art, Yanzhi, dragon dance, youth forum exchanges in sister cities, investigation of urban cultural landscape, visits to universities and venues, city orientation challenges, and local family life experiences, etc., to bloom their youth. In addition to a rich and colorful summer camp physical activity experience, campers and volunteers

Undergraduate voluntary application starts today! These important reminders and suggestions must be read, @ College Entrance Examination Stage | Undergraduate | Volunteer
Undergraduate voluntary application starts today! These important reminders and suggestions must be read, @ College Entrance Examination Stage | Undergraduate | Volunteer

@All college entrance examination candidates, according to the schedule of the college entrance examination, will fill in their undergraduate preferences for all batches except for the comprehensive evaluation batch from 8:00 a.m. to 8:00 p.m. daily from July 1st to 2nd, and from 8:00 a.m. to 12:00 a.m. on July 3rd. The specific contents of this voluntary application include zero voluntary batch, advance batch, art and sports class A batch, local rural special plan batch, special type enrollment, and ordinary batch. The filling method is as follows: Fresh high school graduates in this city will be arranged uniformly by the high school where they are enrolled; Non local fresh high school graduates will be arranged uniformly by the district recruitment office where they apply. It is important to remind candidates that during the voluntary application period from July 1st to 3rd, as the admission of the comprehensive evaluation batch has not yet been completed, candidates who have filled out the comprehensive evaluation batch of voluntary applications still need to carefully fill out other batches of undergraduate voluntary applications

Investment+Services Drive Anti Cancer Drugs into Clinical Practice | Entrepreneurial Stories in Incubators, Second Entrepreneurial Biology for CEOs of Listed Companies | Incubators | Clinical | Incubators
Investment+Services Drive Anti Cancer Drugs into Clinical Practice | Entrepreneurial Stories in Incubators, Second Entrepreneurial Biology for CEOs of Listed Companies | Incubators | Clinical | Incubators

Recently, with the approval of the National Medical Products Administration, the Class 1 innovative drug CC312 developed by Huihe Biotechnology has initiated phase I clinical trials for the treatment of recurrent/refractory CD19 positive B-cell malignant hematological tumors. This is the first domestically and the third globally approved triple specific antibody drug based on CD28 co stimulatory signals to enter clinical practice. When it comes to the development history of this new triple antibody drug, Dr. Zhu Huaxing, the founder of Huihe Biotechnology, still remembers vividly: "Before 2019, there was no triple antibody drug approved for clinical use globally, and some investors couldn't understand CC312. During a critical period of company development, Nokai Xinkang Fund invested 30 million yuan to help us complete Series A financing." Nokai Xinkang is a venture capital fund initiated by Xinze Entrepreneurship Incubator. This incubation company, which has been deeply cultivated in Zhangjiang Science City for many years

"Zidong Taichu" Full Modal Large Model Released, Precisely Positioned 3D Scene, Listening to "Moonlight Song" and Talking about Beethoven Images | Applications | Beethoven
"Zidong Taichu" Full Modal Large Model Released, Precisely Positioned 3D Scene, Listening to "Moonlight Song" and Talking about Beethoven Images | Applications | Beethoven

Not only can you hear Beethoven talk freely in "Moonlight", but you can also achieve precise positioning in three-dimensional scenes, and complete scene analysis through the combination of images and sound. On June 16, at the AI Framework Ecological Summit, the Institute of Automation of the Chinese Academy of Sciences officially released the full mode large model of "Zidong Taichu". This model is the 2.0 version upgraded from the 1.0 version of the 100 billion parameter multimodal model "Zidong Taichu". On the basis of voice, image and text three modes, it adds video, sensor signal, 3D point cloud and other modal data, breaks through the key technologies such as multimodal correlation for cognitive enhancement, and has full modal understanding, generation and correlation capabilities. At the meeting, Xu Bo, the director of the Institute of Automation, presented for the first time in real time the "Zidong Taichu" full modal cognitive model in music understanding

Perseverance is a life experience. Lingling Middle School has weak muscles. Candidates complete the college entrance examination: no matter what difficulties they experience. High school | school | college entrance examination
Perseverance is a life experience. Lingling Middle School has weak muscles. Candidates complete the college entrance examination: no matter what difficulties they experience. High school | school | college entrance examination

"The college entrance examination is an experience in our lives. Looking back on the preparation for the entire senior year of high school, we cannot help but marvel at the preciousness of time. No matter what difficulties we have gone through, I believe that as long as we persist, it will become my lifelong wealth. Today, after completing the morning foreign language listening and speaking test of the college entrance examination, Xiao Song, a senior student of Lingling High School, came out of the examination center of East China University of Science and Technology Affiliated Middle School. The senior year graduate sitting in a wheelchair said," Although academic and life are not small tests for me, I have faced many difficulties in adversity and have never given up on my dream of taking the college entrance examination. "Xiao Song suffered from congenital muscular dystrophy since childhood and entered Lingling High School in high school." Later, the school provided him with classrooms on low floors and close to the toilet, making it easier for him to enter and exit, allowing him to face his academic life with more confidence. At school, I met many kind and enthusiastic people