Our server is supported by SAKURA internet

Reconstructing Shurijo Castle by Google Arts and Culture

Other Models

What is this project for?

On October 31st 2019, Shuri Castle was destroyed by a devastating fire. People all over the world are shocked by this news and we are also deeply saddened. We are researchers, students, and engineers who are working on problems surrounding Computer Vision (CV), Human-Computer Interaction (HCI), Virtual Reality (VR), and other related problems. We started this project in hopes that we can do something to help restore this important piece of cultural heritage, and cheer up others who were also saddened by this tragic incident.

In the field of computer vision, which aims to design computer algorithms to process and understand images, there are technologies that can reconstruct 3D shapes and textures of objects using images and videos. Those can be utilized to preserve pieces of cultural heritage. One example of such project is eHeritage.

Having heard the news of the fire, as people working on computer vision, we hope that we can help to recover these touristic resources by reconstructing a 3D model of Shuri Castle, until the parts that were damaged in the fire are rebuilt. The video above shows the concept of our project. The 3D model in this video is reconstructed using public software from the images we collected by image searching. If there are more photos and videos with a lot of diverse viewpoints, we can create a much more beautiful 3D model. Right now, our dataset is limited and thus our reconstruction is also imperfect. We want to utilize your photos and videos to create a better 3D model and share your memories of Shuri Castle with others who are also saddened by the news. Please share your photos, videos and memories with us for a digital reconstruction of Shuri Castle.

What can we do with the photos / videos collected and the memories shared? We can reconstruct more precise and complete 3D model with more data. With the contents created by the photos / videos, you can see the lost part overlapped on the local landscape through a smart glass or a mobile phone in Okinawa. You can also revive the memories of the past of other people at the place where he/she has been by using the shared memories. With a head-mounted display, you can have the same experience even if you are not at the site in Okinawa. Also, with 3D models reconstructed for each era, you will be able to experience the Shuri Castle and its change over time.

Please share your memories of Shuri Castle!

Why are we working on this?

We are volunteer members who were shocked and heartbroken when we saw the fire news. We felt that we should do something socially meaningful for Shuri Castle with 3D reconstruction technology that is closely related to our expertise, and making this happen by collecting data from everyone who have memories of visiting Shuri Castle. The loss of Shuri Castle is disastrous, but a 3D model restored with everyone’s data will encourage people and allow the castle to remain in their hearts for a long time. Fortunately, we are a group of people with varied specialties. We decided to partake in a project that has good social impact with ideas shared by the members.

In the field of computer vision, there is a technology called Structure from Motion (SfM). This is used to reconstruct 3D shapes using photos and videos from different viewpoints. Our reconstruction of Shuri Castle also uses this technology.

It has already been decades since the early ideas for 3D reconstruction, including SfM, were published, and continuous effort has been made by outstanding researchers to progress the field. Now, thanks to the contributions of great engineers, the technology has developed to a point where we can easily create 3D reconstructions using our computers as the 3D model as shown in the video above. During the International Conference on Computer Vision 2019 (ICCV 2019), Building Rome in a Day, a seminal work on SfM, was awarded the Helmholtz prize for its long term impact in the field of computer vision. This was 2 days before the devastating fire broke out in Okinawa. This work addresses various scalability issues which arise from trying to reconstruct a 3D model from tens of thousands of images. This work allowed the digital reconstruction of the Colosseum of Rome and the city of Dubrovnik from images from the internet.

Several members of our team have participated in ICCV 2019 for their own research, and several of them were the organizers of a workshop for digital preservation of cultural properties at ICCV 2019. When one of the organizers saw the news of the devastating fire, she wondered what she could do to help. When she called out to the researchers and students around her, she got great support and many members have devoted their time to this project since then. After that, we have been discussing what we can do to contribute to society by creating something that will remain in everyone’s hearts.

Why do we spend our work / study time doing these activities? Because we are glad if it helps people. Not only that, it is a pleasure to let people know the utility of our field through this kind of cooperation. Our member’s research areas are computer vision, human-computer interaction, virtual reality, machine learning, and more. We are very happy if we can make you interested in the work we do.

Members

  • Rei Kawakami (Tokyo Institute of Technology/Denso IT Laboratory)
  • Wen Shao (The University of Tokyo)
  • Ikuo Kamei (The University of Tokyo)
  • Hajime Itsukaichi (The University of Tokyo)
  • Takeshi Naemura (The University of Tokyo)
  • Towaki Takikawa (University of Waterloo)
  • Takefumi Hiraki (Osaka University)
  • Yota Egusa (SAKURA internet Inc.)
  • Eduardo Acosta (Raiz New Media)
  • Raul Acosta (Raiz New Media)
  • Karin Kiho (The University of Tokyo)
  • Han Changyo (The University of Tokyo)
  • Yuichi Miyamoto (Party Inc.)
  • Yuzuko Utsumi (Osaka Prefecture University)
  • Katsushi Ikeuchi (Microsoft)
  • Jean Ponce (INRIA / New York University)
  • Jonathan Chemla (Iconem)
  • Yves Ubelmann (Iconem)
  • Miku Sega (gluon)
  • Daiki Sugiyama (UmeeT)
  • Daisaku Kina (Ryukyu history researcher)
  • Ryo Fujiwara (HoloLab Inc. / xRArchi)

Collaborators

  • Yasutaka Furukawa (Simon Fraser University)
  • Yasuaki Kakehi (The University of Tokyo)
  • Wataru Date (Graphic Designer)
  • Furuta Katsumi(Keio University)
  • Martial Hebert (Carnegie Mellon University)
  • Rujirat (Vinitphol) Ishikawa (The University of Tokyo)
  • Marin Kamizono (Graphic Recorder)
  • Marc Pollefeys (ETH Zurich / Microsoft)
  • Johannes Schöenberger (Microsoft)
  • Yuto Kodama (The University of Tokyo)
  • Yinan Wang (The University of Tokyo)
  • Pai-Hsien Hung (Osaka University)
  • Yi-Jhen Lin
  • Tasuku Mizuno (CITY LIGHTS LAW)

References

  • Changchang Wu, “Towards Linear-time Incremental Structure From Motion”, 3DV 2013
  • Changchang Wu, “VisualSFM: A Visual Structure from Motion System”, http://ccwu.me/vsfm/, 2011
  • Changchang Wu, Sameer Agarwal, Brian Curless, and Steven M. Seitz, “Multicore Bundle Adjustment”, CVPR 2011
  • Changchang Wu, “SiftGPU: A GPU implementation of Scale Invaraint Feature Transform (SIFT)”, http://cs.unc.edu/~ccwu/siftgpu, 2007
  • Yasutaka Furukawa, Brian Curless, Steven M. Seitz, and Richard Szeliski, “Towards Internet-scale Multi-view Stereo”, CVPR 2010.
  • Michael Kazhdan, Hugues Hoppe, “Screened poisson surface reconstruction”, ACM Transactions on Graphics (TOG), Volume 32 Issue 3, June 2013
  • P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, G. Ranzuglia, “MeshLab: an Open-Source Mesh Processing Tool”, Sixth Eurographics Italian Chapter Conference, page 129-136, 2008
  • Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and Richard Szeliski. 2011. “Building Rome in a day”. Commun. ACM 54, 10 (October 2011), 105-112.
  • Y. Furukawa and J. Ponce. “Accurate, Dense, and Robust Multi-View Stereopsis”. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8):1362-1376, 2010.
  • Johannes L. Schönberger and Jan-Michael Frahm. “Structure-from-motion revisited.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
  • Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. “Pixelwise view selection for unstructured multi-view stereo.” European Conference on Computer Vision. Springer, 2016.

Acknowledgment

This project is supported by SAKURA internet Inc.
This website uses “Sakura Cloud” and “Sakura Web Accelerator“.
This website uses 3D viewer developed by Sketchfab, Inc.
This project is supported by CapturingReality, developing 3D reconstruction software RealityCapture.
This project uses image archive collected by Okinawa Times.
This project uses Azure environment supported by Microsoft Japan.
This project uses images collected by Konan Act Club at Konan high and Junior high school.
This project’s image collection is supported by Yokohama Fujimigaoka Junior high and high school.
This project’s image collection is supported by Baidu Baike and Baidu Baike Museums Project.

Public support

This project is supported by Okinawa Prefecture.

Media appearances

2019/11/06首里城が3Dモデルに?最先端の画像処理技術を使った「首里城デジタル復元プロジェクト」(BLOGOS)
2019/11/06「全焼した首里城を3Dモデルで復元する」 東大研究者ら有志が写真・動画の提供を呼びかけ 再建までの観光資源に(ITmedia)
2019/11/07首里城をデジタルで復元へ「写真提供してほしい」(NHK)
2019/11/08首里城3Dモデル再現へ みんなの写真求めます 東大研究者ら(The Mainichi)
2019/11/08首里城、写真100万枚でデジタル復元 東大講師ら3D化プロジェクト「再建までの観光資源に」(Okinawa Times)
2019/11/10日本大學首里城重建計畫 向全球募集「100萬」張照片(華視新聞)
2019/11/12デジタルで首里城復元 「再建までの観光資源に」 東大講師ら 写真募り3D化(Rykyu Shimpo)
2019/11/14首里城、3Dで復元…研究者ら思い出の写真100万枚募る(The Yomiuri)
2019/11/15首里城火災 広がる支援の輪(NHK Okinawa 沖縄金曜クルーズ きんくる)
2019/11/15「みんなの首里城をもう一度」首里城デジタル復元プロジェクト発起人の思いに触れる(東大新聞オンライン)
2019/11/28首里城、電気系統から出火か(日経アーキテクチュア)
2019/11/29再建へ 火災からまもなく一か月 首里城蘇らせる取り組み(Nippon TV news every.)
2019/11/29画像集めて首里城3D復元へ 目標100万枚、提供募る(The Asahi)
2019/11/30VRやアーカイブ 首里城再建、デジタル技術を生かせ(The Nikkei)
2019/12/01あなたの写真で首里城がデジタル復元される!?プロジェクト発起人の東大講師に話を聞いてみた。(UmeeT)
2019/12/12「日本のゲーム大好き」世界遺産を多数復元…首里城3Dモデル化に協力する海外専門家たち(The Mainichi)
2019/12/13首里城の3D復元プロジェクトを生んだ小さな奇跡の連なり(ASCII.jp)
2019/12/25Shuri Castle to rise from ashes in virtual resurrection(Kyodo News)
2019/12/25全焼した首里城正殿、3Dモデルでよみがえる 東大の研究者ら、ネットで集めた写真・動画から制作(ITmedia)
2019/12/26首里城正殿 在りし日の姿 思い出の写真などで3D復元(NHK)
2019/12/27首里城デジタル3Dで“復元”(めざましテレビ)
2019/12/29Okinawa’s Shuri Castle to rise from ashes in virtual resurrection (Japan Times)
2019/12/31Shuri Castle: Appeal for images to ‘rebuild’ burned site online (BBC News)
2020/1/31 Reconstructing Shuri Castle, One Picture at a Time (NHK World)
2020/2/1OUR Shurijo: Shuri Castle Digital Reconstruction Project (in Japanese, IPSJ Magaine “Information Processing,” Feb. 2020)
2020/2/5興南中高生がVRを活用し首里城ガイド(RBC)
2020/2/28The Path to Rebuilding Okinawa’s Iconic Castle (NHK World)
2020/3/6Our Shurijo project (Reality Capture)