Do you know what we do?

A little bit. They briefed me, but it is OK if you just...

Basically, we both use technology and digital to make culture more accessible, but in a live way. We don’t use digital technology to be on the Internet or in virtual things, but to augment real places, and to encourage people to be engaged.

What I do at Panthea is the following: Panthea is a company specializing in surtitling, which means that we translate the shows, the live performing arts in several languages.

In real time?

Yes, using multiple reality glasses. The spectators wear the glasses. We use Epson technology. Exactly.

I see that.

The concept is to have the text while you watch the show and you can understand it.

We started with this technology three years ago at the Avignon Festival. That was a co-production with the National Theater of Taipei, the National Theater and Concert Hall. We had the performance of King Lear by Shakespeare in French with supertitles in French, English, and Mandarin. Then the show went to perform here.

Carl, have you worked with like automated translation services?

No.

No, not yet? You’re not currently streaming like from Google Translate?

No, because this is not...

It has to be pre-scripted?

Yes, it doesn’t work for theater because the quality of the translation is not sufficient, and there is a latency time that is not acceptable for theater. It’s even more difficult than translating a book. If the computer is translating it, it’s even more difficult because it’s happening live, so you need the translation at the moment where people are speaking.

For example, in German, the verb is at the end of the sentence so you cannot know beforehand what will be said.

This is my domain, [laughs] so I know that.

What we are using is augmented reality to ease the service for the end user. What the end user needs is to have the text in the right position at the right moment. Everything that technology can do, we use, but what technology cannot do, we do manually.

It doesn’t matter in the sense we are not looking at simplifying things. We are looking at adding a new service, and people can pay for that.

Full accessibility.

Yes.

Is it AR in the sense that if I look away the text disappear, or does the text stay with me?

We can parameter that. At the moment, we don’t see a lot of benefit of having interactivity. It’s good that the text is moving with your eyes, actually. It’s better.

Of course, because people are usually watching this way only anyway.

Not always. Sometimes, things will happen. There will be a blast in the back but the actor continues to speak. You are looking back, and you want to understand what he is saying.

Ah, OK.

In the opera, several people will sing at the same time from different places. You cannot look at both at the same time so you need the text to follow you. The technology allows it if you want.

To have an immersive theater.

It’s also possible to position it. That’s definitely possible. It’s the very beginning. As far as we know, there is no real application of smartglasses yet on the market for the public domain. There are some niche applications for surgery, for maintenance in hostile environments.

Elevator maintenance is the classic example.

Even that, I’m not so sure about the benefits. Super-imposing reality is a help, but I know that when you interview maintenance people they say it’s good enough if they get the information in an earplug. There is a camera so a very skilled technician can analyze the situation, but then he will give his recommendations on the phone not super-imposing it on the screen.

Super precise, "You have to touch here," it very rarely happens actually.

Yes, exactly. I’m not sure because of course technology evolves. It’s exciting on principle, but what I see in theater and the opera is that people need the text, and they have to move their head to find it, and they cannot have all languages at the same time.

It’s very useful, and performing arts is a growing sector with more and more needs. For example, when I was here two years ago, we had a discussion with the National Theater. It’s a leading place for the identity of Taiwan and the shining of its culture, so making it accessible to more people is very useful. You cannot expect everyone to speak Mandarin before they come.

No, not at all. How good is the resolution of this technology? Is it 300 pixels wide? How many pixels?

The resolution is 921,600 HD (horizontal 1280 x vertical 720 dots) x 3.

This technology is now the one that we use, and the National Theater in London has purchased 50 of them to put for all people that are hearing impaired to get the translation.

It’s good enough at least for the hearing impaired people.

It’s good. We have...

Maybe not sign language though. I mean...

We could have it in the future. For the moment we don’t, because sign language takes a lot of space in the field of vision. It’s complicated, but we are going to test probably next season with a theater in Gothenburg.

I ask, because early next year sign language will be an official language in Taiwan. It will be given the same status as other written languages. Of course, as you said, it’s not always possible to give room to a sign language, but even in a simplified form, it’s very much...

It’s definitely possible. Our software Spectitular(r) manages text and sound and image, but having video is not a problem. It’s all cloud-based, so it’s collaborative.

There’s also people that prepare the translations who can do it... If the specialist of sign language is based in Taipei, and it’s happening in Taichung, you could have someone who is doing it in an online platform, and it’s automatically available.

This greatly interests me, because it will be a real need, and need will be by law.

The big hurdles we see for the adoption of this is that it’s very new, because it’s one of the first applications in the world for smartglasses that is really useful. It’s a pioneer thing, so we need to find customers that want to take that risk to be the first. It’s a big hurdle at the moment.

The second hurdle is the financing of it. We are talking to public institutions whose mission is to promote culture, so it ends to be accessible to as many people as possible, but a lot of them have budget constraints, so they are trying to go around the law. They say, "We do the minimum possible," but they are not very motivated to do it.

What about the other end, like social enterprise and charities for deaf people?

We are talking to them. In the UK, it’s the first example that there is. It’s not happening with us, but we are in close discussion with them, and there is a sponsor, I think standard sponsor. Accenture is the company who sponsored the content, the programing, and then there is a body for art and cultural accessibility that has financed the National Theater for a certain number of shows per year.

For the moment, I did not see a theater or an opera that was motivated enough on their own to decide to finance it. We do some tests. We did one at the Opera Garnier this year, the French National Opera, and they are interested, but they want to find financial resources for that.

I’m an entrepreneur, I’m not a public financing person, but it’s happening now. What we see is that people get extremely excited and not the geek people. Not people like you and me, but the older people. People that cannot hear anymore.

This summer at the opera we had... The most amazing thing was an old couple, probably 80 years old. They could hardly walk. They used these super futurist glasses.

They were incredibly excited because they said, "We could read it because it’s written... It’s, it’s high resolution, and it’s, uh, we can choose the brightness. And so, it works," which is, for us, the promise of a use case that really matches with a need and with people that have money to pay for the tickets as well. There is a great social benefit to it.

This is what we are at. We do many other things but this is our most advanced technology. That’s what we’re most excited about.

Great. I see Timescope has a Oculus Go as well.

I have one at home.

I’m sure.

Probably in the office, I need better equipments.

I see you have a Vive. You have a Gear.

[laughs] Yes.

I have no doubt you know about VR.

What do you do?

Before explaining exactly what I do, I often start by a question. If you had the very cool power to step back in time, which place, which time would you love to see? What would it be for you?

The beginning of the universe, of course. [laughs]

That’s the first time I have this answer. [laughs]

Because then I can know whether there are multiverses, like why was the universe created? Just a great physics question.

You’re interested in the why...

[laughs]

That’s a very scientific dream. Most people in France say the Revolution, the Middle Age. That’s very interesting.

We actually pretty much do this. My company is called Timescope. We develop time travel experiences thanks to virtual reality. We do two things actually.

We invented a hardware, which is called the Timescope machine, that you see here. It’s a VR terminal that works as a self-service system. You arrive in front of the machine. You are let’s say in a time now in a historical place. You say, "I want to go back to that time." You pick a content. You adjust the headset to your height. Then you can start a 360-degree experience and go back in time.

This is very smart because then you avoid the translation problem. You only have rotation. People will feel much better.

Exactly. You don’t have to put the headset. You don’t have to wear it.

Just discrete.

You do the 360.

You don’t get so sick.

That’s right.

No translation.

This is one of the design that will avoid the VR sick. Congrats.

Because you are holding the machine. Thank you.

We do the hardware. We also do the content. That’s our specific thing that we do. We invented the Timescope machine that is now installed into 30 places in France, Paris, Le Havre, Arras, north of France, soon in the south of France.

We also produce 3D content which is actually three kinds of content, content which are historical reconstructions where we do the whole production process from the generalizers of the historical sources to the 3D content at the end of...

You have AI to hallucinate color and details?

We use mainly CGI.

Ah, it’s not really photogrammetry from old pictures. It’s more inspired by old pictures.

Exactly.

[laughs]

It would be great if we could technically use...

Because there’s a AI lab in Taiwan that takes very old or bad resolution, like 2040p, 240p. It use AI to hallucinogen 4K or 8K...

Whoa, that’s great.

...by imagining details basically. [laughs]

That’s interesting. [laughs]

You’re mostly CGI.

We gather all the historical sources. Then we model them. We are the specific touch about the CG that we use, that we produce, is that is very photorealistic and story-tells.

We also work on urban projects to show the future of locations, that can have past or that cannot be very rich in terms of past, on urban projects that need to be explained to the people. The Timescope is also a way to explain to the people how it will be.

When we were talking about fake news, it’s very different. But it’s also a way to avoid rumors and to speak in a very powerful way to the people. You know that VR is a medium which is very, very strong.

This is also you or is this somebody else?

No. I know I have to check this out because I know that I saw that there is a Timescope that...

This is SomaSoft. This is not you.

No, this is not me.

Very confusing branding. [laughs]

I’m the one owning the Timescope brand. [laughs]

Maybe you can do something about it. [laughs]

Definitely.

You don’t have an Oculus version or you do, but it’s not on the store?

Oh, yeah.

Is it not on the market of Oculus?

This is not on the Oculus market. It’s on the terminals that we have. We don’t put it online because usually, the experiences are made to be tried on-site directly.

I know. You will have to file a trademark [laughs] case if you are going to put them.

I know I need to go for them.

Maybe not now.

Exactly. Would you like to try?

Of course.

Let me see if I have battery left in my headset.

Meanwhile, have you worked with VR before or it’s all AR?

It’s all AR. We don’t do VR because we take people to performing arts. We don’t use VR at all.

Because the natural extension will be people watching at home the theater that’s streaming into the devices.

I had this very, very interesting meeting at YouTube in New York City three years ago about this. We talked to the YouTube VR guru. I don’t remember her name. She’s very well-known in this field.

She was paid by Google to experiment shooting in virtual reality. I seen the results. She said, "When I filmed the concert and I showed it to people, they didn’t...the experience was pretty poorly rated because they compared it to the experience in the concert hall. And they said, ’Yeah, it’s not as good as it. It’s kind of faking the experience.’"

Then she said, "The most rewarding result we had was I was filming stones on the beach..."

Natural tracks.

"...because you could access to details that you would not normally see." A bit like in a game where you can go, and then you explore something from the close...She said this is when we measure the satisfaction of the end users. This is where it’s most exciting.

That they probably don’t need subtitles, or just don’t...

No. This is to explain that allowing people through VR or AR to experience something at home has to be proven yet to be a benefit.

So far, we’ve found maybe the planetarium really worked, like seeing how the solar system evolves...

It’s something you cannot see better in a different way, right?

Exactly, yes.

It’s unique value proposition. I guess that’s the key thing. It’s something you cannot do differently. She’s the connection.

I’m not a designer of how it works. I focus only on the use case. I think on Timescope as well. In fact, when there is a connection with something that justifies your presence, it’s a huge added value. Then you can leave something that nobody else can leave.

[French] Yes. You are ready for me? No?

Sorry, I have no battery any more. I put it all on during the whole day !

Ah, OK. Live demo, it’s always like that.

I’m going to demo you these videos so that you can see the kind of contents that we produce. To give you an idea of it. For example, that’s Paris, the river bank, 400 years ago.

That’s exactly what the city hall looked like. The Galeries Lafayette, if you’ve ever come to Paris, you might have seen the gallery.

Of course.

The beginning of the 20th century, that’s the Eiffel Tower, which is the equivalent, I would say, of the 101 Tower. From this point of view, we have four timescopes.

Oh, this is beautiful.

Whoa. [laughs]

[laughs]

We work on making it really realistic. Paris at the...

The World Exposition.

...the World Exposition, exactly. When the...

Rafael was first killed, yes.

...was read, yes. Then many other places in France. Sometimes it’s also very strong and very dramatic when we use World War I, when we reconstruct the trenches. This is for Kering, the luxury brand, where they are located.

That gives you an idea of the kind of content.

This is great. I want to show you the Taiwan Digital Asset Library, the TTAL. The idea that I got from a participatory budgeting experience I had in Paris. I think it was 2015, or something. I was staying in the 3ème, and there was a petition in the participatory budgeting that says let’s make a 3D version of everything.

I full voted for it, because I don’t have to be a citizen. I only have to provide a hotel address, so I voted for it. It didn’t get the budget.

Then I become Digital Minister. I can do whatever I want. Now we have a collaboration between the Ministry of Culture. I don’t know whether you’ve talked to their people, but they’re now very all-in in VR. They now use point cloud and videogrammetry, photogrammetry to scan all the historical buildings.

They open source the content, using Creative Commons. Anyone can not just walk into a old temple, but actually do a lot of investive experiences. Using old photos, they can very easily do alternate texturing...

That’s cool.

...and make an easier pipeline to the kind of timescope that you’re producing.

We’re offering this service and content for free, so every filmmaker around the world can use Taiwan as their backdrop. I think that this too have a lot of synergy.

It’s a easier timeline for us to produce, but you offer something unique that you can install on Taipei 101. [laughs]

Definitely. I think that Taipei 101 is really like the observatory. You’re pointing at 101 Tower with big nice view would be such a cool experience to see how Taipei has changed over...

The next logical question is how expensive is it.

It really depends on how...

Assuming we have the content. Assuming we produce the content.

Assuming that’s possible for you to produce the content. I know you have a 3D studios. I have met some of them.

Yeah?

That’s possible.

We recommend if that’s the case, to give insights to the production so that they can produce something which has the potential...

Of course, you could become like consultants or something.

In terms of budget, to give you an idea. For a Timescope machine, usually depending on the number of timescopes, depending on the engagement period, which means more rental, we are between 1,000€ to 10,000€ a month.

That sounds OK.

1 to 10 gives you an idea. If it’s more, our main contracts are on a long-term basis, four years usually, since it’s installed on a historical location. It’s designed to stay in outdoor spaces.

Of course.

Usually that’s where we stand.

Then I think this is a very good combination because there is already a lot of projects using ordinary Oculus that’s going on in Taiwan. Exactly as you said, if they compare it to the real thing, it’s disappointing.

Deceptive, yeah.

Then they take out the glass and find the reality to be much better.

If they choose the language themselves, and they hold a machine. It’s like riding in something. We know that it will be a prepared experience, and people will subjectively rate it as better.

We have 96 percent of people who rate the experience as "Wow" and "Cool."

That’s good, because it’s the same as our text filing system, ha-ha...

...which we also co-create with people. Anything that involve a action by the person themself is better.

Definitely.

If it’s all passive, it’s just like watching a movie, then it doesn’t work...

Completely.

...because it’s low resolution, bad movie. [laughs] An active component I think is very important.

It’s fascinating to learn about your work, and I think the Ministry of Culture, in particular, will have a lot to talk about.

I will be in the C-LAB. You’ve been to the Social Innovation Lab, right?

Yeah.

The app, and the VR space. You remember it’s part of a larger place?

Yes.

Yes.

The Larger Place is now converted into the C-LAB, the Taiwan Contemporary Culture Lab, which you did not see, because it is really new. It just opened a couple of months ago.

It’s really new, but it is really good that we have this place dedicated for contemporary culture experimentation because it doesn’t have to make economic sense. People can just commission any kind of event and it’s sponsored by the Minister of Culture.

For example, two days later, I’m going to participate in Sleep 79, which is a lot of people just spending a lot of prime time in the carts that are shaped like this. I’m going to read out Mozilla Common Voice for two hours from 9:00 to 11:00, for people to sleep while I read for the AI corpus.

Obviously, performative.

That’s cool. That’s interesting.

Things like that can be done in this huge open space that it doesn’t make any economic sense. But the Minister of Culture is very willing to sponsor this kind of work, so it’s perfect for piloting.

If you want to convince a institution to buy 5,000 copies, they will not think about. But if it is just 100 or 50 copies for an experimental pilot, then it’s well within their small procurement budget. I think this is a great help.

You think in our case, it would be the Minister of Culture who would be our direct plant, or it would be like a mix between you and the Ministry of Culture?

I think the Ministry of Culture has all the content. For you, this would be a higher approval rating delivery device for the project they’re already doing anyway.

I don’t think they will actually commission you to do a historical storytelling of Taiwan in the old indigenous places because they’re paying this amount for real, true, huge teams to do a meticulous creation that’s not just historically correct but culturally significant.

They’re making into like Netflix movies and things. All the derivative parts, they have that already. They will be interested, I think, in this particular vehicle of delivery. You can provide consultant because their movies are two hours or three hours. Obviously, nobody is going to stand there for three hours...

...no matter how good the content is. [laughs]

They will recommend to do two minutes.

Exactly. Which two minutes to choose? Which two minutes to show which angle, even though they do film in 360, but which initial angle to project?

I think that is your specialty is like a choreographer or a director. I will keep an eye on the possible events where the Minister of Culture showcase their historical overlays. I think that is the better combination.

Another possibility is starting next year, we will have what we call original revitalization project, where we encourage all the townships of about 50K to 100K people to find its historical identity and historical stories. The idea is that people if they care about their history in that township, even if they go to Taipei for college, they will go back to make their home community better.

If during basic education. You don’t give them something to do to associate with the local history. When they go to Taipei, they don’t come back anymore. [laughs] That is another possibility, which will not be the Ministry of Culture. It will be up to each municipal government.

Municipality, yeah.

They will have a lot of budget and care to curate this kind of history. In which case your other specialty, which is taking old photos and doing modeling. That becomes useful to them, because they don’t have the...

You want us to do modeling?

Yeah. They don’t have the ready specialty like the Ministry of Culture’s flagship projects do. They only have local universities to work with. That is one possibility.

I think the next step would be, I mean the closest one, would be to talk to the Ministry of Culture about our technologies...

Or just visit CLAB. You will see all the experiments going on.

[French]

We have a French tech night.

We have a French tech night tomorrow.

Tomorrow, I think.

I know.

It’s not international.

I don’t know, but...

I don’t know.

Tech for Good, is that the name or no...

I don’t remember.

No. I think it’s another thing.

It’s a different thing.

Then I don’t know.

The CLAB is now Ministry of Culture’s premier place to showcase emergent technology that interacts with culture.

In the case of Panthea, we have two contacts here that are in charge of self-acting for the national theater, with Cami, who we worked with them three years ago on the Avignon project.

If we were to imagine something here, we would have people to interact with to hire for this project for the content, and to collaborate with people with the organizations of hearing impairment. We would not start out of the blue from the distance, which is very good.

Another possibility is to start a small project, but enlist the help of one of our international startup hubs. There is one in Linkou, in New Taipei City, called the Startup Terrace. There’s a lot of companies working in that dimension as well.

The other way is to think outside of Taipei, because I’m sure you are connected with the Taipei theme anyway. There is another city called Kaohsiung. It’s in the south part of Taiwan. They are actually in charge of most of the 3D modeling and the other industries, even though they’re not very loud about it, but they’re actually very active in terms of AVR.

The Kaohsiung city government has a space dedicated to immersive experiences. It showcases a place they call the Kosmos, K-O-S-M-O-S, and they just started. I think they’re also be very willing to look into showcases similar to the natural theater one.

Do you know if there is something in Taichung?

In Taichung, there is of course the Taichung Opera House.

That’s why I’m asking. I’m wondering if there is also a hub for innovation and tech.

That is kind of independently operated, the Taichung Opera. If you do it in an accessibility angle, I can connect you with people from Taichung City that cares about the elders. We had our social enterprise submit in Taichung.

I know it is a priority for them to increase participation of the elderly and the otherwise disabled people so that they reintegrate into the society instead of being alone at home. From that angle, I know the person, but from the tech angle, I don’t really know.

Do you know the people from Tower 101?

Yes, of course.

If you could connect to us would be good.

The previous chair of 101, Christina Sung of SERT.tw, I think is perhaps a good contact, because, first, she knows the right people, but also that she really cares about social entrepreneurship. She manages one of our independent social innovation investment board.

She has also access to people who really want to make the experience better for the arts and the society, rather than just from a purely financial return perspective. She’s one of the leading impact investors, so I think it’s a easier angle, if she think it’s a good fit.

Thank you.

Usually you can take 10 days to edit this transcript until you can put on all the relevant URLs, the films, the YouTube that you want to show or whatever. As long as both of you are ready, even before the 10 days, you can tell me and I will publish online.

Once we publish this transcript, then the introductions can happen in a public way, because I don’t do exclusives, it will need to be in public knowledge.

If you introduce, it’s...

It’s a link to our transcript...

To our...?

...full text that you can click online. Everybody can see it. If you’re interested, here is this email that you can write to.

We need to create a specific URL?

No, you don’t have to do anything. I will do that.

I’m a radically transparent minister. The way we work is very simple. On my name card, there is a website, PDIS.tw. Once you go there, you will see the English version. Basically, this is everything that I did and everything that I had a conversation with.

For example, people from RedHat just visited. If you click "RedHat Visit," you will see exactly who said what, where, why. I did that for all the meetings, even internal meetings that I’m a chair of.

Did we say smart things? [laughs]

That’s why you’re given 10 days to edit, so you can edit to make yourself sound smarter. [laughs]

OK.

It’s true for like more than 3,000 people that I spoke with as a minister. This is also a good machine learning process to understand the language alignment and automated translation, and stuff.

Once our conversation gets published here, I will then send an email with the published link.

I get it. I understand.

If you are ready before 10 days, just let me know so we can publish it. Is that OK with you?

Yes, definitely. Thank you so much.

Thank you so much.

Thank you so much.