Overview and Thesis Topic
While the first year typically has a focus on exploration and finding a research topic, my thesis topic is already as firm as the fate of a codfish lying in the freezer of a fishing boat, it is only a matter of how to cook it. As such, the exploration I had are mostly in adjacent fields related to my thesis.
The thesis is Implicit Optical System Variation (name here is for illustration purposes only and may not be an exact representation of the final thesis). Digital sensor and technology makes the use of vintage lenses popular, but as these lenses are no longer being produced, they will one day die out or becoming so prohibitive that only rich collectors could afford. The thesis aims to find a away to faithfully digitalize the vintage lenses so that images can still be created as if looked through the vintage lens.
More information about the thesis can be seen in the picture (research poster) below or in the Handbook.
Aside from historical preservation purposes, the target audience of the thesis product contains media production such as animation (either 2D or 3D) and post process special effects, which typically require an additional pass of editing to emulate the optical artifacts in order to blend with the live action footages. While not directly planned in this thesis, given the increasingly higher demand of photo-realistic rendering effect of video games, it is also possible to adapt the digitalized lens into video games.
The wide array of target user also means that most areas of motion picture production / animation / video games are, at least in some degree, related to the thesis. For this reason, the section below may seems to be overly broad for some.
More info about the project can be found in the handbook.
Procedural Process
I attempted to create a prototype of the thesis application using Python, I quickly realized 2 things:
Python is not fast enough.
The per-pixel convolution and the need to operate on at least FHD resolution demands a tremendous amount of computation. While Python is fairly decent in applications of artificial intelligence, I was not able to find developed packages for mass image convolution, especially not for moving images.A pure programming environment is hard to develop with.
Using only an programming IDE means that I have to run it for every tiny modifications and wait a while to see the result, which is both painstaking and counterproductive. Another way is needed for faster iteration and seeing the results update in real-time.
I concluded that a node-based development software will be most idea, similar to Nuke, Shadergraph in Unity, Graph Editor in Unreal, Hypershade in Maya, and Houdidni.
This prompted me to take Prof. Lewis’ Procedural Animation class, learning about animation and special effect using Houdini, but also trying to have a better understanding of how node-based logic work and when do they excel.
Rigid Body Collision
An interesting I learned in rigid body simulation is how the shattering is actually created. Instead of using a single area as shatter source, there are typically several layers of shatter sources, some shatters into small pieces while others become bigger ones, thus building the layer effect.
Fluid/particle effect
The Navier-Stokes equation provided a solid foundation of abstracting fluid into small individual particles, simulate each one and get a convincing fluid effect. One can say that’s an instance of reductionism applied successfully.
Animation Without Key
I came to the sad conclusion that the same animation logic in Unity and Unreal does not work as well in Houdini, thus decided to switch to the more procedural way of animating. Creating data from randomized parameters and generate continuous functions, using them to animate the position and direction of the plane. At the same time, taking derivatives of the function, with which to animate the control surfaces.
Back to my own thesis, I opted to start trying to create a prototype of the application during Professor Kyoung Swearingen’s 6400 class, using C++ and Qt.
C++ is chosen for the speed, and Qt is used as it is a widely received GUI system for C++. The goal of this application is to be an development environment with which I can test the result of the digital lens simulation interactively and observe the result in real-time.
During the process I also realized that my knowledge of light field and wave front is not sufficient to build this application for now, so I have decided to not code the light field part during the 6400 class, instead focus mainly on the UI and thinking about the feature set. Currently, the software only has the preliminary UI components established and the node graph function built, no node or calculation has been coded, despite some attempts.
The finished software will ideally look like this:
I plan to talk to more faculties in the Department of Physics and the Department of Astronomy in the coming semester, hopefully get to know better about optical simulation before starting to code the implementation. This application will be one of the main topic in my second year.
Animation and Interactive Media
Like discussed in the overview, the digitalized lens from the thesis can be used in moving image productions and, by extension, possibly video games.
However, one of the core ideas of the thesis is that it aims at professional production, not amateur video editing. The key difference between professional and amateur in this context is pipeline and information. For example, special effects rendered in a professional pipeline typically came in several dozens of layers/channels, albedo, emission, motion vector, z-depth, ambient occlusion, etc., and then composited carefully together. Whereas amateurs may opt to simply use stock video elements. Professional pipeline offers more information for downstream to work with, which is paramount for this thesis to create optical artifacts with physical accuracy.
But the information in professional productions are, of course, not publicly available. To acquire similar level of information and detail, I decided to do it myself, walk through the pipeline and make corresponding assets for each step. This decision is made because:
I have the technical ability to produce most of the assets.
Walk through the process helps me understand where and how my thesis can be placed in the pipeline.
It generates the asset that will be needed to test my thesis.
Below are a myriad of projects during the past 2 semester involving moving images and interactive media.
Virtual Environment
Exhibition room for Professor Addy’s VR class.
Whether or not VR is within range of this thesis is not exactly clear, since all VR videos so far have almost an infinite depth of field and little to no optical artifacts. Since the lenses are used in moving image primary for subject separation and emotion, it could be interesting to see how it may play out in VR.
Hand Drawn Animation
Result of a tutorial for DESIGN 2330, a course I TA-ed for.
While still incomplete, it was worked on with proper layer separation in preparation for future export, editing, and compositing.
Virtual Environment
Japanese workspace built in 3D. This is part of a language learning program built in Unity using VR. Trying to be as faithful to the real Japanese office workspace to emulate the feeling one might have when first entering such place and work there.
Facial Recognition and Capture
From the same Japanese learning program above. In addition to the software part, this recognition and capture is also recorded for replay during the game. Additional voice recording is also needed for syncing the capture with sound (due to animation key is captured on its own), this thus almost becomes a motion picture production project with the amount of ADR.
I have also been working on a video project with a fair amount of CGI elements in it. Unfortunately, due to a series of events I have not been able to get the actors all together on set and finish the live action part. Without which the CGI also have been moving slowly.
In fairness, the scope of this project is far beyond a team of one, as the amount of modeling, sculpting, rigging, texturing, lighting, special effects and simulation is on par with a legit motion picture production sequence. There is a possibility that this sequence will take the next two years to fully finish and be used as another test subject of my thesis.
Optical Design
One cannot escape optical design and physics when the thesis topic involving optical system simulation.
During Professor Lewis’ 6400 class, I spent a project learning computer aided optical design using Zemax OpticStudio. A common software in optical design, roughly the equivalent of AutoCAD in CAD (in fact its UI almost look like AutoCAD).
However, as pointed out in the research poster, Zemax OpticStudio is an “explicit” optical system simulation, whereas my thesis requires the simulation to be implicit. This software will likely be used as a tool of validation, testing my result with its and see how accurate mine in compare is.
While I did consider taking the optics class offered by the department of Physics, its 3000 level definition and my rather specific demand makes me wonder if it is worthy of the time. I ended up spending a fair amount of time learning optical physics and optical design from some textbooks.
The sources include the Introduction to Aberrations in Optical Imaging Systems by José Sasián, The Cine Lens Manual by Jay Holben and Christopher Probst, and The Ultimate Guide to Lens Design Forms: The types of optical systems in a lens designer’s toolbox by Kats Ikeda. Most of which I am still reading, since optical design itself is worth of at least 4 years of undergraduate degree and often is a master or PhD by itself.
With more knowledge in optics and combined with my prior experience in photo/video, 15 aberrations are believed to be important for the thesis to model:
There are, of course, difficulties in modeling these. Diffraction, for example, cannot be replicated with geometric optics and is thus near impossible to simulate using light fields. This and other challenges will be the focus of my second year and possibly also part of the third year as I communicate more with the Physics and Astronomy department.
Misc
Hegel philosophy has left a deep mark on me, “Was vernünftig ist, das ist wirklich; und was wirklich ist, das ist vernünftig”, what is reasonable is real, and what is real is reasonable. A thing and everything came into being for a reason, a sequence of events prior whose causal relationship brought forth of it.
Who am I to accuse another people lack of dinning etiquette if he was raised in an insecure environment with food not even guaranteed? I would have been the same if raised in the same den and saw what he had seen. I am no more intelligent or civilized, all that I have been doing was denying another’s misfortune with my privilege.
In the same way, I am the biological sum of my past, not my nature, fate, or talent (or the lack there of). Should another being happened to share the exact same experience, grow up in the exact same environment, then he would be in the same place as I do now. But again, who is he at that point but me?
I encountered two profound minds in the past two semester.
Professor Mathew Coleman from the Department of Geography showed up in Professor Münch’s 6200 class, talking about his research. Among the things he talked about include some rides with the police, and one of the seemingly kind officers ended up being the person committing unreasonable and rather unjust violence days later. During Q&A, I asked if herd effect, the diminishing individuality and the sequential loss of accountability can be an explanation of the violence, to which Professor Coleman gave neither an positive nor an negative answer. Instead, Professor Coleman hinted that while environmental factors may be affecting the police officer’s behavior, it might not be as significant, the officer can be kind and violent at the same time, yet both are true to his character.
It took me days to fully understand but when I did, I felt shocked and embarrassed. For Professor Coleman was right, it was too naïve of me to assume a man having a immovable character and reject the complexity of people. I hate this species as much as I hope and know how it can be so much better, duality does not mean mutual-exclusiveness.
The second great mind I came into was during Professor Proulx’s 6300 class, in which students are asked to read several papers, one of which by Edgar Morin. In his paper From the Concept of System to the Paradigm of Complexity (translated by Sean Kelly), Edgar Morin demonstrated a true mastery of knowing, referencing examples from several different fields from quantum mechanics, physics, biology, sociology, etc., and all were done in a meaningful way that respects the source and remaining faithful, but also contribute to his arguments. This is the first ever paper that made me feel that the argument in the paper is not only supported by logic and evidences, but also the words and sentences of the paper itself, a piece of artwork that managed to be both representational and non-representational.
For me, one of the most astounding arguments in this paper is “Complexity cannot be simplified”. Its brilliance stems not from it being metaphorically symmetric, but how it expressed a feeling that I have long since holding but never able to verbalize. In many group projects I have proposed objections to others’ ideas not because they sound bad, but because I know from practice that the idea will not work at a certain stage of implementation. For me, ideas cannot exists in vacuum and keep existing without the interference of reality, but I have trouble telling this to others. Despite being written three decades ago, the paper from Edgar Morin materialized my vague feeling and offered it solid conceptual framework.
Social event has been something I dread, aside from being highly introverted, it was also because I believed that having less social connection will impact less people when I die early for sure or encounter/commit some accidents. But the past 2 semester had me quickly came into the realization that this will not work.
I live in a society, and regardless of how much I wish I am not, I am still a part of it. At times, even without statement, I am not myself but an representative of another person, organization, community. I am no longer the sole accountant of my words and actions, there are others who may bear blame of my mistake. The choices I make, even if it appears to be entirely personal and inconsequential to me, have meanings to other people, some not even related to the event. To properly address the complexity of this invisible net, I become everything but free.
Trying to live as a human has resulted in nothing but pain for me in the past, so I tried to objectify myself, measure my actions and guide it with nothing but net utility. This worked really well as I have been relatively free from the disappointment and disgusts that came with human interaction. But either because of the graduate standing, or the fact that this degree leans closer to humanity compare to my former major, it seems that I have to start being human again.