Cornell University researchers have created an interface that allows users to handwrite and sketch within computer code — a challenge to conventional coding, which typically relies on typing.
The pen-based interface, called Notate, lets users of computational, digital notebooks open drawing canvases and handwrite diagrams inside lines associated with traditional, digitized computer program code.
Powered by a deep learning model, the interface bridges handwritten plus textual programming contexts: notation in the particular handwritten diagram can reference textual code and vice versa. For instance, Notate recognizes handwritten programming symbols, like “n, ” and then links them up in order to their typewritten equivalents.
“A system like this would be great for data science, specifically with sketching plots and charts that then inter-operate along with textual program code, ” said Ian Arawjo, lead author of the paper and doctoral student in the field of information science. “Our work shows that the current infrastructure of programming is actually holding us back. People are ready for this type associated with feature, but developers of interfaces with regard to typing code need to take note of this and support images and graphical interfaces inside code. ”
Arawjo also said the work demonstrates a new path forward by introducing artificial intelligence-powered, pen-based coding at a time when drawing tablets are becoming more widely used.
“Tools want Notate are important because they open us up to new ways to think about what programming will be, and how different tools and representational practices can change that perspective, inch said Tapan Parikh, associate professor of information science plus paper co-author.