I'm a Ph.D. student at **Brown University**,
advised by Stefanie Tellex in the H2R Lab.
I graduated from the **University of Washington**
in June 2018, with both M.S. and B.S. in Computer Science, and minor in Mathematics.
At UW, advised by Andrzej Pronobis and Rajesh Rao,
I conducted research on mobile robot navigation, as well as proposing new methods to use Sum-Product Networks for
graph modeling and large-scale semantic mapping.
Prior to that, I worked on using Myo to control a robotic arm
advised by Vikash Kumar
and Emo Todorov. (**CV**)

My Chinese name is 郑开宇 (simplified), or 鄭開宇 (traditional).

## Preprints

New!
**From Pixels to Buildings: End-to-end Probabilistic Deep Networks for Large-scale Semantic Mapping**
Kaiyu Zheng, Andrzej Pronobis

[

pdf]
[

show abstract]
[

hide video]

We introduce TopoNets, end-to-end probabilistic deep networks for modeling
semantic maps with structure reflecting the topology of large-scale
environments. TopoNets build unified deep networks spanning multiple levels of
abstraction and spatial scales, from pixels representing geometry of local
places to high-level descriptions representing semantics of buildings. To this
end, TopoNets leverage complex spatial relations expressed in terms of
arbitrary, dynamic graphs. We demonstrate how TopoNets can be used to perform
end-to-end semantic mapping from partial sensory observations and noisy
topological relations discovered by a robot exploring large-scale office
spaces. We further illustrate the benefits of the probabilistic representation
by generating semantic descriptions augmented with valuable uncertainty
information and utilizing likelihoods of complete semantic maps to detect
novel and incongruent environment configurations.

## Publications

New!
**Learning Graph-Structured Sum-Product Networks for Probabilistic Semantic Maps**
Kaiyu Zheng, Andrzej Pronobis, Rajesh P. N. Rao

*AAAI Conference on Artificial Intelligence (AAAI 2018).* **oral presentation.**
[

pdf]
[

bibtex]
[

slides]
[

project]
[

show abstract]
[

show video]

We introduce Graph-Structured Sum-Product Networks
(GraphSPNs), a probabilistic approach to structured prediction
for problems where dependencies between latent variables
are expressed in terms of arbitrary, dynamic graphs.
While many approaches to structured prediction place strict
constraints on the interactions between inferred variables,
many real-world problems can be only characterized using
complex graph structures of varying size, often contaminated
with noise when obtained from real data. Here, we focus on
one such problem in the domain of robotics. We demonstrate
how GraphSPNs can be used to bolster inference about semantic,
conceptual place descriptions using noisy topological
relations discovered by a robot exploring large-scale office
spaces. Through experiments, we show that GraphSPNs consistently
outperform the traditional approach based on undirected
graphical models, successfully disambiguating information
in global semantic maps built from uncertain, noisy
local evidence. We further exploit the probabilistic nature of
the model to infer marginal distributions over semantic descriptions
of as yet unexplored places and detect spatial environment
configurations that are novel and incongruent with
the known evidence.

**Learning Large-Scale Topological Maps Using Sum-Product Networks**
Kaiyu Zheng

*Senior Thesis*, 2017
[

pdf]
[

bibtex]
[

show abstract]

In order to perform complex actions in human environments, an autonomous robot needs the ability
to understand the environment, that is, to gather and maintain spatial knowledge. Topological map
is commonly used for representing large scale, global maps such as floor plans. Although much work
has been done in topological map extraction, we have found little previous work on the problem
of learning the topological map using a probabilistic model. Learning a topological map means
learning the structure of the large-scale space and dependency between places, for example, how
the evidence of a group of places influence the attributes of other places. This is an important step
towards planning complex actions in the environment. In this thesis, we consider the problem of
using probabilistic deep learning model to learn the topological map, which is essentially a sparse
undirected graph where nodes represent places annotated with their semantic attributes (e.g. place
category). We propose to use a novel probabilistic deep model, Sum-Product Networks (SPNs) [20],
due to their unique properties. We present two methods for learning topological maps using SPNs:
the place grid method and the template-based method. We contribute an algorithm that builds SPNs
for graphs using template models. Our experiments evaluate the ability of our models to enable
robots to infer semantic attributes and detect maps with novel semantic attribute arrangements.
Our results demonstrate their understanding of the topological map structure and spatial relations
between places

## Presentations

New!
**An Introduction to Semantic Mapping in Robotics**
together with Eric Rosen

*Brown Robotics*
— on 3/22/2019 in Providence, RI, US.
[

slides]
[

show abstract]

In order for robots to perform complex tasks, they
require a representation of the world that enables task
execution. Important aspects of the world model that are
often studied are mapping and localization, place
classification, and object detection, which can be used to
perform spatial inference and follow navigational
commands. Acquiring these world characteristics is
difficult for an autonomous agent. Semantic mapping is one
solution to this problem; Semantic maps tie together
geometric representations of the world (metric map) with
semantic representations of the world (topological
maps). In this presentation, we describe the motivation
for why these world characteristics are useful for
completing complex tasks and review current
state-of-the-art literature for autonomously generating
these representations. Lastly, we will discuss our current
research on semantic maps, such as TopoNets and
action-oriented semantic maps.

**Probabilistic Semantic Mapping Using Graph-Structured Sum-Product Networks**
Kaiyu Zheng, Andrzej Pronobis, Rajesh P. N. Rao

*UW Computer Science & Engineering Industry Affiliates Research Day*
— on 11/15/2017 in Seattle, WA, US.
[

poster]
[

slides]
[

show abstract]

Graph-structured data appears in wide range of domains, from social network analysis, to computer vision and robotics. In this
talk, we introduce Graph-Structured Sum-Product Networks (GraphSPNs), a probabilistic approach to modeling structured data,
where dependencies between latent variables are expressed in terms of arbitrary, dynamic graphs. While many approaches to
structured prediction place strict constraints on the interactions between inferred variables (e.g. grids and sequences), many
real-world problems can only be characterized using complex graph structures of varying size, often contaminated with noise when
obtained from real data. Here, we focus on one such problem in the domain of robotics. We demonstrate how GraphSPNs can be used
to bolster inference about semantic, conceptual place descriptions using noisy topological relations discovered by a robot
exploring large-scale office spaces. Through experiments, we show that GraphSPNs consistently outperform the traditional
approach based on undirected graphical models, successfully disambiguating information in global semantic maps built from
uncertain, noisy local evidence. We further exploit the probabilistic nature of the model to infer marginal distributions over
semantic descriptions of as yet unexplored places and detect spatial environment configurations that are novel and incongruent
with the known evidence.

## More about Me

Email: kaiyu_zheng [at] brown [dot] edu

Github: zkytony

Dumping ground for more projects, writings, interests.