A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures

Neural Computation, Journal Year: 2019, Volume and Issue: 31(7), P. 1235 - 1270

Published: May 22, 2019

Recurrent neural networks (RNNs) have been widely adopted in research areas concerned with sequential data, such as text, audio, and video. However, RNNs consisting of sigma cells or tanh are unable to learn the relevant information input data when gap is large. By introducing gate functions into cell structure, long short-term memory (LSTM) could handle problem long-term dependencies well. Since its introduction, almost all exciting results based on achieved by LSTM. The LSTM has become focus deep learning. We review variants explore learning capacity cell. Furthermore, divided two broad categories: LSTM-dominated integrated networks. In addition, their various applications discussed. Finally, future directions presented for

Language: Английский

---

Continual lifelong learning with neural networks: A review

Neural Networks, Journal Year: 2019, Volume and Issue: 113, P. 54 - 71

Published: Feb. 10, 2019

Humans and animals have the ability to continually acquire, fine-tune, transfer knowledge skills throughout their lifespan. This ability, referred as lifelong learning, is mediated by a rich set of neurocognitive mechanisms that together contribute development specialization our sensorimotor well long-term memory consolidation retrieval. Consequently, learning capabilities are crucial for computational systems autonomous agents interacting in real world processing continuous streams information. However, remains long-standing challenge machine neural network models since continual acquisition incrementally available information from non-stationary data distributions generally leads catastrophic forgetting or interference. limitation represents major drawback state-of-the-art deep typically learn representations stationary batches training data, thus without accounting situations which becomes over time. In this review, we critically summarize main challenges linked artificial compare existing approaches alleviate, different extents, forgetting. Although significant advances been made domain-specific with networks, extensive research efforts required robust on robots. We discuss well-established emerging motivated factors biological such structural plasticity, replay, curriculum intrinsic motivation, multisensory integration.

Language: Английский

---

Deep learning with coherent nanophotonic circuits

Nature Photonics, Journal Year: 2017, Volume and Issue: 11(7), P. 441 - 446

Published: June 12, 2017

Language: Английский

---

Relational inductive biases, deep learning, and graph networks

arXiv (Cornell University), Journal Year: 2018, Volume and Issue: unknown

Published: Jan. 1, 2018

Artificial intelligence (AI) has undergone a renaissance recently, making major progress in key domains such as vision, language, control, and decision-making. This been due, part, to cheap data compute resources, which have fit the natural strengths of deep learning. However, many defining characteristics human intelligence, developed under much different pressures, remain out reach for current approaches. In particular, generalizing beyond one's experiences--a hallmark from infancy--remains formidable challenge modern AI. The following is part position paper, review, unification. We argue that combinatorial generalization must be top priority AI achieve human-like abilities, structured representations computations are realizing this objective. Just biology uses nature nurture cooperatively, we reject false choice between "hand-engineering" "end-to-end" learning, instead advocate an approach benefits their complementary strengths. explore how using relational inductive biases within learning architectures can facilitate about entities, relations, rules composing them. present new building block toolkit with strong bias--the graph network--which generalizes extends various approaches neural networks operate on graphs, provides straightforward interface manipulating knowledge producing behaviors. discuss support reasoning generalization, laying foundation more sophisticated, interpretable, flexible patterns reasoning. As companion released open-source software library networks, demonstrations use them practice.

Language: Английский

---

Building machines that learn and think like people

Behavioral and Brain Sciences, Journal Year: 2016, Volume and Issue: 40

Published: Nov. 24, 2016

Recent progress in artificial intelligence has renewed interest building systems that learn and think like people. Many advances have come from using deep neural networks trained end-to-end tasks such as object recognition, video games, board achieving performance equals or even beats of humans some respects. Despite their biological inspiration achievements, these differ human crucial ways. We review cognitive science suggesting truly human-like learning thinking machines will to reach beyond current engineering trends both what they how it. Specifically, we argue should (1) build causal models the world support explanation understanding, rather than merely solving pattern recognition problems; (2) ground intuitive theories physics psychology enrich knowledge is learned; (3) harness compositionality learning-to-learn rapidly acquire generalize new situations. suggest concrete challenges promising routes toward goals can combine strengths recent network with more structured models.

Language: Английский

---

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Published: July 1, 2017

When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress discover short-comings. Existing benchmarks for question answering help, but have strong biases models exploit correctly without reasoning. They also conflate multiple sources of error, making it hard pinpoint model weaknesses. We present a dataset range reasoning abilities. It contains minimal has detailed annotations describing the kind each requires. use this variety modern systems, providing novel insights into their abilities limitations.

Language: Английский

---

On the Opportunities and Risks of Foundation Models

arXiv (Cornell University), Journal Year: 2021, Volume and Issue: unknown

Published: Jan. 1, 2021

AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and adaptable to wide range downstream tasks. We call these foundation underscore their critically central yet incomplete character. This report provides thorough account opportunities risks models, ranging from capabilities language, vision, robotics, reasoning, human interaction) technical principles(e.g., model architectures, training procedures, data, systems, security, evaluation, theory) applications law, healthcare, education) societal impact inequity, misuse, economic environmental impact, legal ethical considerations). Though based standard deep learning transfer learning, results in new emergent capabilities,and effectiveness across so many tasks incentivizes homogenization. Homogenization powerful leverage but demands caution, as defects inherited by all adapted downstream. Despite impending widespread deployment we currently lack clear understanding how they work, when fail, what even capable due properties. To tackle questions, believe much critical research will require interdisciplinary collaboration commensurate fundamentally sociotechnical nature.

Language: Английский

---

Deep Learning in Mobile and Wireless Networking: A Survey

IEEE Communications Surveys & Tutorials, Journal Year: 2019, Volume and Issue: 21(3), P. 2224 - 2287

Published: Jan. 1, 2019

The rapid uptake of mobile devices and the rising popularity applications services pose unprecedented demands on wireless networking infrastructure. Upcoming 5G systems are evolving to support exploding traffic volumes, real-time extraction fine-grained analytics, agile management network resources, so as maximize user experience. Fulfilling these tasks is challenging, environments increasingly complex, heterogeneous, evolving. One potential solution resort advanced machine learning techniques, in order help manage rise data volumes algorithm-driven applications. recent success deep underpins new powerful tools that tackle problems this space. In paper, we bridge gap between research, by presenting a comprehensive survey crossovers two areas. We first briefly introduce essential background state-of-the-art techniques with networking. then discuss several platforms facilitate efficient deployment onto systems. Subsequently, provide an encyclopedic review research based learning, which categorize different domains. Drawing from our experience, how tailor environments. complete pinpointing current challenges open future directions for research.

Language: Английский

---

Memory devices and applications for in-memory computing

Nature Nanotechnology, Journal Year: 2020, Volume and Issue: 15(7), P. 529 - 544

Published: March 30, 2020

Language: Английский

---

The rise of deep learning in drug discovery

Drug Discovery Today, Journal Year: 2018, Volume and Issue: 23(6), P. 1241 - 1250

Published: Jan. 31, 2018

Over the past decade, deep learning has achieved remarkable success in various artificial intelligence research areas. Evolved from previous on neural networks, this technology shown superior performance to other machine algorithms areas such as image and voice recognition, natural language processing, among others. The first wave of applications pharmaceutical emerged recent years, its utility gone beyond bioactivity predictions promise addressing diverse problems drug discovery. Examples will be discussed covering prediction, de novo molecular design, synthesis prediction biological analysis.

Language: Английский

---