new icn messageflickr-free-ic3d pan white
View allAll Photos Tagged deep+learning

we are in a drought time , which is actually very nice , i prefer sunny days ( who doesn't )

we should be in our rainy season and yet we have sunny days

 

sadly we shall pay for this in the summer with major water restrictions

 

machinelearningmastery.com/inspirational-applications-dee...

With Peter Diamandis. Here is a bullet point summary of our fireside chat from Next Big Future:

 

Steve Jurvetson is deeply interested and excited by deep learning

 

It is the future of all engineering

 

Apply anywhere you have a lot of data and make sense of it

 

It is the biggest advance in engineering since the scientific method

 

Deep learning can be used update and optimize deep learning

  

Spacex and Falcon Heavy

 

Talk about the returning the rocket stages

 

The big thing is the heavy use of simulation to get a lot of things to work first time

 

Spacex has a lot of modularity.

 

For the 9 rockets on the Falcon 9, two can fail and the mission can still succeed

 

Methane is used for the the Raptor engine

Natural gas is the cheapest right now on earth

It will good for Earth and Mars

 

Next year land a one way Red Dragon trip

 

2024 launch and 2025 land on Mars

  

2.9 cents per kwh for solar for some sunny countries

 

Nuclear regulatory problems is why they will hold back on investing in nuclear but Steve thinks we (the world- civilization) needs new nuclear

 

Machine learning, autonomy and robotics

 

Biggest customer for Rethink robotics is China.

China has 100% employee turnover.

 

Spacex rockets will all be robots

Planes will be robots

Planes are flown by computer now except for taxiing on the runway

 

All vehicles will be electric

all power will be solar

all vehicles will be autonomous

 

Every new product that surprises you will have deep learning in it

 

Dwave's Geordie Rose has a new Quantum AI company Kindred

 

ANd <a href=Steve Jurvetson is deeply interested and excited by deep learning

 

It is the future of all engineering

 

Apply anywhere you have a lot of data and make sense of it

 

It is the biggest advance in engineering since the scientific method

 

Deep learning can be used update and optimize deep learning

  

Spacex and Falcon Heavy:

 

The big thing is the heavy use of simulation to get a lot of things to work first time

 

Spacex has a lot of modularity.

 

For the 9 engines on the Falcon 9, two can fail and the mission can still succeed

 

Methane is used for the the Raptor engine

Natural gas is the cheapest right now on earth

It will good for Earth and Mars

 

Next year land a one way Red Dragon trip

 

Humans? 2024 launch and 2025 land on Mars

  

2.9 cents per kwh for solar for some sunny countries

 

Nuclear regulatory problems is why they will hold back on investing in nuclear but Steve thinks we (the world- civilization) needs new nuclear

 

Machine learning, autonomy and robotics

 

Biggest customer for Rethink robotics is China.

China contract manufacturers have 100% annual employee turnover.

 

Spacex spacecraft are all robots

Planes will be robots

Planes are flown by computer now except for taxiing on the runway

 

All vehicles will be electric

all power will be solar

all vehicles will be autonomous

 

Every new product that surprises you will have deep learning in it

 

Dwave's Geordie Rose has a new Quantum AI company Kindred

 

And Singularity Hub wrote on the observation that Engineering Will Soon Be ‘More Parenting Than Programming’

Congratulations to Intel on their acquisition of Nervana. This photo is from the last board meeting at our offices; the Nervana founders — from right to left: Naveen Rao, Amir Khosrowshahi and Arjun Bansal — pondered where on the wall they may fall during M&A negotiations.

 

We are now free to share some of our perspectives on the company and its mission to accelerate the future with custom chips for deep learning.

 

I’ll share a recap of the Nervana story, from an investor’s perspective, and try to explain why machine learning is of fundamental importance to every business over time. In short, I think the application of iterative algorithms (e.g., machine learning, directed evolution, generative design) to build complex systems is the most powerful advance in engineering since the Scientific Method. Machine learning allows us to build software solutions that exceed human understanding, and shows us how AI can innervate every industry.

 

By crude analogy, Nervana is recapitulating the evolutionary history of the human brain within computing — moving from the logical constructs of the reptilian brain to the cortical constructs of the human brain, with massive arrays of distributed memory and iterative learning algorithms.

 

Not surprisingly, the founders integrated experiences in neuroscience, distributed computing, and networking — a delightful mélange for tackling cognitive computing. Ali Partovi, an advisor to Nervana, introduced us to the company.

 

We were impressed with the founding team and we had a prepared mind to share their enthusiasm for the future of deep learning. Part of that prepared mind dates back to 1989, when I started a PhD in EE focusing on how to accelerate neural networks by mapping them to parallel processing computers. Fast forward 25 years, and the nomenclature has shifted to machine learning and the deep learning subset, and I chose it as the top tech trend of 2013 at the Churchill Club VC debate (video). We were also seeing the powerful application of deep learning and directed evolution across our portfolio, from molecular design to image recognition to cancer research to autonomous driving.

 

All of these companies were deploying these simulated neural networks on traditional compute clusters. Some were realizing huge advantages by porting their code to GPUs; these specialized processors originally designed for rapid rendering of computer graphics have many more computational cores than a traditional CPU, a baby step toward a cortical architecture. I first saw them being used for cortical simulations in 2007. But by the time of Nervana’s founding in 2014, some (e.g., Microsoft’s and Google’s search teams) were exploring FPGA chips for their even finer-grained arrays of customizable logic blocks. Custom silicon that could scale beyond any of these approaches seemed like the natural next step. Here is a page from Nervana’s original business plan (Fig. 1 in comments below).

 

The march to specialized silicon, from CPU to GPU to FPGA to ASIC, had played out similarly for Bitcoin miners, with each step toward specialized silicon obsoleting the predecessors. When we spoke to Amazon, Google, Baidu, and Microsoft in our due diligence, we found a much broader application of deep learning within these companies than we could have imagined prior, from product positioning to supply chain management.

 

Machine learning is central to almost everything that Google does. And through that lens, their acquisition, and new product strategies make sense; they are not traditional product line extensions, but a process expansion of machine leaning (more on that later). They are not just playing games of Go for the fun of it. Recently, Google switched their core search algorithms to deep learning, and they used Deep Mind to cut data center cooling costs by a whopping 40%.

 

The advances in deep learning are domain independent. Google can hire and acquire talent and delight in their passionate pursuit of game playing or robotics. These efforts help Google build a better brain. The brain can learn many things. It is like a newborn human; it has the capacity to learn any of the languages of the world, but based on training exposure, it will only learn a few. Similarly, a synthetic neural network can learn many things.

 

Google can let the Brain team find cats on the Internet and play a great game of Go. The process advances they make in building a better brain (or in this case, a better learning machine) can then be turned to ad matching, a task that does not inspire the best and the brightest to come work for Google.

 

The domain independence of deep learning has profound implications on labor markets and business strategy. The locus of learning shifts from end products to the process of their creation. Artifact engineering becomes more like parenting than programming. But more on that later; back to the Nervana story.

 

Our investment thesis for the Series A revolved around some universal tenets: a great group of people pursuing a product vision unlike anything we had seen before. The semiconductor sector was not crowded with investor interest. AI was not yet on many venture firms’ sectors of interest. We also shared with the team that we could envision secondary benefits from discovering the customers. Learning about the cutting edge of deep learning applications and the startups exploring the frontiers of the unknown held a certain appeal for me. And sure enough, there were patterns in customer interest, from an early flurry in medical imaging of all kinds to a recent explosion of interest in the automotive sector after Tesla’s Autopilot feature went live. The auto industry collectively rushed to catch up.

 

Soon after we led the Series A on August 8, 2014, I found myself moderating a deep learning panel at Stanford with Nervana CEO Naveen Rao.

 

I opened with an introduction to deep learning and why it has exploded in the past four years (video primer). I ended with some common patterns in the power and inscrutability of artifacts built with iterative algorithms. We see this in biology, cellular automata, genetic programming, machine learning and neural networks.

 

There is no mathematical shortcut for the decomposition of a neural network or genetic program, no way to “reverse evolve” with the ease that we can reverse engineer the artifacts of purposeful design.

 

The beauty of compounding iterative algorithms — evolution, fractals, organic growth, art — derives from their irreducibility. (More from my Google Tech Talk and MIT Tech Review)

 

Year 1. 2015

Nervana adds remarkable engineering talent, a key strategy of the first mover. One of the engineers figures out how to rework the undocumented firmware of NVIDIA GPUs so that they run deep learning algorithms faster than off-the-shelf GPUs or anything else Facebook could find. Matt Ocko preempted the second venture round of the company, and he brought the collective learning of the Data Collective to the board.

 

Year 2. 2016 Happy 2nd Birthday Nervana!

The company is heads down on chip development. They share some technical details (flexpoint arithmetic optimized for matrix multiplies and 32GB of stacked 3D memory on chip) that gives them 55 trillion operations per second on their forthcoming chip, and multiple high-speed interconnects (as typically seen in the networking industry) for ganging a matrix of chips together into unprecedented compute fabrics. 10x made manifest. See Fig. 2 below.

 

And then Intel came knocking.

With the most advanced production fab in the world and a healthy desire to regain the mantle of leading the future of Moore’s Law, the combination was hard to resist. Intel vice president Jason Waxman told Recode that the shift to artificial intelligence could dwarf the move to cloud computing. “I firmly believe this is not only the next wave but something that will dwarf the last wave.” But we had to put on our wizard hats to negotiate with giants.

 

The deep learning and AI sector have heated up in labor markets to relatively unprecedented levels. Large companies are recently paying $6–10 million per engineer for talent acquisitions, and $4–5M per head for pre-product startups still in academia. For the Masters students in a certain Stanford lab, they averaged $500K/yr for their first job offer at graduation. We witnessed an academic turn down a million dollar signing bonus because they got a better offer.

 

Why so hot?

The deep learning techniques, while relatively easy to learn, are quite foreign to traditional engineering modalities. It takes a different mindset and a relaxation of the presumption of control. The practitioners are like magi, sequestered from the rest of a typical engineering process. The artifacts of their creation are isolated blocks of functionality defined by their interfaces. They are like blocks of magic handed to other parts of a traditional organization. (This carries over to the customers too; just about any product that you experience in the next five years that seems like magic will almost certainly be built by these algorithms).

 

And remember that these “brain builders” could join any industry. They can ply their trade in any domain. When we were building the deep learning team at Human Longevity Inc. (HLI), we hired the engineering lead from the Google’s Translate team. Franz Och pioneered Google’s better-than-human translation service not by studying linguistics, grammar, or even speaking the languages being translated. He focused on building the brain that could learn the job from countless documents already translated by humans (UN transcripts in particular). When he came to HLI, he cared about the mission, but knew nothing about cancer and the genome. The learning machines can find the complex patterns across the genome. In short, the deep learning expertise is fungible, and there are a burgeoning number of companies hiring and competing across industry lines.

 

And it is an ever-widening set of industries undergoing transformation, from automotive to agriculture, healthcare to financial services. We saw this explosion in the Nervana customer pipeline. And we see it across the DFJ portfolio, especially in our newer investments. Here are some examples:

 

• Learning chemistry and drug discovery: Here is a visualization of the search space of candidates for a treatment for Ebola; it generated the lead molecule for animal trials. Atomwise summarizes: “When we examine different neurons on the network we see something new: AtomNet has learned to recognize essential chemical groups like hydrogen bonding, aromaticity, and single-bonded carbons. Critically, no human ever taught AtomNet the building blocks of organic chemistry. AtomNet discovered them itself by studying vast quantities of target and ligand data. The patterns it independently observed are so foundational that medicinal chemists often think about them, and they are studied in academic courses. Put simply, AtomNet is teaching itself college chemistry.”

 

• Designing new microbial life for better materials: Zymergen uses machine learning to predict the combination of genetic modifications that will optimize product yield for their customers. They are amassing one of the largest data sets about microbial design and performance, which enables them to train machine learning algorithms that make search predictions with increasing precision. Genomatica had great success in pathway optimization using directed evolution, a physical variant of an iterative optimization algorithm.

 

• Discovery and change detection in satellite imagery: Planet and Mapbox. Planet is now producing so much imagery that humans can’t actually look at each picture it takes. Soon, they will image every meter of the Earth every day. From a few training examples, a convolutional neural net can find similar examples globally — like all new housing starts, all depleted reservoirs, all current deforestation, or car counts for all retail parking lots.

 

• Automated driving & robotics: Tesla, Zoox, SpaceX, Rethink Robotics, etc.

 

• Visual classification: From e-commerce to drones to security cameras and more. Imagen is using deep learning to radically improve medical image analysis, starting with radiology.

 

• Cybersecurity: When protecting endpoint computing & IOT devices from the most advanced cyberthreats, AI-powered Cylance is proving to be a far superior and adaptive approach versus older signature-based antivirus solutions.

 

• Financial risk assessment: Avant and Prosper use machine learning to improve credit verification and merge traditional and non-traditional data sources during the underwriting process.

 

• And now for something completely different: quantum computing. For a wormhole peek into the near future, our quantum computing company, D-Wave Systems, powered a 100,000,000x speedup in a demonstration benchmark for Google, a company that has used D-Wave quantum computers for over a decade now on machine learning applications.

 

So where will this take us?

Neural networks had their early success in speech recognition in the 90’s. In 2012, the deep learning variant dominated the ImageNet competitions, and visual processing can now be better done by machine than human in many domains (like pathology, radiology and other medical image classification tasks). DARPA has research programs to do better than a dog’s nose in olfaction.

 

We are starting the development of our artificial brains in the sensory cortex, much like an infant coming into the world. Even within these systems, like vision, the deep learning network starts with similar low level constructs (like edge-detection) as foundations for higher level constructs like facial forms, and ultimately, finding cats on the internet with self-taught learning.

 

But the artificial brains need not limit themselves to the human senses. With the internet of things, we are creating a sensory nervous system on the planet, with countless sensors and data collecting proliferating across the planet. All of this “big data” would be a big headache but for machine learning to find patterns in it all and make it actionable. So, not only are we transcending human intelligence with multitudes of dedicated intelligences, we are transcending our sensory perception.

 

And it need not stop there. It is precisely by these iterative algorithms that human intelligence arose from primitive antecedents. While biological evolution was slow, it provides an existence proof of the process, now vastly accelerated in the artificial domain. It shifts the debate from the realm of the possible to the likely timeline ahead.

 

Let me end with the closing chapter in Danny Hillis’ CS book The Pattern on the Stone: “We will not engineer an artificial intelligence; rather we will set up the right conditions under which an intelligence can emerge. The greatest achievement of our technology may well be creation of tools that allow us to go beyond engineering — that allow us to create more than we can understand.”

 

-----

Here is some early press:

Xconomy(most in-depth), MIT Tech Review, Re/Code, Forbes, WSJ, Fortune.

Touring the awesome Reddit HQ where my son is a summer intern programming and optimizing deep learning models for their Anti-Evil team. #ProudDaddy

Here it is on Soundcloud.

 

From interviewer Rainer Sternfeld: This is the 20th episode – as you know, every tenth episode we make is a special where I talk to someone who is of Estonian descent yet doesn’t speak the seemingly unintelligible language, or is a big friend of Estonia who is contributing to the success of Estonia.

 

We’re recording this on March 24 2017, and my guest today is an Estonian-American polymath, a world renowned venture capitalist, and the Estonia’s first e-resident outside Europe – Steve Jurvetson. In his day job, he invests in bold human endeavors in quantum computing, deep learning, electric cars, rockets, synthetic biology, genomics, robotics, and other areas.

 

In this podcast you’ll hear us cover a wide variety of brain-stimulating topics:

 

His technology-infused, Estonian-subtext upbringing in Arizona

How chip design and computing is undergoing a fundamental shift using biomimicry?

Why learning 9 programming languages is not as hard as 9 human languages, and what advice does he give to young people starting out in technology?

How does he think about the future of humanity in the light of accelerating rich-poor gap, automation, and why will robots be the slaves, not humans?

and his thoughts on why Estonia is competitive on the world stage.

Fasten your seatbelts!

 

Quotes

“If you didn’t understand evolution, and somebody explains it to you, you have to take your ego down a notch. You have to say: “Wait a minute. So humanity is not the endpoint of purposeful design? Wait – we’re just kind of an accident?!””

 

“I think we are currently in the middle of a major renaissance in how we do computation and how we actually think of engineering in general. I think it is shifting profoundly, almost as profoundly as when we first came up with the concept of the scientific method as a way to accumulate knowledge as a species over time. Something as profound is happening in the field of machine intelligence.”

 

“What fascinates me is that our humanity’s capacity to compute has compounded over 120 years and across multiple technology modes including mechanical devices etc. The main takeaway for me that is so powerful is there is I think a reflection here of a huge phenomenon, even bigger than computers themselves, which is humanity’s information reserve — our knowledge, our learning is compounding.”

 

“In terms of advice, first of all, I think that everyone should learn computer science. Do it young, do it early, do it often. Most importantly, I would encourage people, once they have had any taste of CS, to force themselves to play around with neural networks, whatever they will call it in the future. The core of it is neural networks patterned on the brain.”

 

“It sort of clicked for me that there are power laws in income (meaning it looks in and there are power laws in the number of companies that succeed in the information age businesses. As businesses succeed, they become information-centric and global, it tends to be winner takes all dynamic. Couple that with the notion that I strongly believe every business becomes an information business over time, just at different rates of speed. … The worries around AI should be centered on the concentration of power and I think OpenAI is spot on to say let’s look to Google, should one company be that powerful?”

 

from Memokraat

My most recent investment and board seat came out of stealth today. Mythic is taking a radically novel approach to building low-power, low-cost neural nets that can make any product more intelligent. Imagine any device having a voice interface like the Amazon Echo, but with local processing. It would be more responsive and secure with ~100x power and cost savings (at a comparable mfg process node). And no internet connection needed. Consider a Roomba or microwave oven. A rich voice interface could cost less than the clumsy buttons it replaces. Imagine a Dropcam with local intelligence to avoid the daily false alarms. Drones that could better track their targets.

 

Mythic implements the basic computation of machine intelligence — matrix multiply and add — by using a standard flash memory array and modifying the peripheral circuitry so that the memory cells store analog values for activation levels (a) for each neuron, and the weights (w) of each synapse. In the digital domain, multipliers and adders take many transistors, each consuming power and time.

 

Bottom line: Mythic can do an 8-bit multiply and add in a single transistor!

 

Computation and memory are unified, as in the brain. And flash is a non-volatile memory that consumes no power in standby mode.

 

Couple some local intelligence to each sensor and “The internet of things is becoming the sensory cortex of the planet” — TechCrunch

 

“'When you start to see these hints that we re-create evolutionary biology in a computer, you get the same basic building blocks, the same developmental milestones. It’s kind of spooky!' When discussing Mythic, Jurvetson compared the startup to Minecraft…” — VentureBeat

 

I was drawing a loose analogy to the redstone wire in Minecraft implementing a digital OR. Mythic uses a common memory readout wire for current accumulation. Addition in the analog domain is instantaneous, and just takes a wire. It needs none of the transistors or time consumed by a digital carry-adder. With just inverters and OR logic to build with, here is an 8-bit CPU that my son built back in grade school.

 

From the NVIDIA CEO: “We’ll soon see the power of computing increase way more than any other period. There will be trillions of products with tiny neural networks inside.”

 

The march to specialized silicon, from CPU to GPU to FPGA to ASIC is now going further, to analog and quantum processing. At a high level, we are recapitulating our evolutionary computational march in silicon, and an ever-growing percentage of our compute will be massively parallel and in-memory processing, just like our cortex. I wrote a blog post about when Intel acquired our deep learning chip company Nervana.

 

Now it’s time to get Mythic.

Note the pivotal AI moment in 2012 when a neural net run on GPUs crushed the ImageNet competition. Deep Learning and non-CPU architectures blossomed.

 

From the AI Index 2019: "Prior to 2012, AI results closely tracked Moore’s Law, with compute doubling every two years. Post-2012, compute has been doubling every 3.4 months (for a net increase of ~10,000,000x since then).

 

The y-axis of the chart shows the total amount of compute, in petaflop/s-days, used to train selected results. A petaflop-day (pf-day) consists of performing 10^15 neural net operations per second for one day, or a total of about 10^20 operations.

 

In the past year and a half, the time required to train a network on cloud infrastructure has fallen from about three hours in October 2017 to about 88 seconds in July, 2019. During the same period, the cost to train such a system has fallen similarly."

 

— p.65 of the AI Index from Stanford HAI (and the OpenAI graph of the recent red datapoints)

The Silicon Graphics head in my office was my muse.

 

I just finished reading a fascinating summary by Lin & Tegmark of the tie between the power of neural networks / deep learning and the peculiar physics of our universe. The mystery of why they work so well may be resolved by seeing the resonant homology across the information-accumulating substrate of our universe, from the base simplicity of our physics to the constrained nature of the evolved and grown artifacts all around us. The data in our natural world is the product of a hierarchy of iterative algorithms, and the computational simplification embedded within a deep learning network is also a hierarchy of iteration. Since neural networks are symbolic abstractions of how the human cortex works, perhaps it should not be a surprise that the brain has evolved structures that are computationally tuned to tease apart the complexity of our world.

 

Does anyone know about other explorations into these topics?

 

Here is a collection of interesting plain text points I extracted from the math in Lin & Tegmark’s article:

 

"The exceptional simplicity of physics-based functions hinges on properties such as symmetry, locality, compositionality and polynomial log-probability, and we explore how these properties translate into exceptionally simple neural networks approximating both natural phenomena such as images and abstract representations thereof such as drawings. We further argue that when the statistical process generating the data is of a certain hierarchical form prevalent in physics and machine-learning, a deep neural network can be more efficient than a shallow one. Various “no-flattening theorems” show when these efficient deep networks cannot be accurately approximated by shallow ones without efficiency loss."

 

This last point reminds me of something I wrote in 2006: "Stephen Wolfram’s theory of computational equivalence suggests that simple, formulaic shortcuts for understanding evolution (and neural networks) may never be discovered. We can only run the iterative algorithm forward to see the results, and the various computational steps cannot be skipped. Thus, if we evolve a complex system, it is a black box defined by its interfaces. We cannot easily apply our design intuition to the improvement of its inner workings. We can’t even partition its subsystems without a serious effort at reverse-engineering." — 2006 MIT Tech Review

 

Back to quotes from the paper:

Neural networks perform a combinatorial swindle, replacing exponentiation by multiplication: if there are say n = 106 inputs taking v = 256 values each, this swindle cuts the number of parameters from v^n to v×n times some constant factor. We will show that this success of this swindle depends fundamentally on physics: although neural networks only work well for an exponentially tiny fraction of all possible inputs, the laws of physics are such that the data sets we care about for machine learning (natural images, sounds, drawings, text, etc.) are also drawn from an exponentially tiny fraction of all imaginable data sets. Moreover, we will see that these two tiny subsets are remarkably similar, enabling deep learning to work well in practice.

 

Increasing the depth of a neural network can provide polynomial or exponential efficiency gains even though it adds nothing in terms of expressivity.

 

Both physics and machine learning tend to favor Hamiltonians that are polynomials — indeed, often ones that are sparse, symmetric and low-order.

 

1. Low polynomial order

For reasons that are still not fully understood, our universe can be accurately described by polynomial Hamiltonians of low order d. At a fundamental level, the Hamiltonian of the standard model of particle physics has d = 4. There are many approximations of this quartic Hamiltonian that are accurate in specific regimes, for example the Maxwell equations governing electromagnetism, the Navier-Stokes equations governing fluid dynamics, the Alv ́en equations governing magnetohydrodynamics and various Ising models governing magnetization — all of these approximations have Hamiltonians that are polynomials in the field variables, of degree d ranging from 2 to 4.

 

2. Locality

One of the deepest principles of physics is locality: that things directly affect only what is in their immediate vicinity. When physical systems are simulated on a computer by discretizing space onto a rectangular lattice, locality manifests itself by allowing only nearest-neighbor interaction.

 

3. Symmetry

Whenever the Hamiltonian obeys some symmetry (is invariant under some transformation), the number of independent parameters required to describe it is further reduced. For instance, many probability distributions in both physics and machine learning are invariant under translation and rotation.

 

Why Deep?

What properties of real-world probability distributions cause efficiency to further improve when networks are made deeper? This question has been extensively studied from a mathematical point of view, but mathematics alone cannot fully answer it, because part of the answer involves physics. We will argue that the answer involves the hierarchical/compositional structure of generative processes together with inability to efficiently “flatten” neural networks reflecting this structure.

 

A. Hierarchical processes

One of the most striking features of the physical world is its hierarchical structure. Spatially, it is an object hierarchy: elementary particles form atoms which in turn form molecules, cells, organisms, planets, solar systems, galaxies, etc. Causally, complex structures are frequently created through a distinct sequence of simpler steps.

 

We can write the combined effect of the entire generative process as a matrix product.

 

If a given data set is generated by a (classical) statistical physics process, it must be described by an equation in the form of [a matrix product], since dynamics in classical physics is fundamentally Markovian: classical equations of motion are always first order differential equations in the Hamiltonian formalism. This technically covers essentially all data of interest in the machine learning community, although the fundamental Markovian nature of the generative process of the data may be an in-efficient description.

 

Summary

The success of shallow neural networks hinges on symmetry, locality, and polynomial log-probability in data from or inspired by the natural world, which favors sparse low-order polynomial Hamiltonians that can be efficiently approximated. Whereas previous universality theorems guarantee that there exists a neural network that approximates any smooth function to within an error ε, they cannot guarantee that the size of the neural network does not grow to infinity with shrinking ε or that the activation function σ does not become pathological. We show constructively that given a multivariate polynomial and any generic non-linearity, a neural network with a fixed size and a generic smooth activation function can indeed approximate the polynomial highly efficiently.

 

The success of deep learning depends on the ubiquity of hierarchical and compositional generative processes in physics and other machine-learning applications.

 

And thanks to Tech Review for the pointer to this article:

 

“We’ll soon see the power of computing increase way more than any other period. There will be trillions of products with tiny neural networks inside.”

 

Intel has long ago ceded leadership for Moore’s Law. And so, understandably, they have trumpeted the end of Moore’s Law for many years. To me, it sounds a lot like Larry Ellison’s OpEd declaring the end of innovation in enterprise software, just before cloud computing and SaaS took off. In both cases, the giants missed the organic innovation bubbling up all around them.

 

For the past seven years, it has not been Intel but NVIDIA that has pushed the frontier of Moore’s processor performance/price curve. For a 2016 data point, consider the NVIDIA Titan GTX. It offers 10^13 FLOPS per $1K (11 trillion calculations per second for $1,200 list price), and is the workhorse for deep learning and scientific supercomputing today. And they are sampling much more powerful systems that should be shipping soon. The fine-grained parallel compute architecture of a GPU maps better to the needs of deep learning than a CPU. There is a poetic beauty to the computational similarity of a processor optimized for graphics processing and the computational needs of a sensory cortex, as commonly seen in neural networks today.

 

Here are some of Huang’s provocative prognostications from WSJ.D Live (and some photos I took of him):

 

It turns out we created an AI company. We power deep learning algorithms in the car. To drive autonomously, the car needs to do perception, reasoning, planning, and learning. These are a big part of AI.

 

Deep Learning has taken perception to a level that is superhuman. We have eyes around the car. We are never intoxicated or angry. It can look around corners and see things you can’t see, helping you even when you are driving.

 

To drive autonomously, we have to predict where everything will be in near future. We do path planning and detect objects. But we need to invert the logic. When humans drive, we don’t keep checking “there is no tree in the way; there is no boat in the way” and so on. We need to train on what is safe, not on all the things to avoid.

 

Elon is right. The AI gets better and better over use. It needs road miles.

 

We want to turn the car into an AI itself. I want to talk to it and have it respond with a sultry voice. You can ask to make a call for you from calendar.

 

The autonomous car improves its driving over time, whereas the human capability decreases over time as we age.

 

The AI should get a drivers license.

 

Computing will increase at product of Moore’s Law and Metcalfe’s Law. We’ll soon see the power of computing increase way more than any other period.

 

AI takes a different process, a new software approach, new tools, a new computational architecture. Microsoft just announced Cognitive Studio / CNTK yesterday.

 

To enable AI, we will shift deep learning from CPUs to a new computing architecture with GPUs.

 

We will embed inference engines into everything: little robots, sensors, into the factory, the machines building the machines. There will be trillions of products with tiny neural networks inside.

 

Every search query that uses Hadoop today will move to deep learning. Every query will invoke a billion calculations when we add AI to our apps.

My talk on deep learning and AI, followed by my interview of Naveen Rao, head of AI at Intel, can be seen starting at minute 54 of this facebook video.

 

Conference schedule.

I moderated a legendary panel of machine intelligence pioneers in Toronto today: Russ Salakhutdinov (the new head of AI at Apple), Rich Sutton (Univ. of Alberta, author of Reinforcement Learning), Geoff Hinton (Univ. of Toronto, Google, DL patron saint), and Yoshua Bengio (Univ. of Montreal and head of MILA),

 

All were deeply inspired by the biology of the brain and wanted to focus their future work on deeper biomimicry.

 

Ruslan: focusing on embedding memory for natural language understanding, dialog, Q&A and deep reinforcement learning. Unsupervised, one-shot and transfer learning.

 

Sutton: “AIs will compete and cooperate with us just like other people, but with greater diversity and asymmetries. So, we need to set up mechanisms (social, legal, political, cultural) to ensure this works out well. Inevitably, conventional humans will be less important. Step 1 – Lose your sense of entitlement. Step 2 – Include AIs in your circle of empathy.”

 

Hinton: Wants to find a better axon activation function (sigmoid and RELU being used for so long) and a better model of the synapse (with instantaneous shifts and retained state). His motivation is to understand the brain better. The only reason he applies his work to deep learning is that Google does a better job funding his research than medical research sources. Current neural nets are not good at coincidences or detecting sameness in time (the XOR problem). Turing did not believe the brain was a turing machine. He did work on neural networks.

 

Event Agenda. Thanks to Shivon for the photo.

Bing Xu, co-founder of SenseTime at the Goldman Sachs Private Innovative Company Conference (PICC) in Las Vegas:

 

“AI is the key economic driver for next 20 years of economic growth!”

 

“We have the #1 AI supercomputer dedicated to Deep Learning with 15,000 GPUs, 157 PetaFLOPS. That is the weapons that we have."

 

"AI is the new engine for industry upgrades" — Chinese Government

 

SenseTime raised $2B in 2018 from Silverlake, Softbank and Alibaba as a private company. They raised $600M at a $3B valuation in April and then $700M at a $7B valuation from Softbank. “When Silverlake did the C+ round, Softbank paid attention. They were looking for an algorithm company and never found one until us.”

 

“We have 160 PhDs in AI area and 3000 staff, adding 200-300 per month. For the past three years the average revenue growth has been 400%/yr.”

 

I took a short video of his AI superpower claims.

With my DFJ colleagues Mo & Maryanna & Mythic. Not surprisingly, this research group develops applications for deep learning throughout the company.

 

I opened with our 20-year connection to the company. From p.10. of Silicon Dragon (where they interviewed the Baidu founders about the early days):

 

"The Baidu saga began in Silicon Valley in 1997, when Baidu co-founder Eric Xu and Li were introduced by Li's wife, Melissa... Not only was Xu entrepreneurial, he was well connected to key investors in the valley. Xu made those connections in 1998 and 1999 while filming a documentary called "A Journey to Silicon Valley," which later aired on the Chinese national television network CCTV. In some 120 hours of taping, Xu interviewed Valley legends such as Cisco Systems CEO John Chambers, Albert Yu of Intel, and venture capitalists Steve Jurvetson of Draper Fisher Jurvetson and Bob King of Peninsula Capital. After watching Xu's documentary at Stanford, Li was energized. He asked Xu to lunch the next day to discuss his brainstorm. At a Chinese restaurant in Sunnyvale, California, Li pitched Xu to become his partner.”

 

“Just a few weeks after returning to their homeland, the cofounders opened their shop doors on January 18, 2000 in Beijing.”

 

“The serious-minded cofounders named their young firm Baidu, which translates as 'seek for truth' from an ancient Song Dynasty poem.”

 

“Xu relates how engineers were hired straight out of China’s top universities. The new recruits were cheered on with examples from the Valley to persuade them to work hard, grow the company, and get rich. ‘We share the overall vision with them. We brainwash them about why Silicon Valley has become so famous,’ says Xu. ‘We took the secrets of the Silicon Valley culture: tolerance of failure and tolerance of differences and free flow of information with no barriers. We tell them we have a democratic environment and that they, as bright engineers, can turn their ideas into products. These are the fundamentals of creating innovative products,’ Xu tells me. ‘It was a tremendous startup experiment. We were not sure we were going to make it.’

 

Indeed, they almost didn’t. Xu relates how Baidu failed with three near clones of successful U.S. Internet businesses (Inktomi, Akamai, Verity) before finally turning a profit with a fourth plan: a Google-like search engine designed for consumers.”

 

We were investors during that exploration period, becoming the largest shareholder in Baidu, back then and through the IPO. Our initial $8M investment is worth 25% of Baidu today (as if held).

With co-founder and CTO, Dave Fick today.

 

Of all the Series A investments that I have led over the past 23 years, I have never seen as many inbound investment offers for the Series B. Investor interest in edge AI and deep learning in general has skyrocketed after the Intel Nervana acquisition. And the Mythic approach is truly unique, with an in-memory analog compute architecture that more closely mimics the power of the human cortex, and affords a 100x cost and power advantage over digital chips (i.e., every other chip out there).

 

I am also delighted that Rene Haas (President of the IP Product Group at ARM) has joined the board. He has the perfect background from ARM, NVIDIA and Tensilica.

 

And Lockheed Martin signed up as an early customer.

 

News: Xconomy, VentureBeat, TechCrunch

 

www.mythic-ai.com

At this first presentation since their acquisition by Intel, Amir Khosrowshahi, co-founder of Nervana, was sweetly spunky "We provide tools for you to weaponize against each other, and hopefully make money for Intel."

 

He gave a good overview of deep learning applications in finance, with examples in order book substructure analysis and semantic sentiment analysis

 

From the AI in Fintech Forum, hosted by the Stanford Institute for Computational & Mathematical Engineering ("We do big math").

Experiment with Deep learning AI image generation reveals striking truth. If the AI starts with messaging me with what looks like bottled fish and other biological material, no wonder we all meatbags will end up canned.

© Dan McCabe

 

Processing my "Knotty Mask" with Deep Dream Generator.

 

You can find the original here:

flic.kr/p/29J4Ww4

 

When I look at the original photo, I see a single face. In this image, I see a number of facial parts (eyes, mouth, etc.), but no single face.

The Hive at Nanyang Technological university, Singapore

 

Went with a more gritty and dark processing for this shot to show the great details hidden in the structure. These textures are more commonly found in the construction phase of a building and are rarely left as is for the entire building. Simply amazing just to look at it..

 

Canon 5d Mk 3 | Canon 17mm TS-E f4L | ISO 100 | f 13.0 | focal length 17mm (2 tile shifted, 4 exposure digital blend vertorama) 13112016

 

--

 

© Jonathan Danker

My images are posted here for your enjoyment only. All rights are reserved. Please contact me @ jonathan@anselmedia.com if you are interested in using one of my images for any reason.

 

As an initiative to make my works more exposed and for you, the viewer to have a more easier experience, I have created a facebook page for my photography.

 

Please visit, like it and comment and I hope to see you there!

www.facebook.com/ANSELmedia

Thanks for the support and encouragement!

 

MINIMAL - Soho Build -no snow-

5. Apple Fall Gold Leaf Cherries w/ Cup

Apple Fall Coffee & Muffin

Nutmeg. Romanov's Shabby Table Dark

Deep learning with Elastic Averaging SGD. Zhang, Choromanska, LeCun arxiv.org/abs/1412.6651 #stat

some experiments with deepdream generator

Apple Fall Make-up Brush Assortment w/ Horn Jar

West Village Classic Watering Can - Steel

Stealing from other programs is old, human style to write code. A machine learning process has gained the chance to write its own code. The software, called DeepCoder, fixed general problems of the kind set by programming competitions.

 

Originally published on: Waffal

waffal.com/seo/artificial-intelligence-learn-by-stealing/

"... the menacing crisis that represents the tragedy of not being willing to overcome it."

 

"... l'unica crisi pericolosa, è la tragedia di non voler lottare per superarla."

 

Albert Einstein

West Village Classic Watering Can - Steel

Click the below link to know the Best Way to Know More Better about Deep Learning Vs Machine Learning...

Visit: bit.ly/2QZbN0N

1 3 4 5 6 7 ••• 79 80