JavaScript Iterators and generators
Iterating over lists of objects is a very common (if not daily) problem that developers confront with at work. With the introduction of ES6, iterators and generators have officially been added to JavaScript to help developers solve this particular issue.
Iterators give the developers the superpower of consuming a list of items, one at a time, and proceeding to the next item only after finishing consuming and processing it.
Generators are a special kind of function that give the developer the superpower of producing data and pausing execution until called again.
Altough they might seem to be the same thing, they are not. They are strongly connected with each other, but not the same. Altough iterators are very powerful, their definition and the maintenance of their internal logic can be very challenging. Generators then come in place to provide a less error-prone and more efficient way of using iterators, hiding the complexity of using iterators behind the hood.
After reading this article about iterators and generators, you should be able to answer the following questions:
- What are iterators and iterables?
- What are generators? How are they connected to iterators?
- How does the "for ... of" Loop work behind the scenes?
How does the "for ... of" Loop know how to go through an Array, Map, Set or any ITERABLE Data Structure?
Tip: It does not know ANYTHING about the type of the data structure that it iterates over.
Answer: "for ... of" Loops only work on ITERABLES. Iterables have ITERATORS. "for ... of" Loops use these iterators. To better understand what these are, let's define an iterable and iterate over it using the "for ... of" Loop.
Iterable object definition
const MAX_LENGTH = 5
const myIterable = {
length: 0,
[Symbol.iterator]: () => ({
next: () => length++ < MAX_LENGTH
? { value: Math.random(), done: false }
: { value: undefined, done: true }
})
}
for(const item of myIterable){
console.log(item)
}
Our iterable object will give access to 5 random numbers, through its iterator.
Stepwise explanation:
1. We define our iterable as an object, that has a special property, accessible through the "[Symbol.iterator]" key - defined using Symbols, makes the property unique and generic. More about Symbols: Symbols - MDN
2. This special key gives access to a method that returns the actual ITERATOR object.
3. This iterator object is used to access our ITERABLE's items one by one, using it's inner method "next()".
Once you understand the Explanation, you will be ready to write your own "for... of" Loop using both basic loops: for and while.
Note that the first value is created only on the first call of the next() method.
Iterable object definition
const iterator = myIterable[Symbol.iterator]()
let item = iterator.next()
while(item.value){
console.log(item.value)
item = iterator.next()
}
for(let i = iterator.next(); i.value; i = iterator.next()){
console.log(i.value)
}
Actually, the [Symbol.iterator] key of the iterable gives access to a generator function - a function that returns an iterator that spits values each time its next() method is called. You can view this as a PAUSABLE FUNCTION THAT RETURNS MULTIPLE VALUES, one at a time. By "pausable" I mean that it returns values only when the user wants it to do so. Until that, it waits.
JavaScript provided generator functions
Iterable object definition
function* numberGenerator() {
let numbersGenerated = 0;
while(numbersGenerated++ < MAX_LENGTH){
yield Math.random()
}
}
const numbers = numberGenerator()
for(const number of numbers){
console.log(number)
}
Note 1: asterix ("*")" symbol after the function keyword (or before function name)
Note 2: yield keyword, used for "returning" the desired next value.
This generator functions are actually syntactic sugar, using iterators behind the hood. They are as well PAUSABLE FUNCTIONS. Therefore, we can replace the iterator generator method from our iterable with the generator function:
const myIterable = {
length: 0,
[Symbol.iterator]: function* numberGenerator() {
let numbersGenerated = 0;
while(numbersGenerated++ < MAX_LENGTH){
yield Math.random()
}
}
}
JavaScript async iterators
const myIterable = {
length: 0,
[Symbol.asyncIterator]: async function* numberGenerator() {
let numbersGenerated = 0;
while(numbersGenerated++ < MAX_LENGTH){
yield Math.random()
}
}
}
Our iterable now gives access to its values through an async generator function. In the above example, you cannot immediately see its value because we are yielding random numbers, but imagine that we are yielding different API call results. But we cannot use a basic "for ... of" loop, because async iterators return promises of the yielded values/objects. We instead have to use the "for await ... of" loop like this:
for await (const item of myIterable){
console.log(item)
}
Note that we have to use this inside an async function, because we are using await.
Conclusion
We can now write our own practical and more advanced iterables that give us sequential access to the desired data.
The problem I am trying to solve is the following: we want to create a car selling website and we have an API that contains hundreds of millions of records that we need to display on our website. Due to the huge size of the records, we can not get them through a single request and we need to create a mechanism to access them sequentially.
const searchCars = () => {
const fetchCarsByPageAndTag = async (page, tag) =>
await axios.get('https://api.flickr.com/services/rest/' +
'?method=flickr.photos.search' +
'&api_key=' + API_KEY +
'&page=' + page +
'&tags=' + tag +
'&format=json' +
'&nojsoncallback=1')
.then(response => response.data)
.then(body => body.photos.photo)
.then(photos => photos.map(photo =>
`https://farm${photo.farm}.staticflickr.com/` +
`${photo.server}/${photo.id}_${photo.secret}_q.jpg`
))
.catch(error => console.error(error));
return {
[Symbol.asyncIterator]: async function* (){
while(true){
let pageNumber = 1;
for(const generatedUrl of await fetchCarsByPageAndTag('cars', pageNumber)){
yield generatedUrl
}
pageNumber++
}
}
}
}
(async function() {
for await(let car of searchCars()){
console.log(car)
}
})()
Explanation: we are iterating over our iterable returned by the searchCars() function using async iterators. How does the searchCars() method work? The async generator function of the returned iterable waits on the first yield call being prepared to request the first page of cars to our API, yielding the first page after the API responds. After that, it automatically increments the pageNumber and makes another request and yields its response. In the above example, while(true) means that we are incrementing pageNumber an infinite number of times, meaning that we are supposing that our API has an infinite number of pages. Of course it's impossible, but for the sake of understanding, let's pretend it's not. The code will still run, because we are not able to consume all pages of the API so quick, but at some point, they will end.
In this article, we learned what iterators, async iterators and generators are and how and when we can use them. We also learned how to make any object iterable and how to write our own data streaming service using async iterators. Pretty cool, right?