Edit

Cette page permet de soumettre une suggestion de modifications à la page (typiquement correction orthographique ou erreur de contenu). Le contenu est tapé (essentiellement) en markdown.

This page allows the reader to suggest modifications to the page they read. Typically spelling mistakes or suggestions to update the content. The format is essentially standard markdown.

(Optional) Votre nom, your name: (Optional) Votre mail, your email:

# Reiterability

With [Bruno Guillon](http://researchers.lille.inria.fr/~guillonb/)
 we developed a nice class we call *reiterable* within a larger
(and hopefully soon to be disclosed) project.

For people not at ease with iterators and generators, you can
read [an introduction here](https://pymbook.readthedocs.io/en/latest/igd.html) or the official documentation [here](https://docs.python.org/3/tutorial/classes.html#iterators)

## An example of Generator

Lets say that you are into number theory and you want to manipulate the sequences of prime
numbers. Of course it is infinite, but nevertheless you can manipulate an iterator
over all primes. For instance:

```python
def primes():
	Primes = []
	n = 2
	while True:
		if all(n % p for p in Primes):
			Primes.append(n)
			yield n
		n += 1

P = primes()
# P is an iterator over all primes!
for i in range(10):
	print(next(P), end=", ")
```

The iterator is useful itself since we can map or filter it:

```python
P = primes()
P2 = filter(lambda e:e%4 == 3, P)
# P2 is the infinite sequence of primes
# whose congruent to 3 mod 4.
for i in range(10):
	print(next(P2), end=", ")
```

One classical caveat of iterators (which is actually a feature)
is that they are designed to not be reusable.
For instance:

```python
P = primes()
P2 = filter(lambda e:e%4 == 3, P)
P3 = filter(lambda e:e%4 == 3, P)
print(next(P3))
print(next(P2))
```

In particular, we have an inter-dependencies between
the definitions of iterators that is highly prone of mistake.
In the case of Primes, you could (and should) regenerate the
iterators for each new definition:

```python
P2 = filter(lambda e:e%4 == 3, primes())
P3 = filter(lambda e:e%4 == 3, primes())
print(next(P3))
print(next(P2))
```

When building complicated code, iterator regeneration can become
a nightmare and make requirement to inspect the object to check if
it is iterable or iterator.

# Reiterability

In many situations (but certainly not all) where iterators are useful,
it is possible to maintain the information of *how* the iterator has been generated.
For prime numbers, it is not hard to provide a way of restarting the computation.
It is actually the case with all generators: we can simply regenerate the iterator.

Hence, we have the possibility to have iterators that can be *reset*. In the [reiterable.py](reiterable.py)
we introduce a decorator of generators that simply does that. Instead of generating iterators,
it will generate iterators with reset (called reiterator).

```python
from reiterable import reiterable
@reiterable
def primes():
	Primes = []
	n = 2
	while True:
		if all(n % p for p in Primes):
			Primes.append(n)
			yield n
		n += 1

P = primes()
for i in range(10):
	print(next(P), end=', ')
P.reset()
print()
print("After the reset:", next(P))
```

The reiterator does not store much more information than the initial iterator.
We simply remember the generator and regenerate internally the iterator.
Reiterators are **not** iterators. Their `__iter__` method actually generates a new
reiterator starting from the beginning.

```python
i = 0
for p in P:
	print(p, end=", ")
	i += 1
	if i == 10:
		break
print()
print("We can do that again")
for p in P:
	print(p, end=", ")
	i += 1
	if i == 20:
		break
```

Every function that call `__iter__` will regenerate an iterator from the current position.
If we redo the previous code, but after popping the first prime number:

```python
next(P) # We pop 2
i = 0
for p in P:
	print(p, end=", ")
	i += 1
	if i == 10:
		break
print()
print("We can do that again")
for p in P:
	print(p, end=", ")
	i += 1
	if i == 20:
		break
```

## Extra operations on reiterators

On Reiterator, we don't lose information by iterating. So we can actually move
freely within the sequence of elements without really consuming the elements.
For that reason, we support the full slices of lists. Of course, reaching for the
last element of an infinite reiterator is a really bad idea.

```python
P.reset()
print("The 5th Prime", P[5])
```

If we pop the first element, then the 5th
is the 6th prime.

```python
next(P)
print("The 6th Prime", P[5])
```

Slice allow to have a lookahead.

```python
print("The next elements", list(P[:5]))
```
We didn't pop from the sequence yet, it is just a lookahead.
```python
print(next(P))
print("The next elements", list(P[:10]))
```
We can also look behind
```python
print(P.previous())
```

Although these operations are supported, in term of complexity they are not
efficient. Each time we copy an iterator, we have to re-execute the generator
from the beginning to the current point. So it should be used lightly.

## Functional operations

We can of course perform `map` and `filter` on reiterators. It will generate
new iterators that will be independent from the initial generators:

```python
P.reset()
P2 = filter(lambda e: e%4 == 1, P)
print(next(P2))
print(next(P)) # It didn't get consummed.
```

Unfortunately, it is not possible to stack many layers of filter and map by keeping
the good behaviour of regeneration.

```python
P3 = map(lambda e:e**2, P2)
print(next(P3))
print(next(P2)) # P2 get consummed :(
```

There are no reasonable ways of overloading the action
of functional operators like `filter` and `map`.
Thus, as alternative, we can call `.filter` and `.map` as class methods.
The reiterators obtained are independents.

```python
P2 = P.filter(lambda e: e%4 == 1)
P3 = P2.map(lambda e: e**2)
print(list(P2[:5]))
print(next(P3))
print(list(P2[:5])) # Not consumed
```

We also overload `accumulate` but use the semantic from the standard
version in `itertools` starting from python 3.8.

```python
P.reset()
AdjacentPrimes = P.accumulate(lambda x,y: (x+(y,))[-2:], ())
print(list(AdjacentPrimes[:5]))
```

Now we can even filter that:

```python
TwinPrimes = AdjacentPrimes[1:].filter(lambda e:e[1]-e[0] == 2)
print(list(TwinPrimes[:10]))
```

We can now look ahead, even rather faraway without any trouble:

```python
print(TwinPrimes[1000]) # It took around 2seconds
```

We can of course nest even more functional operators.

```python
P4 = TwinPrimes.filter(lambda e:e[0] % 4 == 1)
print(list(P4[:10]))
```

# Practical use-cases

So reiterability is a nice feature with a rather poor complexity
hidden behind. It allows to play with infinite sequences nicely without
all the technicalities of iterators. But can it be used in practical reasonable situations?

Actually some algorithms requires to be able to go through dataset few times. If the dataset
didn't fit in main memory, then it is become annoying quickly.
With our decorator, it is become really easy.

Let us produce a file with some useless data.
```python
from random import sample

with open("/tmp/file", "w") as f:
	for i in range(1000):
		f.write(f"{i}\n")
```

Now we can easily construct a reiterator over lines:

```python
@reiterable
def lines():
	with open("/tmp/file") as f:
		yield from f
Lines = lines().map(lambda e:int(e.strip()))
print(list(Lines[:10]))
print(list(Lines[8:20:2]))
```

Actually, we package this code snippet to provides a reiterable
through files seen as lines.

```python
from reiterable import ReitLinesFiles
Lines = ReitLinesFiles("/tmp/file")
print(list(Lines[:10]))
print(list(Lines[8:20:2]))
```

Database access are another situation where reiterability can
simplify a lot. We thus create another small function helper
that should work for any database connector respecting
python db-api.

```python
import sqlite3
from reiterable import ReitDBTable
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE A (x, y)")
db.executemany("INSERT INTO A VALUES (?,?)", enumerate(range(1000)))
ATable = ReitDBTable(db, "SELECT * FROM A")
print(list(ATable[:5]))
print(list(ATable.filter(lambda e:e[0] % 3 == 0)[:5]))
```