In this post, I'll show you how to build a lightweight data catalog on top of dbt artifacts using Algolia.


A lightweight alternative to Amundsen for your dbt project


## Table of contents

## Overview

If you're using [`dbt`](https://www.getdbt.com/), chances are you've noticed
that it generates and saves one or more _artifacts_ with every invocation.

In this post, I'll show you how to get started with dbt artifacts, and how
to parse them to unlock applications valuable to your team and your use case.

Whether that's just for a fun Friday afternoon learning
session, or whether that's your first foray
at building a Data Governance tool using dbt,
I hope you'll find this post useful, and if you do, let me know [on twitter](https://twitter.com/louis_guitton)!

## When are Artifacts Produced

![dbt](/images/dbt-artifacts/dbt.png 'dbt')

A word of warning: dbt's current minor version as of writing is v0.18.1 and multiple improvements to artifacts are coming in dbt's next version v0.19.0, but that doesn't change the content of this post.

> dbt has produced artifacts since the release of dbt-docs in v0.11.0. Starting in dbt v0.19.0, we are committing to a stable and sustainable way of versioning, documenting, and validating dbt artifacts.\
> Ref: https://next.docs.getdbt.com/reference/artifacts/dbt-artifacts/

[The artifacts currently generated](https://docs.getdbt.com/reference/dbt-artifacts) are JSON files called
`manifest.json`, `catalog.json`, `run_results.json` and `sources.json`.
They are used to power the docs website and other dbt features.

[Different dbt commands](https://docs.getdbt.com/reference/dbt-commands) generate different artifacts, so I've summarised that in the table below:

|             | debug | init | compile | run | test | deps | snapshot | clean | seed | docs | source | run-operation | rpc | list |
| ----------- | ----- | ---- | ------- | --- | ---- | ---- | -------- | ----- | ---- | ---- | ------ | ------------- | --- | ---- |
| manifest    |       |      | x       | x   | x    |      |          |       |      | x    |        |               |     | x    |
| run results |       |      |         | x   | x    |      | x        |       | x    | x    |        |               |     |      |
| catalog     |       |      |         |     |      |      |          |       |      | x    |        |               |     |      |
| sources     |       |      |         |     |      |      |          |       |      |      | x      |               |     |      |

Of course, `dbt docs` is the command that refreshes most artifacts (makes sense, since they were initially introduced to power the docs site).
But it's interesting to note that `manifest` can be refreshed by other commands than the usual suspects `dbt run` or `dbt test` too.

### Available Data in dbt artifacts

`Manifest`:

> Today, dbt uses this file to populate the [docs site](https://docs.getdbt.com/docs/building-a-dbt-project/documentation), and to perform [state comparison](https://next.docs.getdbt.com/docs/guides/understanding-state/). Members of the community have used this file to run checks on how many models have descriptions and tests.

`Run Results`:

> In aggregate, many run_results.json can be combined to calculate average model runtime, test failure rates, the number of record changes captured by snapshots, etc.

`Catalog`:

> Today, dbt uses this file to populate metadata, such as column types and table statistics, in the [docs site](https://docs.getdbt.com/docs/building-a-dbt-project/documentation).

`Sources`:

> Today, dbt Cloud uses this file to power its [Source Freshness visualization](https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-snapshotting-source-freshness/).

`graph.gpickle`:

> Stores the networkx representation of the dbt resource DAG.

## Parsing Artifacts from the Command Line with `jq`

![jq command line](/images/dbt-artifacts/jq.png 'jq command line')

To get started with parsing dbt artifacts for your own use case,
I suggest to use [`jq`](https://stedolan.github.io/jq/), the
lightweight and flexible command-line JSON processor.
This way, you can try out your ideas, explore the available data
without writing much code at first.

`jq` Cheat sheet:

<details>

```shell
→ tldr jq
  jq

  A command-line JSON processor that uses a domain-specific language.
  More information: https://stedolan.github.io/jq.

  - Output a JSON file, in pretty-print format:
    jq . file.json

  - Output all elements from arrays (or all the values from objects) in a JSON file:
    jq '.[]' file.json

  - Read JSON objects from a file into an array, and output it (inverse of jq .[]):
    jq --slurp . file.json

  - Output the first element in a JSON file:
    jq '.[0]' file.json

  - Output the value of a given key of the first element in a JSON text from stdin:
    cat file.json | jq '.[0].key_name'

  - Output the value of a given key of each element in a JSON text from stdin:
    cat file.json | jq 'map(.key_name)'

  - Output the value of multiple keys as a new JSON object (assuming the input JSON has the keys key_name and other_key_name):
    cat file.json | jq '{my_new_key: .key_name, my_other_key: .other_key_name}'

  - Combine multiple filters:
    cat file.json | jq 'unique | sort | reverse'

  - Output the value of a given key to a string (and disable JSON output):
    cat file.json | jq --raw-output '"some text: \(.key_name)"'
```

</details>

In particular, you will need to make use of some of the
built-in operators like `to_entries` and `map`.

Here is a command to grab the materialisation of each model

```shell
→ cat target/manifest.json | jq '.nodes | to_entries | map({node: .key, materialized: .value.config.materialized})'
[
  {
    "node": "model.jaffle_shop.dim_customers",
    "materialized": "table"
  },
  {
    "node": "model.jaffle_shop.stg_customers",
    "materialized": "view"
  }
]
```

You can then for example store that into a file by piping the output

```shell
cat target/manifest.json | jq '.nodes | ...' > my_data_of_interest.json
```

## Parsing Artifacts from Python with `pydantic`

![pydantic](/images/dbt-artifacts/pydantic.png 'pydantic')

Once you get a better idea of what data you need,
you might want to develop more custom logic around dbt artifacts.
This is where python shines: you can write a script with the logic you need.
You can install and import great python libraries.
For instance, you could use `networkx` to run graph algorithms
on your dbt DAG.

You will then need to parse the dbt artifacts in python.
I recommend to use the great [pydantic](https://pydantic-docs.helpmanual.io/)
library: among other things, it allows to parse JSON files
with very concise code that lets you focus on high-level parsing logic.

Here is an example logic to parse `manifest.json`:

```python showlinenumbers
import json
from typing import Dict, List, Optional
from enum import Enum

from pydantic import BaseModel, validator


class DbtResourceType(str, Enum):
    model = 'model'
    analysis = 'analysis'
    test = 'test'
    operation = 'operation'
    seed = 'seed'
    source = 'source'


class DbtMaterializationType(str, Enum):
    table = 'table'
    view = 'view'
    incremental = 'incremental'
    ephemeral = 'ephemeral'
    seed = 'seed'


class NodeDeps(BaseModel):
    nodes: List[str]


class NodeConfig(BaseModel):
    materialized: Optional[DbtMaterializationType]


class Node(BaseModel):
    unique_id: str
    path: Path
    resource_type: DbtResourceType
    description: str
    depends_on: Optional[NodeDeps]
    config: NodeConfig


class Manifest(BaseModel):
    nodes: Dict["str", Node]
    sources: Dict["str", Node]

    @validator('nodes', 'sources')
    def filter(cls, val):
        return {k: v for k, v in val.items() if v.resource_type.value in ('model', 'seed', 'source')}


if __name__ == "__main__":
    with open("target/manifest.json") as fh:
        data = json.load(fh)

    m = Manifest(**data)
```

Once you've got the `Manifest` class, you can use it in your custom logic.
For example, in our use case from above where we want to check
for model materialization, we can do:

```python
>>> m = Manifest(**data)
>>> [{"node": node, "materialized": n.config.materialized.value} for node, n in m.nodes.items()]
[
  {
    "node": "model.jaffle_shop.dim_customers",
    "materialized": "table"
  },
  {
    "node": "model.jaffle_shop.stg_customers",
    "materialized": "view"
  }
]
```

## Example Application 1: Detecting a Change in Materialization

Let's say you want to check that no materialisation has changed
before you run `dbt run`. This is useful because some materialization
changes require a `--full-refresh`.
You could achieve the change detection with the following commands:

```shell
→ cat target/manifest.json | jq '.nodes | to_entries | map({node: .key, materialized: .value.config.materialized})' > old_state.json
→ # code change: let's say one model materialization is changed from table to view
→ dbt compile
→ cat target/manifest.json | jq '.nodes | to_entries | map({node: .key, materialized: .value.config.materialized})' > new_state.json
→ diff old_state.json new_state.json
12c12
<     "materialized": "table"
---
>     "materialized": "view"
```

## Example Application 2: Compute Model Centrality with `networkx`

Once you've parsed the `manifest.json`, you have at your disposal
the graph of models from your project.
You could explore off-the-shelf graph algorithms provided
by [`networkx`](https://networkx.org/), and see if any of the insights you get are valuable.

For example, [nx.degree_centrality](https://networkx.org/documentation/stable//reference/algorithms/generated/networkx.algorithms.centrality.degree_centrality.html) can give you
the list of models that are "central" to your project.
You can use that e.g. to priotise maintenance efforts.
In the future, you could imagine a `dbt docs` search that
prioritises results based on this metric as a very simple PageRank proxy.

Once you've written the pydantic code from above, this turns out
to be possible in a very small amount of lines.

```python
import networkx as nx

# ... pydantic code from above for Manifest class

class GraphManifest(Manifest):
    @property
    def node_list(self):
        return list(self.nodes.keys()) + list(self.sources.keys())

    @property
    def edge_list(self):
        return [(k, d) for k, v in self.nodes.items() for d in v.depends_on.nodes]

    def build_graph(self) -> nx.Graph:
        G = nx.Graph()
        G.add_nodes_from(self.node_list)
        G.add_edges_from(self.edge_list)
        return G


if __name__ == "__main__":
    with open("target/manifest.json") as fh:
        data = json.load(fh)

    m = GraphManifest(**data)
    G = m.build_graph()
    nx.degree_centrality(G)
```

## Example Application 3: Graph visualisation

Provided you use python 3.8+, there is another dbt artifact that can
be interesting to you: `graph.gpickle`.
Instead of parsing `manifest.json` and building the graph yourself,
you can deserialize the networkx graph built by dbt itself.

All it takes is 2 lines!

That's hard to beat, but note that you will rely
on the internal graph definition of `dbt` and won't be able to
customise it. For example, tests will be nodes on your graph now.

```python
import networkx as nx

G = nx.read_gpickle("target/graph.gpickle")
```

Nevertheless, this can be useful for example for a quick visulisation
using [pyvis](https://pyvis.readthedocs.io/en/latest/):

```python
from pyvis.network import Network

nt = Network("500px", "1000px", notebook=True)
nt.from_nx(G)
nt.show("nx.html")
```

<embed
  type="text/html"
  src="/other/nx.html"
  width="800"
  height="600"
  style={{ display: 'flex', margin: 'auto' }}
/>

## Resources

1. [dbt (data build tool) - Transform data in your warehouse](https://www.getdbt.com/)
1. [dbt Artifacts | docs.getdbt.com](https://docs.getdbt.com/reference/dbt-artifacts/)
1. [dbt Artifacts | next.docs.getdbt.com](https://next.docs.getdbt.com/reference/artifacts/dbt-artifacts/)
1. [dbt Command reference | docs.getdbt.com](https://docs.getdbt.com/reference/dbt-commands)
1. [jq, a lightweight and flexible command-line JSON processor.](https://stedolan.github.io/jq/)
1. [jq Manual | Builtin operators and functions](https://stedolan.github.io/jq/manual/#Builtinoperatorsandfunctions)
1. [pydantic docs](https://pydantic-docs.helpmanual.io/)
1. [NetworkX — NetworkX documentation](https://networkx.org/)
1. [networkx.algorithms.centrality.degree_centrality — NetworkX 2.5 documentation](https://networkx.org/documentation/stable//reference/algorithms/generated/networkx.algorithms.centrality.degree_centrality.html)
1. [Interactive network visualizations — pyvis 0.1.3.1 documentation](https://pyvis.readthedocs.io/en/latest/)


In this post, I'll show you how to get started with dbt artifacts, and how to parse them to unlock applications valuable to your use case.


How to Parse dbt Artifacts


## What is the trax deep learning framework

[`trax`](https://trax-ml.readthedocs.io/en/latest/) is coming out of the Google Brain team and is the latest iteration after almost a decade of work on TensorFlow, machine translation, and Tensor2Tensor. Being a new-comer in a somewhat crowded space (`keras`, `pytorch`, `thinc`), it has been able to learn from those APIs.
In particular:

- it is very concise
- it runs on `TensorFlow` backend
- it uses `Jax` to speed up tensor-based computation (instead of numpy)

## Text Classification on the AG News Dataset

The [AG's news topic classification dataset](https://www.tensorflow.org/datasets/catalog/ag_news_subset) is a dataset of 120,000 training samples of news articles that are of one of 4 classes.

It has been used as a text classification benchmark in numerous papers, like [[1509.01626]](https://arxiv.org/abs/1509.01626). You can see the papers using that dataset on [paperswithcode.com](https://paperswithcode.com/sota/text-classification-on-ag-news)

![Text Classification on AG News](/images/trax/text-classification-on-ag-news.png "AG News Benchmark (Text Classification) | Papers With Code")

On top of being a well-understood dataset, it has the advantage of being available on [TensorFlow Datasets](https://www.tensorflow.org/datasets). Therefore we will use it in the rest of this post.

## Prepare the dataset using trax.data.TFDS and data.Serial

`trax` needs generators of data. Each element is a tuple (input, target) or (input, target, weight) (usually weight is =1 because all examples have the same importance).

```python
train_stream = data.TFDS(
    'ag_news_subset',
    keys=('description', 'label'),
    train=True
)()
eval_stream = data.TFDS(
    'ag_news_subset',
    keys=('description', 'label'),
    train=False
)()
```

```python
data_pipeline = data.Serial(
    data.Tokenize(vocab_file='en_8k.subword', keys=[0]),
    data.Shuffle(),
    data.FilterByLength(max_length=2048, length_keys=[0]),
    data.BucketByLength(boundaries=[  32, 128, 512, 2048],
                             batch_sizes=[512, 128,  32,    8, 1],
                             length_keys=[0]),
    data.AddLossWeights()
)
train_batches_stream = data_pipeline(train_stream)
eval_batches_stream = data_pipeline(eval_stream)
```

## Build the Text Classification Model using tl.Serial

`trax` is really concise, you can use the library of layers available.

```python
model = tl.Serial(
    tl.Embedding(vocab_size=8192, d_feature=50),
    tl.Mean(axis=1),
    tl.Dense(4),
    tl.LogSoftmax()
)
```

## Model Training using training.Loop

For training, there is the concept of a "task" which wraps the data, the optimiser, the metrics etc...

```python
# Training task.
train_task = training.TrainTask(
    labeled_data=train_batches_stream,
    loss_layer=tl.CrossEntropyLoss(),
    optimizer=trax.optimizers.Adam(0.01),
    n_steps_per_checkpoint=500,
)

# Evaluaton task.
eval_task = training.EvalTask(
    labeled_data=eval_batches_stream,
    metrics=[tl.CrossEntropyLoss(), tl.Accuracy()],
    n_eval_batches=20  # For less variance in eval numbers.
)

# Training loop saves checkpoints to output_dir.
output_dir = os.path.expanduser('~/output-dir/')
!rm -rf {output_dir}
training_loop = training.Loop(model,
                              train_task,
                              eval_tasks=[eval_task],
                              output_dir=output_dir)

# Run 2000 steps (batches).
training_loop.run(2000)
```

## Look at the Predictions

```python
inputs, targets, weights = next(eval_batches_stream)

example_input = inputs[0]
expected_class = targets[0]
example_input_str = trax.data.detokenize(example_input, vocab_file='en_8k.subword')
print(f'example input_str: {example_input_str}')
sentiment_log_probs = model(example_input[None, :])  # Add batch dimension.
print(f'Model returned class probabilities: {np.exp(sentiment_log_probs)}')
print(f'Expected class: {expected_class}')
```

## References

1. [google/trax: Trax — Deep Learning with Clear Code and Speed](https://github.com/google/trax)
1. [Trax Tutorials — Trax documentation](https://trax-ml.readthedocs.io/en/latest/)
1. [ag_news_subset  |  TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/ag_news_subset)
1. [[1509.01626] Character-level Convolutional Networks for Text Classification](https://arxiv.org/abs/1509.01626)
1. [AG News Benchmark (Text Classification) | Papers With Code](https://paperswithcode.com/sota/text-classification-on-ag-news)
1. [TensorFlow Datasets](https://www.tensorflow.org/datasets)


In this post, I show you how to train a model with trax, the very concise deep learning framework. I showcase text classification on the AG News dataset.

How to Train a Text Classification Model with trax


In this article, I'll discuss how to monitor the latency and code performance of a FastAPI service.

## API Monitoring vs API Profiling

Monitoring is essentially **collecting data in the background** of your application for the purpose of helping diagnosing issues, helping debugging errors, or informing on the latency of a service.

For example, at the infrastructure level, you can monitor CPU and memory utilization. For example, at the application level, you can monitor errors, code performance or database querying performance. For a more complete introduction to monitoring and why it's necessary, see [this excellent post from Full Stack Python](https://www.fullstackpython.com/monitoring.html).

In this post, we fill focus on **Application Performance Monitoring (APM)** for a FastAPI application.

### Error Tracking

In this post, I will not talk about monitoring application errors and warnings. For this purpose, check Sentry, it has great ASGI support and [will work out of the box](https://docs.sentry.io/platforms/python/guides/asgi/) with your FastAPI service.

### API Profiling

Profiling is a code best-practice that is not specific to web development. From the [python docs on profiling](https://docs.python.org/3/library/debug.html) we can read :

> the profilers run code and give you a detailed breakdown of execution times, allowing you to identify bottlenecks in your programs. Auditing events provide visibility into runtime behaviors that would otherwise require intrusive debugging or patching.

You can of course apply profiling in the context of a FastAPI application. In which case you might find [this timing middleware](https://fastapi-utils.davidmontague.xyz/user-guide/timing-middleware/) handy.

However, with this approach, the timing data is logged to stdout. You can use it in development to to find bottlenecks, but in practice looking at the logs in production to get latency information is not the most convenient.

## Available Tools for Application Performance Monitoring (APM)

As will all things, there are many options. Some are open source, some are SaaS businesses.
Most likely you or your organisation are already using one or more monitoring tools, so I'd suggest starting with the one you know.
The tools on the list below don't do only APM, and that's what makes it harder to understand sometimes. Example application monitoring tools you might have heard of:

- [New Relic](https://docs.newrelic.com/docs/apm) (commercial with parts open source)
- [Datadog](https://docs.datadoghq.com/tracing/) (commercial with parts open source)
- StatsD (open source)
- Prometheus (open source)
- OpenTelemetry (open source)

This list is not exhaustive, but let's note [**OpenTelemetry**](https://opentelemetry.io/) which is the
most recent on this list and is now the de-facto standard for application monitoring metrics.

At this point, choosing a tool doesn't matter, let's rather understand what an APM tool does.

## The 4 Steps of Monitoring

![4 steps of monitoring](/images/fastapi-monitoring/4_steps_monitoring.png "The 4 components of monitoring")

1. It all starts with your application code. You instrument your
   service with a library corresponding to your app's language
   (in our case python). This is the `monitoring client library`.
   Monitoring client library examples:

   - [newrelic/newrelic-python-agent: New Relic Python Agent](https://github.com/newrelic/newrelic-python-agent)
   - [DataDog/dd-trace-py: Datadog Python APM Client](https://github.com/DataDog/dd-trace-py)
   - [open-telemetry/opentelemetry-python: OpenTelemetry Python API and SDK](https://github.com/open-telemetry/opentelemetry-python)

2. Then the `monitoring client library` sends each individual call to the `monitoring server daemon` over the network (UDP in particular, as opposed to TCP or HTTP).

3. The `monitoring server daemon` is listening to monitoring events coming from the applications. It packs the incoming data into batches and regularly sends it to the `monitoring backend`.

4. The `monitoring backend` has usually 2 parts: a data processing application and a visualisation webapp. It turns the stream of monitoring data into human-readable charts and alerts.
   Examples:
   - app.datadoghq.com
   - one.newrelic.com

![monitoring backend](/images/fastapi-monitoring/monitoring_backend.png "The 2 components of the monitoring backend")

## The problem with monitoring ASGI webapps

ASGI is a relatively new standard for python web servers. As with
every new standard, **it will take some time for all tools in the
ecosystem to support it**.

Given the 4 steps of monitoring laid out above, a problem arise if
the `monitoring client library` doesn't support ASGI. For example,
this is the case with NewRelic at the moment (see [ASGI - Starlette/Fast API Framework · Issue #5 · newrelic/newrelic-python-agent](https://github.com/newrelic/newrelic-python-agent/issues/5) for more details). I looked at Datadog too and saw that ASGI is also not supported at the moment.

On the open source side of the tools however, OpenTelemetry had
great support for ASGI. So I set out to instrument my FastAPI
service with OpenTelemetry.

_Update - Sep 19th, 2020:_
[There seems to be support for ASGI in ddtrace](https://pypi.datadoghq.com/trace/docs/web_integrations.html?highlight=asgi#asgi)

_Update - Sep 22th, 2020:_
[There is now an API in the NewRelic agent to support ASGI frameworks](https://docs.newrelic.com/docs/agents/python-agent/python-agent-api/asgiapplication-python-agent-api), with uvicorn already supported and starlette on the way.

_Update - Oct 23th, 2020:_
The NewRelic python agent now supports Starlette and FastAPI out of the box.

## Instrumenting FastAPI with OpenTelemetry and Jaeger

OpenTelemetry provides a standard for steps 1 (with `Instrumentors`) and 2 (with `Exporters`) from the 4 steps above. One of the big advantages of
OpenTelemetry is that you can send the events to any monitoring
backend (commercial or open source). This is especially **awesome because you can use the same intrumentation setup for `development`, `staging` and `production` environments**.

_Update - May 30th, 2021:_
[Github is now adopting OpenTelemetry](https://github.blog/2021-05-26-why-and-how-github-is-adopting-opentelemetry/)

Note that depending on the language you use for your microservice, your mileage may vary. For example,
there is no NewRelic OpenTelemetry Exporter in Python yet.
But there are OpenTelemetry Exporters for many others, see the list here: [Registry | OpenTelemetry](https://opentelemetry.io/registry/) (filter by language and with type=Exporter).

One of the available backends is [Jaeger: open source, end-to-end distributed tracing](https://www.jaegertracing.io/). (Note that Jaeger is also a monitoring client library that you can
instrument your application with, but here that's not the part
of interest).

![opentelemetry jaeger](/images/fastapi-monitoring/opentelemetry_jaeger.png "Example instrumentation")

Although it's open source and worked really easily, the issue I had with Jaeger was that **it doesn't have any data pipeline yet**.
This means that, in the visualisation webapp, you can browse traces
but you cannot see any aggregated charts. Such a backend is [on their roadmap](https://www.jaegertracing.io/roadmap/) though.

Still, Jaeger is my goto tool for monitoring while in `development`. See the last part for more details.

## Instrumenting FastAPI with OpenTelemetry and Datadog

I couldn't find any open source monitoring backend with a data pipeline that would provide the features I was looking for
(latency percentile plots, bar chart of total requests and errors ...).

It became apparent that that's where commercial solutions like
NewRelic and Datadog shine. I hence set out to try the OpenTelemtry Datadog exporter.

![opentelemetry datadog](/images/fastapi-monitoring/opentelemetry_datadog.png "Example instrumentation")

With this approach, you get a fully featured monitoring backend
that will allow you to have full observability for your microservice.

The 2 drawbacks are:

- you need to deploy the Datadog agent yourself (with docker or on Kuberetes or on whatever environment fits your stack) and this can get a bit involved
- Datadog being a commercial product, this solution will not be free.
  You will have to pay extra attention to the pricing of Datadog (especially if you deploy the Datadog
  agent to Kubernetes 😈).

## Example FastAPI instrumentation using OpenTelementry, Jaeger and DataDog

So how does it look in the code ?
This is how my application factory looks. If you have any questions, feel free to reach out on [twitter](https://twitter.com/louis_guitton) or open a [github issue](https://github.com/louisguitton/guitton.co/issues).
I will not share my instrumentation because it is specific to my application, but imagine that you can define any nested spans and that those traces will sent the same way to Jaeger or to DataDog. This makes it really fast to iterate on your instrumentation code (e.g. add or remove spans), and even faster to find performance bottlenecks in your code.

```python:main.py
"""FastAPI Application factory with OpenTelemetry instrumentation
sent to Jaeger in dev and to DataDog in staging and production."""
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.exporter.datadog import DatadogExportSpanProcessor, DatadogSpanExporter
from opentelemetry.exporter.jaeger import JaegerSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchExportSpanProcessor

from my_api.config import generate_settings
from my_api.routers import my_router_a, my_router_b


def get_application() -> FastAPI:
    """Application factory.

    Returns:
        ASGI application to be passed to ASGI server like uvicorn or hypercorn.

    Reference:
    - [FastAPI Middlewares](https://fastapi.tiangolo.com/advanced/middleware/)
    """
    # load application settings
    settings = generate_settings()

    if settings.environment != "development":
        # opentelemetry + datadog for staging or production
        trace.set_tracer_provider(TracerProvider())
        datadog_exporter = DatadogSpanExporter(
            agent_url=settings.dd_trace_agent_url,
            service=settings.dd_service,
            env=settings.environment,
            version=settings.dd_version,
            tags=settings.dd_tags,
        )
        trace.get_tracer_provider().add_span_processor(
          DatadogExportSpanProcessor(datadog_exporter)
        )
    else:
        # opentelemetry + jaeger for development
        # requires jaeger running in a container
        trace.set_tracer_provider(TracerProvider())
        jaeger_exporter = JaegerSpanExporter(
            service_name="my-app", agent_host_name="localhost", agent_port=6831,
        )
        trace.get_tracer_provider().add_span_processor(
            BatchExportSpanProcessor(jaeger_exporter, max_export_batch_size=10)
        )

    application = FastAPI(
        title="My API",
        version="1.0",
        description="Do something awesome, while being monitored.",
    )
    # Add your routers
    application.include_router(my_router_a)
    application.include_router(my_router_b)

    FastAPIInstrumentor.instrument_app(application)
    return application


app = get_application()
```

## Conclusion

I hope that with this post you've learned:

- the difference between profiling, monitoring, tracking errors
- the architecture of application monitoring
- some of application monitoring tools out there
- that OpenTelemetry allows you to reuse the same instrumentation setup for all your environments, which speeds up the speed at which you can find performance bottlenecks in your application

I've used this setup to get a 10x speed up on one multi-lingual NLP fastapi service I built at OneFootball.

## Resources

1. [StatsD, What It Is and How It Can Help You | Datadog](https://www.datadoghq.com/blog/statsd/)
1. [Monitoring - Full Stack Python](https://www.fullstackpython.com/monitoring.html)
1. [ASGI | Sentry Documentation](https://docs.sentry.io/platforms/python/guides/asgi/)
1. [Debugging and Profiling — Python 3.9.0 documentation](https://docs.python.org/3/library/debug.html)
1. [Timing Middleware - FastAPI Utilities](https://fastapi-utils.davidmontague.xyz/user-guide/timing-middleware/)
1. [APM | New Relic Documentation](https://docs.newrelic.com/docs/apm)
1. [APM & Distributed Tracing - Datadog](https://docs.datadoghq.com/tracing/)
1. [**OpenTelemetry**](https://opentelemetry.io/)
1. [newrelic/newrelic-python-agent: New Relic Python Agent](https://github.com/newrelic/newrelic-python-agent)
1. [DataDog/dd-trace-py: Datadog Python APM Client](https://github.com/DataDog/dd-trace-py)
1. [open-telemetry/opentelemetry-python: OpenTelemetry Python API and SDK](https://github.com/open-telemetry/opentelemetry-python)
1. [Registry | OpenTelemetry](https://opentelemetry.io/registry/)
1. [Jaeger: open source, end-to-end distributed tracing](https://www.jaegertracing.io/)
1. [Getting Started with OpenTelemetry Python — OpenTelemetry Python documentation](https://opentelemetry-python.readthedocs.io/en/stable/getting-started.html)


In this post I explain how to get visibility into your FastAPI application's performance for both development and production environments.

How to monitor your FastAPI service


From July 23 to July 26, I attended Europython 2020 online.
Here, I will go over my notes.

Disclaimer: I've attended some talks that I've not included here.
Since there were at all times 4 tracks running in parallel, I could not attend 75% of the talks.
For these 2 reasons, these notes reflect my personal Europython experience.

## Typing

- [Static typing in python, Dustin Ingram, Google](https://ep2020.europython.eu/talks/9BZ96n9-static-typing-in-python/)

Python is dynamically typed and can be statically typed.
`mypy` originated out of Dropbox (Jukka Lehtosalo) as a tool to statically check python code.
One key design decision was made when Guido van Rossum (the former [BDFL](https://en.wikipedia.org/wiki/Benevolent_dictator_for_life)
of Python) nudged Jukka Lehtosalo to drop using a typing DSL and instead rely on type hints,
which are now common for python 3.6+ . There are other static type checkers like `pytype` (Google) which has improvements over `mypy`.
Static type checking gives free advantages (IDE completion, documentation, error catching).

- [Fullstack type safety, Szymon Pyżalski](https://ep2020.europython.eu/talks/5muMqs3-full-stack-type-safety/)

The gist of this talk is that even if you enforce types in your python backend code, it's not enough because your stack is made of more than python.
SQL is stricter than python which is stricter than JS.
A type safety stack might look like this:
![fullstack_typesafety](/images/europython-2020/fullstack_typesafety.png)
With a tool like `pydantic`, you can marshall incoming JSON, and can annotate your python code with more complex data types. You can statically check your code with `mypy`.
With tools like `fastAPI` (based on `pydantic`) or `strawberry-graphql` you can generate OpenAPI specifications straight from the type annotations.
With tools like `openapi-generator` you can generate TypeScript classes from an OpenApi spec.

- [Tooling for Static Analysis of Python Programs, Serge Sans Paille, Telecom Bretagne](https://ep2020.europython.eu/talks/3k93S3j-tooling-for-static-analysis-of-python-programs/)

Serge Sans Paille leads a community of pythonistas that are hard at work to make static python analysis suck less.
This artform relies on parsing Abstract Syntax Trees (ASTs) reliably (eg: what your linter does).
To that effect, he provides multiple tools like [`gast`](https://github.com/serge-sans-paille/gast) (which has 160k downloads per day !), [`beniget`](https://github.com/serge-sans-paille/beniget) or [`memestra`](https://github.com/QuantStack/memestra).
This tools was definitely above my level but showed that even if python is not meant for static analysis, you can still create great tooling for it.

## Python best practices gotchas

- [How to write multi-paradigm code, Elias Mistler](https://ep2020.europython.eu/talks/83SnxW9-how-to-write-multi-paradigm-code/)

Combining Object Oriented Programming (OOP) and Functional Programming (FP) is powerful, if done right.
One gotcha was to use [`toolz.functoolz.thread_last`](https://toolz.readthedocs.io/en/latest/api.html#toolz.functoolz.thread_last)
to make code readable from left to right. It takes more time to write but is easier to read, as this quote puts it:

> “I would have written a shorter letter, but I did not have the time.” ― Blaise Pascal

```python
raw_example = "7055862"

# bad
values = []

for digit in raw_example:
    values.append(int(digit))

# better
values = tuple(map(int, raw_example))

# good
from toolz.functoolz import thread_last

values = thread_last(raw_example, (map, int), tuple)

# or
values = [int(digit) for digit in raw_example]
```

Another gotcha was to define idempotent FP functions outside classes and use them inside.

- [Writing Zenlike Python, Jason McDonald](https://ep2020.europython.eu/talks/6Le7GKY-writing-zenlike-python/)

Zen of python is great. One gotcha was to use the cartesian product instead of nested for loops:

```python
# bad
alpha = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'

for r in ('WK'):
    for l1 in alpha:
        for l2 in alpha:
            for l3 in alpha:
                print(f"{r}{l1}{l2}{l3}")

# good
import itertools
alpha = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'

for r, l1, l2, l3 in itertools.product('WK', alpha, alpha, alpha):
    print(f"{r}{l1}{l2}{l3}")
```

## NLP

- [Why Transformers Work and RNNs fall short, Vincent D. Warmerdam](https://ep2020.europython.eu/talks/6tgwXVi-why-transformers-work/)

The intuition behind **self-attention** is that it's a re-weighting mechanism on top of pre-computed word vectors, that account for the fact that language is a sequence. `Bank` has a different meaning in the sequence "money in the bank" than in the sequence "the bank of the river". The simplest attention mechanism you can design is by doing a dot product of the word embeddings, normalise them and add them up to get a context-sensitive embedding.

![Self-Attention Block](/images/europython-2020/attention.png "Self-Attention Block")

You can move this idea into a layer, which gets you a so-called `Self-Attention Block` which turns a sequence of word embeddings into context-sensitive embeddings. It's all fine but at this point you are not learning anything because there are no weights to tune. It's when you add those extra layers of learnable weights to the self-attention block that you get a `Transformer Layer`.

The speaker then goes into how Rasa applies the transformer idea to their business problem of dialogue systems. He mentions DIET (Dual Intent and Entity Transformer) which is their way to label entities and classify intents both at the same time (the really cool part is the shared gradient update). He also mentions TED (Transformer Embedding Dialog) which is they way to decide what action to take for the chatbot. The really cool part there is that **you don't have to use word embeddings to use transformers**: TED is not even using text data but instead, it's using a composite feature space.

<div id="paolo-melchiorre"></div>

- [A pythonic full-text search using only Django and PostgreSQL, Paolo Melchiorre](https://ep2020.europython.eu/talks/iGXPHck-a-pythonic-full-text-search/)

The idea of this talk is that instead of using an external enigne like ElasicSearch (based on Apache Lucene), you can use native PostgreSQL's [Full text search](https://www.postgresql.org/docs/9.5/textsearch.html) feature. Because this can be too low-level, the speaker advises to use a web-framework. His choice is on Django, which has great support for fulltext-search. See [`django.contrib.postgres.search`](https://docs.djangoproject.com/en/3.1/ref/contrib/postgres/search/).

- [NLPeasy - a Workflow to Analyse, Enrich, and Explore Textual Data - Philipp Thomann](https://ep2020.europython.eu/talks/6x7ezDb-nlpeasy-a-workflow-to-analyse-enrich-and-explore-textual-data/)

The speaker presents his library that wraps around spacy, elasticsearch and the python docker client to build an end to end NLP pipeline. This is an interesting workflow for ad-hoc analysis or for small projects where you don't need APIs between ElasticSearch, a stream of article and a language model.
Example python notebooks can be found [here](https://github.com/d-one/NLPeasy-workshop).

## Data

- [Real Time Stream Processing for Machine Learning at Massive Scale, Alex Saucedo](https://ep2020.europython.eu/talks/Ccb6D5Z-real-time-stream-processing-for-machine-learning-at-massive-scale/)

I had seen this talk before. Don't connect to kafak streams with anything other than `faust`.

- [How to be Pythonic? Design a Query Language in Python, Cheuk Ho, TerminusDB](https://ep2020.europython.eu/talks/BiuMv57-how-to-be-pythonic-design-a-query-language-in-python/)

The modelling potential of graph databases in inherently bigger than relational DBs. `TerminusDB` is positioned
at the top of the foodchain. It shines especially to build ontologies or knowledge graphs.
The python client is in the making but tries to be very pythonic.
![graph_dbs](/images/europython-2020/graph_dbs.png)
More details [here](https://terminusdb.com/blog/2020/03/02/why-graph-will-win).

- [A deep dive and comparison of Python drivers for Cassandra and Scylla, Alexys Jacob](https://ep2020.europython.eu/talks/a-deep-dive-and-comparison-of-python-drivers-for-cassandra-and-scylla/)

Really advanced talk about ScyllaDB ([used by Discord](https://www.youtube.com/watch?v=MHI29_c8S0k) for example).
ScyllaDB add a key feature on top of Cassandra which is to shard data by CPU on a node (`shard-per-core`)
instead of "just" by node in a cluster (`shard-per-node`).
For both Cassandra and Scylla, when looking for a data row, you can locate the shard where it's stored by
hashing its `partition key`, which gives you a `token`.
For ScyllaDB, this means that you know which CPU will crunch that row so you can optimise the query plan.
If your client implements this, you can preselect the right node to query in the python cassandra driver,
and hence you save network, and get reduced latency (due to naïvely querying the cluster and then letting the
clustering sort out where the data is).
You also get reduced load on the cluster which means you can achieve the same thorughput with a smaller cluster.

- [Making Pandas Fly, Ian Ozsvald](https://ep2020.europython.eu/talks/A7TniMV-making-pandas-fly/)

First advice to make pandas faster is to save RAM to fit more data. For example, cast raw strings as pandas categoricals.
If you encode low cardinality items well, you can get 1% memory footprint for 15x speed up.
He introduced his tool [`dtype_diet`](https://github.com/ianozsvald/dtype_diet) to automatically shrink your pandas footprint.

Second advice was about dropping to numpy to calculate faster. Some pandas types are not part of numpy (like timedeltas, datetimes etc...).
But you can get a 10x speedup.

Pandas has 2 [recommended dependencies](https://pandas.pydata.org/pandas-docs/stable/getting_started/install.html#recommended-dependencies)
`bottleneck` and `numexpr` that will give you another speedup as well.

Finally, you can wrap your slow function with `@njit` from `numba` but not everything is supported.
Or you could scale out of core by using `dask`.

Ultimately the takeaway was that you should worry about *being* performant (writing unit tests, writing time saving docs, ...)

> Get things right first, and worry about speed later

Ref: [Enhancing performance — pandas 1.0.5 documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html)

## Web and APIs

- [API-schema-based testing with schemathesis, Alexander Hultnér](https://ep2020.europython.eu/talks/8XYc942-api-schema-based-testing-with-schemathesis/)

This talks introduced `property based testing (PBT)` as a great way to surface a spectrum
of defects inherent to any application.
For example, making sure the schema you hand in to clients is valid;
or finding simple unhandled errors or more severe logic errors that
arise from **corner cases** not unit tested for.
In practice, it means you're going to catch some errors in `dev`
before your error tracking tool (e.g. Sentry) reports them to you in `stg` or `prod`.
[`hypothesis`](https://github.com/HypothesisWorks/hypothesis) is a powerful, flexible, and easy to use library
for doing property-based testing in python.
[`schemathesis`](https://github.com/kiwicom/schemathesis) is a tool built on top of schemathesis that generates
and runs test cases from an OpenAPI specification.
Based on the [QuickREST Paper, 20 Dec 2019](https://arxiv.org/abs/1912.09686).
If you have a python application, you can integrate `schemathesis` to your `pytest` test suite.
If you don't have a python app, you can run `schemathesis` over HTTP as long as you can point it to the OpenAPI spec.

- [An ASGI Server from scratch, Philip Jones](https://ep2020.europython.eu/talks/4g2en8S-an-asgi-server-from-scratch/)

The `hypercorn` (ASGI server) creator talked us through the new standard ASGI (as opposed to WSGI).
The main advantages are that the new standard enables **asynchronous** execution.
"Asynchronous" refers to concurrent IO-bound code suing `async`/`await` syntax enabled by the `asyncio` library
(example of IO-bound: HTTP requests; example of CPU-bound: compression).
Another advantage is that you can swap out your ASGI server for another (e.g. `uvicorn` for `hypercorn`).
In order to write an ASGI server, you need to translate HTTP messages to ASGI and back.

- [Flasync Await, David Bordeynik, Nvidia](https://ep2020.europython.eu/talks/AFZwiSK-flasync-await/)

After the web server from the previous talk, this talk is about **asynchronous** web framework alternatives to flask.
Although Flask revolutionised webdev in python, it is synchronous; meaning it can be 3 to 4 times slower than asynchronous alternatives.
The talk introduces `sanic` a webframework very similar to Flask in its API but written with `asyncio`.
The speaker shows how easy it is to convert a flask API to sanic due to how close the APIs were designed to be.
Sanic can be an alternative to `FastAPI` although FastAPI does not just offer async, but also
offers typing based data validation, and puts dependency injection as a guiding principle.

The speaker explains that you should benchmark your own use cases with real data and not trust "fake" speed benchmarks.
The goal being to figure out where are the bottlenecks of your application and how to improve them.
His toolbox for benchmarking is `ab (Apache Bench)`, `pytest-benchmark sometimes`, `cProfile`, `snakeviz`,
`Prometheus` & `Grafana`(to get insights on how physical resources are used when deployed to k8s), [`blazemeter`](https://www.blazemeter.com/).

## Macro Trends

- [HTTP/3 – Why should I care?, Miloslav Pojman, Akamai Technologies](https://ep2020.europython.eu/talks/5m4aNGz-http3-why-should-i-care/)

With HTTP/1.1, you can have 6 connections per domain. This means that clients use a connection pool
to download 6 objects at the same time.
If there are more objects to download, they are queued like in a traffic jam.
With HTTP/2, `multiplexing` was introduced which meant that you could download
multiple objects through 1 connection; while also being limited to 1 connection per domain.
It was designed to speed up latency: you bundle things and don’t need to wait for many connections.

HTTP transfers data over TCP.
`TCP` is a protocol layer that handles network unreliabilities for you.
You can write anything to a TCP socket (email POP3, or FTP, or HTTP).
TCP is also responsible of the `head-of-line blocking` (the traffic jam).

There is a TCP alternative called `QUIC` that comes out of Google.
It gives you a reliable bytestream like TCP.
But it has TLC built-in which means it is encrypted by default.
It also improves performance by leveraging independent bytes streams, thus avouding head-of-line blocking.

`HTTP/3` is HTTP, rebuilt on top of QUIC instead of TCP. It can change the Internet as we know it today.
More time is needed for HTTP/3 to come out of Google and be standardized by IETF, let alone be supported by python libraries likes `httpx`.

- [Extending Python with Rust, Mikhail Medvedev](https://ep2020.europython.eu/talks/6wuE8rA-extending-python-with-rust/)

Just like you would extend python code with C for improving speed, you can also extend python with Rust.
Essentially you turn your Rust code into a **dynamic library** (.dll file on Windows, .so on Linux,
or .dylib on Mac) using a python wrappers like [maturin](https://github.com/PyO3/maturin).
You can build the dynamic library with a docker multi-stage build for example.
You can upload it to PyPI (or a private pip server).

- [Diffprivlib: Privacy-preserving machine learning with Scikit-learn, Naoise Holohan, IBM](https://ep2020.europython.eu/talks/6Js4E4r-diffprivlib-privacy-preserving-machine-learning-with-scikit-learn/)

In order to achieve privacy-preserving machine learning, you can add noise to your training data (concept of "bounds").
There is the idea of fixing a privacy budget for a dataset, and then spending it on training your pipeline (concept of "epsilon").


My notes from attending the 2020 Europython conference online. Typing, NLP, Data, Web and other trends.

Notes on Europython 2020


This is the Kubectl Cheat Sheet I wrote for my team at OneFootball. It is designed **to help you use Kubernetes and Helm in practice.** You will learn how to troubleshoot your deployment and your application.

![Kubernetes Cheat Sheet](/images/kubernetes-cheatsheet/kubernetes_cheat_sheet.png "Kubernetes Cheat Sheet")

## Prerequisites: doublecheck your setup

You will need to **check your kubernetes context**, for example you might want to switch between a staging kubernetes cluster and a production cluster.

You will want to double **check your helm versions** too. One ⚠️ thing to pay attention to is whether you have the same version in your client than in your cluster.

```shell
# Kubectl Alias
alias k=kubectl

# check your context
k config current-context

# switch context
k config use-context <context>

# check versions
helm version

# check you see all releases, if not: it might be due to you using
# a higher helm version than on the cluster
helm ls

# convenience command to permanently save the namespace
# for all subsequent kubectl commands in that context
k config set-context --current --namespace=<your-default-namespace>
```

## Use the Kubernetes Dashboard

The Kubernetes Dashboard is a web-based Kubernetes user interface. While most of the commands in this cheat sheet are about the `kubectl` CLI, **using a GUI can be helpful**, especially when you're starting out.

In particular, you can use Dashboard to get an overview of applications running on your cluster, as well as for creating or modifying individual Kubernetes resources, or troubleshoot your containerized application. For example you can scale a Deployment, restart a pod etc...

```shell
k proxy
```

```shell
open https://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
```

## Debbuging the helm deployment

When you're deploying a new application, things can go wrong for many reasons: the pod can't pull the docker image, or the liveliness probes fail for some reason etc... Here are commands to help you debug your deployment.

```shell
# check Kubernetes logs
# We create objects with `helm`, not with `kubectl` directly.
helm status <my-deployment>
# or get your pod name and describe it
k get po -o wide
k describe po <my-pod>

# if you're facing a CrashLoopBackOff, it's useful to access the logs from the previous (crashed) instance using -p
k logs po <my-pod> -p

# profile CPU and memory consumption on nodes
k describe nodes | egrep 'Name:|Resource' -A2 | grep -v Roles | grep -v Labels

# manually scale down and up the deployment, it will use the latest pod
# useful when pod is stuck due to lack of resources,
# or if your chart change was ambassador-only
k scale deployment <my-deployment> --replicas=0
k scale deployment <my-deployment> --replicas=1
```

## Debugging the deployed application

Once the application is deployed, it can still fail. You then need to investigate the application itself. You can do so by checking its logs or port-forwarding the application to your local machine.

```shell
# check the application logs
k logs -f -l app=<my-deployment>
# or get your pod name and check the logs
k get po -o wide
k logs <my-pod>

# ssh into the pod
k exec --stdin --tty <my-pod> -- /bin/sh

# listen on port 5000 on the local machine and forward to port 8501 on the pod
k port-forward <my-pod> 5000:8501
```

## Debugging Ambassador configs

If you're using a reverse proxy in front of your application, things could fail there too. Here are commands to debug your Ambassador configs.

```shell
# open the Ambassador UI
k port-forward svc/ambassador-private-admin -n ambassador 8877
open https://localhost:8877/ambassador/v0/diag/

# watch Ambassador logs
k logs -f -n ambassador \
    -l release=ambassador-private \
    --max-log-requests 6 \
    -c ambassador
```

## Other Useful Bits

```shell
# Retrieve a k8s secret
kubectl get secrets <my-deployment> \
    -o jsonpath="{.data.POSTGRES_PASSWORD}" | base64 --decode
```

```yml
apiVersion: apps/v1
kind: Deployment  # create a deployment of a container
---
apiVersion: v1
kind: Service  # create a service, that allows us to connect to a given deployment
---
apiVersion: v1
kind: PersistentVolume  # create a "disk" that provides space for pods to store data
```

## References

- [kubectl Cheat Sheet | Kubernetes](https://kubernetes.io/docs/reference/kubectl/cheatsheet/)
- [#Helm #Kubernetes #cheatsheet, happy helming!](https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49)
- [Troubleshoot Applications | Kubernetes](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application)
- [Debug Running Pods | Kubernetes](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/)


Get a Kubectl Cheat Sheet to use Kubernetes and Helm in practice. Troubleshoot your deployment and your application. Updated regularly.

Kubernetes Deployment Cheat Sheet


## Existing Options to Track Football Games

First, you can use any activity tracker like a Garmin watch, a Fitbit or an Apple Watch to track you football game as any other outdoor activity. But the issue is that applications like Strava do not support football specifically. You can still use Strava to track your activity around games [like the Football for Fathers club does](https://footballforfathers.co.uk/players-strava), and you can certainly track your games as well this way. But the stats will be running stats, not football specific (e.g. no top speed, no sprint distance ...).

Others have felt the same frustration. In [Laces Out](https://thelacesout.com/tracking-my-movements-on-the-football-pitch-with-fitbit-872726e99809), Pierre Delarroqua explains how he analyses his data. Out of this, he built a [free webapp](https://strava-football-app.herokuapp.com/) where you can connect with Strava and get the same plots. Unfortunately some functionality seems broken.

There are also options you can pay for. [Playertek](https://playr.catapultsports.com/eu/#) in particular has probably the best offering. In particular, they offer an [Apple Watch App](https://playr.catapultsports.com/eu/apple-watch/) that costs €5.99 per month.

If like me, you remain unsatisfied with those options, follow along.

## What Football do I play

At [OneFootball](https://company.onefootball.com/) we are not all football aficionados but for those of us who want to, we can play quite a bit of football.
We play in the mornings for fun and we play in 2 leagues in the evenings, competing against other companies.

| name          | is_competition | game_length | format | pitch_type | location | can_use_walls |
|---------------|----------------|-------------|--------|------------|----------|---------------|
| Morning footy | False          | 1h          | 5v5    | concrete   | outdoors | False         |
| Medienliga    | True           | 1h          | 7v7    | hybrid     | outdoors | False         |
| Bobofuego     | True           | 1h          | 5v5    | astroturf  | indoors  | True          |

My favorite one is the [Medienliga](https://www.medienkicker.org/) where we compete against other media companies, currently in the second division. Here is the summary of the last game.

### Medienliga / 2. Liga / 5. Spieltag - Monday Sep 23, Prenzlauer Berg

The reporters came in late and missed the first 10 minutes of the 19:15 Medienliga game between Media:net and OneFootball. The blues (Media:net) are sitting in the 7th place of the table while the greens (OneFootball) were sitting at the 5th and hoping to climb.

OneFootball being the best attack of the table with **4.5 goals per game**, the reporters should have known that coming late to the stadium would mean missing goals and eventually picking up the game at the 10' minute with 0-1 OneFootball.

Unfortunately, the next 15 minutes would prove to be a struggle: at 13' an own goal brings the blues back at 1-1, at 23' a defensive error leaves the Media:net striker free to arm a volley and score 2-1. These 2 goals came in  "contre le cours du jeu" (against the odds) as in the 5 minutes before that the attacking duo @Alessandro and @Jack were having 2 good chances.

Fast forward to the second half, and 49' minute when @dominik has to leave his partners, injured, leaving them worried. The ambulance can be heard in the background. The medics come in but it turns out that the ambulance is in fact coming for a player from Media:net. Taking advantage of the confusion, media:net scores 1-3, in an attempt that was not really seen by the reporters.

Only 4 minutes later (53') @dbrown subs in and speeds forward. He makes a beautiful cross to @Alexis who arms a volley and scores 3-2. At that point, with 7 minutes to go, hopes were high.

But in a last counter attack (56') that embodied the game - the blues defending really far back, playing a long ball on their striker and taking their chances - the hopes vanished and the score went up to 4-2.

After the game, players were avoiding the mixed zone, and only @Dani came forward "I think we need some data to analyze our performance; I can't say what happened without a Tableau dashboard."

[Link to videos](https://photos.app.goo.gl/bCeMCVYia3Ea9nUM6)

## How I Track and Visualise my Football Games with Strava

I track my games using my activity tracker and Strava.
I can [download the GPX file from Strava](https://support.strava.com/hc/en-us/articles/216918437-Exporting-your-Data-and-Bulk-Export) and then analyse it using python.

The only manual input was to annotate the corners of the big pitch in the Mauerpark.
I then derive the small pitch location (orange) and the goal posts location (target markers) automatically.
I also detect when the warm up ends to make sure I visualise only my game (the blue dot).

Here is the visualisation of our March 9th game (just before confinement): `OneFootball 3 - 1 Exozet`.

<embed type="text/html" src="/images/football-strava/strava_football.html" className="w-full aspect-video"/>

## Next Steps

On top of those nice visualisations, my idea is to also extract football specific stats from this data.

For example, I want to use Bayesian Statistics (a Hidden Markov Model) to automatically tag
offensive and defensive sprints you do as player. You could have 3 states: sprinting towards
the opponent goal (= `attacking`), sprinting towards your own goal (= `defending`), or not sprinting (= `supporting`).
The nice thing with bayesian statistics is that they wouldn't require as many games for training as supervised learning would.
On the other hand, instead of data labelling, you have to do a thorough modelling
and I still need to think more about how I would model this.

Other than stats, we can also generate more football heatmaps using python. And lastly, you could compare your metrics (distance, top speed, sprint distance, power) to your teammates who independently who have tracked their games on Strava too.

If you're interested by this project, let me know on twitter [@louis_guitton](https://twitter.com/louis_guitton).

## References

1. [Players' Strava | Football for Fathers](https://footballforfathers.co.uk/players-strava)
1. [Tracking my movements on the football pitch with Fitbit | by Pierre Delarroqua | Laces Out](https://thelacesout.com/tracking-my-movements-on-the-football-pitch-with-fitbit-872726e99809)
1. [Strava Football App](https://strava-football-app.herokuapp.com/)
1. [Home | GPS Football Tracker For Players | PLAYR](https://playr.catapultsports.com/eu/#)
1. [The PLAYR Apple Watch App | PLAYR](https://playr.catapultsports.com/eu/apple-watch/)
1. [Exporting your Data and Bulk Export – Strava Support](https://support.strava.com/hc/en-us/articles/216918437-Exporting-your-Data-and-Bulk-Export)


I play football for fun and I like to track my football games using Strava aand my GPS watch. In this post I show you how to generate a nice dataviz with python.

Football Strava


I'm working on Entity Linking and Knowledge Bases.
In that context, exporting a relevant part of Wikidata can be really useful to build surface form dictionaries and coocurence probabilities etc...
In order to know which part of Wikidata is relevant to dump, I thought we could query Wikidata (although it seems we can only download the entire dump and filter afterwards).

`SPARQL` is a language to formulate questions (queries) for knowledge databases.
Therefore you can query Wikidata with `SPARQL`.
At first sight, the syntax is not particularly easy and I've gone through this [tutorial](https://m.wikidata.org/wiki/Special:MyLanguage/Wikidata:SPARQL_tutorial).

## SparQL in 5 minutes

- `#`is the comment character
- The `SELECT` clause lists variables that you want returned (variables start with a question mark)
- The `WHERE` clause contains restrictions on them, in the form of SPO triples (_subject_, _predicate_, _object_), e.g. `?fruit hasColor yellow.`
- On Wikidata, items and properties are not identified by human-readable names like “hasColor” (property) or “yellow” (item).
- Instead, Wikidata items and properties are assigned an identifier, that you need to know beforehand.
  - for items, it's a Q-number, e.g. "yellow" is [Q943](https://m.wikidata.org/wiki/Q943)
  - for properties, it's a P-number, e.g. "hasColor" is [P462](https://m.wikidata.org/wiki/Property:P462)
- you can [search](https://m.wikidata.org/wiki/Special:Search) for the itentifiers using `search term` for items and `P:search term` for properties
- but you should rely on **autocompletion** in [query.wikidata.org](https://query.wikidata.org/) by pressing <kbd>Ctrl</kbd> + <kbd>Space</kbd>
- Finally, you need to include prefix namespaces to query the WQDS (Wikidata Query Service). There are many prefixes, one for each namespace in SPARQL :
  - `wd:` for items
  - `wdt:` for properties, pointing to the object
- to doublecheck what prefix links to what resource, use https://prefix.cc/
- to get more than the Wikidata ID as selectable attributes, you need to include them in the WHERE clause using `SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }`

Putting this all together we get:

```sparql
SELECT ?fruit ?fruitLabel
WHERE
{
  # fruit hasColor yellow
  ?fruit wdt:P462 wd:Q943
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}
```

[Try it](https://query.wikidata.org/#SELECT%20%3Ffruit%20%3FfruitLabel%0AWHERE%0A%7B%0A%20%20%23%20fruit%20hasColor%20yellow%0A%20%20%3Ffruit%20wdt%3AP462%20wd%3AQ943%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%22.%20%7D%0A%7D)

- You can further filter this down by adding more triple conditions using `;` character
e.g. you could filter for actual fruits by doing

```sparql
# fruit instance of or subclass of a fruit
?fruit wdt:P31/wdt:P279* wd:Q3314483;
```

- Advanced filters :
  - `p:` for properties, pointing to the subject
  - `ps:` for property statement
  - `pq:` for property qualifier
- You can abbreviate a lot with the `[]` syntax

```sparql
SELECT ?painting ?paintingLabel ?material ?materialLabel
WHERE
{
  # element is a painting
  ?painting wdt:P31/wdt:P279* wd:Q3305213;
  # extract the statement node 'material' (P186)
            p:P186 [
              # get material property statement
              ps:P186 ?material;
              # 'applies to part'(P518) 'painting surface'(Q861259)
              pq:P518 wd:Q861259
            ].
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}
```

- More grammar by example: `ORDER BY`, `LIMIT`

```sparql
SELECT ?country ?countryLabel ?population
WHERE
{
  # instances of sovereign state
  ?country wdt:P31/wdt:P279* wd:Q3624078;
  # hasPopulation populationValue
           wdt:P1082 ?population.
  # filter for english translations
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}
# ASC(?something) or DESC(?something)
ORDER BY DESC(?population)
LIMIT 10
```

- If you add more variables like population above, the query will filter out
countries that don't have a population value. To fix this, use an `OPTIONAL` clause

```sparql
SELECT ?book ?title ?illustratorLabel ?publisherLabel ?published
WHERE
{
  ?book wdt:P50 wd:Q35610.
  OPTIONAL { ?book wdt:P1476 ?title. }
  OPTIONAL { ?book wdt:P110 ?illustrator. }
  OPTIONAL { ?book wdt:P123 ?publisher. }
  OPTIONAL { ?book wdt:P577 ?published. }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}
```

- `FILTER` and `BIND`, see [tutorial section](https://m.wikidata.org/wiki/Wikidata:SPARQL_tutorial#Expressions,_FILTER_and_BIND) for more details

```sparql
SELECT ?person ?personLabel ?age
WHERE
{
  # instance of human
  ?person wdt:P31 wd:Q5;
          wdt:P569 ?born;
          wdt:P570 ?died;
  # died from capital punishment
          wdt:P1196 wd:Q8454.
  BIND(?died - ?born AS ?ageInDays).
  BIND(?ageInDays/365.2425 AS ?ageInYears).
  BIND(FLOOR(?ageInYears) AS ?age).
  FILTER(?age > 90)
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}
```

[Try it](https://query.wikidata.org/#SELECT%20%3Fperson%20%3FpersonLabel%20%3Fage%0AWHERE%0A%7B%0A%20%20%23%20instance%20of%20human%0A%20%20%3Fperson%20wdt%3AP31%20wd%3AQ5%3B%0A%20%20%20%20%20%20%20%20%20%20wdt%3AP569%20%3Fborn%3B%0A%20%20%20%20%20%20%20%20%20%20wdt%3AP570%20%3Fdied%3B%0A%20%20%23%20died%20from%20capital%20punishment%0A%20%20%20%20%20%20%20%20%20%20wdt%3AP1196%20wd%3AQ8454.%0A%20%20BIND%28%3Fdied%20-%20%3Fborn%20AS%20%3FageInDays%29.%0A%20%20BIND%28%3FageInDays%2F365.2425%20AS%20%3FageInYears%29.%0A%20%20BIND%28FLOOR%28%3FageInYears%29%20AS%20%3Fage%29.%0A%20%20FILTER%28%3Fage%20%3E%2090%29%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%22.%20%7D%0A%7D)

- One can select based on a list of items using `VALUES`

```sparql
SELECT ?item ?itemLabel ?mother ?motherLabel
WHERE {
  # A. Einstein or J.S. Bach
  VALUES ?item { wd:Q937 wd:Q1339 }
  # mother of
  OPTIONAL { ?item wdt:P25 ?mother. }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}
```

- The [Label Service extension](https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Label_service) automatically generates labels as follows:
  - `?xxxLabel` as a shortcut for `rdfs:label`
  - `?xxxAltLabel` as a shortcut for `skos:altLabel`
  - `?xxxDescription` as a shortcut for `schema:description`

## Fun SPARQL queries related to football

Get the 🇳🇱 dutch nicknames of a team:

```sparql
# get the dutch nicknames from Bayern München
SELECT ?item ?itemLabel ?itemDescription ?itemAltLabel
WHERE {
  VALUES ?item { wd:Q15789 }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "nl". }
}
```

[Try it](https://query.wikidata.org/#SELECT%20%3Fitem%20%3FitemLabel%20%3FitemDescription%20%3FitemAltLabel%0AWHERE%20%7B%0A%20%20VALUES%20%3Fitem%20%7B%20wd%3AQ15789%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22nl%22.%20%7D%0A%7D)

Get the stadium names of the teams that are part of the Big 5:

```sparql
SELECT ?item ?itemLabel ?venue ?venueLabel ?venueAltLabel
WHERE
{
  ?item wdt:P31/wdt:P279* wd:Q847017;
        wdt:P118 ?league;
        wdt:P115 ?venue.
  # filter for Big 5
  VALUES ?league { wd:Q82595 wd:Q9448 wd:Q13394 wd:Q15804 wd:Q324867 wd:Q206813}.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}
ORDER BY ?league
```

[Try it](https://query.wikidata.org/#SELECT%20%3Fitem%20%3FitemLabel%20%3Fvenue%20%3FvenueLabel%20%3FvenueAltLabel%0AWHERE%0A%7B%0A%20%20%3Fitem%20wdt%3AP31%2Fwdt%3AP279%2a%20wd%3AQ847017%3B%0A%20%20%20%20%20%20%20%20wdt%3AP118%20%3Fleague%3B%0A%20%20%20%20%20%20%20%20wdt%3AP115%20%3Fvenue.%0A%20%20VALUES%20%3Fleague%20%7B%20wd%3AQ82595%20wd%3AQ9448%20wd%3AQ13394%20wd%3AQ15804%20wd%3AQ324867%7D.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%22.%20%7D%0A%7D)

## Solutions to the tutorial exercices

Here are my solution to the exercises in that tutorial.

### Chemical elements

Write a query that returns all chemical elements with their element symbol and atomic number, in order of their atomic number.

```sparql
SELECT ?element ?elementLabel ?symbol ?atomic_number
WHERE
{
  ?element wdt:P31 wd:Q11344;
           wdt:P246 ?symbol ;
           wdt:P1086 ?atomic_number .
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}
ORDER BY ASC(?atomic_number)
```

[Try it](https://query.wikidata.org/#SELECT%20%3Felement%20%3FelementLabel%20%3Fsymbol%20%3Fatomic_number%0AWHERE%0A%7B%0A%20%20%3Felement%20wdt%3AP31%20wd%3AQ11344%3B%0A%20%20%20%20%20%20%20%20%20%20%20wdt%3AP246%20%3Fsymbol%20%3B%0A%20%20%20%20%20%20%20%20%20%20%20wdt%3AP1086%20%3Fatomic_number%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%22.%20%7D%0A%7D%0AORDER%20BY%20ASC%28%3Fatomic_number%29)

### Rivers that flow into the Mississippi

Write a query that returns all rivers that flow directly or indirectly into the Mississippi River.

```sparql
SELECT ?river ?riverLabel
WHERE
{
  ?river wdt:P31 wd:Q4022;
         wdt:P403/wdt:P403* wd:Q1497 .
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}
ORDER BY ASC(?riverLabel)
```

[Try it](https://query.wikidata.org/#SELECT%20%3Friver%20%3FriverLabel%0AWHERE%0A%7B%0A%20%20%3Friver%20wdt%3AP31%20wd%3AQ4022%3B%0A%20%20%20%20%20%20%20%20%20wdt%3AP403%2Fwdt%3AP403%2a%20wd%3AQ1497.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%22.%20%7D%0A%7D%0AORDER%20BY%20ASC%28%3FriverLabel%29%0A%0A%0A)

### References to Le Figaro website

```sparql
SELECT ?ref ?refURL WHERE {
  ?ref pr:P854 ?refURL .
  FILTER (CONTAINS(str(?refURL),'lefigaro.fr')) .
} LIMIT 10
```

[Try it](https://query.wikidata.org/#SELECT%20%3Fref%20%3FrefURL%20WHERE%20%7B%0A%20%20%3Fref%20pr%3AP854%20%3FrefURL%20.%0A%20%20FILTER%20%28CONTAINS%28str%28%3FrefURL%29%2C%27lefigaro.fr%27%29%29%20.%20%20%20%20%20%20%20%0A%7D%20LIMIT%2010)

## Querying WikiData with SparQL and Python

Now that you have developed a SparQL query, here is the simplest way to programatically query WikiData with python:

```txt:requirements.txt
pandas
requests
```

```python:sparql.py
"""SPARQL utils."""
from pathlib import Path
from typing import List
from urllib.parse import urlparse

import pandas as pd
import requests


def query_wikidata(sparql_file: str, sparql_columns: List[str]) -> pd.DataFrame:
    """Query Wikidata SPARQL API endpoint."""
    wikidata_api = "https://query.wikidata.org/sparql"
    query = Path(sparql_file).read_text()
    r = requests.get(wikidata_api, params={"format": "json", "query": query})
    data = r.json()
    df = (
        pd.json_normalize(data, record_path=["results", "bindings"])
        .rename(columns={c + ".value": c for c in sparql_columns})[sparql_columns]
        .assign(q_id=lambda d: d.item.apply(lambda u: Path(urlparse(u).path).stem))
    )
    return df
```


Learn how to fomulate questions to WikiData using SPARQL.


Learn SPARQL in 5 minutes and use it to query WikiData


## spaCy NER in more than 12 Languages

Let's say you want to do Named Entity Recognition (NER) in Russian or Swedish. You first look at NLP APIs.

Unfortunately, **cloud providers only provide NLP models in a few languages**:

- AWS Comprehend covers 12 languages; neither Russian or Swedish are supported [(source)](https://docs.aws.amazon.com/comprehend/latest/dg/supported-languages.html#supported-languages-feature)
- Google Cloud Natural Language covers 11 languages; Russian is supported but not Swedish [(source)](https://cloud.google.com/natural-language/docs/languages#entity_analysis)

Fortunately, you can now expand beyond those 12 languages and **tap into language models in more than 70 languages**.

This week, StanfordNLP released v1.0.0 of `Stanza`, their NLP toolkit in 70+ languages. [Paper](https://paperswithcode.com/paper/stanza-a-python-natural-language-processing)

<blockquote className="twitter-tweet" data-lang="en">
    <p lang="en" dir="ltr">Now also available for your spaCy pipeline: <br></br><br></br>spacy-stanza (formerly spacy-stanfordnlp)
        integrates Stanza by <a href="https://twitter.com/stanfordnlp?ref_src=twsrc%5Etfw">@StanfordNLP</a> – with
        support for more languages and NER 🚀<a href="https://t.co/vAvAIXojNm">https://t.co/vAvAIXojNm</a> <a
            href="https://t.co/h0bYGh4tN5">https://t.co/h0bYGh4tN5</a> <a
            href="https://t.co/1xFyd9il1d">pic.twitter.com/1xFyd9il1d</a></p>&mdash; spaCy (@spacy_io) <a
        href="https://twitter.com/spacy_io/status/1239982230203764738?ref_src=twsrc%5Etfw">March 17, 2020</a>
</blockquote>
<script async src="https://platform.twitter.com/widgets.js" charSet="utf-8"></script>

It was previously known as `StandfordNLP` and apparently went through a little renaming.
Stanza has a meaning borrowed from poetry and starts with "stan" like Stanford I guess:

> In poetry, a stanza is a grouped set of lines within a poem, usually set off from other stanzas
> by a blank line or indentation. Stanzas can have regular rhyme and metrical schemes,
> though stanzas are not strictly required to have either.

What's even more awesome is that Explosion quickly followed with the release of their `spaCy` wrapper to `Stanza`.

```shell
pip install spacy-stanza
```

So in the rest of this article, I will show how to do NER in Russian and Swedish using Stanza and spaCy.

## Spacy vs Stanza

In a [Hacker News post](https://news.ycombinator.com/item?id=22683184) from March 25,
spaCy's author himself writes :

> Our models are quite a bit behind state-of-the-art atm because we're still optimized for CPU.

He then explains the design choices spaCy made in more details before concluding:

> The bottom-line anyone should care about is, "Am I likely to see **a difference in accuracy
> between Stanza and spaCy** on my problem". At the moment I think the answer is "yes".
> (Although spaCy's default models are still cheaper to run on large datasets).
>
> We're a bit behind the current research atm, and the improvements from that research are
> definitely real. We're looking forward to releasing new models, but in the meantime you can
> also use the Stanza models with very little change to your spaCy code, to see if they help on your problem.

## Spacy in Russian: How to Find Lemmas or Entities

Let's try to parse a Russian document. I head over to Wikipedia and look for the [Arshavin page](https://ru.wikipedia.org/wiki/%D0%90%D1%80%D1%88%D0%B0%D0%B2%D0%B8%D0%BD,_%D0%90%D0%BD%D0%B4%D1%80%D0%B5%D0%B9_%D0%A1%D0%B5%D1%80%D0%B3%D0%B5%D0%B5%D0%B2%D0%B8%D1%87).

Arshavin is a famous Russian player that I used to follow when he played for Arsenal.

![arshavin](/images/spacy-stanza-russian-swedish/gun__1416482723_arsh.jpg)

```shell
# Russian requires an additional dependency
pip install pymorphy2==0.8
```

```python
>>> import stanza
>>> from spacy_stanza import StanzaLanguage
>>> from spacy import displacy
>>> stanza.download('ru')
Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/master/resources_1.0.0.json: 116kB [00:00, 2.96MB/s]
2020-03-18 14:38:35 INFO: Downloading default packages for language: ru (Russian)...
Downloading https://nlp.stanford.edu/software/stanza/1.0.0/ru/default.zip: 100%|██████████| 591M/591M [09:11<00:00, 1.07MB/s]
2020-03-18 14:47:55 INFO: Finished downloading models and saved to /Users/louis.guitton/stanza_resources.
>>> snlp = stanza.Pipeline(lang="ru")
>>> nlp = StanzaLanguage(snlp)
2020-03-18 15:05:49 INFO: Loading these models for language: ru (Russian):
=========================
| Processor | Package   |
-------------------------
| tokenize  | syntagrus |
| pos       | syntagrus |
| lemma     | syntagrus |
| depparse  | syntagrus |
| ner       | wikiner   |
=========================

2020-03-18 15:05:49 INFO: Use device: cpu
2020-03-18 15:05:49 INFO: Loading: tokenize
2020-03-18 15:05:49 INFO: Loading: pos
2020-03-18 15:05:50 INFO: Loading: lemma
2020-03-18 15:05:50 INFO: Loading: depparse
2020-03-18 15:05:51 INFO: Loading: ner
2020-03-18 15:05:52 INFO: Done loading processors!
>>> text = """Андре́й Серге́евич Арша́вин (род. 29 мая 1981[4], Ленинград) — российский футболист, бывший капитан сборной России, заслуженный мастер спорта России (2008). Выступал на позициях атакующего полузащитника, второго нападающего, плеймейкера. При тренере Гусе Хиддинке был одним из основных игроков сборной России. В 2008 году после чемпионата Европы 2008 года попал в символическую сборную Европы по версии УЕФА, а 2 декабря, в результате голосования на определение лучшего игрока 2008 года и вручение награды «Золотой мяч» по версии журнала «France Football», занял 6-е место, что является лучшим результатом для российского футболиста после распада СССР. Неоднократно признавался лучшим футболистом России в различных опросах. С 5 декабря 2009 года входит в «Клуб 100 российских бомбардиров»."""
>>> doc = nlp(text)
>>> displacy.render(doc, style="ent")
```

![arshavin](/images/spacy-stanza-russian-swedish/arshavin.png 'Russian document parsed with NER entities')

## Spacy in Swedish: How to Find Keywords

Now, let's try to parse a Swedish document. Once again, I head over to Wikipedia and look for the [Ibrahimovic page](https://sv.wikipedia.org/wiki/Zlatan_Ibrahimovi%C4%87).

Ibrahimovic was playing for Paris while I lived there.

![ibra](/images/spacy-stanza-russian-swedish/PSG-s-Zlatan-Ibrahimovic-during-the-First-League-Champion-PSG-Na.jpg)

```python
>>> stanza.download('sv')
Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/master/resources_1.0.0.json: 116kB [00:00, 3.38MB/s]
2020-03-19 07:16:01 INFO: Downloading default packages for language: sv (Swedish)...
Downloading https://nlp.stanford.edu/software/stanza/1.0.0/sv/default.zip: 100%|██████████| 225M/225M [03:36<00:00, 1.04MB/s]
2020-03-19 07:19:43 INFO: Finished downloading models and saved to /Users/louis.guitton/stanza_resources.
>>> snlp = stanza.Pipeline(lang="sv")
>>> nlp = StanzaLanguage(snlp)
2020-03-19 07:49:43 INFO: Loading these models for language: sv (Swedish):
=========================
| Processor | Package   |
-------------------------
| tokenize  | talbanken |
| pos       | talbanken |
| lemma     | talbanken |
| depparse  | talbanken |
=========================

2020-03-19 07:49:43 INFO: Use device: cpu
2020-03-19 07:49:43 INFO: Loading: tokenize
2020-03-19 07:49:43 INFO: Loading: pos
2020-03-19 07:49:44 INFO: Loading: lemma
2020-03-19 07:49:44 INFO: Loading: depparse
2020-03-19 07:49:45 INFO: Done loading processors!
>>> text = """Zlatan Ibrahimović, född 3 oktober 1981 i Västra Skrävlinge församling i Malmö, är en svensk fotbollsspelare. Ibrahimović har tilldelats Guldbollen elva gånger och anses allmänt vara Sveriges bäste fotbollsspelare genom tiderna. Från 2001 till 2016 spelade han i svenska landslaget, där han med sina 62 mål är den främste målgöraren genom tiderna. Under sina 18 år som utlandsproffs har svensken vunnit ligan vid elva tillfällen med fem olika klubbar i fyra länder och blivit skyttekung i italienska Serie A två gånger och i franska Ligue 1 vid tre tillfällen. Ibrahimović är den ende som spelat för sju olika klubbar i Champions League, där han med sina 48 mål också intar en delad niondeplats i skytteligans maratontabell. Hans främsta internationella merit är segern i Europa League med Manchester United 2016/2017. Svenskens övergång från Inter till Barcelona 2009 var den spanska storklubbens dittills dyraste spelarköp (69 miljoner euro). 2015 var Ibrahimović enligt tidskriften Forbes den 55:e bäst betalda kändisen i världen, med en årsinkomst på 39 miljoner dollar. I september 2018 gjorde Ibrahimović sitt 500:e mål och blev därmed en av 28 spelare i fotbollshistorien som gjort minst 500 mål (landslag och klubblag)."""
```

Contrary to Russian, the Swedish `stanza` model doesn't have a NER model (yet). You can double check available
NER models [here](https://stanfordnlp.github.io/stanza/models.html).

But the really cool thing for Swedish is that noun chunks rules are available in spacy (not the case for Russian)

> Tip: To check if noun chunks are available for one language in spacy, look for a `syntax_iterators.py` file
> in the language folder (eg `spacy.lang.sv`)

Why does it matter whether a document has noun chunks or not ? Well, if `doc.noun_chunks` are available,
you can then use the awesome [pytextrank](https://github.com/DerwenAI/pytextrank) library to extract
the top keywords.

```shell
pip install pytextrank
```

```python
>>> tr = pytextrank.TextRank()
>>> nlp.add_pipe(tr.PipelineComponent, name="textrank", last=True)  # add PyTextRank to the spaCy pipeline
>>> doc = nlp(text)
```

After writing a couple of convenience functions, here are the top 5 TextRank keywords highlighted in HTML:

![zlatan](/images/spacy-stanza-russian-swedish/zlatan.png 'Swedish document parsed with top 5 keywords')

## Resources

1. [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages | Papers With Code](https://paperswithcode.com/paper/stanza-a-python-natural-language-processing)
1. [You can also try out Stanza in spaCy --- Ines updated the spacy-stanfordnlp wrap... | Hacker News](https://news.ycombinator.com/item?id=22683184)
1. [Models - Stanza](https://stanfordnlp.github.io/stanza/models.html)
1. [DerwenAI/pytextrank: Python implementation of TextRank for phrase extraction and summarization of text documents](https://github.com/DerwenAI/pytextrank)
1. [Languages Supported in Amazon Comprehend - Amazon Comprehend](https://docs.aws.amazon.com/comprehend/latest/dg/supported-languages.html#supported-languages-feature)
1. [Language Support | Cloud Natural Language API | Google Cloud](https://cloud.google.com/natural-language/docs/languages#entity_analysis)


I demo how you can use spaCy in 70+ languages by leveraging Stanza, a NLP toolkit by StandfordNLP. I give spaCy NER examples in Russian and Swedish.

How to use spaCy language models in Russian or Swedish


## Querying Athena in python

Getting data from a datalake or an athena instance is not so hard.

Create a virtual environment with the following libraries:

```txt:requirements.txt
jupyterlab
dask[dataframe]
s3fs
PyAthena[SQLAlchemy]
fastparquet
```

Query Athena:

```python:athena_client.py
import pandas as pd
from pyathena import connect

# https://pypi.org/project/PyAthena/
conn = connect(
    aws_access_key_id="<key>",
    aws_secret_access_key="<secret>",
    s3_staging_dir=
    's3://<bucket>/<folder>/',
    region_name='eu-west-1'
)
df = pd.read_sql("SELECT * FROM <database>.<table> limit 10;", conn)
```

Query the datalake:

```python:datalake_client.py
import dask.dataframe as dd

# https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.read_csv
df = dd.read_csv(
    "s3://<bucket>/databases/<database>/<table>/<file>.csv.gz",
    compression='gzip'
)

# https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.read_parquet
df = dd.read_parquet(
    "s3://<bucket>/databases/<database>/<table>/<file>.parquet"
)
```

## How to optimise Athena (speed and cost) to unlock backend serving

On Friday, I had the idea of using Athena as a serving layer for a Machine Learning backend application of mine. The motivation was that by [querying Athena in python](#querying-athena-in-python), it was drastically reducing the amount of logic I had to write (= backend code) and also reducing the infrastructure blueprint of the application (= no database to spin up and maintain - PostgreSQL or DynamoDB). But after implementing quick wins like caching, I still hit a road block where each Athena query takes 7 seconds and scans 10 GB of data.

After a quick Google search, I found out that [this approach is what companies like SimilarWeb do](https://similarweb.engineering/athena-serving-layer/) and that there are further tricks to optimise Athena. I then proceeded into a rabbit hole of Athena, Parquet and ORC.

Notes that start with *[In theory]* are things I haven't tried and collected here for completeness.

### Notes

![Optimise Athena](/images/athena-serving-layer/athena-serving-layer.png)

1. Athena's pricing and performance (= speed) is indexed on the **amount of data scanned** for each query.

1. [Using **columnar storage formats** (Parquet or ORC) is the first improvement step](https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html). They are great because they are column based (great for intensive read queries), have efficient compression (due to column based compression).

    ```python
    df.write.parquet("s3://users.parquet")
    ```

1. How to choose between Parquet and ORC: if you don't have many nested data types, Parquet might be unnecessary. There is a tooling affinity for each format though: Hive and ORC, Spark and Parquet, Kafka and Avro... So given that ORC and Parquet are sensibly similar *at my scale* (note: see point 9 on predicate pushdown), I'd go for Parquet as it's the most convenient when using Spark.

1. **Discard rows and columns that are known to be irrelevant**. For example, if we have a long-tail distribution, consider throwing away those rows that are unlikely to be queried.

1. **Denormalise** the data as much as possible (= load after joins). This will inflate the data (hence increase cost) but will increase performance.

1. Athena support Hive **partitions** so they remain the best way to restrict the amount of data scanned by each query (= [use physical partitions in folders](https://docs.aws.amazon.com/athena/latest/ug/partitions.html)).

    ```python
    df.partitionBy("year", "month", "day").write.parquet(...)
    ```

1. Always turn on Google **snappy compression**. (It's [already the default](https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#configuration) for parquet when using spark)

    ```python
    df.write.option("compression", "snappy").parquet(...)
    ```

1. Partitioning has a cost. As the number of partitions in your table increases, the higher the overhead of retrieving and processing the partition metadata, and the smaller your files. Partitioning too finely can wipe out the initial benefit. In those cases, look at **bucketting** in Hive. Choose a column with high cardinality, frequently used for filtering, like a primary key or user_id.

    ```sql
    CREATE TABLE users_bucketted
    WITH (
        format = 'PARQUET',
        parquet_compression = 'SNAPPY',
        bucketed_by = ARRAY['user_id'],
        bucket_count = 100)
    AS SELECT user_id, user_name, country
    FROM users_not_bucketted;
    ```

    > Note 1: Although spark has a bucketting feature, Athena doesn't support it; so we're forced to use either Hive or a `CREATE TABLE AS` statement in Athena
    > Note 2: choose the number of buckets based on the optimal file size for Athena (= don't go below 128 MB per file, unless you never aggregate)

1. *[In theory]* A (more advanced) feature of Parquet and ORC to not overlook is the use of column indexes, or **`Predicate pushdown`**. This combined with partitions can further improve performance. When an Athena query obtains specific column values from your data, it uses statistics from data block predicates, such as max/min values, to determine whether to read or skip the block.

    > Note: it's unclear if ORC indexing on columns is better than Parquet. Since Dec 2018, Parquet [now supports column indexes](https://issues.apache.org/jira/browse/PARQUET-1201) too. It's unclear if Spark as a backend for Parquet is implementing Parquet column indexes. If it isn't, then it could mean that switching from Parquet to ORC and leverage column indexes would further improve performance and cost.

    ```python
    df.repartition(100, "user_id").write.parquet(...)
    ```

1. One way to optimize the number of blocks to be skipped is to **identify and sort by a commonly filtered column** before writing your ORC or Parquet files. This ensures that the range between the min and max of values within the block would be as small as possible within each block. [This gives it a better chance to be pruned](https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/).

    ```python
    df.sort("user_id").repartition(100, "user_id").write.parquet(...)
    ```

1. *[In theory]* ORC specific notes.

    ```python
    # turn on indexes in the spark config in ORC
    sqlContext.setConf("spark.sql.orc.filterPushdown", "true")
    ```

    ```python
    # turn on bloom filters
    df.write.option("orc.bloom.filter.columns", "user_id").parquet(...)
    ```

1. To check incremental improvements, one would not rely on a Google search like I did, but instead [use table statistics with `ANALYZE TABLE`](https://hadoopsters.com/2017/12/19/how-to-build-optimal-hive-tables-using-orc-and-metastore-statistics/)


Steps to follow to use AWS Athena as a serving layer for a backend service.

Athena serving layer


## Turnkey dockerised setup for an HTTPS Flask API with SSL certificates included

I was wrapping things up for my freelance clients before holidays, and I had to migrate a dockerized flask API to HTTPS.

I knew how to do it with nginx + python on the server, but I researched a reproducible way to do it with nginx + docker.

Then I started pulling the thread and first creating a docker-compsoe setup for nginx + the flask API.  But then you still need to manage your SSL certificates “manually” on the server with certbot.

I ended up with something pretty cool that I can reuse from now on: a docker-compose setup with certbot + nginx + flask, all in one.

## Reusable Solution

The following gist is made of 3 parts:

1. a docker compose setup sith `nginx`, `certbot` and `flask`
1. a setup shell script that manages the SSL certificates generation
1. a nginx configuration supporting HTTPS

You will need to tweak the hardcoded values that point to my service `startup-meter.louisguitton.com`.

[Nginx + HTTPS + Docker | GitHub Gist](https://gist.github.com/louisguitton/6682f1190ef594b82f5804c2e81c50ac)

## References

1. [Nginx and Let’s Encrypt with Docker in Less Than 5 Minutes | by Philipp | Medium](https://medium.com/@pentacent/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71)
1. [How to setup your website for that sweet, sweet HTTPS with Docker, Nginx, and letsencrypt](https://www.freecodecamp.org/news/docker-compose-nginx-and-letsencrypt-setting-up-website-to-do-all-the-things-for-that-https-7cb0bf774b7e/)


In this post, I show my setup for a dockerised HTTPS Flask API. It generates and renews the SSL certificates automatically and includes the nginx config.

A containerised HTTPS Flask API


I left Paris and France in mid 2016 after graduation to come live in Berlin. I have never lived in Lyon, and to be honest I don't know the city really well. But last weekend, I attended a wedding there and I was really surprised by Lyon's love for electric scooters.

## Trotinnette or Electric Scooter

![What is a scooter?](/images/trotinnettes/1_wKR-663OdGwD3wawK3q5iw.png 'Google search scooter images')_What is a scooter?_

Before we carry on, I must admit my confusion as a Frenchman around the usage of the word scooter. For Google — and other American English native speakers I'm sure — [scooter means two things at once](https://dictionary.cambridge.org/dictionary/english/scooter), while in french we have two different words for them. There is on the one hand the “_light two-wheeled motor vehicle smaller than a motocycle_” (**scooter** in french) and on the other hand the “_child’s toy consisting of a footboard and a long steering handle_” (**trottinette** in french). In the rest of this article, I will actually discuss Lyon’s love for electric trottinettes.

## Lyon is not the only one

Mobility is a really interesting space in 2019. In the last few years, I've seen bike sharing (_Velib_ in Paris) and car sharing (_Autolib_ in Paris) evolve into services where you can park anywhere (_Mobike_ and _Drive Now_ in Berlin respectively). Last year in Berlin electric scooters (as in the french scooter) popped up too (_Coup_ in Berlin). The job market followed the trend and suddenly I had friends working for ride-sharing platforms (_Free2Move_ in Berlin). But it's in Lyon that I properly discovered electric trottinettes.

Yes, people use them a lot. Yes, people drive them with no concern for pedestrians' security, using sidewalks or streets at their convenience. I had heard of all that from other cities. But the really mind blowing thing was that in a matter of 30 minutes, I could count not less than seven different e-trottinette brands, and I probably missed one or two.

## Plethora of Brands

[Dott](https://ridedott.com/), [Lime](https://www.li.me), [Wind](https://www.wind.co/), [Voi](https://www.voiscooters.com/), [Tier](https://www.tier.app/), [Flash](https://goflash.com/), [Bird](https://www.bird.co/) … Here is a small selection of pictures and best bits that I found on their websites.

> [Dott] Our electric scooters do not look like any other! They are designed with different colour combinations to better fit into the urban landscape.

<div className="grid grid-cols-3 gap-x-8">
  <img
    src="/images/trotinnettes/1_QeSUR4zn6jXVLzp7OoxeBg.png"
    alt="Dott electric scooters"
  />
  <img
    src="/images/trotinnettes/1_Q5ttT0Pt0Y2F4ojWsdiLIg.png"
    alt="Lime electric scooters"
  />
  <img
    src="/images/trotinnettes/1_dZGcAUE880NNbT7YFNDMyQ.png"
    alt="Tier electric scooters"
  />
</div>
_The physical products look (almost) identical_

Indeed, they don’t look like each other at all. Some of these brands make bold statements like “_reimagining smart mobility_” but to me it seems rather that we’re **reinventing the wheel**. I’ll acknowledge here that I have not downloaded any of the 7 apps, and thus I am commenting only the physical part of those products, ignoring the fact that those brands may differentiate themselves via the app UX, or the scooter’s autonomy…

Apart from the products looking almost identical, I was also surprised by the branding overdose I had while walking around in Lyon. Flashy colours, catchy (english) names (“_Bird_” and “_Wind_” really ?!). Later, when I came home and looked for the brands’ websites, I was not disappointed. The domain names are “trendy”: _tier.app_, _bird.co_ & _wind.co_ and of course _li.me_ — kudos for fitting a pun into your domain name. To be fair, three brands had .com websites and one even looked “normal” (_voiscooters.com_) but two of them had to add a verb : _ridedott.com_ and _goflash.com …_

Beyond the domain names, I was also not disappointed by the landing pages themselves, and of course their traditional stock images.

<div className="grid grid-cols-2 items-center gap-x-8">
  <img src="/images/trotinnettes/0_CblneF2qHtwzY4GC.jpg" alt="Wind electric scooters" />
  <img src="/images/trotinnettes/1_wK6BWcx96yg-OQFi6uUroQ.jpeg" alt="Bird electric scooters" className="row-span-2"/>
  <img src="/images/trotinnettes/1_R1RqptZdSF6VK46dHnoWnw.jpeg" alt="Voi electric scooters"/>
</div>
_Startupy branding overdose_

Enough with the sarcasm. I actually had quite some fun walking in Lyon and counting the e-scooter brands. And comparing their websites was a great exercise. Indeed, what would **you** do if you had to release one more of these brands?

## What Would be the Branding of Your E-Trottinette Startup

As a parting thought, I can’t help but think about these pictures from bike-sharing cemeteries in China. The (probably upcoming) concentration of the e-trotinette market could have the hidden side-effect of creating similar e-trotinette cemeteries.

<div className="grid grid-cols-2 gap-x-8">
<img src="/images/trotinnettes/1_46xNjluwGwL567IuwJMyGA.jpeg" alt="Bike sharing cemetery"/>
<img src="/images/trotinnettes/1_ByxwryGKq6TCZkGsA7MNug.jpeg" alt="Bike sharing cemetery"/>
</div>

_No, those are not flowers, but bicycles in China_

As of writing, e-scooters are forbidden in Berlin. But [law is in the making](https://www.berlin.de/special/auto-und-motor/nachrichten/5560401-2301467-e-scooter-elektro-tretroller-verleih.html) to allow them. It will be interesting to see **if the e-scooter Cupid shots Berlin like he shot Lyon**.


In mid 2019, a tript to Lyon made me realise the rise of electric scooters there. In total, 7 brands were offering the same service.

Lyon's love for electric scooters


Back in 2015, I was living in Brighton, UK, and was working at EDF Energy as a R&D Engineer. The achievement I'm most proud of from that time is something called #PrettyCurious.

## Pretty Curious

Behind that enigmatic hashtag stands a [nationwide program](https://www.edfenergy.com/prettycurious) that EDF Energy kickstarted when I was there.

> Pretty Curious aims to inspire teenage girls to imagine a future where they use STEM – science, technology, engineering and maths - to help make a difference.

[Charles Delalonde](https://www.linkedin.com/in/cdelalonde/), [David Ferguson](https://www.linkedin.com/in/davidfergusonuk/), [Christopher Weeks](https://mrcweeks.com/), [Alastair Byrne](https://www.linkedin.com/in/alastairbyrne/) and myself set out to build hackathons in which girls aged 10 to 15 would come in and learn about electronics, 3D printing, laser cutting and coding while working in groups to invent, develop and pitch products with our help along the way.

We got to replicate that format in 6 different cities in that first year.

<iframe src="https://www.google.com/maps/d/embed?mid=13-sG1idRDPhVgAVLREFB6iayZ6LRk-bn" className="w-full aspect-video"/>

I personally was in charge of developing the different electronics activities and then running the workshops as a facilitator.

## London

<YoutubeEmbed embedId="o3dPmcKPPW8"/>

## Glasgow

<YoutubeEmbed embedId="6tqRe32PU6Q"/>

Looking back at those videos always brings back so many good memories and I do hope we inspired a few vocations.


Teach young girls electronics by running hackathon across the UK.

#PrettyCurious - Electronics Hackathons


## Project Management

Back in engineering school, during the scholar year of 2013-2014, I led a 25 people team to build a robot from scratch with a 2000€ budget. Our client was a [parisian robotics laboratory](https://www.isir.upmc.fr/index.php?op=view_page&id=2&menuid=1&old=N&lang=en).

That project was a year-long class part of my engineering curriculum. The goal of the class was to combine electronics and mechanics with waterfall project management, thus giving us unforgettable memories.

Being the project manager, I had to gather the specifications, derive the Work Breakdown Structure, build a Roadmap, conduct the functional analysis, handle stakeholders and then engage the team weekly to make sure we would deliver on time.

## Mechanics

The mechanics team worked on dimensioning the motors, designing the robot in CAD software, and 3D-printing and assembling the parts.

<div className="grid grid-cols-2 gap-x-4">
  <img src="/images/mecatronics/snakeboard_model.png" alt="The simplified model of a snakeboard" className="h-12"/>
  <img src="/images/mecatronics/snakeboard_assemblage.jpg" alt="Assemblage of snakeboard pieces" className="h-12"/>
  {/* for some reason col-span-2 in tailwindCSS doesn't work */}
  <img src="/images/mecatronics/snakeboard_cad.jpg" alt="CAD software" className="h-12" style={{'grid-column': "span 2 / span 2"}}/>
</div>

## Electronics

The electronics team worked on wireless communications using Xbee and Arduino, dimensioning the batteries, controlling the motors from the Arduino micro-controller, selecting sensors (optical sensors and inertial measurement unit), and finally designing, printing and soldering the 3 PCBs (printed circuit boards).

![PCB](/images/mecatronics/snakeboard_electronics.jpg)

## Movie

Here is the final movie (in french) a few weeks before demo-day. In that video, the snakeboard is not finalised but we finished on time and delivered the robot to the research lab.

<YoutubeEmbed embedId="PJ_oETjYZow"/>


The overview of my Master's Mecatronics project where I lead a 25 people team in design and fabrication of a snakeboard robot.

Louis Guitton - Tech Lead NFT and Principal Machine Learning Engineer at OneFootball

Most Read Articles