QuantStack

Animating a community

Around a open source software

QuantStack
Avatar picture

Johan Mabille

  • Scientific software developer @QuantStack
  • Adjunct Faculty @Dauphine
  • Formerly quant developer at HSBC
  • Co-author of xtensor, xsimd, xeus
fernando
My advisor had a heavily customized awk/sed/bash workflow to manage job submissions and postprocessing of C codes for supercomputing runs... So I used her scripts to run my jobs, and on top of that has added my own layer of Perl, plus a hefty amount of Gnuplot, IDL and Mathematica.

- Fernando Perez

Objectives of the tools @QuantStack

Simplify workflows of scientific software users

We are

  • Core developers of project Jupyter
  • Core developers of the xtensor stack

What is xtensor?

  • A C++ template library for multi-dimensional array manipulation

    xtensor
    • Followings the idioms of the C++ STL

      (iterator pairs, clear value semantics)

    • But also an API similar to that of numpy

What is xtensor?

  • Python bindings to enable xtensor APIs on numpy arrays.

    xtensor-python
  • Julia bindings to enable xtensor APIs on Julia arrays.

    xtensor-julia
  • R bindings to enable xtensor APIs on R arrays.

    xtensor-r
  • Cookiecutter projects for authoring of Python, Julia, and R extensions.

    xtensor-cookiecutter

What is xtensor?

  • BLAS bindings to enable BLAS operations on xtensor expressions.

    xtensor-blas
  • SIMD acceleration kernels.

    xsimd
  • All open-source (BSD License).

Ever heard of numpy ?

Python 3 - numpy

C++ 14 - xtensor


                                    np.array([[3, 4], [5, 6]])

                                    arr.reshape([3, 4])
                                

                                    xt::xarray<double>({{3, 4}, {5, 6}})
                                    xt::xtensor<double, 2>({{3, 4}, {5, 6}})
                                    arr.reshape({3, 4});
                                

                                    np.linspace(1.0, 10.0, 100)
                                    np.logspace(1.0, 10.0, 100)
                                    np.arange(3, 7)
                                    np.eye(4)
                                    np.zeros([3, 4])
                                    np.ones([3, 4])
                                

                                    xt::linspace<double>(1.0, 10.0, 100)
                                    xt::logspace<double>(1.0, 10.0, 100)
                                    xt::arange(3, 7)
                                    xt::eye(4)
                                    xt::zeros<double>({3, 4})
                                    xt::ones<double>({3, 4})
                                

Python 3 - numpy

C++ 14 - xtensor


                                    a[:, np.newaxis]
                                    a[:5, 1:]
                                    a[5:1:-1, :]
                                    np.broadcast(a, [4, 5, 7])
                                    np.vectorize(f)
                                    a[a > 5]
                                    a[[0, 1], [0, 0]]
                                

                                    xt::view(a, xt::all(), xt::newaxis())
                                    xt::view(a, xt::range(_, 5), xt::range(1, _))
                                    xt::view(a, xt::range(5, 1, -1), xt::all())
                                    xt::broadcast(a, {4, 5, 7})
                                    xt::vectorize(f)
                                    xt::filter(a, a > 5)
                                    xt::index_view(a, {{0, 0}, {1, 0}})
                                

                                    np.sum(a, axis=[0, 1])
                                    np.sum(a)
                                    np.prod(a, axis=1)
                                    np.prod(a)
                                    np.mean(a, axis=1)
                                    np.mean(a)
                                

                                    xt::sum(a, {0, 1})
                                    xt::sum(a)
                                    xt::prod(a, {1})
                                    xt::prod(a)
                                    xt::mean(a, {1})
                                    xt::mean(a)
                                

Python 3 - numpy

C++ 14 - xtensor


                                    np.where(a > 5, a, b)
                                    np.where(a > 5)
                                    np.any(a)
                                    np.all(a)
                                    np.logical_and(a, b)
                                    np.logical_or(a, b)
                                

                                    xt::where(a > 5, a, b)
                                    xt::where(a > 5)
                                    xt::any(a)
                                    xt::all(a)
                                    a && b
                                    a || b
                                

                                    np.absolute(a)
                                    np.exp(a)
                                    np.sqrt(a)
                                    np.cos(a)
                                    np.cosh(a)
                                    scipy.special.erf(a)
                                    np.isnan(a)
                                

                                    xt::abs(a)
                                    xt::exp(a)
                                    xt::sqrt(a)
                                    xt::cos(a)
                                    xt::cosh(a)
                                    xt::erf(a)
                                    xt::isnan(a)
                                

The Jupyter Architecture

Jupyter Architecture
  • A well defined protocol
  • Implemented for more than 40 languages

Implementations of the protocol

  • Some kernels have their own implementation of the protocol
    • Python (ipykernel)
    • R (IRkernel)
    • Julia (IJulia)
  • Many others make use of the wrapper kernel approach
    • The Octave and Matlab kernels

On the wrapper kernel approach

  • Dependency on the Python runtime
  • The wrapped interpreter may not have a usable Python API
  • We may need to expose the API of the wrapper to enable widgets

In fact, a kernel is merely an executable implementing an inter-process communication protocol. It should not need a Python runtime.

xeus
  • A modern C++ implementation of the Jupyter protocol
  • Enabling kernel authors to create new kernels without having to deal with the protocol.

Example: Implementing an echo kernel

echo-kernel

What can you do with Xeus?

A kernel for the C++ programming language based on cling

root-logo
Avatar picture

Axel Naumann

Avatar picture

Vassil Vassilev

xtensor and xeus developers

  • QuantStack
  • Ullrich Koethe, Loic Gouarin
  • Patrick Bos
  • Ghislain Antony Vaillant

xtensor and xeus developers

  • Academics
  • Private companies
  • Aligned interests

Adoption of xtensor

  • ~25k downloads on anaconda cloud
  • tiny-dnn
  • vigra (code developer conributing)

Adoption of xeus

  • Very new
  • Deployed at Saclay and Dauphine
  • ~400 students working on a beta

Building and animating the community

  • Github issues
  • Gitter
  • Twitter
  • Conferences

Larger role in the community

  • PyData Paris organizers
  • Leadership in Jupyter project

Community values

  • Inclusiveness
  • Be positive
  • Develop in the open

Develop in the open

  • Clear decision making (no important decision left behind the doors)
  • No private communication channels

Business model

  • Not a freemium
  • Not a dual licensing
  • BSD: impossible to revert

Business model

  • Sponsored development + features in OS projects we are associated with
  • We bet on high adoption of xtensor and xeus

Business model

  • Find more sponsors
  • Get grants from public institutions
  • Get funding from non-profit organizations