Mise en avant

A thousand deployments later…

A bit more than four years after its inception, two years after its real start, a thousand deployments, about 10 000 commits, more than 1200 unit and integration tests, more than 100 REST API endpoints, Arcsecond.io is approaching v1.0. What a journey, again!

The more I code arcsecond.io, the more I love it, and feel confidence for the years to come. Since the beginning, I knew this endeavour would last for multiple years, and not just be a hobby project. After all, I’ve been developing a macOS app called iObserve for more than 8 years. Thus, I built Arcsecond.io with these long-term objectives: maximum reliability, permanent deployment capability, permanent clean and tested code. And of course: utmost care for the feedback of users and astronomers worldwide. For this, I knew I will have to learn and master whatever it needs, whatever its complexity or technological layer, and develop and master the permanent operations of a pretty nice full-stack software system. This article explains the important choices in the technical stack used in arcsecond.io.

Let me start by showcasing a bit one thing I am pretty proud of today. Although I am very happy to have ported the core of my macOS app iObserve on the web (check it out), one of the coolest feature of arcsecond.io today is Worldwide Observing Activities, that is, the possibility to know in real-time what is being observed in the (largest but not only) observatories in the world, and some scientific satellites!

Live observing activities occuring worldwide and in space!

Even better: you can subscribe to it, and be notified right away if something matches your subscription. Say, you want to be notified when the HST starts an observation, or when an Programme is being performed for a given observer, or when a telescope instrument is being used, or when a region of the sky is being observed, etc. Boum. You’ve got mail! How cool is that?

  • Example of a mail for a SWIFT observation
  • Example of a mail for a La Silla Observatory observation

To achieve this result, it is a long effort, and a rather lengthy list of decisions, optimizations, trials and errors. But the result is what I wanted: even if the user experience is not perfect (some links aren’t always here, aren’t always pointing to complete info etc), its potential is clearly enormous. Now, I have to « fill » it with more connectors.

Ok, let’s dive in. We’ll start with the backend, then move on with the infrastructure, the frontend and finally the code organisation.

The backend

Ok, not the most beautiful webpage of the Internet. But its beauty is what’s inside.

The backend server is written with Django 2.2 and Python 3.7 Originally, arcsecond.io started with Django 1.5 or something, and Python 2. The migration to Python 3 was made more than a year ago, and it was clearly a good choice. Almost no third-party libraries broke during the migration. At the beginning, I evaluated the possibility to use Flask instead of Django. But Django comes with a lot of things built-in, in particular its automatic admin portal, which proved to be invaluable over time.

The design choice of the backend was to make a RESTful purely data-centric server. Hence, the rather obvious choice was to use the fantastic Django REST Framework. It is an essential part of arcsecond.io. It is not the purpose of this article to explain REST principles, but simply let me say it is a mindset, and the DRF allows you to deploy this way of doing with lots of ease.

One of the key reason of using Python in the backend was obviously its seamless integration with scientific and astronomical libraries. In particular, it wouldn’t be possible without AstroPy (and its affiliated packages) and NumPy. Once arcsecond.io is making some benefits (hopefully some day!), I will redirect part of it to help fund these projects.

The projects declares 52 direct dependencies (among which django-cors-headers for managing HTTP headers easily, django-cryptography to encrypt some model fields, django-otp to protect the production admin route with 2FA, django-allauth and django-rest-auth to handle the complex process of auth, django-storages to handle multiple storages remotely, psycopg2 to talk to the PostgresQL DB, requests and pyvo to ease the writing of the multiple connectors…).

More importantly, the backend relies oncelery and channels (with the ASGI daphne server), two libraries of crucial importance.

Celery is a distributed task queue. When I was a newbie at the very beginning of this full-stack journey, I asked myself: Why would I need such a thing? Until I realised that a lot, if not most, of the activity of a backend server could be different from serving HTML pages or serialized data. In arcsecond.io the activity is not stable enough to say if the background Celery worker is more important than the Dahne server. However, with permanent parsing of Data Archives, such as ESO’s, Gemini’s and HST’s, as well as Satellite schedules such as Swift’s, and the creation of a permanent flux of observing activities, a great part of the load of the « arcsecond.io » backend is occuring in the background tasks handled by Celery.

The second library is the official Django-supported project to handle more than just HTTP requests, but also WebSockets. And even if we have only one read-only WebSocket route opened in arcsecond.io so far (the live activities here: api.arcsecond.io/activities/live), the migration from a simple WSGI pure-HTTP server (it was gunicorn for a longtime) to channels and daphne, was not a simple one. We had 2 main difficulties. One was that the channels_redis library (providing the backing store for the channels layer of the server) had a bug that was causing the creation of numerous connections in our infrastructure, way beyond what is expected. For a long time we had various errors pointing in various directions, without knowing the true cause of the problem. But once fixed (recently), the result was quite impressive!

Evolution over time of the number of Redis connections used by our channels / daphne server. See how dramatic the effect of a bugfix had.

The second difficulty was to understand the underlying mechanism of WebSockets, and how one could provide automatically and similarly serialized JSON data. In other words, how to use our carefully crafted REST serializers to format the data in the WebSocket tube the same way it was for HTTP requests. We use the open-source (very early-stage but clean and well written) djangochannelsrestframework library for that.

As for the tests, we use the pytest library, with the standard python mocks when necessary, and these additional and very useful libraries:

  • freezegun : to easily write tests anywhere you want in time,
  • factory_boy : to easily create dozens of instances during your tests setup,
  • django-naomi : to easily see in a browser during development what the emails your backend sent look like,
  • vcrpy : to very easily replay the « cassettes », that is, replay real network requests.

The infrastructure

Arcsecond.io has two servers: one for the backend, and one for the frontend. The two are deployed on the well-known PaaS Heroku. Why Heroku? Well, because a colleague of mine, when I was working back in Switzerland, just mentionned it’s the easiest way of deploying a server. And it is still mostly true today, I am very happy of this decision (even if today I would consider using a serverless stack, amazed by the products of zeit.co for instance).

Some of the many advantages of using Heroku:

  • Large marketplace of one-click-installation add-ons for bazillions of auxiliary services (storage, logging, monitoring…)
  • Deploy with a simple git push.
  • A very easy-to-use CLI allowing you to interact and connect to your servers.
  • A very easy way to have your hostnames using certificates (using letsencrypt)

So here is the infrastructure of arcsecond.io:

  • A Daphne server (one Heroku dyno),
  • A Celery instance for a background worker (one Heroku dyno),
  • A celery-beat scheduler (one Heroku dyno),
  • A production PostgresQL DB instance, automatically backed up every hour,
  • A Papertrail instance for logging,
  • A Sendgrid instance for sending emails,
  • A RedisCloud instance for the backing store of the Daphne server,
  • A basic Heroku Redis instance for the HTTP cache,
  • An Express.js server for the frontend (one Heroku dyno),
  • Multiple Amazon S3 buckets for the storage of the data (price follows data volume),
  • The domains are bought on the french cloud provider ovh.com (~40$ / year for a .io domain),
  • I used to pay the Pro level of GitHub to store private repositories, but then it became free,
  • There are also 2 other free Heroku dynos used for the staging environnements (one for the back, one for the front).

These are mostly first or second-level entries of the services, and as soon as some money flows in, I will move up the level of some of them. Thanks to Heroku it will be as simple as moving a cursor!

There is one key part of the infrastructure that is not mentionned yet: CircleCI. The code of the backend is never pushed in production directly. It always goes through the automatic testing and deployment by CircleCI on a staging environment, then on production.

As for the frontend, the code is not going through an external CI service, for one reason. The Javascript code is split in various private NodeJS modules (see below), and I couldn’t wrap my head around the idea to pay an additional Amazon EC2 instance to hold a Verdaccio server, or use the premium mode of npmjs.com « just for 5 small JS libs ». So I use a Verdaccio server that runs on my iMac at home.

So far, I count about 750 deploys for the backend, and 260 for the frontend.

One couldn’t finish this part without mentionning the use of the fantastic crash-reporter Sentry. The richness and the ease of use of this tool is really great, and its help to find and correct bugs is invaluable. Even if Django comes with an automatic emailing when an error occurs (all 500 status code ends up in my Inbox), Sentry helps you read the code stack and identify more clearly what happened. And it also provides very easy management of problems, grouping identical ones, alerting when closed ones reappear etc.

This « software system » is in production since more than a year now, and has handled already quite a few visit spikes (hello ProductHunt...). It is ready to support a much larger load without having to change its underlying organisation.

The frontend

I never imagined I could build up such a complex view in the browser.

The frontend is a Vuejs Single-Page Application (SPA). The discovery of Vue.js was simply a life-saver. Originally, the frontend of arcsecond.io was written with AngularJS, the first version of the now-called Angular frontend framework (wich is one of the Trilogy, with Vuejs and Reactjs). But the codebase was enormous for the result, and hard to maintain. I tried at least 5 times to migrate to Angular 2, but always failed. And the JSX language of Reactjs appeared to me like a frightning thing and a prohibitive cost to pay (I still find it is a terrible idea for readibility).

Then came Vuejs. Simple. Easy to learn. Natural. No cruft, no crazy syntax (simple v- prefixes like v-if, compared to *ngIf for Angular – really, guys?), a textbook example of how a documentation must be written and displayed (yeah, Python my friend, your documentation is damn difficult to work with). Then came the v3 of the Vuejs Command-Line Interface. Oh man. Even the migration of the project from its original shape to the Vuejs-CLI v3 organisation was a breeze. And thanks God, it abstracts away all Webpack insane complexity (for non-frontend developers, webpack is crazy difficult to learn). To me, Vuejs is what the web development should have always been.

The tests are written with Jest, and vue-test-utils. I have made several attemps with Cypress to make end-to-end tests, but I had trouble installing a stable version on my iMac. Moreover, one could go a long way with Jest, so I am concentrating on unit tests for now.

One last piece that could be categorized as « frontend » in the sense of being a consumer of the APIs and their data is the Arcsecond Command-Line-Interface (CLI), open-source and freely available on GitHub. For a long time I thought it was enough to provide REST APIs of interesting data, I would not commit myself to another Python script / module like every scientist on this planet… But reality is stronger, and I am very happy to have build such tool. It helps integrate api.arcsecond.io more easily into custom workflows.

The organisation of the code

To that day, the backend repository contains 4200 commits, and the frontend repository 4900. Yes, I tend to make a lot of microcommits. The project started for real only two years ago, when I was able to really craft a serious beyond-hello-world frontend app for the long-term. The first commit of the Vuejs frontend was made on Tuesday February 28, 2017. But the first commit of the Python backend was on Tuesday Apr 15, 2014! (Always good to start a project on Tuesdays.)

In the code, I have 295 Vuejs components (.vue files) that allow to build a webapp, which weights 3.1 MB in the end. It is not using Server-Side-Rendering (SSR) so far, and it’s pretty difficult to transform it right now (I have special handlers for dynamic routes for Organisation Portals…). But I intend to increase at least the number of pages using prerendering (see SSR link above). On the backend, even if there is only one « arcsecond » Django monolyth app, the code is split into 24 different Django subapps (which all have their own models, serializers, views, urls, sometimes tasks and/or connectors…).

Yes, the backend app is a monolyth, even if the industry is breezing with microservices. Actually, given the amount of relationships between all the models, the real work of development and its operations appeared to me a lot easier within a monolyth, rater than distinct microservices. I am alone (Eric my friend and partner mostly helps with advices, tests and comments), and arcsecond.io is a kind of product that requires an large amount of development first before starting to be useful. I just can’t spend more time to something else than development.

For the IDEs, I use the pro version of PyCharm since the beginning for the backend (~80$/year). For the frontend I tried Sublime, Atom, VSCode, WebStorm over the years, and finally use PyCharm too, since it has an excellent Vuejs plugin. Finally, I couldn’t commit so fast and so easily, and find the history of the changes so quickly without the excellent Git client Tower (~65$/year – that’s quite a lot just to make commits… but I see Tower more like both a debugging and a management tool at the same time).

The GitHub repositories (repos, in the developers jargon..) are organised as follow:

  • A single arcsecond-back repo for the backend, sometimes with the help of forks of open-source libraries,
  • An arcsecond-front repo for the frontend, but using 6 private repos for custom Vuejs libaries,
  • The AA.js open-source repo for the Astronomical Algorithms,
  • The Arcsecond CLI open-source repo,
  • The public-issues repo to collect and work on public issues, found by my or users.
  • The standard-stars-catalogues repo for holding an open-source and reliable source of standard stars catalogues.

That’s it. Thanks for reading down to these lines! I hope you had learn a bit what it takes to start a « small but growing » project that grabs quite a large fraction of your daily awake time.

–– Clear skies to all!

P.S. The beautiful image of the starry night is from G. Hüdepohl (atacamaphoto.com)/ESO.

Dans l’esprit d’un TechLead: une vidéo qui fait date

Le but de ces « posts » sur mon rôle de TechLead est pour moi de mieux le comprendre et le développer, pour les mois (années?) à venir.

Par le plus pur des faux hasard au temps d’Internet (le suivi de certaines catégories de personnes sur Twitter amène forcément à des rencontres utiles), je suis tombé sur une vidéo d’Arnaud Lemaire.

Je ne suis pas un fan de YouTube. Personnellement, je préfère de loin la lecture pour apprendre quoi que ce soit. Mais là, il faut bien le dire, Arnaud explique avec énormément de pertinence et de maturité le métier de développeur à l’intérieur du contexte de cette jeune industrie (contexte qui manque cruellement à la plupart des quelques vidéos que j’ai tenté de visionner…).

De bout en bout, Arnaud déroule un discours censé, pertinent, intelligent. Cette vidéo fera date pour moi, et je vais en reparler sur ce blog.

Dans l’esprit d’un TechLead: décollage

[Access the English version of the article initially published at the end of december 2018]
[Accéder à toute la série d’articles en français]

J’ai tout juste débuté il y a mois à travailler comme « leader technique » dans une équipe de développeurs pour un projet logiciel ambitieux. Ce rôle n’est pas entièrement nouveau pour moi, puisque j’ai travaillé comme « Head of Software » pour une startup durant 2 ans. Mais à ce moment là, j’ai surtout été submergé par les aspects opérationnels du département de la « R&D » (c’est-à-dire 90% de l’entreprise). Cette fois, je peux me concentrer sur ce rôle intéressant uniquement.

J’aimerais prendre l’occasion de cette nouvelle aventure pour partager quelques unes de mes réflexions, en essayant d’explorer autant que possible tous les aspects. C’est-à-dire la technologie, bien sûr, mais aussi, la confiance en soi, la communication, l’architecture, la progression de carrière, le mentoring, la politique…

(Un petit peu de contexte: j’ai 44 ans. J’ai commencé ma carrière comme astrophysicien, puis j’ai basculé dans le développement logiciel en 2011, travaillant pour des petites entreprises, mais aussi pour des organisations plus grandes. J’ai eu l’occasion de faire pas mal de code scientifique, de modélisation, de traitement d’images, mais aussi une application « desktop » professionelle, des bibliothèques scientifiques, des composants applicatifs, des APIs REST et finalement une app front-end. Peu importe les détails, mais disons que j’ai « mangé » passablement de languages, dans passablement de contextes différents, et – le plus important – livré en production passablement de projets.

La première idée qui me vient, quand je regarde en arrière le déroulement de ce premier moi est la suivante: le TechLead doit avoir un sens aigu de l’équilibre. De nombreux espoirs, rationnels ou non, sont souvent mis sur les épaules de ce rôle, de la part de personnes techniques mais aussi non-techniques. Cela était particulièrement le cas que je suis arrivé: un projet qui a 6 développeurs juniors, un scrum master, un architecte, deux « Product Owners » scrum (et un directeur de projet), and qui a commencé déjà depuis 3 mois. Mais tout le monde dans l’équipe de développement et le management alentour partageait le même sentiment que le projet avec un besoin impératif de « séniorité », étant donné le contraste entre la jeunesse de l’équipe et l’ambition du projet (qui est un sujet en soi).

Quand je suis arrivé, j’ai ressenti fortement le besoin d’équilibre entre de nombreuses forces. Mais concentrons-nous sur la partie la plus pertinente, selon moi, durant ce moment spécial que sont les premières semaines.

J’ai ressenti le besoin d’un sens de l’équilibre entre l’urgence de me couler le mieux possible dans la nouvelle organisation, d’un côté, et d’user de mon ignorance des détails pour questionner les hypothèses existantes du projet, d’un autre. Avec un management très ouvert d’esprit (une chose dont l’importance est cruciale), les gens sont souvent très ouverts à entendre quelles sont les choses qu’elles auraient pu faire autrement, ou quelles éventuelles erreurs elles ont fait. C’est un pouvoir que vous avez seulement durant les première semaines, et il n’y en a qu’une quantité limitée. Il vous est donné de parler librement, mais avec prudence malgré tout, pour partager vos impressions.

C’est aussi l’occasion de fournir déjà ce que sera les grandes lignes de votre action. Ce dernier point ne vous permet pas de vous sentir parfaitement dans votre rôle facilement, mais au moins, cela aide à le remplir…

En d’autres termes, si vous entrez dans un environnement essentiellement problématique (comme c’était le cas pour moi), vous devez rapidement révéler les blocages (mais sans les exagérer) et ouvrir des possibles pistes de solutions (sans les imposer!). C’est un équilibre assez fin. Si vous insistez trop sur les problèmes, ou apparaissez trop rapidement impliqué émotionnellement, c’est facile de comprendre que vous ne servirez pas bien votre rôle, ou que vous ne serez simplement pas une part de la solution. Le risque existe d’être poussé hors du projet très peu de temps après votre entrée.

De plus, si vous imposez des solutions, vous perdez rapidement du crédit: comment pouvez-vous avoir une idée de l’ensemble du projet dès le début ? Des problèmes qui n’ont pas été réglé après de nombreuses semaines, voire mois sont forcément interconnectés avec d’autres forces, et impliquent d’autres gens. Les gens présents sont certainement de bonne foi, esayant de faire de leur mieux. Vous ne pouvez pas faire plus que suggérer des solutions, et si possible, des solutions que le management vous a suggérer au début, plus ou moins explicitement (vous êtes nécessairement un peu « utilisé » par quelqu’un pour modifier l’état existant des choses – c’est parfaitement normal).

Si vous invoquez votre « énorme » expérience pour essayer d’imposer vos solutions, vous semblez devoir vous justifier, et cela affaiblit votre position encore neuve. (Les démonstrations de virilité dans des environnements tech auront besoin d’un post dédié). And vous devez rester ouvert aux solutions suggérées par d’autres; solutions qui ne vous satisfont peut-être pas, alors que vous continuez à vous ajuster à un nouveau contexte, une nouvelle place de travail, une nouvelle pile technique… Tout cela est difficile.

Dans le cas opposé d’un environnement essentiellement positif, ces premières semaines sont, je pense, un bon moment pour construire des pistes de développement à la fois technologiques mais aussi humaines, de long-terme. Ce qui est sûr, c’est que dans les deux cas vous pouvez engranger de grandes quantités de crédit, si vous réussissez à acquérir rapidement la confiance de l’équipe de développement en lui fournissant de l’aide rapidement sur des points difficiles (ce qui est finalement le coeur de votre rôle). L’équipe de dev est le terreau sur lequel quelque chose se construit, et c’est à vous de le cultiver.

Comment réussir à ressentir ce sens aigu de l’équilibre ? Il n’y a pas de magie. Personnellement, je ne vois que deux choses: écouter et dormir, dès le premier jour. Ecouter est absolument clé. Ecouter tout le monde, faire attention un peu au language non-verbal, mais surtout pour confirmer des impression, écouter dans toutes les occasions, les réunions, les déjeuners, écouter tout le temps. C’est épuisant. C’est pourquoi, assurez-vous de dormir assez. Vous aurez certainement l’occasion de profiter de temps plus calmes après quelques semaines.

J’espère que vous aurez trouvé cet article intéressant. Il souligne un peu les sujets à venir dans cette série: comment acquérir la confiance d’une équipe, comment gérer la présence de figure autoritaires, l’équilibre entre être fort, et être une partie d’un tout que vous ne contrôlez pas… A bientôt !

Inside the mind of a TechLead: dev or not dev ?

This is one of the main questions that arises when starting to work as a TechLead inside a team, if this role is new to you: « Should I spend time coding with the team? But shouldn’t I also spend time not coding, and coaching the team? »

[Access all articles of this serie in English].

This is one of the main questions that arises when starting to work as a TechLead inside a team, if this role is new to you: « Should I spend time coding with the team? But shouldn’t I also spend time not coding, and coaching the team? »

In fact, in my humble opinion, this question rapidly becomes obsolete. It may be a relevant one when you just start, and the TechLead role is brand new to you. But as time flies, this question becomes less and less relevant, because it is too simple. And also because everybody knows the answer: of course, you must spend time coding! And of course too, you must not spend all your time coding!

The real question is: how do you make my time the most fruitful for your team(s), and around it? The answer is: it depends. It depends on multiple factors, because in addition to take care of your own work inside a collaboration – your main concern as a developer – the TechLead must sense the outside of this collaboration on one side, and how the whole group is moving (is it moving fast/well enough, etc…) on the other side.

And sometimes, to let the team move better or faster, it is by coding. But what the key thing that has changed is this: you don’t code anymore to deliver yourself (as part of a team). Your coding activity becomes a tech-coaching activity: you must teach coding in the projects code. It’s time to produce your best code, in small chunks, in places where the team need to progress somewhere. It is also coding, but a lot more challenging. You won’t be able to come back a lot and re-think what you’ve written. Because it’s better to hand over it to your team.

Understanding Gravitational Waves in 4 steps

You want to understand Einstein’s gravitational waves in 4 steps? No picture, no equations? Easy. Here it is.

Step #1, Year 1905: Einstein publishes the theory of Special Relativity. It says, among other things, and for reasons too long to explain here, that ‘space’ and ‘time’ must not be considered separatedly (as we always do), but actually are the components of one thing, called the … ‘space-time’. Practically, it means you can’t just make computations with space or time. But you must consider both together. Especially if you have a speed close to the speed of light (which is not infinite). In this new theory, that speed of light has a peculiar role: it is absolute. When you measure your speed relative to the source, no matter where you look at (if you look right into the beam or along it), you always get the same value: the same speed of light. Always. Really, always. You simply can’t go faster than that.

Step #2, Year 1907, Berne: The famous « most thoughtful » idea of Einstein: acceleration is indistinguishable from gravitation. Physicists call this the Equivalence Principle. We all stand in out feet on Earth’s surface. We feel gravitation. In fact, this can be re-stated as feeling a permanent acceleration, since Earth physical surface prevent us to follow a free-fall trajectory in space… Astronauts, assuming they have the eyes closed and forget for a second that they are in space, and there is truly no air, follow in fact a free fall trajectory. They don’t feel the gravitation. They feel no acceleration. Hence gravitation, similary to linear motion, can be « cancelled » in the right reference frame. When you travel in a plane, you don’t feel the motion during all the flight right? But people looking at you see clearly you move. Same thing, for acceleration. Einstein imagined this that a man inside a closed elevator having an acceleration equals to the value of Earth gravitation will feel exactly the same as a man not moving, but on Earth standing on his feet. Gravitation is equal to acceleration. And vice-versa.

Step #3, Year 1911, Prague: Imagine the same man inside that elevator going up with a constant acceleration (hence its speed is constantly increasing). Imagine a small hole in the side of the elevator where a beam of light can enter. Since speed of light is not infinite, and size of the elevator is non zero, it takes some time (pretty small, but still) for the light to enter the elevator and reach the other side. Since the elevator is going up, the point reached by the light is thus lower than the entry point. Hence the trajectory of light inside the accelerated elevator is … curved. But remember that acceleration and gravitation are equivalent?

Well, that is General Relativity, ladies and gentlemen: gravitation is curved space-time, period. And space-time is curved because of the presence of matter. All dynamically, which makes things even more beautiful…

Step #4, 1916, : In particule physics we know that when we shake electrons, waves are created. Electromagnetic waves, the other name for « light ». Einstein predicted that when shaking masses, this should also create waves. Gravitational waves. That was the prediction of the existence of gravitational waves. But Einstein never thought possible to detect such waves. Since one must need an enormous, in fact ultra-giganomous, amount of energy released to produce these waves, and even then, the effect would be ridiculously small.

Guess what LIGO detected? A merging of two 30-solar-masses black holes, releasing as much energy in a few microseconds as all stars in the visible universe, and the effect detected on Earth is of the size of one 10 000th that of a proton!

You can’t beat science in the awesomeness.

If you like this post, tell me about it! I may be to prepare some more. This one in particular is a worked out and personnally twisted transcript of the explanations of  french scientist and educator, Etienne Klein. Here is the video (in french).

In the mind of a TechLead : kickoff

I started just a month ago to work as TechLead inside a developers team for a ambitious software project. (…) I would like to take the occasion of this new adventure to share some of my thoughts, trying to explore as many apsects of it as I can. That is, technology, of course. But also, self-confidence, communication, architecture, career paths, mentoring, politics…

[Access all articles of this serie in English].
[Accéder à l’article en français].

I started just a month ago to work as TechLead inside a developers team for a ambitious software project. This role isn’t entirely new to me, since I worked as « Head of Software » for a startup during 2 years. But at that moment, I was overwhelmed with all operational aspects of the « research and development department » (that is, 90% of the company at that moment). This time I can focus on that interesting role.

I would like to take the occasion of this new adventure to share some of my thoughts, trying to explore as many apsects of it as I can. That is, technology, of course. But also, self-confidence, communication, architecture, career paths, mentoring, politics…

(Just a bit of context: I’m 44. I’ve been starting my carreer as an astrophysicist, then switched to software development in 2011, worked for small companies and startups, as well as larger organisation. I did quite a lot of scientific coding, modeling, image processing, but also a professional desktop app, iOS apps, scientific libraries, app components, REST APIs and a front-end webapp. Whatever the details, I’ve eaten quite a few langages, in quite a few contexts, and most importantly delivered in production quite a few products.)

The first idea that come up to me, when I look back to this short month is: a TechLead must have an acute sense of balance. A lot of rational and less rational expectations are put on the shoulders of that role, both from technical and non-technical people. That was particlarly true when I arrived: the project has a team of 6 juniors developers, a scrum master, an architect, two product owners (and a project manager), and was already working since about 3 months. But everybody in the development team and the management around it shared the feeling that the project strongly needed some « seniority », given the contrast between the juniority of the team, and ambition of the project (a subject on its own).

Thus I quite strongly experienced the need of balance between various things. But let’s focus on the most relevant one for this special moment that the few first weeks are.

I mostly felt the need of an acute sense of balance between the urge of fiting myself into the new organisation on one hand, and using my ignorance of the details to challenge existing hypothesis, on the other hand. With a very open-minded management (a thing that I can’t stress more the importance), people are very keen to hear you about what they could have done better or even wrong. This is a power you have during the first weeks, and there is only a very short supply of it. You have to use it wisely to speak out and share your impressions carefuly.

It is also the occasion to provide already what you think will be the overall direction of your action. The latter point don’t make you fit into the role right away, but at least it helps you fill it.

In other words, if you enter a mostly problematic working environment (as it was, in my case), you should rapidly reveal issues (but without exaggerating them!) and draw possible lines of solutions (without imposing them!). That’s a very fine balance. If you insist too much on problems, or appear to be too rapidly emotionaly involved in these problems, it’s easy to understand you might not fit for the role, or that you won’t simply be part of the solution. The risk exists of being pushed out of the project right at the start.

Moreover, if you try to impose solutions, you rapidly loose credit: how can you have a sense of the big picture already now? Problems that haven’t been solved after a few weeks or months are necessarily interconnected with other forces and/or people. People that are certainly of good faith, trying to do their best. You cannot do more than suggesting solutions, and if possible, solutions that are based on hints the management might have given you, more or less explicitely (you are necessarily ‘utilized’ by someone to modify an existing state of things – this is normal).

If you invoke your « huge » experience to try to impose solutions, you appear to justify yourself, and weaken your position already. (Virility demonstrations in tech environments will require its own post.) And you must remain open to solutions suggested by others, and solutions that don’t make you necessarily happy, while at the same time you are still adjusting yourself to a new context, a new place, new people, new tech stack… It’s hard.

In the opposite case of a mostly positive environment, this time is good to draw long-term guidelines of your action too. In both cases, you can earn a large amount of credit if you manage to rapidly acquire trust from the team, and both provide rapid help for existing issues.

How to achieve this sense of balance? There’s no magic. Personally, I can see only two things: listen and sleep, from day one. Listening everything is key. Listen to everybody, pay attention a bit on the body langage, but mostly to confirm impressions, listen in every occasion, meetings, lunchs, listen all the time. This is exhausting. Thus make sure to sleep enough. You’ll probably have time to rest a bit more after a few weeks.

I hope you found this post interesting. It outlines a bit the subjects to come in this serie of posts: how to acquire trust from a team, how to deal with autoritative figures, the balance between being strong, and being a part of something you can’t control… Stay tuned!

The v0.8 software malediction (and resolutions for 2019)…

Life in Software-land follows the rhythm of version updates. And big jumps in numbers is usually reflecting what happens under the hood (except for the Linux kernel sometimes): big changes, big rewrites, big decisions, all of them at the same time…

Life in Software-land follows the rhythm of version updates. And big jumps in numbers is usually reflecting what happens under the hood (except for the Linux kernel sometimes): big changes, big rewrites, big decisions, all of them at the same time. Long ago in my own Astro-Software-land, I was developing my famous macOS app: iObserve, which remained in the swamp of the beta phase for a long serie of months. (The beta phase is this period of time before the confidence on the capability of the mentionned software being actually useful reaches an « acceptable » level for the developer.)

iObserve never reached the v0.9. I had to stop development, rewrite a big part of the app, and jump right into 1.0 (I then submitted it to the Mac App Store).

v0.7.16. That’s the current version of arcsecond.io (see the changelog). Arcsecond.io was meant to become my new flagship software for the years to come. Still, I hit the v0.8 malediction. I hit it hard. I feel that a lot has to be rewritten. The good news is that the backend is solid, I’m very happy with it.

But I need to not only rewrite lots of the front-end, but actually rethink the whole shape of the webapp I intend to build. Too many things in the pipeline, but at the same time, nothing really finished. Damn.

I don’t know why it always happens around the v0.8. A kind of teenager crisis for a software. The time where you have exhausted your young energy, and start to think this isn’t the right direction, and something must change.

It’s hard to convince a developer he didn’t develop the right thing. But when this developer is yourself, and you invested a big chunk of a whole year into it, it is rough. At least, I must say I am not denying it anymore. It took quite a few months to really ask for feedback, but I did it, and the results of the votes are clear: I must continue to build new versions of iObserve…

Everything’s inside the meaning of « new versions »… This is where it is interesting to think as a product maker. What is iObserve ? An app that embedd into the same « workspace » a bunch of finely-crafted algorithms and visualisations combined with professional archival data, to let people think and prepare better their astronomical observations.

(Okay, okay! I heard it. It must work offline too.)

Fine. Let’s do that. But the success of iObserve is also the result of a few unplanned features nobody asked for. Say, I have a few in my pocket.

Resolutions for 2019. Actually, only one, which encompasses all others: focus. Really focus on the right thing, and nothing more.

Clear skies to everyone!

iObserve 1.7

My hope was to bring iObserve users to the web, and let me continue the journey with arcsecond.io. A vote took place in arcsecond.io, and iObserve users have spoken loudly ! They love the app. Well, I must say it’s nice to hear too.

So to the contrary of what I said before to some of you by email, I decided to make the non-trivial update and support for macOS 10.14 Mojave, for the app. The app will continue to be supported, but no new features will be included.

So here it is.

Open the Mac App Store

Arcsecond.io 0.6: Team Size Doubling + iObserve on the web !

This is a milestone for Arcsecond ! I am happy to announce that Eric Depagne, currently astronomer at SAAO/SALT has agreed to join his forces to arcsecond.io. His expertise with Python will help a lot on the backend, but I don’t expect him to remain within the boundaries of it!

Moreover, we releases iObserve on the web ! Not as feature-rich as the macOS app yet. But already new features that will belong only to Arcsecond: Night Logs (in prep) and Data !

Freely regsiter in www.arcsecond.io and tell us what you think !